1.1. General Introduction

In this paper, we deal with divergence measures estimation using essentially wavelets density function estimation. Let 𝒫 be a class of probability measures on ℝ^d, d ≥ 1, a divergence measure on 𝒫 is a function

such that 𝒟(ℚ, ℚ) = 0 for any ℚ such that (ℚ, ℚ) in the domain of application of 𝒟.

The function 𝒟 is not necessarily an application. And if it is, it is not always symmetrical and it does neither have to be a metric. A great number of them are based on probability density functions (pdf). So let us suppose that any ℚ ∈ 𝒫 admits a pdf f_ℚ with respect to a σ-finite measure ν on (ℝ^d, ℬ(ℝ^d)), which is usually the Lebesgue measure λ_d (with λ₁ = λ) or a counting measure on ℝ^d.

We may present the following divergence measures.

1. (1)

   The L22-divergence measure :

   (1.2)𝒟L2(ℚ,𝕃)=∫ℝd(fℚ(x)−f𝕃(x))2dν(x).

2. (2)

   The family of Renyi’s divergence measures indexed by α ≠ 1, α > 0, known under the name of Renyi-α :

   (1.3)𝒟R,α(ℚ,𝕃)=1α−1log(∫ℝdfℚα(x)f𝕃1−α(x)dν(x)).

3. (3)

   The family of Tsallis divergence measures indexed by α ≠ 1, α > 0, also known under the name of Tsallis-α :

   (1.4)𝒟T,α(ℚ,𝕃)=1α−1(∫ℝdfℚα(x)f𝕃1−α(x)−1)dν(x);

4. (4)

   The Kullback-Leibler divergence measure

   (1.5)𝒟KL(ℚ,𝕃)=∫ℝdfℚ(x)log(fℚ(x)/f𝕃(x))dν(x).

   The latter, the Kullback-Leibler measure, may be interpreted as a limit case of both the Renyi’s family and the Tsallis’ one by letting α → 1. As well, for α near 1, the Tsallis family may be seen as derived from 𝒟_R,α(ℚ, 𝕃) based on the first order expansion of the logarithm function in the neighborhood of the unity.

From this small sample of divergence measures, we may give the following remarks.

1. (a)

   The L22-divergence measure is both an application and a metric on 𝒫², where 𝒫 is the class of probability measures on ℝ^d such that

   ∫ℝdfℚ2(x)dν(x)<+∞.

2. (b)

   For example, for both the Renyi and the Tsallis families, we may have integrability problems and lack of symmetry. From this sample tour, we have to be cautious, when speaking about divergence measures as applications and/or metrics. In the most general case, we have to consider the divergence measure between two specific probability measures as a number or a real parameter.

Originally, divergence measures came as extensions and developments of information theory that was first set for discrete probability measures. In such a situation, the boundedness of these discrete probability measures above zero and below +∞ was guaranteed. That is, the following assumption holds :

• Boundedness Assumption (BD). There exist two finite numbers 0 < κ₁ < κ₂ < +∞ such that

  (1.6)κ1≤fℚ,f𝕃≤κ2.
  If Assumption (1.6) holds, we do not have to worry about integrability problems, especially for Tsallis, Renyi and Kullback-Leibler measures, in the computations arising in the estimation theories. This explains why Assumption (1.6) is systematically used in a great number of works in that topic, for example, in [Singh and Poczos (2014)], [Krishnamurthy et al.2014], [Hall(1987)], to cite a few. But instead of Assumption (1.6), we use the following

• Modified Boundedness Condition : There exist 0 < κ₁ < κ₂ < +∞ and a compact domain D as large as possible such that

  (1.7)κ1≤fℚ1D,f𝕃1D≤κ2.
  This implies that the modified divergence measure, denoted by 𝒟^(m), is applied to the modified pdf’s :
  fℚ(m)=D1−1fℚ1D  and  fℙ(m)=D2−1f𝕃1D,
  where D₁ and D₂ are the integrals of f_ℚ and f_𝕃 of D, respectively. Based on this technique, that we apply in case of integrability problems, we will suppose, when appropriate, that Assumption (1.6) holds on a compact set D.

Although we are focusing on the aforementioned divergence measures in this paper, it is worth mentioning that there exist quite a few number of them. Let us cite for example the ones named after : Ali-Silvey or f-divergence [Topsoe(2000)], Cauchy-Schwarz, Jeffrey divergence (see [Evren(2012)]), Chernoff (See [Evren(2012)]), Jensen-Shannon (See [Evren(2012)]). According to [Cichocki and Amari(2010)], there is more than a dozen of different divergence measures in the literature. In a longer version of this paper (see [Ba et al.(2017)]), some important applications of them are highlighted with there references. The reader, who is interested by a so important review topic is referred to that paper.

In the next subsection, we describe the frame in which we place the estimation problems we deal in this paper.

1.2. Statistical Estimation

The divergence measures may be applied to two statistical problems among others.

• (A)

  First, it may be used as a fitting problem as described here. Let X₁, X₂,.... a sample from X with an unknown probability distribution ℙ_X and we want to test the hypothesis that ℙ_X is equal to a known and fixed probability ℙ₀. Theoretically, we can answer this question by estimating a divergence measure 𝒟(ℙ_X, ℙ₀) by a plug-in estimator 𝒟(ℙX(n),ℙ0) where, for each n ≥ 1, ℙ_X is replaced by an estimator ℙX(n) of the probability law, which is based on sample X₁, X₂, ..., X_n, to be precised.

  From there establishing an asymptotic theory of Δn=𝒟(ℙX(n),ℙ0)−𝒟(ℙX,ℙ0) is thought to be necessary to conclude.

• (B)

  Next, it may be used as tool of comparing for two distributions. We may have two samples and wonder whether they come from the same probability measure. Here, we also may two different cases.

• (B1)

  In the first, we have two independent samples X₁,X₂,.... and Y₁,Y₂,.... respectively from a random variable X and Y. Here the estimated divergence 𝒟(ℙX(n),ℙY(m)), where n and m are the sizes of the available samples, is the natural estimator of D(ℙ_X, ℙ_Y) on which depends the statistical test of the hypothesis : ℙ_X = ℙ_Y.

• (B2)

  But the data may also be paired (X, Y), (X₁, Y₁), (X₂, Y₂),..., that is X_i and Y_i are measurements of the same case i = 1,2,... In such a situation, testing the equality of the margins ℙ_X = ℙ_Y should be based on an estimator ℙ(X,Y)(n) of the joint probability law of the couple (X, Y) based on the paired observations (X_i,Y_i), i = 1,2,...,n.

We did not encounter the approach (B2) in the literature. In the (B1) approach, almost all the papers used the same sample size, at the exception of [Poczos and Jeff(2011)], for the double-size estimation problem. In our view, the study case should rely on the available data so that using the same sample size may lead to a loss of information. To apply their method, one should take the minimum of the two sizes and then loose information. We suggest to come back to a general case and then study the asymptotic theory of 𝒟(ℙX(n),ℙY(m)) based on samples X₁,X₂,..,X_n. and Y₁,Y₂,...,Y_m. In this paper, we will systematically use arbitrary samples sizes.

In the context of the situation (B1), there are several papers dealing with the estimation of the divergence measures. As we are concerned in this paper by the weak laws of the estimators, our review on that problematic did return only of a few results. Instead, the literature presented us many kinds of results on almost-sure efficiency of the estimation, with rates of convergences and laws of the iterated logarithm, L^p (p = 1,2) convergence, etc. To be precise, [Dhakher et al.2016] used recent techniques based on functional empirical process to provide a series of interesting rates of convergence of the estimators in the case of one-sided approach for the class de Renyi, Tsallis, Kullback-Leibler to cite a few. Unfortunately, the authors did not address the problem of integrability, taking for granted that the divergence measures are finite. Although the results should be correct under the boundedness assumption BD we described earlier, a new formulation in that frame would be welcome.

The paper of [Krishnamurthy et al.2015] is exactly what we want to do, except that it is concentrated on the L²-divergence measure and used the Parzen approach. Instead, we will handle the most general case of ϕ-divergence measure and will use the wavelets probability density estimators.

In the context of the situation (B1), we may cite first the works of [Krishnamurthy et al.2014] and [Singh and Poczos (2014)]. They both used divergence measures based on probability density functions and concentrated on Renyi-α, Tsallis-α and Kullback-Leibler. In the description of the results below, the estimated pfd’s - f and g - are usually in a periodic Hőlder class of a known smoothness s..

Specifically, [Krishnamurthy et al.2014] defined Renyi and Tsallis estimators by correcting the plug-in estimator and established that, as long as 𝒟_R,α(f, g) ≥ c and 𝒟_T,α(f, g) ≥ c, for some constant c > 0, then

[Poczos and Jeff (2011)] used a k–nearest-neighbor approach to prove that if |α − 1| < k, (α ≠ 1) then There has been a recent interest in deriving convergence rates for divergence estimators ([Moon and Hero(2014)], [Krishnamurthy et al.2014]). The rates are typically derived in terms of smoothness s of the densities :

The estimator of [Liu et al. 2012] converges at rate n−ss+d, achieving the parametric rate when s > d.

Similarly, [Sricharan et al.(2012)] showed that when s > d a k-nearest-neighbor style estimator achieves the rate n^−2/d (in absolute error) ignoring logarithmic factors. In a follow up work, the authors improved this result to O(n^−1/2) by using a set of weak estimators, but they required s > d orders of smoothness. One can also see [Singh and Poczos (2014)], [Kallberg and Seleznjev(2012)] for other contributions.

The majority of the aforementioned articles worked with densities in Hőlder classes, whereas our work applies for densities in the Besov classes.

Here, we will focus on divergence measures between absolutely continuous probability laws with respect to the Lebesgue measure. As well, our results applied to the approaches (A) and (B1) defined above. As a sequence, we estimate divergence measures by their plug-in counterparts, meaning that we replace the probability density functions (pdf) in the expression of the divergence measure by a nonparametric estimators of the pdf ’s. From now, we have on our probability space, two independent sequences :

• (-)

  a sequence of independent and identically distributed random variables with common pdf f_ℙX :

  (1.8)X1,X2,…

• (-)

  a sequence of independent and identically distributed random variables with common pdf g_ℙY :

  (1.9)Y1,Y2,…

  To make the notations more simple, we write

  f=fℙXand g=fℙY.

  We focus on using pdf ’s estimates provided by the wavelets approach. We will deal on the Parzen approach in a forthcoming study. So, we need to explain the frame in which we are going to express our results.

We also wish to get, first, general laws for an arbitrary functional of the form

where ϕ(x,y) is a measurable function of on (x,y)∈ℝ+2 which we will make the appropriate conditions. The results on the functional J(f, g), which is also known under the name of ϕ-divergence, will lead to those on the particular cases of the Renyi, Tsallis, and Kullback-Leibler measures.

The exposure of all our results will be given in three a series of three papers. This paper is devoted to the foundations the general asymptotic theory and the exposition of the mains theoretical tools concerning the ϕ-forms. The second paper will deal with important key divergence measures and symmetrized estimators. Finally a third paper will focus on the proofs.

1.3. Wavelets estimation of pdf’s

To begin with the wavelets theory and its statistical applications, we say that the wavelets setting involves two functions φ and ψ in L₂(ℝ) respectively called father and mother such that

is a orthonormal basis of L₂(ℝ). We adopt the following notation, for j ≥ 0, k ∈ ℤ : Thus, any function f in L₂(ℝ) is characterized by its coordinates in the orthonormal basis, in the form with for j ≥ 0, k ∈ ℤ, For an easy introduction to the wavelets theory and to its applications to statistics, see for instance [Hardle et al.1998], [Daubechies(1992)], [Blatter(1998)], etc. In this paper we only mention the unavoidable elements of this frame.

Based on the orthonormal basis defined below, the following Kernel function is introduced

For any j ≥ 1 fixed, called a resolution level, we define and for measurable function h, we define the operator projection K_j of h onto the space V_j of L₂(ℝ) (spanned by 2^j/2φ(2^j(.) − k)), by Therefore we can write, for all x ∈ ℝ, In the frame of this wavelets theory, for each n ≥ 1, we fix the resolution level depending on n and denoted by j = j_n, and we use the following estimator of the pdf f associated to X, based on the sample of size n from X, as defined in (1.8), As well, in a two samples problem, we will estimate the pdf g associated to Y, based on a sample of size n from Y, as defined in (1.9), by The aforementioned estimator is known under the name of linear wavelets estimators.

Before we give the main assumptions on the wavelets we are working, we have to define the concept of weak differentiation. Denote by 𝒟(ℝ) the class of functions from ℝ to ℝ with compact support and infinitely differentiable. A function f : ℝ → ℝ is weak differentiable if and only if there exists a function g : ℝ → ℝ locally integrable (on compact sets) such that, for any ϕ ∈ 𝒟(ℝ), we have

In such a case, g is called the weak derivative function of f and denoted f^[1]. If the first weak derivative has itself a weak derivative, ans so forth up to the p − 1-th derivative, we get the p-th derivative function f^[p]. Now we may expose the four assumptions we require on the wavelets.

• Assumption 1.. The wavelets φ and ψ are bounded and have compact support and either (i) the father wavelet φ has weak derivatives up to order T in L_p(ℝ)(1 ≤ p ≤ ∞) or (ii) the mother wavelet ψ associated to φ satisfies ∫ x^mψ(x)dx = 0 for all m = 0,...,T.

  and

• Assumption 2. φ : ℝ → ℝ is of bounded p-variation for some 1 ≤ p < ∞ and vanishes on (B₁, B₂]^c for some −∞ < B₁ < B₂ < ∞.

  Wavelets generators with compact supports are available in the literature. We may cite those named after Daubechies, Coiflets and Symmlets (See [Hardle et al.(1998)]). The cited generators fulfill our two main assumption.

  Under Assumption 2, the summation over k, in (1.12), is finite since only a number of the terms in the summation are non zeros (see [Giné and Nickl(2009)]).

• Assumption 3. There exists a non-negative symmetrical and continuous function Φ(t) of t ∈ ℝ with a compact support 𝒦 such that :

  ∀(x,y)∈ℝ2,|K(x,y)|≤Φ(x−y).
  The fourth assumption concerns the resolution level we choose. We set for once an increasing sequence (j_n)_n≥1 such that

• Assumption 4. lim_n→+∞ n^−1/42^jn = 1.

  By the way, we have as n → ∞, and

  (1.15)jn2jnn+2−t/jn≈14log2lognn3/4+n−t/4→0,  ∀t>0jnloglogn→∞  and   supn≥n0(j2n−jn)=14.
  These conditions allow the use the [Giné and Nickl(2009)]’s results.

We also denote

where ‖h‖_∞ stands for sup_x∈_D(h) |h(x)|, and D(h) is the domain of application of h.

In the sequel we suppose the densities f and g belong to the Besov space ℬ∞,∞t(ℝ). We will say a word of simple conditions under which our pdf’s do belong to such spaces.

Suppose that the densities f and g belong to ℬ∞,∞t(ℝ), that φ satisfies Assumption 2, and φ, ψ satisfy Assumption 1. Then Theorem 3 [Giné and Nickl(2009)] implies that the rates of convergence a_n, b_n and c_n are of the form

almost-surely and converge all to zero at this rate (with 0 < t < T).

In order to establish the asymptotic normality of the divergences estimators, we need this key tool concerning the wavelets empirical process denoted by 𝔾n,Xw(h), where h∈ℬ∞,∞t(ℝ) and defined as follows by

where ℙn,Xw(h)=ℙn,X(Kjn(h))=1n∑i=1nKjn(h)(Xi) and 𝔼_X(h) = ∫ h(x) f(x)dx denotes the expectation of the measurable function h with respect to the probability distribution function ℙ_X. The superscript w refers to wavelets. We have since, by Fubini’s Theorem, We are ready to give our results on the functional J introduced in Formula (1.10).

2.1. Main Results

Here, we present a general asymptotic theory of a class of divergence measures estimators including the Renyi and Tsallis families and the Kullback-Leibler ones.

Actually, we gather them in the ϕ-divergence measure form. We will obtain a general frame from which we will derive a number of corollaries. The assumption (1.6) will be used in the particular cases to ensure the finiteness of the divergence measure as mentioned in the beginning of the article. However, in the general results, the assumption (1.6) is part of the general conditions.

We begin to state a result as a general tool for establishing asymptotic normality and related to the wavelets empirical process, which we will use for establishing the asymptotic normality of divergence measures.