A Nearest neighbor Approach to Estimating Divergence Between Continuous Random Vectors

1. Introduction

The Shannon (or differential) entropy of a continuously distributed random variable (r.v.) X with probability density function (pdf) f is widely used in probability theory and information theory as a measure of uncertainty. It is defined as the negative mean of the logarithm of the density function, i.e.,

$$H\left(f\right)=-E_f[lnf\left(\mathit{X}\right)]$$

(1)

k-Nearest neighbor (knn) density estimators were proposed by Mack and Rosenblatt [1]. Penrose and Yukich [2] studied the laws of large numbers for k-nearest neighbor distances. The nearest neighbor entropy estimators when

$\mathit{X}\in {\mathbb{R}}^p$

were studied by Kozachenko and Leonenko [3]. Singh et al. [4] and Leonenko et al. [5] extended these estimators using k-nearest neighbors. Mnatsakanov et al. [6] studied knn entropy estimators for variable rather than fixed k. Eggermontet et al. [7] studied the kernel entropy estimator for univariate smooth distributions. Li et al. [8] studied parametric and nonparametric entropy estimators for univariate multimodal circular distributions. Neeraj et al. [9] studied knn estimators of circular distributions for the data from the Cartesian product, that is,

${[0,2\pi )}^p$

. Recently, Mnatsakanov et al. [10] proposed an entropy estimator for hyperspherical data based on the moment-recovery (MR) approach (see also Section 4.3).

In this paper, we propose k-nearest neighbor entropy, cross-entropy and KL-divergence estimators for hyperspherical random vectors defined on a unit p-hypersphere

${\mathbb{S}}^{p-1}$

centered at the origin in p-dimensional Euclidean space. Formally,

$${\mathbb{S}}^{p-1}=\left\{\mathbf{x}\in {\mathbb{R}}^p:\parallel \mathbf{x}\parallel =1\right\}$$

(2)

The surface area

$S_p$

of the hypersphere is well known:

$S_p=\frac{2{\pi }^{p/2}}{\Gamma \left(\frac{p}{2}\right)}$

, where Γ is the gamma function. For a part of the hypersphere, the area of a cap with solid angle ϕ relative to its pole is given by Li [11] (cf. Gray [12]):

$$S\left(\varphi \right)=\frac{1}{2}{S}_p\left[1-sgn(cos\varphi )I_{{cos}^2\varphi }\left(\frac{1}{2},\frac{p-1}{2}\right)\right]$$

(3)

where sgn is the sign function, and

$I_{\mathit{x}}(\alpha ,\beta )$

is the regularized incomplete beta function.

For a random vector from the unit circle

${\mathbb{S}}^1$

, the von Mises distribution vM

$(\mathbf{\mu },\kappa )$

is the most widely used model:

$$f_{\mathrm{vM}}(\mathit{x};\mathbf{\mu },\kappa )=\frac{1}{2\pi I_0\left(\kappa \right)}{e}^{\kappa {\mathbf{\mu }}^T\mathit{x}}$$

where T is the transpose operator,

$\left|\right|\mathbf{\mu }\left|\right|=1$

and

$\kappa \ge 0$

are the mean direction vector and concentration parameters, and

$I_0$

is the zero-order modified Bessel function of the first kind. Note that the von Mises distribution has a single mode. The multimodal extension to the von Mises distribution is the so-called generalized von Mises model. Its properties are studied by Yfantis and Borgman [13] and Gatto and Jammalamadaka [14].

The generalization of von Mises distribution onto

${\mathbb{S}}^{p-1}$

is the von Mises-Fisher distribution (also known as Langevin distribution) vMF

${}_p(\mathbf{\mu },\kappa )$

with pdf,

$$f_p(\mathit{x};\mathbf{\mu },\kappa )=c_p\left(\kappa \right)e^{\kappa {\mathbf{\mu }}^T\mathit{x}}$$

(4)

where the normalization constant is

$$c_p\left(\kappa \right)={\displaystyle \frac{{\kappa }^{p/2-1}}{{\left(2\pi \right)}^{p/2}{I}_{p/2-1}\left(\kappa \right)}}$$

and

$I_{\nu }\left(\mathit{x}\right)$

is the ν-order modified Bessel function of the first kind. See Mardia and Jupp [15] (p. 167) for details.

Since von Mises-Fisher distributions are members of the exponential family, by differentiating the cumulant generating function, one can obtain the mean and variance of

${\mathbf{\mu }}^T\mathit{X}$

:

$$\begin{array}{cc}\hfill E_f\left[{\mathbf{\mu }}^T\mathit{X}\right]& =A_p\left(\kappa \right)\hfill \end{array}$$

and

$$\begin{array}{cc}\hfill V_f\left[{\mathbf{\mu }}^T\mathit{X}\right]& =A_p^{\prime }\left(\kappa \right)\hfill \end{array}$$

where

$A_p\left(\kappa \right)=I_{p/2}\left(\kappa \right)/I_{p/2-1}\left(\kappa \right)$

, and

$A_p^{\prime }\left(\kappa \right)=\frac{d}{d\kappa }{A}_p\left(\kappa \right)=1-A_p{\left(\kappa \right)}^2-(p-1)/\kappa A_p\left(\kappa \right)$

. See Watamori [16] for details. Thus the entropy of

$f_p$

is:

$$H\left(f_p\right)=-E_f[ln{f}_p\left(\mathit{X}\right)]=-ln{c}_p\left(\kappa \right)-\kappa E_f\left[{\mathbf{\mu }}^T\mathit{X}\right]=-ln{c}_p\left(\kappa \right)-\kappa A_p\left(\kappa \right)$$

(5)

and

$$V_f[ln{f}_p\left(\mathit{X}\right)]={\kappa }^2{V}_f\left[{\mathbf{\mu }}^T\mathit{X}\right]={\kappa }^2{A}_p^{\prime }\left(\kappa \right)$$

(6)

Spherical distributions have been used to model the orientation distribution functions (ODF) in HARDI (High Angular Resolution Diffusion Imaging). Knutsson [17] proposed a mapping from (

$p=3$

) orientation to a continuous and distance preserving vector space (

$p=5$

). Rieger and Vilet [18] generalized the orientation in any p-dimensional spaces. McGraw et al. [19] used vMF

${}_3$

mixture to model the 3-D ODF and Bhalerao and Westin [20] applied

${\mathrm{vMF}}_5$

mixture to 5-D ODF in the mapped space. Entropy of the ODF is proposed as a measure of anisotropy (Özarslan et al. [21], Leow et al. [22]). McGraw et al. [19] used Rényi entropy for the

${\mathrm{vMF}}_3$

mixture since it has a closed form. Leow et al. [22] proposed an exponential isotropy measure based on the Shannon entropy. In addition, KL-divergence can be used to measure the closeness of two ODF's. A nonparametric entropy estimator based on knn approach for hyperspherical data provides an easy way to compute the entropy related quantities.

In Section 2, we will propose the knn based entropy estimator for hyperspherical data. The unbiasedness and consistency are proved in this section. In Section 3, the knn estimator is extended to estimate cross entropy and KL-divergence. In Section 4, we present simulation studies using uniform hyperspherical distributions and aforementioned vMF probability models. In addition, the knn entropy estimator is compared with the MR approach proposed in Mnatsakanov et al. [10]. We conclude this study in Section 5.

2. Construction of knn Entropy Estimators

Let

$\mathit{X}\in {\mathbb{S}}^{p-1}$

be a random vector having pdf f and

${\mathit{X}}_1,{\mathit{X}}_2,\dots ,{\mathit{X}}_n$

be a set of i.i.d. random vectors drawn from f. To measure the nearness of two vectors x and

$\mathbf{y}$

, we define a distance measure as the angle between them:

$\varphi =arccos\left({\mathit{x}}^T\mathbf{y}\right)$

and denote the distance between

${\mathit{X}}_i$

and its k-th nearest neighbor in the set of n random vectors by

${\varphi }_i:={\varphi }_{n,k,i}$

.

With the distance measure defined above and without loss of generality, the naïve k-nearest neighbor density estimate at

${\mathit{X}}_i$

is thus,

$$f_n\left({\mathit{X}}_i\right)=\frac{k/n}{S\left({\varphi }_i\right)}$$

(7)

where

$S\left({\varphi }_i\right)$

is the cap area as expressed by (3).

Let

$L_{n,i}$

be the natural logarithm of the density estimate at

${\mathit{X}}_i$

,

$$L_{n,i}=ln{f}_n\left({\mathit{X}}_i\right)=ln\frac{k/n}{S\left({\varphi }_i\right)}$$

(8)

and thus we construct a similar k-nearest neighbor entropy estimator (cf. Singh et al. [4]):

$$\begin{array}{c}\hfill H_n\left(f\right)=-\frac{1}{n}\sum _{i=1}^n[L_{n,i}-lnk+\psi \left(k\right)]=\frac{1}{n}\sum _{i=1}^{n}ln\left[nS\left({\varphi }_i\right)\right]-\psi \left(k\right)\end{array}$$

(9)

where

$\psi \left(k\right)=\frac{{\Gamma }^{\prime }\left(k\right)}{\Gamma \left(k\right)}$

is the digamma function.

In the sequel, we shall prove the asymptotic unbiasedness and consistency of

$H_n\left(f\right)$

.

2.1. Unbiasedness of $H_n$

To prove the asymptotic unbiasedness, we first introduce the following lemma:

Lemma 2.1. For a fixed integer $k<n$, the asymptotic conditional mean of $L_{n,i}$ given ${\mathit{X}}_i=\mathit{x}$, is

$$E\left[\underset{n\to \infty }{lim}{L}_{n,i}\right|{\mathit{X}}_i=\mathit{x}]=lnf\left(\mathit{x}\right)+lnk-\psi \left(k\right)$$

(10)

Proof.

$\forall \ell \in \mathbb{R}$

, consider the conditional probability

$$\begin{array}{cc}\hfill P\{L_{n,i}<\ell |{\mathit{X}}_i=\mathit{x}\}& =P\{f_n\left({\mathit{X}}_i\right)<e^{\ell }|{\mathit{X}}_i=\mathit{x}\}\hfill \end{array}$$

$$\begin{array}{cc}& =P\{S\left({\varphi }_i\right)>\frac{k}{n}{e}^{-\ell }\}\hfill \end{array}$$

(11)

Equation (11) implies that there are at most k samples falling within the cap

$C_i$

centered at

${\mathit{X}}_i=\mathit{x}$

with area

$S_{c_i}=\frac{k}{n}{e}^{-\ell }$

.

If we let

$$p_{n,i}={\int }_{C_i}f\left(\mathbf{y}\right)d\mathbf{y}$$

and

${\mathbf{Y}}_{n,i}$

be the number of samples falling onto the cap

$C_i$

, then

${\mathbf{Y}}_{n,i}\sim BIN(n,p_{n,i})$

, is a binomial random variable. Therefore,

$$P\{L_{n,i}<\ell |{\mathit{X}}_i=\mathit{x}\}=P\{{\mathbf{Y}}_{n,i}<k\}$$

If we let

$\frac{k}{n}\to 0$

as

$n\to \infty$

, then

$p_{n,i}\to 0$

as

$n\to \infty$

. It is reasonable to consider the Poisson approximation of

${\mathbf{Y}}_{n,i}$

with mean

${\lambda }_{n,i}=n{p}_{n,i}=\frac{k{e}^{-\ell }}{S_{c_i}}{p}_{n,i}$

. Thus, the limiting distribution of

${\mathbf{Y}}_{n,i}$

is a Poisson distribution with mean:

$${\lambda }_i=\underset{n\to \infty }{lim}{\lambda }_{n,i}=k{e}^{-\ell }\underset{n\to \infty }{lim}\frac{p_{n,i}}{S_{c_i}}=k{e}^{-\ell }f\left(\mathit{x}\right)$$

(12)

Define a random variable

$L_i$

having the conditional cumulative density function,

$$F_{L_i,\mathit{x}}\left(\ell \right)=\underset{n\to \infty }{lim}P\{L_{n,i}<\ell |{\mathit{X}}_i=\mathit{x}\}$$

then

$$F_{L_i,\mathit{x}}\left(\ell \right)=\sum _{j=0}^{k-1}\frac{{\left[kf\left(\mathit{x}\right)e^{-\ell }\right]}^j}{j!}{e}^{-kf\left(\mathit{x}\right)e^{-\ell }}$$

By taking derivative w.r.t. ℓ, we obtain the conditional pdf of

$L_i$

:

$$f_{L_i,\mathit{x}}\left(\ell \right)=\frac{{\left[kf\left(\mathit{x}\right)e^{-\ell }\right]}^k}{(k-1)!}{e}^{-kf\left(\mathit{x}\right)e^{-\ell }}$$

(13)

The conditional mean of

$L_i$

is

$$E\left[L_i\right|{\mathit{X}}_i=\mathit{x}]={\int }_{-\infty }^{\infty }\ell ·\frac{{\left[kf\left(\mathit{x}\right)e^{-\ell }\right]}^k}{(k-1)!}{e}^{-kf\left(\mathit{x}\right)e^{-\ell }}d\ell$$

By change of variable,

$z=kf\left(\mathit{x}\right)e^{-\ell }$

,

$$\begin{array}{cc}\hfill E\left[L_i\right|{\mathit{X}}_i=\mathit{x}]& ={\int }_0^{\infty }[lnf\left(\mathit{x}\right)+lnk-lnz]\frac{z^{k-1}}{(k-1)!}{e}^{-z}dz\hfill \end{array}$$

$$\begin{array}{cc}& =lnf\left(\mathit{x}\right)+lnk-{\int }_0^{\infty }lnz\frac{z^{k-1}}{(k-1)!}{e}^{-z}dz\hfill \end{array}$$

$$\begin{array}{cc}& =lnf\left(\mathit{x}\right)+lnk-\psi \left(k\right)\hfill \end{array}$$

(14)

☐

Corollary 2.2. Given ${\mathit{X}}_i=\mathit{x}$, let ${\eta }_{n,k,\mathit{x}}:=nS\left({\varphi }_i\right)=k{e}^{-L_{n,i}}$, then $ln{\eta }_{n,k,\mathit{x}}=lnk-L_{n,i}$ converges in distribution to $ln{\eta }_{k,\mathit{x}}=lnk-L_i$, and

$$E[ln{\eta }_{k,\mathit{x}}]=-lnf\left(\mathit{x}\right)+\psi \left(k\right)$$

Moreover, ${\eta }_{k,\mathit{x}}$ is a gamma r.v. with the shape parameter k and the rate parameter $f\left(\mathit{x}\right)$ .

Theorem 2.3. If a pdf f satisfies the following conditions: for some

$ϵ>0$

,

$\left(A_1\right):{\int }_{{\mathbb{S}}^{p-1}}{|lnf\left(\mathit{x}\right)|}^{1+ϵ}f\left(\mathit{x}\right)d\mathit{x}<\infty$

,

$\left(A_2\right):{\int }_{{\mathbb{S}}^{p-1}}{\int }_{{\mathbb{S}}^{p-1}}{\left|ln[1-I_{{\left({\mathit{x}}^T\mathbf{y}\right)}^2}(\frac{1}{2},\frac{p-1}{2})]\right|}^{1+ϵ}f\left(\mathit{x}\right)f\left(\mathbf{y}\right)d\mathit{x}d\mathbf{y}<\infty$

, then the estimator proposed in (9) is asymptotically unbiased.

Proof. According to Corollary 2.2 and condition (

$A_2$

), we can show (see (16)–(22)) that for almost all values of

$\mathit{x}\in {\mathbb{S}}^{p-1}$

, there exists a positive constant C such that

$\left(i\right)E\left[\right|ln{\eta }_{n,k,\mathit{x}}{|}^{1+ϵ}]<C$

for all sufficiently large n.

Hence, applying the moment convergence theorem [23] (p. 186), it follows that

$$\underset{n\to \infty }{lim}E[ln{\eta }_{n,k,\mathit{x}}]=E[ln{\eta }_{k,\mathit{x}}]=-lnf\left(\mathit{x}\right)+\psi \left(k\right)$$

for almost all values of

$\mathit{x}\in {\mathbb{S}}^{p-1}$

. In addition, using Fatou's lemma and condition (

$A_1$

), we have that

$$\begin{array}{cc}& \underset{n\to \infty }{lim\; sup}{\int }_{{\mathbb{S}}^{p-1}}{\left|E\left[ln{\eta }_{n,k,\mathit{x}}\right]\right|}^{1+ϵ}f\left(\mathit{x}\right)d\mathit{x}\hfill \\ & \le {\int }_{{\mathbb{S}}^{p-1}}\underset{n\to \infty }{lim\; sup}{\left|E\left[ln{\eta }_{n,k,\mathit{x}}\right]\right|}^{1+ϵ}f\left(\mathit{x}\right)d\mathit{x}\hfill \\ & ={\int }_{{\mathbb{S}}^{p-1}}{|-lnf\left(\mathit{x}\right)+\psi \left(k\right)|}^{1+ϵ}f\left(\mathit{x}\right)d\mathit{x}\hfill \\ & \le C_{ϵ}\left({\int }_{{\mathbb{S}}^{p-1}}{|-lnf\left(\mathit{x}\right)|}^{1+ϵ}f\left(\mathit{x}\right)d\mathit{x}+{\left|\psi \left(k\right)\right|}^{1+ϵ}\right)<\infty \hfill \end{array}$$

where

$C_{ϵ}$

is a constant. Therefore,

$$\begin{array}{cc}& \underset{n\to \infty }{lim}E\left[H_n\left(f\right)\right]=\underset{n\to \infty }{lim}{E}_f[ln\left(nS\left({\varphi }_i\right)\right)]-\psi \left(k\right)\hfill \\ & =\underset{n\to \infty }{lim}{\int }_{{\mathbb{S}}^{p-1}}E\left[ln{\eta }_{n,k,\mathit{x}}\right]f\left(\mathit{x}\right)d\mathit{x}-\psi \left(k\right)\hfill \\ & ={\int }_{{\mathbb{S}}^{p-1}}\underset{n\to \infty }{lim}E\left[ln{\eta }_{n,k,\mathit{x}}\right]f\left(\mathit{x}\right)d\mathit{x}-\psi \left(k\right)\hfill \\ & ={\int }_{{\mathbb{S}}^{p-1}}E\left[ln{\eta }_{k,\mathit{x}}\right]f\left(\mathit{x}\right)d\mathit{x}-\psi \left(k\right)\hfill \\ & ={\int }_{{\mathbb{S}}^{p-1}}[-lnf\left(\mathit{x}\right)+\psi \left(k\right)]f\left(\mathit{x}\right)d\mathit{x}-\psi \left(k\right)\hfill \\ & =H\left(f\right)\hfill \end{array}$$

To show

$\left(i\right)$

, one can follow the arguments similar to those used in the proof of Theorem 1 in [24]. Indeed, we can first establish

$\left(ii\right)E\left[\right|ln{\eta }_{2,1,\mathit{x}}{|}^{1+ϵ}]<C.$

Namely, we justify that

$\left(i\right)$

is valid when

$n=2$

and

$k=1$

. But the inequality

$\left(ii\right)$

follows immediately from the condition

$\left(A_2\right)$

and

$$\begin{array}{cc}\hfill E\left[{\left|ln{\eta }_{2,1,\mathit{x}}\right|}^{1+ϵ}\right]& =E\left[{\left|ln\left[2S\left({\varphi }_{1,2}\right)\right]\right|}^{1+ϵ}|{\mathit{X}}_1=\mathit{x}\right]\hfill \end{array}$$

$$\begin{array}{cc}& =E\left[{\left|ln\left(S_p\left[1-sgn\left({\mathit{x}}^T{\mathit{X}}_2\right)I_{{\left({\mathit{x}}^T{\mathit{X}}_2\right)}^2}(\frac{1}{2},\frac{p-1}{2})\right]\right)\right|}^{1+ϵ}\right]\hfill \end{array}$$

$$\begin{array}{cc}& \le C_{ϵ}{|ln{S}_p|}^{1+ϵ}+C_{ϵ}{|ln2|}^{1+ϵ}+C_{ϵ}{E}_f\left[{\left|ln\left[1-I_{{\left({\mathit{x}}^T{\mathit{X}}_2\right)}^2}(\frac{1}{2},\frac{p-1}{2})\right]\right|}^{1+ϵ}\mathbf{1}({\mathit{x}}^T{\mathit{X}}_2>0)\right]\hfill \end{array}$$

$$\begin{array}{cc}& =C_{ϵ}\left({|ln{S}_p|}^{1+ϵ}+{|ln2|}^{1+ϵ}\right)+\frac{1}{2}{C}_{ϵ}{E}_f\left[{\left|ln\left[1-I_{{\left({\mathit{x}}^T{\mathit{X}}_2\right)}^2}(\frac{1}{2},\frac{p-1}{2})\right]\right|}^{1+ϵ}\right]\hfill \end{array}$$

(15)

Here

${\varphi }_{1,2}=arccos\left({\mathit{X}}_1^T{\mathit{X}}_2\right)$

and

$\mathbf{1}(·)$

is the indicator function.

Now let us denote the distribution function of

${\eta }_{n,k,\mathit{x}}$

by

$$\begin{array}{cc}& G_{n,k,\mathit{x}}\left(u\right)=P({\eta }_{n,k,\mathit{x}}\le u)=P(nS\left({\varphi }_{n,k,1}\right)\le u|{\mathit{X}}_1=\mathit{x})\hfill \\ & =1-\sum _{j=0}^{k-1}\left(\genfrac{}{}{0pt}{}{n-1}{j}\right){\left({\int }_{C_{\mathit{x}}\left({\varphi }_n\left(u\right)\right)}f\left(\mathbf{y}\right)d\mathbf{y}\right)}^j{\left(1-{\int }_{C_{\mathit{x}}\left({\varphi }_n\left(u\right)\right)}f\left(\mathbf{y}\right)d\mathbf{y}\right)}^{n-1-j}\hfill \end{array}$$

where

${\varphi }_n\left(u\right)=S^{-1}\left(\frac{u}{n}\right)$

and

$C_{\mathit{x}}\left(\varphi \right)$

is a cap

$\{\mathbf{y}\in {\mathbb{S}}^{p-1}:{\mathbf{y}}^T\mathit{x}\ge cos\varphi \}$

with the pole x and base radius

$sin\varphi$

. Note also that the functions

$S\left(\varphi \right)$

(see (3)) and

${\varphi }_n\left(u\right)=S^{-1}\left(\frac{u}{n}\right)$

are both increasing functions.

Now, one can see (cf. (66) in [24]):

$$\begin{array}{cc}& E\left[\right|ln{\eta }_{n,k,\mathit{x}}{|}^{1+ϵ}]\le I_1+I_2+I_3\hfill \end{array}$$

(16)

where

$$\begin{array}{cc}& I_1=(1+ϵ){\int }_0^1{\left(ln\frac{1}{u}\right)}^{ϵ}{u}^{-1}{G}_{n,k,\mathit{x}}\left(u\right)du\hfill \end{array}$$

$$\begin{array}{cc}& I_2=(1+ϵ){\int }_1^{\sqrt{n}}{(lnu)}^{ϵ}{u}^{-1}(1-G_{n,k,\mathit{x}}\left(u\right))du\hfill \end{array}$$

$$\begin{array}{cc}& I_3=(1+ϵ){\int }_{\sqrt{n}}^{n{S}_p}{(lnu)}^{ϵ}{u}^{-1}(1-G_{n,k,\mathit{x}}\left(u\right))du\hfill \end{array}$$

It is easy to see that for sufficiently large n and almost all

$\mathit{x}\in {\mathbb{S}}^{p-1}$

:

$$\begin{array}{cc}& I_1<(1+ϵ)f\left(\mathit{x}\right)\Gamma (1+ϵ)<\infty \hfill \end{array}$$

(17)

and

$$\begin{array}{cc}& I_2\le (1+ϵ)\sum _{j=0}^{k-1}{\left[\underset{\mathbf{y}\in {\mathbb{S}}^{p-1}}{sup}f\left(\mathbf{y}\right)\right]}^{j}f{\left(\mathit{x}\right)}^{-j-ϵ}\Gamma (j+ϵ)<\infty \hfill \end{array}$$

(18)

(cf. (89) and (85) in [24], respectively).

Finally, let us show that

$I_3\to 0$

as

$n\to \infty$

. For each x with

$f\left(\mathit{x}\right)>0$

, if we choose a

$\delta \in (0,f(\mathit{x}\left)\right)$

, then for all sufficiently large n,

$\sqrt{n}{\int }_{C_{\mathit{x}}\left({\varphi }_n\left(\sqrt{n}\right)\right)}f\left(\mathbf{y}\right)d\mathbf{y}>f\left(\mathit{x}\right)-\delta$

, since the area of

$C_{\mathit{x}}\left({\varphi }_n\left(\sqrt{n}\right)\right)$

is equal to

$\frac{1}{\sqrt{n}}$

. Using arguments similar to those used in (69)–(72) from [24], we have

$$\begin{array}{cc}\hfill I_3\le & (1+ϵ)n^{k-1}k{e}^{-(n-k-1)(f\left(\mathit{x}\right)-\delta )\frac{1}{\sqrt{n}}}\hfill \end{array}$$

$$\begin{array}{cc}& \times {\int }_{\sqrt{n}}^{n{S}_p}{(lnu)}^{ϵ}{u}^{-1}\left(1-{\int }_{C_{\mathit{x}}\left({\varphi }_n\left(u\right)\right)}f\left(\mathbf{y}\right)d\mathbf{y}\right)du\hfill \end{array}$$

(19)

The integral in (19) after changing the variable,

$t=\frac{2u}{n}$

, takes the form

$$\begin{array}{cc}& {\int }_{\frac{2}{\sqrt{n}}}^{2S_p}{\left(ln\frac{nt}{2}\right)}^{ϵ}{t}^{-1}(1-G_{2,1,\mathit{x}}\left(t\right))dt\hfill \end{array}$$

$$\begin{array}{cc}& =\left({\int }_{\frac{2}{\sqrt{n}}}^1+{\int }_1^{2S_p}\right){\left(ln\frac{nt}{2}\right)}^{ϵ}{t}^{-1}(1-G_{2,1,\mathit{x}}\left(t\right))dt\hfill \end{array}$$

(20)

since

${\varphi }_n\left(\frac{nt}{2}\right)=S^{-1}\left(\frac{t}{2}\right)={\varphi }_2\left(t\right)$

and

$1-{\int }_{C_{\mathit{x}}\left({\varphi }_2\left(t\right)\right)}f\left(\mathbf{y}\right)d\mathbf{y}=1-G_{2,1,\mathit{x}}\left(t\right))$

. The first integral in the right side of (20) is bounded as follows:

$${\int }_{\frac{2}{\sqrt{n}}}^1{\left(ln\frac{nt}{2}\right)}^{ϵ}{t}^{-1}(1-G_{2,1,\mathit{x}}\left(t\right))dt\le \frac{\sqrt{n}}{2}{\left(ln\frac{n}{2}\right)}^{ϵ}$$

(21)

while for the second one, we have

$${\int }_1^{2S_p}{\left(ln\frac{nt}{2}\right)}^{ϵ}{t}^{-1}(1-G_{2,1,\mathit{x}}\left(t\right))dt\le C_{ϵ}{\left(ln\frac{n}{2}\right)}^{ϵ}E\left[{\eta }_{2,1,\mathit{x}}\right]+C_{ϵ}{\left(ln\frac{n}{2}\right)}^{ϵ}B$$

(22)

where

$$B={\int }_1^{2S_p}{(lnt)}^{ϵ}{t}^{-1}(1-G_{2,1,\mathit{x}}\left(t\right))dt=\frac{1}{1+ϵ}E{|ln{\eta }_{2,1,\mathit{x}}|}^{1+ϵ}$$

Combination of (15)–(22) and

$\left(ii\right)$

yields

$\left(i\right)$

.

Remark. Note that

$$\begin{array}{cc}\hfill 1-I_{t^2}(\frac{1}{2},\frac{p-1}{2})& \approx \frac{1}{B(\frac{1}{2},\frac{p-1}{2})}{\left(t^2\right)}^{-\frac{1}{2}}{(1-t^2)}^{\frac{p-1}{2}}\hfill \\ & \approx \frac{2^{\frac{p-1}{2}}}{B(\frac{1}{2},\frac{p-1}{2})}{(1-t)}^{\frac{p-1}{2}}\text{as}t\uparrow 1\hfill \end{array}$$

where

$B(·,·)$

is the beta function. Hence, in the conditions

$\left(A_j\right),j=2$

, 4, 6 and 8, the difference

$1-I_{{\left({\mathit{x}}^T\mathbf{y}\right)}^2}(\frac{1}{2},\frac{p-1}{2})$

can be replaced by

$1-{\mathit{x}}^T\mathbf{y}$

.

2.2. Consistency of $H_n$

Lemma 2.4. Under the following conditions: for some

$ϵ>0$

,

$\left(A_3\right):{\int }_{{\mathbb{S}}^{p-1}}{|lnf\left(\mathit{x}\right)|}^{2+ϵ}f\left(\mathit{x}\right)d\mathit{x}<\infty$

,

$\left(A_4\right):{\int }_{{\mathbb{S}}^{p-1}}{\int }_{{\mathbb{S}}^{p-1}}{\left|ln[1-I_{{\left({\mathit{x}}^T\mathbf{y}\right)}^2}(\frac{1}{2},\frac{p-1}{2})]\right|}^{2+ϵ}f\left(\mathit{x}\right)f\left(\mathbf{y}\right)d\mathit{x}d\mathbf{y}<\infty$

,

the asymptotic variance of $L_{n,i}$ is finite and equals $V_f[lnf\left(\mathit{X}\right)]+{\psi }_1\left(k\right)$, where ${\psi }_1\left(k\right)$ is the trigamma function.

Proof. The conditions

$\left(A_3\right)$

and

$\left(A_4\right)$

, and the argument similar to the one used in the proof of Theorem 2.3, yields

$$\underset{n\to \infty }{lim}E\left[L_{n,i}^2\right|{\mathit{X}}_i=\mathit{x}]=E\left[L_i^2\right|{\mathit{X}}_i=\mathit{x}]$$

Therefore, it is sufficient to prove that

$V_f\left[L_i\right]=V_f(lnf\left(\mathit{X}\right))+{\psi }_1\left(k\right)$

. Similarly to (14), we have

$$\begin{array}{cc}\hfill E\left[L_i^2\right|{\mathit{X}}_i=\mathit{x}]& ={\int }_0^{\infty }{[lnf\left(\mathit{x}\right)+lnk-lnz]}^2\frac{z^{k-1}}{(k-1)!}{e}^{-z}dz\hfill \end{array}$$

$$\begin{array}{cc}& ={[lnf\left(\mathit{x}\right)+lnk]}^2-2[lnf\left(\mathit{x}\right)+lnk]\psi \left(k\right)+{\Gamma }^{\prime \prime }\left(k\right)/\Gamma \left(k\right)\hfill \end{array}$$

(23)

Since

${\Gamma }^{\prime \prime }\left(k\right)/\Gamma \left(k\right)={\psi }^2\left(k\right)+{\psi }_1\left(k\right)$

,

$$\begin{array}{cc}\hfill E\left[L_i^2\right|{\mathit{X}}_i=\mathit{x}]& ={[lnf\left(\mathit{x}\right)+lnk-\psi \left(k\right)]}^2+{\psi }_1\left(k\right)\hfill \end{array}$$

(24)

After some algebra, it can be shown that

$$\begin{array}{cc}\hfill V_f\left[L_i\right]& =E_f\left[{(lnf\left(\mathit{X}\right))}^2\right]-{\left(E_f[lnf\left(\mathit{X}\right)]\right)}^2+{\psi }_1\left(k\right)\hfill \end{array}$$

$$\begin{array}{cc}& =V_f[lnf\left(\mathit{X}\right)]+{\psi }_1\left(k\right)\hfill \end{array}$$

(25)

☐

Lemma 2.5. For a fixed integer $k<n$, $L_{n,i}$ are asymptotically pairwise independent.

Proof. For a pair of random variables

$L_{n,i}$

and

$L_{n,j}$

with

$i\ne j$

and

${\mathit{X}}_i\ne {\mathit{X}}_j$

, following the similar argument for Lemma 2.1,

$C_i$

and

$C_j$

shrink as n increases. Thus, it is safe to assume that

$C_i$

and

$C_j$

are disjoint for large n, and

$L_{n,i}$

and

$L_{n,j}$

are independent. Hence Lemma 2.5 follows. ☐

Theorem 2.6. Under the conditions $\left(A_1\right)$ through $\left(A_4\right)$, the variance of $H_n\left(f\right)$ decreases with sample size n, that is

$$\underset{n\to \infty }{lim}{V}_f\left[H_n\left(f\right)\right]=0$$

(26)

and $H_n\left(f\right)$ is a consistent estimator of $H\left(f\right)$.

Theorem 2.6 can be established by using Theorem 2.3 and Lemmas 2.4 and 2.5, and

$$\underset{n\to \infty }{lim}{V}_f\left[H_n\left(f\right)\right]=\underset{n\to \infty }{lim}\frac{1}{n}\{V_f[lnf\left(\mathit{X}\right)]+{\psi }_1\left(k\right)\}=0$$

For a finite sample, the variance of

$H_n\left(f\right)$

can be approximated by

$\frac{1}{n}\{V_f[lnf\left(\mathit{x}\right)]+{\psi }_1\left(k\right)\}$

. For instance, for the uniform distribution,

$V_f[lnf\left(\mathit{x}\right)]=0$

and

$V\left[H_n\left(f\right)\right]\approx {\psi }_1\left(k\right)/n$

and for a vMF

${}_p(\mathbf{\mu },\kappa )$

,

$V\left[H_n\left(f\right)\right]\approx \frac{1}{n}[{\kappa }^2{A}_p^{\prime }\left(\kappa \right)+{\psi }_1\left(k\right)]$

. See the illustration in Figure 1. The simulation was done with sample size

$n=1000$

and the number of simulations was

$N=10,000$

. Since

${\psi }_1\left(k\right)$

is a decreasing function, the variance of

$H_n\left(f\right)$

decreases when k increases.

Figure 1. Variances of

$H_n\left(f\right)$

by simulation and approximation.

Figure 1. Variances of

$H_n\left(f\right)$

by simulation and approximation.

3. Estimation of Cross Entropy and KL-divergence

3.1. Estimation of Cross Entropy

The definition of cross entropy between continuous pdf's f and g is,

$$H(f,g)=-\int f\left(\mathit{x}\right)lng\left(\mathit{x}\right)d\mathit{x}$$

(27)

Given a random sample of size n from f, {

${\mathit{X}}_1,{\mathit{X}}_2,\dots ,{\mathit{X}}_n$

}, and a random sample of size m from g, {

${\mathbf{Y}}_1,{\mathbf{Y}}_2,\dots ,{\mathbf{Y}}_m$

}, on a hypersphere, denote the knn density estimator of g by

$g_m$

. Similarly to (7),

$$g_m\left({\mathit{X}}_i\right)=\frac{k/m}{S\left({\phi }_i\right)}$$

(28)

where

${\phi }_i$

is the distance from

${\mathit{X}}_i$

to its k-th nearest neighbor in {

${\mathbf{Y}}_1,{\mathbf{Y}}_2,\dots ,{\mathbf{Y}}_m$

}. Analogously to the entropy estimator (9), the cross entropy can be estimated by:

$$H_{n,m}(f,g)=\frac{1}{n}\sum _{i=1}^{n}lnS\left({\phi }_i\right)+lnm-\psi \left(k\right)$$

(29)

Under the conditions

$\left(A_1\right)$

–

$\left(A_4\right)$

, for a fixed integer

$k<min(n,m)$

, one can show that

$H_{n,m}(f,g)$

is asymptotically unbiased. Moreover, by similar reasoning applied for

$H_n\left(f\right)$

, one can show that

$H_{n,m}(f,g)$

is also consistent and

$V\left[H_{n,m}(f,g)\right]\approx \frac{1}{n}\{V_f[lng\left(\mathit{x}\right)]+{\psi }_1\left(k\right)\}$

. For example, when both f and g are vMF with the same mean direction and different concentration parameters,

${\kappa }_1$

and

${\kappa }_2$

, respectively, the approximate variance will be

$\frac{1}{n}[{\kappa }_2^2{A}_p^{\prime }\left({\kappa }_1\right)+{\psi }_1\left(k\right)]$

. Figure 2 shows the approximated and simulated variance of the knn estimators for cross entropy are close to each other and both decrease with k. The simulation is done with sample size

$n=m=1000$

and the number of simulations was

$N=10,000$

.

Figure 2. Variances of

$H_{n,m}$

by simulation and approximation.

Figure 2. Variances of

$H_{n,m}$

by simulation and approximation.

3.2. Estimation of KL-Divergence

KL-divergence is also known as relative entropy. It is used to measure the similarity of two distributions. Wang et al. [24] studied the knn estimator of KL-divergence for distributions defined on

${\mathbb{R}}^p$

. Here we propose the knn estimator of KL-divergence of continuous distribution f from g defined on a hypersphere. The KL-divergence is defined as:

$$KL(f\parallel g)=E_f[lnf\left(\mathit{X}\right)/g\left(\mathit{X}\right)]=\int f\left(\mathit{x}\right)ln\frac{f\left(\mathit{x}\right)}{g\left(\mathit{x}\right)}d\mathit{x}$$

(30)

Equation (30) can also be expressed as

$KL(f\parallel g)=H(f,g)-H\left(f\right)$

. Then the knn estimator of KL-divergence is constructed as

$H_{n,m}(f,g)-H_n\left(f\right)$

, i.e.,

$$K{L}_{n,m}(f\parallel g)=\frac{1}{n}\sum _{i=1}^{n}ln\frac{f_n\left({\mathit{X}}_i\right)}{g_m\left({\mathit{X}}_i\right)}=\frac{1}{n}\sum _{i=1}^{n}ln\frac{S\left({\phi }_i\right)}{S\left({\varphi }_i\right)}+ln\frac{m}{n}$$

(31)

where

$g_m\left({\mathit{X}}_i\right)$

is defined as in (28). Besides, for finite samples, the variance of the estimator,

$V\left[K{L}_{n,m}\right]$

, is approximately

$\frac{1}{n}\{V_f[lnf\left(\mathit{X}\right)]+V_f[lng\left(\mathit{X}\right)]-2{\mathrm{Cov}}_f[lnf\left(\mathit{X}\right),lng\left(\mathit{X}\right)]+2{\psi }_1\left(k\right)\}$

. When f and g are vMF as mentioned above, with concentration parameter

${\kappa }_1$

and

${\kappa }_2$

, respectively, we have:

$$\begin{array}{cc}& V_f[lnf\left(\mathit{X}\right)]={\kappa }_1^2{A}_p^{\prime }\left({\kappa }_1\right)\hfill \\ & V_f[lng\left(\mathit{X}\right)]={\kappa }_2^2{A}_p^{\prime }\left({\kappa }_1\right)\hfill \end{array}$$

and

$$\begin{array}{cc}& {\mathrm{Cov}}_f[lnf\left(\mathit{X}\right),lng\left(\mathit{X}\right)]={\kappa }_1{\kappa }_2{A}_p^{\prime }\left({\kappa }_1\right)\hfill \end{array}$$

So the approximate variance is

$\frac{1}{n}[{({\kappa }_1-{\kappa }_2)}^2{A}_p^{\prime }\left({\kappa }_1\right)+2{\psi }_1\left(k\right)]$

. Figure 3 shows the approximated and simulated variance of the knn estimators for KL-divergence. The approximation for von Mises-Fisher distribution is not as good as the one for uniform distributions. This could be due to the modality of von Mises-Fisher distributions or the finitude of sample sizes. The larger the sample size, the closer the approximation is to the true value.

Figure 3. Variances of

$K{L}_{n,m}$

by simulation and approximation.

Figure 3. Variances of

$K{L}_{n,m}$

by simulation and approximation.

In summary, we have

Corollary 3.1. (1) Under conditions $\left(A_1\right),\left(A_2\right)$ and for some $ϵ>0$,

$\left(A_5\right):{\int }_{{\mathbb{S}}^{p-1}}{|lng\left(\mathit{x}\right)|}^{1+ϵ}f\left(\mathit{x}\right)d\mathit{x}<\infty$

,

$\left(A_6\right):{\int }_{{\mathbb{S}}^{p-1}}{\int }_{{\mathbb{S}}^{p-1}}{\left|ln[1-I_{{\left({\mathit{x}}^T\mathbf{y}\right)}^2}(\frac{1}{2},\frac{p-1}{2})]\right|}^{1+ϵ}f\left(\mathit{x}\right)g\left(\mathbf{y}\right)d\mathit{x}d\mathbf{y}<\infty$

,

for a fixed integer $k<min(n,m)$, the knn estimator of KL-divergence given in (31) is asymptotically unbiased.

(2) Under condition $\left(A_3\right),\left(A_4\right)$ and for some $ϵ>0$,

$\left(A_7\right):{\int }_{{\mathbb{S}}^{p-1}}{|lng\left(\mathit{x}\right)|}^{2+ϵ}f\left(\mathit{x}\right)d\mathit{x}<\infty$

,

$\left(A_8\right):{\int }_{{\mathbb{S}}^{p-1}}{\int }_{{\mathbb{S}}^{p-1}}{\left|ln[1-I_{{\left({\mathit{x}}^T\mathbf{y}\right)}^2}(\frac{1}{2},\frac{p-1}{2})]\right|}^{2+ϵ}f\left(\mathit{x}\right)g\left(\mathbf{y}\right)d\mathit{x}d\mathbf{y}<\infty$

,

for a fixed integer $k<min(n,m)$, the knn estimator of KL-divergence given in (31) is asymptotically consistent.

To prove the last two corollaries, one can follow the similar steps proposed in Wang et al. [24].

4. Simulation Study

To demonstrate the proposed knn entropy estimators and assess their performance for finite samples, we conducted simulations for the uniform distribution and von Mises-Fisher distributions with the p-coordinate unit vector,

${\mathbf{e}}_p$

, as the common mean direction for

$p=3$

and 10. For each distribution, we drew samples of size

$n=100$

, 500 and 1000. All simulations were repeated

$N=10,000$

times. Bias, standard deviation (SD) and root mean squared error (RMSE) were calculated.

4.1. Bias and Standard Deviation

Figure 4, Figure 5, Figure 6, Figure 7, Figure 8 and Figure 9 show simulated bias and standard deviation of the proposed entropy, cross-entropy and KL-divergence estimators along different k. The pattern for the standard deviation is clear. It decreases sharply then slowly as k increases. This is consistent with the variance approximations described in Section 2 and Section 3. The pattern for bias is diverse. For uniform distributions, the bias term is very small. When the underlying distribution has a mode, for example, vMF models used in the current simulations, the relation between bias and k becomes complex and the bias term can be larger for larger k values.

Figure 4.

$\left|\mathrm{Bias}\right|$

(dashed line) and standard deviation (solid line) of entropy estimate

$H_n$

for uniform distributions.

Figure 4.

$\left|\mathrm{Bias}\right|$

(dashed line) and standard deviation (solid line) of entropy estimate

$H_n$

for uniform distributions.

Figure 5.

$\left|\mathrm{Bias}\right|$

(dashed line) and standard deviation (solid line) of entropy estimate

$H_n$

for vMF

${}_p({\mathbf{e}}_p,1)$

distributions.

Figure 5.

$\left|\mathrm{Bias}\right|$

(dashed line) and standard deviation (solid line) of entropy estimate

$H_n$

for vMF

${}_p({\mathbf{e}}_p,1)$

distributions.

Figure 6.

$\left|\mathrm{Bias}\right|$

(dashed line) and standard deviation (solid line) of cross entropy estimate

$H_{n,m}$

for uniform distributions.

Figure 6.

$\left|\mathrm{Bias}\right|$

(dashed line) and standard deviation (solid line) of cross entropy estimate

$H_{n,m}$

for uniform distributions.

Figure 7.

$\left|\mathrm{Bias}\right|$

(dashed line) and standard deviation (solid line) of cross entropy estimate

$H_{n,m}$

for

$f={\mathrm{vMF}}_p({\mathbf{e}}_p,1)$

and

$g=$

uniform distributions.

Figure 7.

$\left|\mathrm{Bias}\right|$

(dashed line) and standard deviation (solid line) of cross entropy estimate

$H_{n,m}$

for

$f={\mathrm{vMF}}_p({\mathbf{e}}_p,1)$

and

$g=$

uniform distributions.

Figure 8.

$\left|\mathrm{Bias}\right|$

(dashed line) and standard deviation (solid line) of KL-divergence estimate

$K{L}_{n,m}$

for uniform distributions.

Figure 8.

$\left|\mathrm{Bias}\right|$

(dashed line) and standard deviation (solid line) of KL-divergence estimate

$K{L}_{n,m}$

for uniform distributions.

Figure 9.

$\left|\mathrm{Bias}\right|$

(dashed line) and standard deviation (solid line) of KL-divergence estimate

$K{L}_{n,m}$

for

$f={\mathrm{vMF}}_p({\mathbf{e}}_p,1)$

and

$g=$

uniform distributions.

Figure 9.

$\left|\mathrm{Bias}\right|$

(dashed line) and standard deviation (solid line) of KL-divergence estimate

$K{L}_{n,m}$

for

$f={\mathrm{vMF}}_p({\mathbf{e}}_p,1)$

and

$g=$

uniform distributions.

4.2. Convergence

To validate the consistency, we conducted simulations of different sample size n from 10 to 100,000 for the distribution models used above. Figure 10 and Figure 11 shows the estimates and theoretical values of entropy, cross-entropy and KL-divergence for different sample sizes with

$k=1$

and

$k=\lfloor lnn+0.5\rfloor =$

2–12, respectively. The proposed estimators converge to the corresponding theoretical values quickly. Thus the consistency of these estimators are verified. The choice of k is an open problem for knn based estimation approaches. These figures show that using lager k, e.g., the logarithm of n, for lager n, is giving a slightly better preference.

Figure 10. Convergence of estimates with sample size n using the first nearest neighbor. For vMF

${}_p$

,

$\kappa =1$

.

Figure 10. Convergence of estimates with sample size n using the first nearest neighbor. For vMF

${}_p$

,

$\kappa =1$

.

Figure 11. Convergence of estimates with sample size n using

$k=\lfloor lnn+0.5\rfloor$

nearest neighbors. For vMF

${}_p$

,

$\kappa =1$

.

Figure 11. Convergence of estimates with sample size n using

$k=\lfloor lnn+0.5\rfloor$

nearest neighbors. For vMF

${}_p$

,

$\kappa =1$

.

4.3. Comparison with the Moment-Recovered Construction

Another entropy estimator for hyperspherical data was developed recently by Mnatsakanov et al. [10] using MR approach. We call this estimator the MR entropy estimator and denote it by

$H_n^{\left(MR\right)}\left(f\right)$

:

$$H_n^{\left(MR\right)}\left(f\right)=-\frac{1}{n}\sum _{i=1}^{n}ln{P}_{n,t}\left({\mathit{X}}_i\right)+lnS(arccost)$$

(32)

where

$P_{n,t}\left({\mathit{X}}_i\right)$

is the estimated probability of the cap

$\{\mathbf{y}\in {\mathbb{S}}^{p-1}:{\mathbf{y}}^T{\mathit{X}}_i\ge t\}$

defined by the revolution axis

${\mathit{X}}_i$

and t is the distance from the cap base to the origin and acts as a tuning parameter. Namely, (see Mnatsakanov et al. [10]),

$$P_{n,t}\left({\mathit{X}}_i\right)=\frac{1}{n-1}\sum _{j=1,j\ne i}^n\sum _{k=\lfloor nt\rfloor +1}^n\left(\genfrac{}{}{0pt}{}{n}{k}\right){\left({\mathit{X}}_j^T{\mathit{X}}_i\right)}^k{(1-{\mathit{X}}_j^T{\mathit{X}}_i)}^{n-k}$$

(33)

Via simulation study, the empirical comparison between

$H_n\left(f\right)$

and

$H_n^{\left(MR\right)}\left(f\right)$

was done for the uniform and vMF distributions. The results are presented in Table 1. The values of k and t listed in the table are the optimal ones in the sense of minimizing RMSE. Z-tests and F-tests (at

$\alpha =0.05$

) were performed to compare the bias, standard deviation (variance) and RMSE (MSE) between the knn estimators and corresponding MR estimators. In general, for uniform distributions, there are no significant difference for biases. Among other comparisons, the differences are significant. Specifically, knn achieves slightly smaller bias and RMSE values than those of the MR method. The standard deviations of knn method are also smaller for the uniform distribution but larger for vMF distributions than those based on MR approach.

Table 1. Comparison of knn and moment methods by simulations for spherical distributions.

Table 1. Comparison of knn and moment methods by simulations for spherical distributions.

Method		knn				MR
p	n	k	bias	SD	RMSE	t	bias	SD	RMSE
Uniform:
3	100	99	0.00500	0.00147	0.00521	0.01	0.00523	0.01188	0.01298
3	500	499	0.00100	0.00013	0.00101	0.01	0.00107	0.00233	0.00257
3	1000	999	0.00050	0.00005	0.00050	0.01	0.00051	0.00120	0.00130
10	100	99	0.00503	0.00130	0.00520	0.01	0.00528	0.01331	0.01432
10	500	499	0.00100	0.00011	0.00101	0.01	0.00102	0.00264	0.00283
10	1000	999	0.00050	0.00004	0.00050	0.01	0.00052	0.00130	0.00140
vMF${}_p(e_p,1)$:
3	100	71	0.01697	0.05142	0.05415	0.30	0.02929	0.04702	0.05540
3	500	337	0.00310	0.02336	0.02356	0.66	0.00969	0.02318	0.02512
3	1000	670	0.00145	0.01662	0.01668	0.74	0.00620	0.01658	0.01770
10	100	46	0.02395	0.02567	0.03511	0.12	0.02895	0.02363	0.03737
10	500	76	0.00702	0.01361	0.01531	0.40	0.01407	0.01247	0.01881
10	1000	90	0.00366	0.01026	0.01089	0.47	0.01115	0.00907	0.01437

5. Discussion and Conclusions

In this paper, the knn based estimators for entropy, cross-entropy and Kullback-Leibler divergence are proposed for distributions on hyperspheres. Asymptotic properties such as unbiasedness and consistency are proved and validated by simulation studies using uniform and von Mises-Fisher distribution models. The variances of these estimators decrease with k. For uniform distributions, variance is dominant and bias is negligible. When the underlying distributions are modal, the bias can be large if k is large. In general, we conclude that the behavior of knn and MR entropy estimators have similar performance in terms of root mean square error.

Acknowledgements and Disclaimer

The authors thank the anonymous referees for their helpful comments and suggestions. The research of Robert Mnatsakanov was supported by NSF grant DMS-0906639. The findings and conclusions in this report are those of the author(s) and do not necessarily represent the views of the National Institute for Occupational Safety and Health.

austinhistalle.blogspot.com

Source: https://www.mdpi.com/1099-4300/13/3/650/htm

Share this post