Sự khác biệt giữa khoảng cách Bhattacharyya và phân kỳ KL

33

Tôi đang tìm kiếm một lời giải thích trực quan cho các câu hỏi sau:

Trong thống kê và lý thuyết thông tin, sự khác biệt giữa khoảng cách Bhattacharyya và phân kỳ KL, là thước đo của sự khác biệt giữa hai phân phối xác suất rời rạc?

Họ có hoàn toàn không có mối quan hệ và đo khoảng cách giữa hai phân phối xác suất theo cách hoàn toàn khác nhau?

— Viên ngọc
nguồn

36

Các hệ số Bhattacharyya được định nghĩa là

D_{B} (p, q) = \int \sqrt{p (x) q (x)} d x

$D_B(p,q) = \int \sqrt{p(x)q(x)}\,\text{d}x$

d_{H} (p, q)

$d_H(p,q)$

d_{H} (p, q) = = {1 - D_{B} (p, q)}^{1 / 2}

$d_H(p,q)=\{1-D_B(p,q)\}^{1/2}$

d_{K L} (p ‖ q) \geq 2 d_{H}^{2} (p, q) = 2 {1 - D_{B} (p, q)} .

$d_{KL}(p\|q) \geq 2 d_H^2(p,q) = 2 \{1-D_B(p,q)\}\,.$

Tuy nhiên, đây không phải là câu hỏi: nếu khoảng cách Bhattacharyya được xác định là sau đó

d_{B} (p, q) \overset{def}{=} - \log D_{B} (p, q),

$d_B(p,q)\stackrel{\text{def}}{=}-\log D_B(p,q)\,,$

\begin{aligned} d_{B} (p, q) = - \log D_{B} (p, q) & = - \log \int \sqrt{p (x) q (x)} d x \\ \overset{def}{=} - \log \int h (x) d x \\ = - \log \int \frac{h (x)}{p (x)} p (x) d x \\ \leq \int - \log {\frac{h (x)}{p (x)}} p (x) d x \\ = \int \frac{- 1}{2} \log {\frac{h^{2} (x)}{p^{2} (x)}} p (x) d x \\ = \int \frac{- 1}{2} \log {\frac{q (x)}{p (x)}} p (x) d x = \frac{1}{2} d_{K L} (p ‖ q) \end{aligned}

$\begin{align*}d_B(p,q)=-\log D_B(p,q)&=-\log \int \sqrt{p(x)q(x)}\,\text{d}x\\ &\stackrel{\text{def}}{=}-\log \int h(x)\,\text{d}x\\ &= -\log \int \frac{h(x)}{p(x)}\,p(x)\,\text{d}x\\ &\le \int -\log \left\{\frac{h(x)}{p(x)}\right\}\,p(x)\,\text{d}x\\ &= \int \frac{-1}{2}\log \left\{\frac{h^2(x)}{p^2(x)}\right\}\,p(x)\,\text{d}x\\ &= \int \frac{-1}{2}\log \left\{\frac{q(x)}{p(x)}\right\}\,p(x)\,\text{d}x= \frac{1}{2}d_{KL}(p\|q) \end{align*}$

d_{K L} (p ‖ q) \geq 2 d_{B} (p, q) .

${d_{KL}(p\|q)\ge 2d_B(p,q)\,.}$

- l o g (x) \geq 1 - x 0 \leq x \leq 1,

$-log(x)\ge 1-x\qquad\qquad 0\le x\le 1\,,$ enter image description here

we have the complete ordering

d_{K L} (p ‖ q) \geq 2 d_{B} (p, q) \geq 2 d_{H} (p, q)^{2} .

${d_{KL}(p\|q)\ge 2d_B(p,q)\ge 2d_H(p,q)^2\,.}$

— Xi'an
nguồn

2

Brilliant! This explanation should be the one I am looking for eagerly. Just one last question: in what case (or what kinds of P and Q) will the inequality becomes equality?

— JewelSue

1

Given that the

- \log (\cdot)

$-\log(\cdot)$ function is strictly convex, I would assume the only case for equality is when the ratio

p (x) / q (x)

$p(x)/q(x)$ is constant in

x

$x$ .

— Xi'an

5

And the only case when

p (x) / q (x)

$p(x)/q(x)$ is constant in

x

$x$ is when

p = q

$p=q$ .

— Xi'an

8

I don't know of any explicit relation between the two, but decided to have a quick poke at them to see what I could find. So this isn't much of an answer, but more of a point of interest.

For simplicity, let's work over discrete distributions. We can write the BC distance as

d_{BC} (p, q) = - \ln \sum_{x} (p (x) q (x))^{\frac{1}{2}}

$d_\text{BC}(p,q) = - \ln \sum_x (p(x)q(x))^\frac{1}{2}$

and the KL divergence as

d_{KL} (p, q) = \sum_{x} p (x) \ln \frac{p (x)}{q (x)}

$d_\text{KL}(p,q) = \sum_x p(x)\ln \frac{p(x)}{q(x)}$

Now we can't push the log inside the sum on the $\text{BC}$ distance, so let's try pulling the log to the outside of the $\text{KL}$ divergence:

d_{KL} (p, q) = - \ln \prod_{x} {(\frac{q (x)}{p (x)})}^{p (x)}

$d_\text{KL}(p,q) = -\ln \prod_x \left( \frac{q(x)}{p(x)} \right)^{p(x)}$

Let's consider their behaviour when $p$ is fixed to be the uniform distribution over $n$ possibilities:

d_{KL} (p, q) = - \ln n - \ln {(\prod_{x} q (x))}^{\frac{1}{n}} d_{BC} (p, q) = - \ln \frac{1}{\sqrt{n}} - \ln \sum_{x} \sqrt{q (x)}

$d_\text{KL}(p,q) = -\ln n - \ln \left(\prod_x q(x)\right)^\frac{1}{n} \qquad d_\text{BC}(p,q) = - \ln \frac{1}{\sqrt{n}} - \ln\sum_x \sqrt{q(x)}$

On the left, we have the log of something that's similar in form to the geometric mean. On the right, we have something similar to the log of the arithmetic mean. Like I said, this isn't much of an answer, but I think it gives a neat intuition of how the BC distance and the KL divergence react to deviations between $p$ and $q$ .

— Andy Jones
nguồn