Question

How do to find the values of p and q so that we get the maximum entropy if I have this equation?

Answer 1

\[X = -p*\log_2 (p)-q \log_2{q} - (1-p-q) \log_2 (1-p-q)\]

Answer 2

that's the expression for the first order base 2 entropy

Answer 3

well, i know a derivative is useful to find min/max points

Answer 4

With respect to what variable though?

Answer 5

p and q, unless otherwise noted, assume multivariate
\[X = -p*\log_2 (p)-q \log_2{q} - (1-p-q) \log_2 (1-p-q)\]
\[X_p = -2\log_2 (p)+2\log_2 (1-p-q)\]
hard to see thru the logs

Answer 6

so I take partial derivative for p then for q and equate them to 0?

Answer 7

and then solve for q and p with the two equations.. Can I do that?

Answer 8

yes, essentially; and there is a formula to determine if its a saddle point as well; but its essentially the partials set to zero

Answer 9

how would I know that I didn't get a minimum?

Answer 10

i also recall that the gradF points to the highest point ...

Answer 11

if the second partials are signed; they follow the same rule for single variable stuff

Answer 12

Ah okay this is not a calculus course lol so I just know the basics.. if I get a positive value for both p and q then it's the highest point?

Answer 13

F(x,y), gradient F is the vector <Fx,Fy>

when the gradient is equal to zero, we are either at a min/max or a saddle point

Answer 14

i believe the test is:

D = FxxFyy - (Fxy)^2

but i forget the "rules" for distinguishing the Ds

Answer 15

Oh okay but to solve for p and q would I do Fx =0 then Fy=0

Answer 16

\[X_p = \log_2 (1-p-q)-\log_2 (p)\]
i think i had a spurious little 2 in there at first

Xp = 0 when (1-p-q)/p = 1 right?

Answer 17

yes

Answer 18

Xp = 0 when (1-p-q)/p = 1 ; how did you get this?

Answer 19

ahh sorry for the questions I'm jsut trying to find the max value of that equation

Answer 20

partial with respect to p; and the fact that ln(a)-ln(b) = ln(a/b)

Answer 21

and since ln(1) = 0, then a/b = 1

Answer 22

it looks like you end up with 2 equations in 2 unknowns .... if i see it right

Answer 23

i know if i try to latex this im gonna get a snap page just about the end of it :/

Answer 24

sorry I tried it but why doesn't the partial of p for (-(1-p-q)*log2(1-p-q) ) give you zero?

Answer 25

\[X = -p\frac{\ln(p)}{\ln2}-q \frac{\ln{q}}{\ln2} - (1-p-q) \frac{\ln (1-p-q)}{\ln2}\]
\[X_p = -\frac{\ln(p)}{\ln2}-\frac{1}{\ln2} + \frac{\ln (1-p-q)}{\ln2}+\frac{1}{\ln2}\]
\[X_p = \frac{1}{\ln2}\left( \ln (1-p-q)-\ln(p)\right)\]
\[X_p = \frac{1}{\ln2} \ln (\frac{1-p-q}{p})=0\]
when equal to ln(1)

Answer 26

Xq has a similar setup

Answer 27

Ohhhhhhhhh I see where I went wrong

Answer 28

1-p-q = p
1= 2p + q
q = 1-2p

1-p-q = q
1 = p + 2q

1 = p + 2(1-2p)

1 = p + 2 - 4p

1 = 2 - 3p

-1 = - 3p

1/3 = p
-------------------

1-1/3 - q = 1/3

2/3 - 1/3 = q = 1/3

Answer 29

whoa thanks so much for your help!!! I need to freshen up on my calculus and derivatives

Answer 30

good luck :)