Cognitive Science > Statistical Natural Language Processing > 2

Jump to: navigation, search

[edit] Properties of probability distributions

  • Normalization
    0 <= Pr(w) <= 1
  • Additivity
    LaTex: Pr(E_1 \cup E_2) = Pr(E_1) + Pr(E_2) if E1 & E2 are mutually exclusive
  • Conditional Probability
    • not symmetric
      LaTex: Pr(E_1|E_2) = \frac {Pr(E_1 \cap E_2)} {Pr(E_2)}
    • Chain rule
      LaTex: Pr(E_1 \cap E_2) = Pr(E_1|E_2) * Pr(E_1)
      mutually exclusive: LaTex: Pr(E_1 \cap E_2) = Pr(E_1) * Pr(E_2)
  • Bayes' theorem
    LaTex: Pr(C|E) = \frac {Pr(E|C) * Pr(C)}{Pr(E)}

[edit] Information theory

  • Amount of info
    -log2Pr(E)
    • in bits
    • associated with an event E
  • Joint Info
    • LaTex: -log_2Pr(E \cap R)
  • Entropy
    • average info provided
    • measure of homogeneity
      LaTex: H[C] = -\sum_x Pr(X = x) * log_2 Pr(X = x)
      • H[C] is max if all values of C have same prob
Source: http://wiki.stuhlmueller.info/Cognitive_Science/Statistical_Natural_Language_Processing/2