Information entropy

For a discrete random variable X with outcome probabilities pᵢ, Shannon entropy is H(X)=−Σpᵢlog₂pᵢ. The probabilities sum to one; a zero-probability term contributes zero by continuity. Entropy describes uncertainty before the outcome is known. A list of possible outcomes alone is insufficient: their probabilities matter.

The self-information of an outcome with probability p is −log₂p. Rare outcomes therefore carry more self-information when they occur, but entropy weights each contribution by its probability. It is an average over the distribution, not the largest possible surprise and not a quality score assigned to an individual message.

References: [1] [2]

A fair coin has probabilities one-half and one-half. Either outcome supplies one bit of self-information, so its entropy is one bit. Now suppose heads has probability one-quarter and tails three-quarters. The entropy becomes −0.25log₂0.25−0.75log₂0.75, approximately 0.811 bits.

In this second model, heads supplies two bits of self-information and tails about 0.415 bits. Their probability-weighted average is smaller than the fair coin's entropy, even though the rarer outcome is more surprising. A certain outcome has zero entropy. For a fixed finite number of possible outcomes, equal probabilities maximize entropy; having two alternatives does not by itself guarantee one bit of average information.

For comparison, two independent fair tosses have four equally likely ordered outcomes and entropy of two bits. Repeating the same recorded outcome twice would not create that independent uncertainty.

References: [2]

The logarithm's base sets the unit: base two gives bits, while natural logarithms give natural units, commonly called nats. Changing the base rescales the answer. A bit as an information unit must also be distinguished from a stored binary digit: the latter's average information depends on its probability distribution.

Entropy gives a benchmark for lossless coding. For an independent, identically distributed discrete source, suitable long-block coding can approach its entropy in average bits per source symbol. This does not allocate a fractional physical digit to every individual symbol. Correlated sequences require attention to conditional entropy or entropy rate rather than simply adding separate symbol entropies.

References: [1] [2]

Shannon's communication problem concerns reproducing a selected message, not assessing its semantic merit. Meaningless symbols may have high entropy under a model, while a crucial but already certain announcement may resolve no uncertainty. The entropy calculation does not establish that a claim is true.

A statistical-physical connection arises when the probabilities describe physical microstates. Using natural logarithms and multiplying by Boltzmann's constant yields an entropy expression in joules per kelvin. Such a connection requires specifying the system, its states and its constraints. A probability distribution over arbitrary text is not automatically a thermodynamic description, and neither a file's bit count nor its information entropy can simply be identified with heat. The shared mathematical form is useful precisely when the underlying interpretation is made explicit.

References: [1] [3]