Modeling possible messages
A communication system must accommodate possible messages before knowing which one will be sent. Information theory studies those alternatives and their probabilities rather than only the message eventually selected. Shannon’s 1948 framework separated sources, encoding, channels, reception, and noise, allowing different devices to be analyzed through common mathematical structures.
References: [1]
Entropy and uncertainty
For a discrete random variable, entropy averages uncertainty using the probabilities of its possible outcomes. Two equally likely alternatives have one bit of entropy; a nearly predictable outcome contributes less average surprise. Entropy depends on a distribution, not simply a file’s byte count or the truth and quality of its contents.
Compression and predictability
Compression exploits repeated structure and unequal probabilities, assigning economical representations to predictable patterns. Source coding asks how concise lossless representation can become under an assumed source model. Failure to compress one string with one method does not establish that the string lacks structure under every possible model.
Reliability and channel limits
Noise can make received signals differ from transmitted ones. Channel coding introduces organized redundancy to support recovery, while capacity expresses a reliable-rate limit under a model. Separating semantics makes the transmission problem analyzable; it does not claim that meaning or practical use is unimportant outside that engineering question.
References: [1]