Shannon's Information Theory: How Much Information Can Communication Carry?

In July and October 1948, Claude Shannon of Bell Telephone Laboratories published A Mathematical Theory of Communication in two parts in the Bell System Technical Journal. Telephones, telegraphs, radio and television used different equipment, yet all sent messages from one location to another. Channels had limited bandwidth and signals were disturbed by noise. Engineers needed to decide how to encode messages, how much power to use and how far transmission speed could be increased.

Nyquist had earlier analysed the relationship between telegraph speed and frequency range, while Ralph Hartley's 1928 paper developed a logarithmic measure of information from the number of possible signals. Shannon inherited these questions and brought message probabilities, correlations between symbols and noise into a unified analysis. The subject shifted from the performance of one circuit to limits shared by a class of communication systems.

References: [1] [2]

The paper's communication system comprised an information source, transmitter, channel, receiver and destination, with a separate noise source. The source produced messages, the transmitter converted them into suitable signals, the channel could be a wire, radio-frequency band or another physical path, and the receiver reconstructed messages from the incoming signal. Separating these stages allowed coding schemes and channel performance to be discussed individually and then assessed together.

Telegraph text, for example, became interruptions of electrical current, from which letters were recovered at the far end. Comparing two schemes required more than counting switches per second: it also required knowing how many possibilities each symbol represented and which sequences actually occurred. Information here was an engineering quantity. A message could be reconstructed accurately and still contain false facts; the channel model did not judge truth.

References: [1] [3]

Shannon treated a discrete source as a process producing symbols with specified probabilities and used entropy to express average uncertainty. Two equally likely outcomes provide one bit of information per choice; if one outcome is nearly certain, less uncertainty needs to be removed on average. The base-two bit placed the selection capacities of different physical devices on a common scale.

This did not mean that a sentence with fewer letters necessarily contained less information. Natural-language letters are not equally probable, and their combinations follow patterns, making some content predictable from context. In the paper's model, compression could exploit these regularities, giving common cases shorter average representations and reserving longer codewords for rarer ones. Redundancy therefore had precise uses: some could be removed to economize on transmission, while other redundancy could help resist errors.

References: [1] [3]

For a noisy channel, Shannon distinguished transmitted from received symbols and described confusion through conditional probabilities. Under a specified model, channel capacity set the boundary for reliable transmission rates. The key result was that below capacity, with suitable coding and sufficiently long code blocks, error probability could be made arbitrarily small. Above that boundary, more ingenious coding alone could not maintain the same reliability objective.

This was an existence and limit result, not delivery on publication day of a machine reaching the limit. Engineers still had to design encoders and decoders that were computationally practical, manufacturable and acceptable in delay. One intuitive trade-off was to add structured redundancy to distinguish messages that noise might otherwise make similar, at the cost of more transmission resources for the same original data.

The paper also studied continuous signals. For a particular band-limited Gaussian-noise model, capacity was constrained jointly by bandwidth, signal power and noise power. Increasing power and expanding the frequency band were not indistinguishable alternatives, providing a calculable basis for comparing modulation methods.

References: [1] [3]

In 1949, the University of Illinois Press published The Mathematical Theory of Communication under the names of Shannon and Warren Weaver, combining Shannon's paper with Weaver's explanatory essay. The mathematical work reached a broader readership. Weaver discussed wider issues of communication, while Shannon's model remained most directly concerned with technical relationships among symbol selection, coding and reconstruction.

Subsequent research in compression, error correction and digital communication continually sought practical ways to approach the theoretical limits. The 1948 contribution identified what could be economized, what was attainable under noise and which limits would not disappear through replacement of one component. It supplied a common language for equipment design and allowed researchers to compare a specific system with clearly defined limits.

References: [1] [3]