Concept
Information theory
Overview
Information theory is the mathematical study of the quantification, storage, and communication of a particular type of mathematically defined information. The field was established and formalized by Claude Shannon in the 1940s, though early contributions were made in the 1920s through the works of Harry Nyquist and Ralph Hartley.
Information theory was initially formed in the context of telecommunication but soon found a wide range of other applications. It is now at the intersection of mathematics, statistics and computer science, and has applications in diverse fields ranging from electrical engineering and physics to neurobiology.
As a simple example of the concept, if one flips a fair coin and does not yet know the outcome (heads or tails), then one lacks a certain amount of information. After looking at the coin, one gains information about the outcome. For a fair coin, the probability of either heads or tails is 1/2 and the amount of information is expressed as
−
log
2
(
1
/
2
)
{\displaystyle -\log _{2}(1/2)}
= 1 bit of information.
A key concept in information theory is entropy. In Shannon's formulation, entropy is equal to the lack of information about an event. In the above coin flip example, the entropy in the case where you don't know the outcome is 1 bit. When you know the outcome after the coin has landed, the entropy is zero because you have gained one bit.
Information theory has been used in a wide range of applications, such as source coding/data compression (e.g. for ZIP files), and channel coding/error detection and correction (e.g. for DSL). Its impact has been crucial to the success of the Voyager missions to deep space, the invention of the compact disc, the feasibility of mobile phones and the development of the Internet and artificial intelligence. The theory has also found applications in other areas, including statistical inference, cryptography, neurobiology, perception, signal processing, linguistics, the evolution and function of molecular codes (bioinformatics), thermal physics, molecular dynamics, black holes, quantum computing, information retrieval, intelligence gathering, plagiarism detection, pattern recognition, anomaly detection, the analysis of music, art creation, imaging system design, study of outer space, the dimensionality of space, and epistemology.
Overview adapted from Wikipedia (CC BY-SA 4.0; see the article history for authors).
Possible passages
Found by search and not yet checked by a person; they may only mention the idea in passing.
The knowledge of the elementary fact enables us to put the problem in an equation. Nothing remains but to deduce from this by combination the complex fact that can be observed and verified. This is what is called _integration_, and is the business of the mathematician. It may be asked why, in physical sciences, generalization so readily takes the mathematical form. The reason is now easy to see. It is not only…
Henri Poincaré, The Foundations of Science
States what Information theory is.
The theory of continuity, with which we shall be occupied in the present lecture, is, in most of its refinements and developments, a purely mathematical subject--very beautiful, very important, and very delightful, but not, strictly speaking, a part of philosophy. The logical basis of the theory alone belongs to philosophy, and alone will occupy us to-night. The way the problem of continuity enters into philosophy…
Bertrand Russell, Our Knowledge of the External World as a Field for Scientific Method in Philosophy
States what Information theory is.
Simplicity and objectivity do not have to be secured through elaborate and labored metaphysical construction. The data are simple and the hypothesis objective in so far as they accomplish the work where unto they are called--the removal of conflict, ambiguity, and inhibition in conduct and affection. In the experimental conception of inference it is clear that the principles of formal logic must play their role…
John Dewey, Creative Intelligence
States what Information theory is.
Related ideas
- Artificial intelligence
- Alan Turing
- Internet
- Giulio Tononi
- Mathematics
- Music
- Cryptography
- Information
- Gerald Edelman
- Neuroscience
- Linguistics
- Physics
- University of Illinois Press
- Probability theory
- Conditional probability
- Perception
- Statistics
- Space
- Semiotics
- Scientific American
- Thermodynamics
- Transistor
- Umberto Eco
- Bit
Place in the tradition
Part of Formal science.