[For a more detailed version with the relevant mathematics, and a 2026 addendum on why an AI is named after Claude Shannon, read Information Entropy.]
Information and Uncertainty are quantities that describe any decision making process - like selecting one or more objects from a set of objects. Both these ideas and much more, related to communication theory, were introduced by Claude E. Shannon in his seminal paper “A mathematical theory of communication,” Bell Syst. Tech. J. (1948) [pdf].
For example, consider the case of one of the two boxes shown below, containing a ball in it. You are asked to find which box the ball is by asking me questions. I will answer only with a ‘yes’ or a ‘no’, much like the Twenty Questions game we play in our college cultural festival.
Figure 1
Before you ask any question to me, you have no clue as to which box contains the ball. In other words, you are UNCERTAIN and have no INFORMATION about the system so that you can DECIDE on the correct answer of which box contains the ball. Once you ask questions and receive answers - in this case, one is enough - you gain INFORMATION and are no more UNCERTAIN, for making your decision. This implies that the decision that has to be made by you between the two boxes will depend on some reason and not purely a random event.
More generally, Information and Uncertainty are quantities (because, they are attributed with numerical values) that could be used to describe the process of picking out a ’set of symbols’ from a relatively large sequence of symbols.
If the set of symbols that form the main sequence is the ‘alphabet’ of a Language, then the constituent individual symbols represent the letters of that Language. The random sequence baabbcac in English language is an example. It should be reminded however that the words Information and Uncertainty, as defined here, DO NOT try to quantitatively describe ‘meaning’ of a set of letters (forming possible words) of a language. Neither do they answer mathematically, what is the ‘grammar’ of a language.
Simply put, in view of a decision making process, Uncertainty decrease leads to Information increase.
Now, the question is, how to measure this Uncertainty?
For this we need to introduce the concept of the ‘real’ bit. A BIT is defined as the amount of Information necessary to decide between two equal choices.
As we will see later, unlike in computer parlance, the above concept of a bit here can be associated with real numbers.
In the Ball and Box example, for the simplest initial case of 2 boxes, you HAVE to ask AT LEAST one question to me, to decide on the correct choice of box. In other words, you require one bit of information to decide between a choice of two. Similarly, in the example of 4 boxes and a ball shown in Figure 2 below, for the existence of the ball in any of the boxes being equally likely, you require at least 2 bits of information. That is, you can be sure of the correct answer of which box the ball is ONLY after 2 questions.
Figure 2
The number of bits provides the OPTIMAL number of questions, taking into account all possible eventualities, that leads to the answer.
To summarize, we need at least one question to be asked (and receive a yes or no answer for it) before we decide in which box the ball is in Figure 1. Making a choice between 2 equi-probable symbols (or events) in Figure 1, requires AT LEAST one bit of Information. Similarly, for the same choice to be made out of 4 equi-probable symbols or events (Figure 2), at least 2 bits of Information are necessary. Extending, for 8 equally likely symbols (or events, as in the box and ball example above), we should require at least 3 bits of Information to make a choice (decision).
For the Box and Ball example, this implies at least 3 questions are necessary to decide correctly, if there are 8 boxes with one of them containing the ball. In other words, the Uncertainty of the system is 3 bits BEFORE any question is asked and reduces to zero, once the answer is known. In case of Figure 2, the Uncertainty is 2 bits and reduces to zero, once the answer is known. The process of reduction in Uncertainty leads to gain in Information, that is from zero knowledge of which box has the ball to the case of complete knowledge of where the ball actually is.
Uncertainty DECREASE ⟶ Information INCREASE
The above concept of Uncertainty is also known as Information Entropy or Shannon Entropy, an idea which revolutionized communication theory (quiz your electronics and communications engineer friend on terms like signal to noise ratio, channel capacity etc.).
Calling this Uncertainty (measured in bits per symbol or event) as Information Entropy has unfortunately led to many confusions, because of wrong comparisons and allusions with the Thermodynamic Entropy (measured in Joules per Kelvin).
The story(*) goes that Shannon, after having described his concept in mathematical form (see Information Entropy), asked his friend John von Neumann what to call it, and von Neumann told him to call it entropy, as nobody is sure of what it is and he (Shannon) would always have the advantage in any debate over his theory. So Shannon named it Information Entropy, although he also used the term Uncertainty to denote the same idea in his paper.
Shannon’s name has since travelled further than he might have guessed: an AI assistant called Claude is widely taken to be named after him. Why that is fitting is told in Section 6 of Information Entropy.
***
[For a more detailed version with the relevant mathematics, read Information Entropy.]
(*) see Myron Tribus and Edward C. McIrvine, Energy and Information, Sci. Amer., vol. 225, no. 3, pp. 179–188, Sep. 1971