The approach I have called Kaplan Information Theory, or KIT, starts with the observation that there would be no information without differences. That is, humans, or any intelligent entity, could not perceive a world unless we could perceive and draw distinctions between this and that. Therefore, most generally, the notion of difference and the quantification of differences is the essence of meaningful information.
The approach I have called Kaplan Information Theory, or KIT, starts with the observation that there would be no information without differences. That is, humans, or any intelligent entity, could not perceive a world unless we could perceive and draw distinctions between this and that. Therefore, most generally, the notion of difference and the quantification of differences is the essence of meaningful information.
Classical Information Theory, the body of work that begins with Claude Shannon’s 1948 paper A Mathematical Theory of Communication, fits within KIT as one dimension of difference, because Shannon defines information as the difference between what was observed and what was expected.
The greater the difference between what was received and what one could probabilistically expect to receive, the greater the information contained in a message.
Shannon’s information formulation made sense when determining how to maximize the information sent over copper wires from a sender to a receiver. This was his problem at Bell Labs when he wrote his classic paper. In that context, measuring the difference between what the receiver expected to see and what the receiver saw made complete sense as a rigorous definition of information, with practical implications for a channel’s capacity to carry information.
However, differences in expectation are only one type of difference that can be measured.
Let us consider some intuitive concepts about information. We commonly say some events carry information if it is news, that is, if it was previously unknown to a particular recipient, even if it is not generally surprising. Something is new to me and carries information. It is well known to you and carries no new information. So, there is a relative aspect of information that is not explicitly part of classical theory. Assuming a different probability distribution of expected events for each entity could solve this problem, but that seems cumbersome.
Also, while the amount of information is sometimes proportional to the number of words in a message, there are situations in which fewer words convey more information. For example, Blaise Pascal wrote in 1657 that he had made a letter longer only because he had not had the time to make it shorter, implying that fewer words would have conveyed more useful information. Again, Classical Information Theory can be contorted to say that Pascal’s shorter letter was somehow less expected and therefore contained more information, but this seems counterintuitive.
Is there a more general theory of information that can more naturally account for the fact that sometimes surprising information is not necessarily relevant or valuable, or that a longer string of words has less useful information, or that commonly known and expected facts could still carry high information content if they are especially relevant?
KIT considers any difference between two events, datasets, categories, or informational units to be a valid measure of the information content. Generally, distinguishable events, objects, or categories of information only exist to the degree that differences exist. An infinite string of 1s contains no information. An infinite string of 0s contains no information. Zero has meaning only if 1 is possible and 1 sometimes exists. Similarly, 1 has meaning only if 0 is a possibility and 0 sometimes exists.
Seeing a 1 after an incredibly long sequence of 0s carries much information, not just because it is unexpected, but be a difference!
Thinking of information as a measure of difference is more general than considering information as a measure of surprise. Surprise is just one type of difference, whereas any difference, even non-surprising ones, contains information.
For example, consider two datasets. Where the two sets intersect, there is no new information. However, the sum of the non-overlapping areas of the sets, known as the Symmetric Difference in set theory, represents the new information contained in the datasets, relative to each other. Picture the two datasets, A and B, as a pair of overlapping circles. The region where the circles overlap is what both datasets already have in common. Everything outside that overlap, in either circle, is the Symmetric Difference.
The datasets can contain events that have already occurred, such as static snapshots of existing events, or their content can change over time. Some examples might help.
AI #1 knows everything in the Encyclopedia Britannica. AI #2 knows everything in Wikipedia. AI #3 is the combined knowledge of AI #1 and AI #2. The intersection between Wikipedia and Britannica represents things that both AIs know. The intersection contains no new information for either AI #1 or AI #2. However, the Symmetric Difference, namely, the knowledge in Britannica and not in Wikipedia, plus the knowledge in Wikipedia and not in Britannica, represents the new knowledge of AI #3. Time is not relevant in this example. The new information can be calculated by comparing the static information across the two AI datasets.
In contrast, consider the same two AIs, except that this time each continues to add to its knowledge. Now the informational calculations must consider the static encyclopedias and any new information that has been added to AI over time. So, the intersection and Symmetric Difference are constantly changing over time.
The information in these examples is still a matter of difference. Still, in this case, it is not the difference between what was expected and what was observed, as in Classical Information Theory, but rather the difference between two static sets of information or two sets of information that continue to change over time.
Specifically, KIT enables methods that account not only for how surprising an event is, but also for how much an event differs from another event and how relevant it is to the goals of an intelligent entity. The ideas of quantifying differences in knowledge and goal relevance, and of quantifying how unlikely an event is, represent key distinctions between KIT and Classical Information Theory.
The next post sets out the dimensions along which those differences can be measured. It explains why the same instructions for making fire carry vastly different amounts of information depending on who else already knows how.
This series draws on White Paper 6: Catalysts for Growth of SuperIntelligence. Read it in full to see how every piece fits together!
If this made you think, subscribe to Superintelligence at read.superintelligence.com so you don’t miss what comes next. And if someone in your life needs to understand where AI is heading, send this to them.