Fractal Press

I’m beginning to suspect that the notion of entropy outside the physical domain is simply a mathematical construct and may have nothing really to do with “information” as the word is generally accepted (unless you change the numeraire and define information as entropy). I claim that we can work with it as a mathematical construct (like the notion of variance), but to try and assign a deeper meaning or a connection to information may be going too far. (This does not preclude me from claiming that the mathematical construct called entropy is increasing).

The size of a zipped-file containing the text of Eco’s Foucault’s Pendulum does not represent the information contained within that text. The notion of information is relative to the reader and is at least tangentially related to the concept of “meaning.”

Shannon’s entropy is a measure only of the probability of a specific message amongst all possible messages. It is not a function of the transmitter or the receiver, only the channel. But, for someone who speaks no Korean, the Korean translation of Foucault’s Pendulum is meaningless, even though the entropy is probably the same. So entropy does not represent meaning or information or the lack of meaning or uncertainty or any such thing except in a narrowly defined sense as described by Shannon.

Here is a quote from Shannon’s original paper:

The fundamental problem of communication is that of reproducing at one point either exactly or approximately a message selected at another point. Frequently the messages have meaning; that is, they refer to or are correlated according to some system with certain physical or conceptual entities. These semantic aspects of communication are irrelevant to the engineering problem. The significant aspect is that the actual message is one selected from a set of possible messages

I repeat for emphasis:“These semantic aspects of communication are irrelevant to the engineering problem.” When we talk about information, all we’re talking about is the semantic aspect, not the engineering aspect. When a new discovery is made and it enhances our understanding of the world it increases the information content in the universe, but it is not directly related Shannon’s entropy. It does increase the size of our libraries, but so do many other things that are not considered new forms of information, such as a book containing a purely random stream of numbers that just hasn’t been recorded before. Entropy offers us no way to distinguish between the two and hence may not be useful as a direct measure of information.

How does this relate to my earlier thesis about information, entropy, and kurtosis? I’m not sure yet, but I’m thinking about it. I think it only requires a change in terminology to remain consistent. That is, I can remove all references to “information” and work only with the math of entropy and still have the essay be meaningful.

Nassim Taleb’s notebook discusses the “Parabole Chinoise” [In french, google translate does an ok job].

More notes from my Travelogue:

==

In the Suan Pakkad Palace Museum in Bangkok, there is a letter. The curator wouldn’t let me photograph it, but I was able to copy it out by hand. The letter talks about a major archeological find in Ban Chiang, Thailand. It is now a UNESCO World Heritage Site. It is a major site for understanding prehistoric settlements in Asia and an important source of clues for the cultural, technological, and social evolution of humans. The author of the letter discovered the site.

The letter, to his family:

August 12, 1966 …

Here is a short message about my feelings on Ban Chiang: In reflecting on the Ban Chiang find, I became more humble and spiritual, less scientific and confident of human dominance over fate.To use a Thai reference, some Theveda must have been responsible for the Ban Chiang find for I did not consciuosly seek out ancient Thai history. In the summer of 1966, I was looking for villages in the Isan in which to interview villagers for my senior thesis at Harvard College.A chance acquaintance sent me to Ban Chiang. I could have gone to any of hundreds of villages.Why did I end up in Ban Chiang? Luck.

Then, I had been in the village, walking around, for some time and never saw the pottery in the ground. But on this day I lost control and tripped right where there were many pots in the ground.When I fell forward, face down on the ground, there was a pot just under my nose. I had to see it. It was as if some force outside me wanted me and not somebody else to find the pots. So, as soon as I realized that they were beautiful and old, I became the recipient of a sense of responsibility that it was my fate, my duty, to tell the world about this find.The important event, and my task, had not been planned at all. Just so, many important things happen to people in life which are not planned.

Stephen B.Young

A very multi-faceted story. Ranging across countries, disciplines, and belief systems.

Nassim Taleb’s latest interview with The Sunday Times is very well done and talks about the underemphasized issue of behavior in the absence of Black Swans. I had been thinking about the same topic over the last few days. Here are my thoughts:

The part of TBS that discusses being a fool in the right places is equally important as the part that discusses being conservative in Extremistan domains. In that respect, the chapter titled “Half and Half, or How to Get Even With the Black Swan” is my favorite and contains a succinct and personal kernel of the whole book. I think even the folks who understand the Extremistan ideas did not take away all that there is in the book on the other aspect: behavior in Mediocristan.

Let us consider a system of thought where “there is no logical reason to prefer one action over another.” In Extremistan domains, where the unknown plays a large role, it is beneficial to lean towards this system of thought. For example, if you are asked to invest in N securities, it is best to either invest in all of them, or none (if that’s a choice), instead of tunneling and showing a preference for a few over others. In this domain it is useful to value all choices equally beforehand, so we should not prefer one action over another.

However, it is possible to take this too far and apply this system of thought to Mediocristan domains. This will clearly lead to inaction. In a transaction involving exchange of goods, money, or time, a person who treats all choices as equal gains nothing from entering the transaction. He will exchange Object X for Object Y, but if he values X and Y equally, he gains nothing from the transaction. En fait, he will incur a transaction fee and come out at a loss. This would imply he is better off doing nothing at all, if he truly values everything equally (including his time). For example, if he knows that an MBA is valued fairly (it probably is, given the competition) and he values money and the MBA education equally, he has no reasons to apply at all.

So Taleb rails against the application of Mediocristan tools in Extremistan domains, but he is careful to point out that Extremistan thinking should not be brought into Mediocristan domains and he stresses the opposite. Many miss this point, and he emphasizes:

Few understand the beauty in the story of Apelles; in fact, most people exercise their error avoidance by repressing the Apelles in them.

If one decides to be hyperconservative in Extremistan domains, it becomes necessary to become hyperaggressive in Mediocristan domains. Otherwise, a part of us, the part designed to take risks, will be amputated.

So it becomes important to cultivate tastes, likes, and dislikes in harmless things. There are perils to being too open minded and valuing everything equally. The process of living in Mediocristan is trading-off what one doesn’t mind losing in order to acquire what one desires. For someone who has no particular likes or dislikes, this becomes very difficult and can lead to inaction. The stronger the likes and dislikes, the easier it will be to make decisions and trade-offs. These trade-offs are acceptable because the worst-case-scenario is clipped, these decision cannot hurt us more than a certain amount. A little diversification will wash the negative consequences out.

Creating an emotional differential between harmless choices in Mediocristan is what drives us - it generates action. This can be cultivated. It is difficult to put this in terms of “Gaussian risks,” so something may be lost in translation when the difference between Mediocristan and Extremistan is introduced.

Both takeaways from the book are important. Applying the one without the other leaves you lacking something important.

I am making some more progress with the entropy angle. But there is enormous confusion in terminology and fundamental concepts when it comes to entropy.

In various journal papers, “entropy” has been taken to mean information, randomness, disorder, uncertainty, increased order. In others, it is “negentropy” that takes these meanings.

A note on the NIH website talks about “Information is not Entropy, Information is not Uncertainty,” and has a nice quote:

The story goes that Shannon didn’t know what to call his measure so he asked von Neumann, who said `You should call it entropy … [since] … no one knows what entropy really is, so in a debate you will always have the advantage’

More as I clarify my thinking and stick to one set of terminology and concepts.