It's amazing how much of the statistical approach to language and the capacity to predict next "tokens" is already contained in Shannon's Mathematical Theory of Communication (1948). His interests were compression and dealing with noise (so, how to reconstruct a partially distorted message), but the basic approach that LLMs are using is fully there.
The only thing that prevent him making this a center piece in his theory is that "the labor [in creating the frequency tables (aka training the model)] involved becomes enormous".
felix stalder
npub1jw…464n7
2026-10-16 09:26:48 UTC
Author Public Key
npub1jwl9282n05rdgvtk4qu06x6cwfamcykdntwhdqzu6euqx0vympyqn464n7Seen on
wss://relay.mostr.pubPublished at
2026-10-16 09:26:48 UTCEvent JSON
{
"id": "342457e9bf96eb84add8ff48204e91026c2ca3a1d3424cad4e5b8608576f1c79",
"pubkey": "93be551d537d06d43176a838fd1b58727bbc12cd9add76805cd678033d84d848",
"created_at": 1792142808,
"kind": 1,
"tags": [
[
"proxy",
"https://tldr.nettime.org/users/festal/statuses/117449871105711269",
"activitypub"
],
[
"client",
"Mostr",
"31990:6be38f8c63df7dbf84db7ec4a6e6fbbd8d19dca3b980efad18585c46f04b26f9:mostr",
"wss://relay.ditto.pub"
]
],
"content": "It's amazing how much of the statistical approach to language and the capacity to predict next \"tokens\" is already contained in Shannon's Mathematical Theory of Communication (1948). His interests were compression and dealing with noise (so, how to reconstruct a partially distorted message), but the basic approach that LLMs are using is fully there. \n\nThe only thing that prevent him making this a center piece in his theory is that \"the labor [in creating the frequency tables (aka training the model)] involved becomes enormous\".",
"sig": "07f98e4a816a09a087274802d149af473dd09c68a1afee65a338606796aadbd8cf44004753ee92e5bf68c1b74327771df86ebd211ba47f0e39f6428a73ebbb45"
}