Peter Bloem on Nostr: I'll try to keep it brief. :) It's true that offline statistical analysis can only ...

I'll try to keep it brief. :)

It's true that offline statistical analysis can only uncover correlation not causation, so that's a limit to what a purely autoregressive LLM can do.

There are two important counters though:
- LLMs aren't trained just autoregressively. They are also trained by RL, which does admit causal analysis to emerge. Admittedly, most of the power comes from the pre-training, but this removes the impossibility argument.
- A huge amount of behaviors don't require understanding of the causality of the training data. Including, and this is I admit is counter-intuitive, running causal analysis in-context. An LLM could learn to run programs from a suffiently large dataset, and those programs could include existing causal analysis algorithms, which it could then apply in context to data.
- Modern models also have a kind of internal monologue. This pushes its behavior quite far from "sampling from the data distribution", especially coupled with RL.