"Scaling up and shaping up large language models increased their tendency to provide sensible yet incorrect answers at difficulty levels humans cannot supervise, highlighting the need for a shift in AI design towards reliability, according to a paper in Nature."
https://www.nature.com/articles/s41586-024-07930-y
Something something confirmation bias