Benjamin Paassen

Junior Professor for Knowledge Representation and Machine Learning at Bielefeld University; affiliate at DFKI EdTec Lab; co-host of "Autonomie und Algorithmen" podcast; researches machine learning for education; supporter of queer feminism 🏳️‍🌈; preferred pronouns: they/them; role model: Sailor Mercury; views are my own

Location: Bielefeld
Languages: German, English
Preferred Pronouns: they/them
Homepage: https://bpaassen.gitlab.io/

Last Notes

2026-10-07 19:43:36 UTC

- reply

Yes, happy to. It's this one: https://doi.org/10.1037/tmb0000210

2026-10-06 18:30:21 UTC

- reply

But our core argument still stands and has survived peer review. So to cite our conclusion: For research questions that aim to characterize human judgments, experiences, or behavior, however, human participants cannot be replaced.

2026-10-06 18:30:17 UTC

- reply

2. We have strongly revised our theory section, incorporating more arguments now and referencing cognitive architecture work.
3. We have a lot more supplementary analysis to look deeper into the patterns in our data.
And, sadly, our title had to become much less concise.

2026-10-06 18:30:01 UTC

- reply

1. We have now evaluated more LLMs, including more recent ones. Our core effect is still maintained: under rewording, the correlation between human responses and LLM responses significantly reduces. But we also find differences between LLMs: Older and smaller LLMs tend to just ignore the rewordings, whereas newer LLMs do react to them - but not in the same way humans do.

2026-10-06 18:28:25 UTC

- reply

I want to thank all authors for their tireless work on this - but also our reviewers for some helpful discussion and feedback. Some highlights of what we changed since the preprint:

2026-10-06 18:28:08 UTC

- reply

Our core argument is that LLMs should not be regarded as reliable simulators of human participants because subtle changes to the input text can lead to very different reactions from humans and LLMs. This is also shown in our key figure below. The x axis corresponds to different moral scenarios, the y axis to the reaction if we subtly change the text of the scenario. You can see that the reactions of LLMs and humans are clearly very different for most scenarios (exact correlations in the paper).
https://storage.gra.cloud.ovh.net/v1/AUTH_91eb37814936490c95da7b85993cc2ff/bildungsocial/media_attachments/files/117/395/375/879/777/707/original/1ee48146afd8fa7a.png

2026-10-06 18:27:40 UTC

- reply

Well, it turns out, it has. The journal "Technology, Mind, and Behavior" just published our paper under the new title "Exploring the Limits of Large Language Models as a Reflection of Human Cognition: An Illustration in the Context of Moral Judgment".
https://doi.org/10.1037/tmb0000210

2026-10-06 18:27:36 UTC

Over one year ago, we got a lot of response for our preprint "Large language models do not simulate human psychology". While psychologists seemed to be somewhat puzzled that such claims are made in the first place, some colleagues from computer science or application reacted quite skeptically to our claim - we even got some comments that our paper would never make it through peer review.

2026-03-05 09:59:31 UTC

- reply

I agree with that. One sometimes also can hear/read/see how scientists try to make the role they are currently taking more explicit - but it is still imperfect, I feel, and lines can be blurry.

2026-03-05 09:53:24 UTC

- reply

For myself, I find it at least somewhat helpful to think of science as informing us about the consequences (including uncertainty estimates, ranges, etc.) of political decisions but politics has to make the decision itself. In that sense, science is "apolitical" or "neutral". But a lot of political actors try to occlude/mislead about the consequences of their political program. And in such cases, I think, science has to take a clear stance in opposition.

2025-08-12 15:09:16 UTC

- reply

But there are also plenty of criticisms. Empirically, we know that LLMs are biased (in different ways than humans), their answers vary much less compared to human answers, and they make mistakes that humans wouldn't make. (5/12)

2025-08-12 15:09:11 UTC

- reply

Some papers claim that. Dillion et al. (2023) let LLMs respond to morality questions and they did indeed respond like average humans. Binz et al. (2025) fine-tuned an LLM to respond like humans to psych questions. (4/12)
https://doi.org/10.1016/j.tics.2023.04.008
https://doi.org/10.1038/s41586-025-09215-4

2025-08-12 15:08:14 UTC

- reply

LLMs as participants can only work if LLMs really respond like humans to psychological questions. Do they? (3/12)

2025-08-12 15:08:10 UTC

- reply

Our starting point is a trend in psychology: Recruiting human participants for psychological studies is a lot of effort and expensive. Wouldn't it be great if we could just use LLMs as participants, instead? N = 1000 for just a few dollars. (2/12)

2025-08-12 15:08:06 UTC

Large Language models (LLMs) do not simulate human psychology. That's the title of our new paper, available as preprint today (1/12):
https://arxiv.org/abs/2508.06950