Starting to seem like "Obviously you have to double-check AI output" is just a way to scapegoat the operators. Sort of like self-driving cars that require a human driver to sit with their hands on the wheel and their feet near the brake pedal - you're not going to catch a lot of errors because the safety factor, the human, is not actually engaged in the process but instead sitting passively watching something else do the work for extended periods.
https://www.theregister.com/2025/05/15/anthopics_law_firm_blames_claude_hallucinations/