AI has a greater chance of failure the longer it works on a task
I was browsing over this research paper by Toby Ord who talks about comparing an LLM working on a lengthy task as compared to a human working on the same lengthy task. The longer it takes to complete the task, the greater the odds of failure for the LLM, in fact the odds of failure increase exponentially with time. Naturally, larger LLMs with more computing power can go for longer, but the exponential odds of failure trend are the same no matter what.
So for example you might be able to get the LLM to work on a task for 1 hour with a 8-billion parameter model and expect it will succeed at the task 50% of the time. Maybe you can get 2 hours out of a 16-billion parameter model where you can expect it will succeed at the task 50% of the time (I am guessing these parameter sizes). But after that “half life” the odds of the model succeeding taper-off to zero percent.
I haven’t read the little details yet, like how do you judge when the task is done (I presume when the LLM claims that it has finished the job), or do these tests allow multiple prompts (I presume it is just fed an input once and allowed to churn on that input until it believes it is finished). So that makes me wonder, could you solve this problem, increase the AI’s rate of success, if you combine it with classical computing methods? For example, perhaps you can ask the LLM to list the steps it would perform to complete a task, then parse the list of steps into new prompts and feed-back each of those steps into the LLM as a second prompt to produce a new list of sub tasks? Could you keep breaking-down the tasks and feeding them back into the AI to increase it’s odds of success?
It is an interesting research question. I am also interested to see how much energy and water this takes as compared to a human working on the same task, including the caloric intake of food, and perhaps the energy used to harvest, process, and deliver the food.
#tech #AI #LLM #ComputerScience
Ramin Honary
npub105…puwdv
2025-06-19 01:34:06 UTC
Author Public Key
npub1053m9ka5kf6r3rlcr3kmstaswqk2ewqduv2dvtjtud3p38r0dxvqepuwdvSeen on
wss://relay.ditto.pubPublished at
2025-06-19 01:34:06 UTCEvent JSON
{
"id": "514687401500734e311855249719e8b2b99d1cde9c7fddc75bcf378ffc888c91",
"pubkey": "7d23b2dbb4b274388ff81c6db82fb0702cacb80de314d62e4be362189c6f6998",
"created_at": 1750296846,
"kind": 1,
"tags": [
[
"t",
"ai"
],
[
"t",
"computerscience"
],
[
"t",
"llm"
],
[
"t",
"tech"
],
[
"proxy",
"https://fe.disroot.org/objects/7777cca1-f50c-4cb3-a084-b976519fd649",
"activitypub"
],
[
"client",
"Mostr",
"31990:6be38f8c63df7dbf84db7ec4a6e6fbbd8d19dca3b980efad18585c46f04b26f9:mostr",
"wss://relay.mostr.pub"
]
],
"content": "AI has a greater chance of failure the longer it works on a task\n\nI was browsing over this research paper by Toby Ord who talks about comparing an LLM working on a lengthy task as compared to a human working on the same lengthy task. The longer it takes to complete the task, the greater the odds of failure for the LLM, in fact the odds of failure increase exponentially with time. Naturally, larger LLMs with more computing power can go for longer, but the exponential odds of failure trend are the same no matter what.\n\nSo for example you might be able to get the LLM to work on a task for 1 hour with a 8-billion parameter model and expect it will succeed at the task 50% of the time. Maybe you can get 2 hours out of a 16-billion parameter model where you can expect it will succeed at the task 50% of the time (I am guessing these parameter sizes). But after that “half life” the odds of the model succeeding taper-off to zero percent.\n\nI haven’t read the little details yet, like how do you judge when the task is done (I presume when the LLM claims that it has finished the job), or do these tests allow multiple prompts (I presume it is just fed an input once and allowed to churn on that input until it believes it is finished). So that makes me wonder, could you solve this problem, increase the AI’s rate of success, if you combine it with classical computing methods? For example, perhaps you can ask the LLM to list the steps it would perform to complete a task, then parse the list of steps into new prompts and feed-back each of those steps into the LLM as a second prompt to produce a new list of sub tasks? Could you keep breaking-down the tasks and feeding them back into the AI to increase it’s odds of success?\n\nIt is an interesting research question. I am also interested to see how much energy and water this takes as compared to a human working on the same task, including the caloric intake of food, and perhaps the energy used to harvest, process, and deliver the food.\n\n#tech #AI #LLM #ComputerScience",
"sig": "a094a06a0c5e19de35dc61f0558793de62ea6d247b798dd5249df84d5df5154fbf76db82c94b7608646f09441267d0a50a05577e2d9f8f9bda2b71cf61de7d5d"
}