yes, project euler is around since about 10 years and started with ca. 160 problems. I also believe that most LLMs have seen solutions, however probably mostly in python. I don't think this is too influencing to other programming languages, so benchmarks to java etc. may be more significant.
My target is to evaluate the ability of LLMs to perform in an auto-coder engine and there I would not mind if those models perform because of some 'cheating'.
Michael Christen / YaCy Search
npub1v5…eldnk
2024-12-25 21:27:31 UTC
in reply to nevent1q…7mex
Author Public Key
npub1v5nnv7827xa34r5fmvppkq69vpgjm27cva4dle3s53st2gg8cetqkeldnkSeen on
wss://relay.momostr.pinkPublished at
2024-12-25 21:27:31 UTCEvent JSON
{
"id": "48638326054c326867bd36652c88aa036e5cc4f2376e281031b96dd9eeba51de",
"pubkey": "65273678eaf1bb1a8e89db021b034560512dabd8676adfe630a460b52107c656",
"created_at": 1735162051,
"kind": 1,
"tags": [
[
"p",
"65273678eaf1bb1a8e89db021b034560512dabd8676adfe630a460b52107c656"
],
[
"e",
"8c07e6892e7b8432401a242f9344de1991cfbb0d658b8d2073eb57fcfb81dead",
"",
"root",
"65273678eaf1bb1a8e89db021b034560512dabd8676adfe630a460b52107c656"
],
[
"p",
"d2d44c47f97006febd0a2ec34de85482c6c2e01f23529f89d5b4f7168185153c"
],
[
"e",
"4987ffffac005ed087a6c7956e9ef25b8a71a6025f836199e3bc0c0dc7bb668d",
"",
"reply",
"d2d44c47f97006febd0a2ec34de85482c6c2e01f23529f89d5b4f7168185153c"
],
[
"proxy",
"https://sigmoid.social/@orbiterlab/113715580178379910",
"web"
],
[
"proxy",
"https://sigmoid.social/users/orbiterlab/statuses/113715580178379910",
"activitypub"
],
[
"L",
"pink.momostr"
],
[
"l",
"pink.momostr.activitypub:https://sigmoid.social/users/orbiterlab/statuses/113715580178379910",
"pink.momostr"
],
[
"-"
]
],
"content": "yes, project euler is around since about 10 years and started with ca. 160 problems. I also believe that most LLMs have seen solutions, however probably mostly in python. I don't think this is too influencing to other programming languages, so benchmarks to java etc. may be more significant.\n\nMy target is to evaluate the ability of LLMs to perform in an auto-coder engine and there I would not mind if those models perform because of some 'cheating'.",
"sig": "2b90b87f133b99a5b9af2661e7d2d53c7bdcc37e1f146e3bb3a2ca2c9141d88aa5bc4cc272a25566fde840b9f578c202d568125243cddf4f3c0544c5418e493b"
}