We just tested* a legal LLM and at least about 1 3rd of the cases it returned were not relevant to the question or the cases did not exist.
I tested it filing in citations only and it seemed to do that reasonably well but it still got two wrong. This was just looking up the actual citations of a case with a given name.
(Edit: *I should clarify that this thing came bundled in another service we already have. So this is not a service we sought out. It was shoved into my face.)
Jonathan
npub1xp…4ek9h
2026-06-25 12:17:46 UTC
in reply to nevent1q…q9jy
Author Public Key
npub1xpdxyg4ufg24vp9g66tx5klhtm6k5ccd6yd5sykhgljgl5f9kukqx4ek9hSeen on
wss://relay.momostr.pinkPublished at
2026-06-25 12:17:46 UTCEvent JSON
{
"id": "cf6d7173d722e9ec3a9253e652cec15641ff183eafd94f450b0488932861b94d",
"pubkey": "305a6222bc4a155604a8d6966a5bf75ef56a630dd11b4812d747e48fd125b72c",
"created_at": 1782389866,
"kind": 1,
"tags": [
[
"p",
"305a6222bc4a155604a8d6966a5bf75ef56a630dd11b4812d747e48fd125b72c"
],
[
"e",
"cc6c22a9b654d22d2a44eede3de68b7740db9b232471502c3b4294bdf46a9e0a",
"",
"root",
"305a6222bc4a155604a8d6966a5bf75ef56a630dd11b4812d747e48fd125b72c"
],
[
"e",
"358f74b0fbfb7173075630f8920f139b049f1428cd32c6083353fd59759241e8",
"",
"reply",
"305a6222bc4a155604a8d6966a5bf75ef56a630dd11b4812d747e48fd125b72c"
],
[
"proxy",
"https://mastodon.social/@jmcrookston/116810702263013629",
"web"
],
[
"proxy",
"https://mastodon.social/users/jmcrookston/statuses/116810702263013629",
"activitypub"
],
[
"L",
"pink.momostr"
],
[
"l",
"pink.momostr.activitypub:https://mastodon.social/users/jmcrookston/statuses/116810702263013629",
"pink.momostr"
],
[
"-"
]
],
"content": "We just tested* a legal LLM and at least about 1 3rd of the cases it returned were not relevant to the question or the cases did not exist.\n\nI tested it filing in citations only and it seemed to do that reasonably well but it still got two wrong. This was just looking up the actual citations of a case with a given name.\n\n(Edit: *I should clarify that this thing came bundled in another service we already have. So this is not a service we sought out. It was shoved into my face.)",
"sig": "1aec76fec9b86f51909df3e954957fc9463062b527e1467f37cadad737c5e41a8716842f460926227c5000758cf336b551ad30a876d5b90a2388ab24a490c815"
}