If you're running local LLMs on a 16-24GB CUDA GPU this might interest you: "Raymond's fork" but with hot-swappable MTP meaning when there's room for it you get faster token generation, and when the VRAM is better used for the context it's ejected without a trace.
I used an LLM to make an LLM faster which I believe means I should start working at Cyberdyne Systems.
https://github.com/troed/llama.cpp-adaptive-kv-streaming
#LLM
Troed Sångberg
npub192…vy8wk
2026-09-13 06:51:15 UTC
Author Public Key
npub192kepsf3ypyzgsds9r93amadzta0efs05ftuxhl84m04e9xxhy9qevy8wkSeen on
wss://relay.momostr.pinkPublished at
2026-09-13 06:51:15 UTCEvent JSON
{
"id": "00f59cd0411f40e1f3551ee23c816524eb39d00d946c0ec54fcc52905bf220c7",
"pubkey": "2aad90c13120482441b028cb1eefad12fafca60fa257c35fe7aedf5c94c6b90a",
"created_at": 1789282275,
"kind": 1,
"tags": [
[
"t",
"llm"
],
[
"proxy",
"https://swecyb.com/@troed/117262403217943146",
"web"
],
[
"proxy",
"https://swecyb.com/users/troed/statuses/117262403217943146",
"activitypub"
],
[
"L",
"pink.momostr"
],
[
"l",
"pink.momostr.activitypub:https://swecyb.com/users/troed/statuses/117262403217943146",
"pink.momostr"
],
[
"-"
]
],
"content": "If you're running local LLMs on a 16-24GB CUDA GPU this might interest you: \"Raymond's fork\" but with hot-swappable MTP meaning when there's room for it you get faster token generation, and when the VRAM is better used for the context it's ejected without a trace.\n\nI used an LLM to make an LLM faster which I believe means I should start working at Cyberdyne Systems.\n\nhttps://github.com/troed/llama.cpp-adaptive-kv-streaming\n\n#LLM",
"sig": "a11f1f5a7ae1f061a8e7926219f0d940101e0d8f8d41496a333016f8f3716c5135e8c8d164b2b654edeb89c594ba000aee0c6921ef6a1ee97a7094a9f9d94263"
}