For anyone using self-hosted LLMs, I've had a lot of success with unsloth/qwen3.6-35b-a3b Q3_K_M model.
Inference speed is quite good, ~90 tok/sec. It does self context-expansion ("thinking"), visual input (which I haven't tested yet), and fits in less than 20GB VRAM.
I don't think speculative decoding works (yet?) to speed up inference, but I'm not sure how much impact that would ultimately have given that it's an MoE model and inference is only performed on a subset anyway.
Matt Hamilton
npub1gx…74lmh
2026-05-06 20:18:53 UTC
Author Public Key
npub1gx2gqstph00y56zgeshhme8dyt4kk4tgt0dypqdk49qwravk37jse74lmhPublished at
2026-05-06 20:18:53 UTCEvent JSON
{
"id": "3ab9877fca090d803285489eb5e206952d5a47807809b02f7cce665a47b22d52",
"pubkey": "4194804161bbde4a6848cc2f7de4ed22eb6b55685bda4081b6a940e1f5968fa5",
"created_at": 1778098733,
"kind": 1,
"tags": [
[
"proxy",
"https://noauthority.social/@eriner/116529478565860824",
"web"
],
[
"proxy",
"https://noauthority.social/users/eriner/statuses/116529478565860824",
"activitypub"
],
[
"L",
"pink.momostr"
],
[
"l",
"pink.momostr.activitypub:https://noauthority.social/users/eriner/statuses/116529478565860824",
"pink.momostr"
],
[
"-"
]
],
"content": "For anyone using self-hosted LLMs, I've had a lot of success with unsloth/qwen3.6-35b-a3b Q3_K_M model.\n\nInference speed is quite good, ~90 tok/sec. It does self context-expansion (\"thinking\"), visual input (which I haven't tested yet), and fits in less than 20GB VRAM.\n\nI don't think speculative decoding works (yet?) to speed up inference, but I'm not sure how much impact that would ultimately have given that it's an MoE model and inference is only performed on a subset anyway.",
"sig": "41ddf71c5bd7c9b7115b9c9243a00d751a9311d57bfb54d89a042015a9ae0ebe90a26e8294a0738aa5103b92ee992d735c3644ffeb45633677acd1da5598d104"
}