You can run Qwen 3.8 27b with 16-64gb vram depending on the quant. The 8bit version that would need 64gb is comparable to Opus 4.6 from less than a year ago. While 64gb vram is still expensive, but definitely in consumer grade already. Meanwhile, Alibaba is already working on a dedicated chip for running Qwen that doesn’t even need a ton of vram.
There are also interesting efforts like DwarfStar where they quantized parts of the model, so the reasoning layers stay at high qunats but the routing ones drop to low. And this lets you run a really large model locally.
And I do expect there will likely be more push towards making models single-purpose too. For example, you probably can train one to just work with a specific programming language really well. The holy grain would be to extract the reasoning parts of the model, and then build the context around that at runtime. Then you could have a fairly small model that could be adopted to a wide range of problems as needed.
You can run Qwen 3.8 27b with 16-64gb vram depending on the quant. The 8bit version that would need 64gb is comparable to Opus 4.6 from less than a year ago. While 64gb vram is still expensive, but definitely in consumer grade already. Meanwhile, Alibaba is already working on a dedicated chip for running Qwen that doesn’t even need a ton of vram.
There are also interesting efforts like DwarfStar where they quantized parts of the model, so the reasoning layers stay at high qunats but the routing ones drop to low. And this lets you run a really large model locally.
And I do expect there will likely be more push towards making models single-purpose too. For example, you probably can train one to just work with a specific programming language really well. The holy grain would be to extract the reasoning parts of the model, and then build the context around that at runtime. Then you could have a fairly small model that could be adopted to a wide range of problems as needed.