This is really damn cool chip design. The liberation of technology for the proletariat by China really just keeps rolling along. When the AI crash hits the financial markets, China will be ready with commodity solutions.
I’d say self hosted models are the future, not local ones. As far as I’m aware, hardware requirements haven’t been going down, and hardware keeps getting more expensive, meaning you’re probably better off renting compute from a data center than you are shelling out tens of thousands for AI-capable hardware.
Or maybe I’m just not up to date on stuff. I only have a distilled deepseek R1 model that I barely ever touch, running on a 4060ti 16gb. I find single-purpose ML models a lot more useful and cost-efficient for homelab stuff than a general purpose LLM.
You can run Qwen 3.8 27b with 16-64gb vram depending on the quant. The 8bit version that would need 64gb is comparable to Opus 4.6 from less than a year ago. While 64gb vram is still expensive, but definitely in consumer grade already. Meanwhile, Alibaba is already working on a dedicated chip for running Qwen that doesn’t even need a ton of vram.
There are also interesting efforts like DwarfStar where they quantized parts of the model, so the reasoning layers stay at high qunats but the routing ones drop to low. And this lets you run a really large model locally.
And I do expect there will likely be more push towards making models single-purpose too. For example, you probably can train one to just work with a specific programming language really well. The holy grain would be to extract the reasoning parts of the model, and then build the context around that at runtime. Then you could have a fairly small model that could be adopted to a wide range of problems as needed.
This is really damn cool chip design. The liberation of technology for the proletariat by China really just keeps rolling along. When the AI crash hits the financial markets, China will be ready with commodity solutions.
Exactly, local models are obviously the future, and China is building it.
I’d say self hosted models are the future, not local ones. As far as I’m aware, hardware requirements haven’t been going down, and hardware keeps getting more expensive, meaning you’re probably better off renting compute from a data center than you are shelling out tens of thousands for AI-capable hardware.
Or maybe I’m just not up to date on stuff. I only have a distilled deepseek R1 model that I barely ever touch, running on a 4060ti 16gb. I find single-purpose ML models a lot more useful and cost-efficient for homelab stuff than a general purpose LLM.
You can run Qwen 3.8 27b with 16-64gb vram depending on the quant. The 8bit version that would need 64gb is comparable to Opus 4.6 from less than a year ago. While 64gb vram is still expensive, but definitely in consumer grade already. Meanwhile, Alibaba is already working on a dedicated chip for running Qwen that doesn’t even need a ton of vram.
There are also interesting efforts like DwarfStar where they quantized parts of the model, so the reasoning layers stay at high qunats but the routing ones drop to low. And this lets you run a really large model locally.
And I do expect there will likely be more push towards making models single-purpose too. For example, you probably can train one to just work with a specific programming language really well. The holy grain would be to extract the reasoning parts of the model, and then build the context around that at runtime. Then you could have a fairly small model that could be adopted to a wide range of problems as needed.