I was not expecting AI to be a catalyst to reinvigorate the game modding landscape, but AI tech seems to be filled with unintended consequences. Hopefully these types of processes can be achieved with local hardware running open models in the near future. Micro-scale community compute could completely facilitate a departure from the tech behemoths’ walled gardens.
This is really damn cool chip design. The liberation of technology for the proletariat by China really just keeps rolling along. When the AI crash hits the financial markets, China will be ready with commodity solutions.
I’d say self hosted models are the future, not local ones. As far as I’m aware, hardware requirements haven’t been going down, and hardware keeps getting more expensive, meaning you’re probably better off renting compute from a data center than you are shelling out tens of thousands for AI-capable hardware.
Or maybe I’m just not up to date on stuff. I only have a distilled deepseek R1 model that I barely ever touch, running on a 4060ti 16gb. I find single-purpose ML models a lot more useful and cost-efficient for homelab stuff than a general purpose LLM.
You can run Qwen 3.8 27b with 16-64gb vram depending on the quant. The 8bit version that would need 64gb is comparable to Opus 4.6 from less than a year ago. While 64gb vram is still expensive, but definitely in consumer grade already. Meanwhile, Alibaba is already working on a dedicated chip for running Qwen that doesn’t even need a ton of vram.
There are also interesting efforts like DwarfStar where they quantized parts of the model, so the reasoning layers stay at high qunats but the routing ones drop to low. And this lets you run a really large model locally.
And I do expect there will likely be more push towards making models single-purpose too. For example, you probably can train one to just work with a specific programming language really well. The holy grain would be to extract the reasoning parts of the model, and then build the context around that at runtime. Then you could have a fairly small model that could be adopted to a wide range of problems as needed.
I was not expecting AI to be a catalyst to reinvigorate the game modding landscape, but AI tech seems to be filled with unintended consequences. Hopefully these types of processes can be achieved with local hardware running open models in the near future. Micro-scale community compute could completely facilitate a departure from the tech behemoths’ walled gardens.
Qwen 3.8 is actually fairly capable already. You do need a beefy machine to run it, but it’s very much in consumer grade at this point. My expectation is that within a few years we’ll be able to buy something like this, hopefully at a cost of a raspberryPi to run it https://wccftech.com/alibabas-tsmc-built-5nm-risc-v-chip-xuantie-c950-now-runs-qwen-3-8-27b-model-natively-unlocking-massive-vertical-integration-tailwinds/
This is really damn cool chip design. The liberation of technology for the proletariat by China really just keeps rolling along. When the AI crash hits the financial markets, China will be ready with commodity solutions.
Exactly, local models are obviously the future, and China is building it.
I’d say self hosted models are the future, not local ones. As far as I’m aware, hardware requirements haven’t been going down, and hardware keeps getting more expensive, meaning you’re probably better off renting compute from a data center than you are shelling out tens of thousands for AI-capable hardware.
Or maybe I’m just not up to date on stuff. I only have a distilled deepseek R1 model that I barely ever touch, running on a 4060ti 16gb. I find single-purpose ML models a lot more useful and cost-efficient for homelab stuff than a general purpose LLM.
You can run Qwen 3.8 27b with 16-64gb vram depending on the quant. The 8bit version that would need 64gb is comparable to Opus 4.6 from less than a year ago. While 64gb vram is still expensive, but definitely in consumer grade already. Meanwhile, Alibaba is already working on a dedicated chip for running Qwen that doesn’t even need a ton of vram.
There are also interesting efforts like DwarfStar where they quantized parts of the model, so the reasoning layers stay at high qunats but the routing ones drop to low. And this lets you run a really large model locally.
And I do expect there will likely be more push towards making models single-purpose too. For example, you probably can train one to just work with a specific programming language really well. The holy grain would be to extract the reasoning parts of the model, and then build the context around that at runtime. Then you could have a fairly small model that could be adopted to a wide range of problems as needed.
emphasis mine. all ram is theirs already.
it’s cool tech, but until it can be much closer to ours, i don’t think it’d be wise to rely so heavily on it.
a lot of stuff would already crumble if humans suddenly needed to maintain it manually again, for any likely reason.
Give Chinese companies a couple of years to ramp up production, and I guarantee they will do with RAM what they’ve done with solar and EVs.
on one hand yes, they said that they’d do that and i believe it.
on the other, like solar and EVs, usians will try their darndest to block me from having it.
haha yeah, gonna have to smuggle ram into the country 🤣