Huawei's New Atlas 300i Duo 96g Deepseek Ai Gpu Server Inference Card Fan Cooler Video Acceleration Graphic Card For Workstation - Buy Atlas 300i Duo 96g
huaweis Gpu
server Gpu
hua Wei Gpu Server
hua Wei Gpu
new Huawei Gpu Card
fan-cooled Graphics Card
workstation Graphics Card Product on Alibaba.com
You can run llama.cpp on CPU. LLM inference doesn’t need any features only GPUs typically have, that’s why it’s possible to make even simpler NPUs that can still run the same models. GPUs just tend to be faster. If the GPU in question is not faster than an equally priced CPU, you should use the CPU (better OS support).
Edit: I looked at a bunch real-world prices and benchmarks, and read the manual from Huawei and my new conclusion is that this is the best product on the market if you want to run a model at modest speed that doesn’t fit in 32GB but does in 96GB. Running multiple in parallel seems to range from unsupported to working poorly, so you should only expect to use one.
Original rest of the comment, made with the assumption that this was slower than it is, but had better drivers: The only benefit to this product over CPU is that you can slot multiple of them and they parallelise without needing to coordinate anything with the OS. It’s also a very linear cost increase as long as you have the PCIe lanes for it. For a home user with enough money for one or two of these, they would be much better served spending the money on a fast CPU and 256GB system RAM.
If not AI, then what use case do you think this serves better?
The point is that the GPU is designed for parallel computation. This happens to be useful for graphics, AI, and any other problem that can be expressed as a lot of independent calculations that can be executed in parallel. It’s a completely different architecture from a traditional CPU. This particular card is meant for running LLM models, and it will do it orders of magnitude faster than running this stuff on a CPU.
300i https://www.bilibili.com/video/BV15NKJzVEuU/
M4 https://github.com/itsmostafa/inference-speed-tests
It’s comparable to an M4, maybe a single order of magnitude faster than a ~1000 euro 9960X, at most, not multiple. And if we’re considering the option of buying used, since this is a brand new product and less available in western markets, the CPU-only option with an EPYC and more RAM will probably be a better local LLM computer for the cost of 2 of these and a basic computer.
I agree with your conclusion, but these are LPDDR4X, not DDR4 SDRAM. It’s significantly faster. No fans should also be seen as a positive, since they’re assuming the cards aren’t going to melt. It costs them very little to add visible active cooling to a 1000+ euro product.
According to this article https://www.hardware-corner.net/huawei-atlas-300i-duo-96gb-llm-20250830/
This card consists of two processors with a bandwidth of 204GB/s each
Compare that with the RTX 3090 which has 936GB/s bandwidth,
It really negates the extra memory capacity that will heavily bottleneck the processors.
That’s still faster than your expensive RGB XMP gamer RAM DDR5 CPU-only system, and you can depending on what you’re running saturate the buses independently, doubling the speed and matching a 5060 or there about. I disagree that you can categorise the speed as negating the capacity, as they’re different axis. You can run bigger models on this. Smaller models will run faster on a cheaper Nvidia. You aren’t getting 5080 performance and 6x the RAM for the same price, but I don’t think that’s a realistic ask either.
PCI-E 3.0, DDR4 memory, no drivers, no fans You would be better off any DDR4 CPU with a bunch of ram
When you definitely know the difference between what a CPU and a GPU does.
You can run llama.cpp on CPU. LLM inference doesn’t need any features only GPUs typically have, that’s why it’s possible to make even simpler NPUs that can still run the same models. GPUs just tend to be faster. If the GPU in question is not faster than an equally priced CPU, you should use the CPU (better OS support).
Edit: I looked at a bunch real-world prices and benchmarks, and read the manual from Huawei and my new conclusion is that this is the best product on the market if you want to run a model at modest speed that doesn’t fit in 32GB but does in 96GB. Running multiple in parallel seems to range from unsupported to working poorly, so you should only expect to use one.
Original rest of the comment, made with the assumption that this was slower than it is, but had better drivers:
The only benefit to this product over CPU is that you can slot multiple of them and they parallelise without needing to coordinate anything with the OS. It’s also a very linear cost increase as long as you have the PCIe lanes for it. For a home user with enough money for one or two of these, they would be much better served spending the money on a fast CPU and 256GB system RAM.If not AI, then what use case do you think this serves better?The point is that the GPU is designed for parallel computation. This happens to be useful for graphics, AI, and any other problem that can be expressed as a lot of independent calculations that can be executed in parallel. It’s a completely different architecture from a traditional CPU. This particular card is meant for running LLM models, and it will do it orders of magnitude faster than running this stuff on a CPU.
300i https://www.bilibili.com/video/BV15NKJzVEuU/
M4 https://github.com/itsmostafa/inference-speed-tests
It’s comparable to an M4, maybe a single order of magnitude faster than a ~1000 euro 9960X, at most, not multiple. And if we’re considering the option of buying used, since this is a brand new product and less available in western markets, the CPU-only option with an EPYC and more RAM will probably be a better local LLM computer for the cost of 2 of these and a basic computer.
M4 is a SoC architecture so it’s not directly comparable. It combines multiple chips for CPU and GPU that share memory on a single chip.
For 2000$ it “claims” to do 140 TOPS of INT8
When a Intel Core Ultra 7 265K does 33 TOPS of INT8 for 284$
Don’t get me wrong, I would LOVE to buy a chinese GPU at a reasonnable price but this isn’t even price competitive with CPUs let alone GPUs.
Again, completely different purposes here.
Alright, lets compare it to another GPU.
According to this source , the RTX 4070 costs about 500$ and does 466 TOPS of INT8
I dont know if TOPS is a good measurement tho (I dont have any experience with AI benchmarking)
Now go look at the amount of VRAM it has.
I agree with your conclusion, but these are LPDDR4X, not DDR4 SDRAM. It’s significantly faster. No fans should also be seen as a positive, since they’re assuming the cards aren’t going to melt. It costs them very little to add visible active cooling to a 1000+ euro product.
According to this article
https://www.hardware-corner.net/huawei-atlas-300i-duo-96gb-llm-20250830/
This card consists of two processors with a bandwidth of 204GB/s each Compare that with the RTX 3090 which has 936GB/s bandwidth, It really negates the extra memory capacity that will heavily bottleneck the processors.
That’s still faster than your expensive RGB XMP gamer RAM DDR5 CPU-only system, and you can depending on what you’re running saturate the buses independently, doubling the speed and matching a 5060 or there about. I disagree that you can categorise the speed as negating the capacity, as they’re different axis. You can run bigger models on this. Smaller models will run faster on a cheaper Nvidia. You aren’t getting 5080 performance and 6x the RAM for the same price, but I don’t think that’s a realistic ask either.