Wow! thx for all the works here. I wish more people can see the methodology.
My Mac is only 16GB 😅 We need to find one of those people above who have big budget
https://www.ebay.com/itm/257712960417
Yi Cui
onekq
AI & ML interests
Benchmark, Code Generation Model
Recent Activity
repliedto their post about 1 hour ago
My take on device-side inference: it's all about high bandwidth memory (thinking about it, this holds for the cloud too).
MacBooks enjoy incidental capacity of apple silicon, but per-device RAM is too low (16 to 24GB), only sufficient for a decent SLM. 512GB is the highest you can go (Kimi K2*). Counting MLX downloads of Kimi K2* on Huggingface, I estimate the user base to be <25K.
On the other hand, the newly debuted DGX station (Nvidia) has 748GB, which can fit in the latest Kimi, DS, and Qwen. Also the quantization options of CUDA is way better than MLX.
For high-end inferencing, I place my bet on workstations over Macs. posted an update about 19 hours ago
My take on device-side inference: it's all about high bandwidth memory (thinking about it, this holds for the cloud too).
MacBooks enjoy incidental capacity of apple silicon, but per-device RAM is too low (16 to 24GB), only sufficient for a decent SLM. 512GB is the highest you can go (Kimi K2*). Counting MLX downloads of Kimi K2* on Huggingface, I estimate the user base to be <25K.
On the other hand, the newly debuted DGX station (Nvidia) has 748GB, which can fit in the latest Kimi, DS, and Qwen. Also the quantization options of CUDA is way better than MLX.
For high-end inferencing, I place my bet on workstations over Macs.Organizations
replied to their post about 1 hour ago
posted an update about 19 hours ago
Post
65
My take on device-side inference: it's all about high bandwidth memory (thinking about it, this holds for the cloud too).
MacBooks enjoy incidental capacity of apple silicon, but per-device RAM is too low (16 to 24GB), only sufficient for a decent SLM. 512GB is the highest you can go (Kimi K2*). Counting MLX downloads of Kimi K2* on Huggingface, I estimate the user base to be <25K.
On the other hand, the newly debuted DGX station (Nvidia) has 748GB, which can fit in the latest Kimi, DS, and Qwen. Also the quantization options of CUDA is way better than MLX.
For high-end inferencing, I place my bet on workstations over Macs.
MacBooks enjoy incidental capacity of apple silicon, but per-device RAM is too low (16 to 24GB), only sufficient for a decent SLM. 512GB is the highest you can go (Kimi K2*). Counting MLX downloads of Kimi K2* on Huggingface, I estimate the user base to be <25K.
On the other hand, the newly debuted DGX station (Nvidia) has 748GB, which can fit in the latest Kimi, DS, and Qwen. Also the quantization options of CUDA is way better than MLX.
For high-end inferencing, I place my bet on workstations over Macs.
- 2 replies
posted an update 4 days ago
Post
144
My biggest takeaway from Ben Thompson interview is the legacy of this round of "bubble".
For railroad, it was the track, and dot com, fiber. Compared to these two, GPUs depreciate too fast.
The answer is power plants.
For railroad, it was the track, and dot com, fiber. Compared to these two, GPUs depreciate too fast.
The answer is power plants.
posted an update 7 days ago
Post
1999
Back of the envelope calculation: Ox Alpha has been out for a week with 130K users and 7T tokens burned.
Method 1: Assuming Ox Alpha is the GLM 5.3 class, it has roughly the same active param count as DeepSeek V4 Pro (~40B vs 49B), derive the cost by the floor price DS has ever published.
Method 2: Assuming the users are concentrated within an 8-hour working window each day, derive number of H800 nodes needed (~700 nodes at $2/GPU-hour).
In both methods, I assume 90/10 IO split and 60% cache hit. Both methods come to $2M.
I think the ROI is awesome: (1) publicity and (2) data harvesting.
Method 1: Assuming Ox Alpha is the GLM 5.3 class, it has roughly the same active param count as DeepSeek V4 Pro (~40B vs 49B), derive the cost by the floor price DS has ever published.
Method 2: Assuming the users are concentrated within an 8-hour working window each day, derive number of H800 nodes needed (~700 nodes at $2/GPU-hour).
In both methods, I assume 90/10 IO split and 60% cache hit. Both methods come to $2M.
I think the ROI is awesome: (1) publicity and (2) data harvesting.
- 1 reply
replied to their post 8 days ago
Whoever this is, they give away so much compute for a model launch
replied to their post 8 days ago
Most likely
replied to their post 8 days ago
Lots of SLMs, like the 27b one from Qwen
posted an update 12 days ago
Post
106
I for one believe DeepMind will ship one more frontier model.
In any case GCP is irrelevant here. GDM should still have more compute than DeepSeek, Kimi, or GLM, maybe all of them combined.
In any case GCP is irrelevant here. GDM should still have more compute than DeepSeek, Kimi, or GLM, maybe all of them combined.
posted an update 15 days ago
Post
3229
Lots of attentions are now on GPU residual value. I find car analogy to be useful.
* Both new and used cars can do the same job (you can run the latest model on 6-year-old A100)
* New cars are more efficient (higher performance per power draw)
* It's not the year, but mileage and maintenance
The last point is undeveloped. We need Carfax and KBB for GPUs.
And there will be lemon GPUs.
* Both new and used cars can do the same job (you can run the latest model on 6-year-old A100)
* New cars are more efficient (higher performance per power draw)
* It's not the year, but mileage and maintenance
The last point is undeveloped. We need Carfax and KBB for GPUs.
And there will be lemon GPUs.
- 1 reply
posted an update 19 days ago
Post
2763
This is an easy-to-remember pattern.
GLM and Kimi are in Beijing, DeepSeek and Qwen are in Hangzhou.
OpenAI and Anthropic are in SF, xAI and Meta are in the peninsula.
In both China and the Bay Area, token price of one area is ~1/3 of the other area.
GLM and Kimi are in Beijing, DeepSeek and Qwen are in Hangzhou.
OpenAI and Anthropic are in SF, xAI and Meta are in the peninsula.
In both China and the Bay Area, token price of one area is ~1/3 of the other area.
- 1 reply
posted an update 21 days ago
Post
1754
I think contributor pricing is a good idea. But things will get sophisticated very quickly.
* data sharing tiers from exclusive to syndication
* provenance and lineage
* scoping such as duration and geofencing
and their levels of enforcement (best practices, standards, laws)
* data sharing tiers from exclusive to syndication
* provenance and lineage
* scoping such as duration and geofencing
and their levels of enforcement (best practices, standards, laws)
posted an update 22 days ago
Post
2667
Looking forward to the Qwen 3.8 model drop, and congratulations on joining the trillion parameters club.
But my eyes are on the promised 27B model. Small (<50B) models decline on OpenRouter, because they are being run on local devices. If people see the family trees here on HF they will understand.
But my eyes are on the promised 27B model. Small (<50B) models decline on OpenRouter, because they are being run on local devices. If people see the family trees here on HF they will understand.
- 2 replies
replied to their post 24 days ago
That's a brilliant strategy. Mystery is why Kimi and GLM didn't follow it.
replied to their post 24 days ago
You are right. It's margin, simple as that.
Dax (OpenCode) has a tweet with lots of discussions.
posted an update 25 days ago
Post
2433
DeepSeek plans to raise token prices. I don't think this is because they are bleeding, but they are overwhelmed. If your price is 1/10 of your affiliate vendors, you can't leverage their resources. Markup is the only way to diverge traffic away.
Sadly I haven't found discussions on differentiators enabling DS to balance cost at such low prices. All software solutions (that we know of) are accessible by other vendors. If you attribute it to electricity or hardware, you can't explain why GLM and Kimi charge so much for their APIs.
This is where our attention should be (but distracted by things above).
Sadly I haven't found discussions on differentiators enabling DS to balance cost at such low prices. All software solutions (that we know of) are accessible by other vendors. If you attribute it to electricity or hardware, you can't explain why GLM and Kimi charge so much for their APIs.
This is where our attention should be (but distracted by things above).
- 5 replies
posted an update 26 days ago
Post
144
Jeff Dean departure is 100% unexpected. It's impossible to process this.
And yes, it matters more than other exec shakeups today.
And yes, it matters more than other exec shakeups today.
posted an update 27 days ago
Post
2355
Many developers discovered that the native DeepSeek API has higher cache-hit rate than neocloud APIs hosting the same DS models.
My speculation is that DS aggressively kills its old models. It has released 18 models thus far, and only 2 are being served now (v4 pro and flash).
This is tough to customers who don't want to upgrade (migrate or leave), but effectively boost the serving capacity to the same model, i.e. more woods behind fewer arrows.
My speculation is that DS aggressively kills its old models. It has released 18 models thus far, and only 2 are being served now (v4 pro and flash).
This is tough to customers who don't want to upgrade (migrate or leave), but effectively boost the serving capacity to the same model, i.e. more woods behind fewer arrows.
posted an update about 1 month ago
Post
2227
There has been a leaked memo (now struck down) from the founder of DeepSeek. I'm not here to circulate it, but comment on the minimum-effort evolutionary path he proposed.
LLM->CoT->Agent->Self-improvement->Singularity->Physical
This makes sense to me: even at the agent stage I learn world models much faster than when I learned LLM at the LLM stage.
But this means humans are still needed beyond the digital singularity, until robots can close their own loop: eval, manufacturing, self improvement, i.e. physical singularity.
LLM->CoT->Agent->Self-improvement->Singularity->Physical
This makes sense to me: even at the agent stage I learn world models much faster than when I learned LLM at the LLM stage.
But this means humans are still needed beyond the digital singularity, until robots can close their own loop: eval, manufacturing, self improvement, i.e. physical singularity.