Cast by coolbeans1r.eth
1 reply
But those gpu's aren't even good and your fantastic cpu is nearly at idle?
It's perfect for what I need 🤷♂️ Im not some who wants max tokens/sec I want to run good size models efficiently since I live off grid. It's all about what makes sense, and powerful GPUs don't in my case.
I totally get you. I've got what's probably a similar machine for similar reasons. But my point is that GPU-maxing on this chipset isn't optimal for most models, because (while being a very nice mobile GPU) it's a relatively small and inefficient part of your total capacity on the machine. Your system has a strix halo NPU, which is meant to spread the load across your 65W + 115W processors efficiently already. Depending on your model family, there's a selection of various runtime configs/wrappers which can help orchestrate work across all of your efficient AI-optimized cores (with the 115w GPU doing occasional bursts of logistics work). You also get better memory management from not having fiddle with GPU-centric optimizations. ..It's still an 8-core NPU, so you're still stuck with low practical context limits (or loooong wait times). But IMO that's easier to supplement with 2-4b models on smartphones (for small calls) and a litellm router (for big calls). But you still get 8 cores for whatever else.
