Cast by rathermercurial.eth
Show the cast this replies toBut those gpu's aren't even good and your fantastic cpu is nearly at idle?
1 reply
It's perfect for what I need 🤷♂️ Im not some who wants max tokens/sec I want to run good size models efficiently since I live off grid. It's all about what makes sense, and powerful GPUs don't in my case.
I totally get you. I've got what's probably a similar machine for similar reasons. But my point is that GPU-maxing on this chipset isn't optimal for most models, because (while being a very nice mobile GPU) it's a relatively small and inefficient part of your total capacity on the machine. Your system has a strix halo NPU, which is meant to spread the load across your 65W + 115W processors efficiently already. Depending on your model family, there's a selection of various runtime configs/wrappers which can help orchestrate work across all of your efficient AI-optimized cores (with the 115w GPU doing occasional bursts of logistics work). You also get better memory management from not having fiddle with GPU-centric optimizations. ..It's still an 8-core NPU, so you're still stuck with low practical context limits (or loooong wait times). But IMO that's easier to supplement with 2-4b models on smartphones (for small calls) and a litellm router (for big calls). But you still get 8 cores for whatever else.
Tl;dr your system should go faster and use 1/3 the power using your NPU instead of gpumaxxing
Meh, I think you're trippin' broski. 🤷♂️ We must have different expectations of performance. No big deal 👌 what are you running? There's not many unified memory PC setups at the moment if you are running something similar.
