Cast by coolbeans1r.eth
Show the cast this replies toIt's perfect for what I need 🤷♂️ Im not some who wants max tokens/sec I want to run good size models efficiently since I live off grid. It's all about what makes sense, and powerful GPUs don't in my case.
2 replies
I totally get you. I've got what's probably a similar machine for similar reasons. But my point is that GPU-maxing on this chipset isn't optimal for most models, because (while being a very nice mobile GPU) it's a relatively small and inefficient part of your total capacity on the machine. Your system has a strix halo NPU, which is meant to spread the load across your 65W + 115W processors efficiently already. Depending on your model family, there's a selection of various runtime configs/wrappers which can help orchestrate work across all of your efficient AI-optimized cores (with the 115w GPU doing occasional bursts of logistics work). You also get better memory management from not having fiddle with GPU-centric optimizations. ..It's still an 8-core NPU, so you're still stuck with low practical context limits (or loooong wait times). But IMO that's easier to supplement with 2-4b models on smartphones (for small calls) and a litellm router (for big calls). But you still get 8 cores for whatever else.
Tl;dr your system should go faster and use 1/3 the power using your NPU instead of gpumaxxing
Meh, I think you're trippin' broski. 🤷♂️ We must have different expectations of performance. No big deal 👌 what are you running? There's not many unified memory PC setups at the moment if you are running something similar.
Not trippin. AMD makes a separate "AI Engine" (the part that's supposed to run the models) which is a separate wafer from your GPU's. amd.com/en/products/processors/laptop/ryzen/ai-300-series/amd-ryzen-ai-max-385.html My main reason for mentioning it is that my setup is solar-assisted also, and I also setup my Ai mini-pc (minisforum um890) in gpu-only mode when I got it. It was running at 1/3 speed and using 3x power than it should have on paper. Anyway not a performance nitpick so much as "I've got one of those new tools too, and I think you're holding it the wrong way"... Like you said, unified systems are new and there's a lot of misconceptions to unlearn (like skeumorphically selecting a gpu-only runtime even when we paid the extra $ for an AI CPU)
