Cast by rathermercurial.eth
Show the cast this replies toI totally get you. I've got what's probably a similar machine for similar reasons. But my point is that GPU-maxing on this chipset isn't optimal for most models, because (while being a very nice mobile GPU) it's a relatively small and inefficient part of your total capacity on the machine. Your system has a strix halo NPU, which is meant to spread the load across your 65W + 115W processors efficiently already. Depending on your model family, there's a selection of various runtime configs/wrappers which can help orchestrate work across all of your efficient AI-optimized cores (with the 115w GPU doing occasional bursts of logistics work). You also get better memory management from not having fiddle with GPU-centric optimizations. ..It's still an 8-core NPU, so you're still stuck with low practical context limits (or loooong wait times). But IMO that's easier to supplement with 2-4b models on smartphones (for small calls) and a litellm router (for big calls). But you still get 8 cores for whatever else.
2 replies
Tl;dr your system should go faster and use 1/3 the power using your NPU instead of gpumaxxing
Meh, I think you're trippin' broski. 🤷♂️ We must have different expectations of performance. No big deal 👌 what are you running? There's not many unified memory PC setups at the moment if you are running something similar.
Not trippin. AMD makes a separate "AI Engine" (the part that's supposed to run the models) which is a separate wafer from your GPU's. amd.com/en/products/processors/laptop/ryzen/ai-300-series/amd-ryzen-ai-max-385.html My main reason for mentioning it is that my setup is solar-assisted also, and I also setup my Ai mini-pc (minisforum um890) in gpu-only mode when I got it. It was running at 1/3 speed and using 3x power than it should have on paper. Anyway not a performance nitpick so much as "I've got one of those new tools too, and I think you're holding it the wrong way"... Like you said, unified systems are new and there's a lot of misconceptions to unlearn (like skeumorphically selecting a gpu-only runtime even when we paid the extra $ for an AI CPU)
Broski, I think you misunderstand what the NPU is for. Your system is limited by data throughput significantly more than mine is, so your need to run on the CPU is more of a product of hardware limitations(necessity) and not one of optimum performance. You have about 60+gbps of bandwidth, I am running 256gbps 🤷♂️
