ANotSoI

As I mentioned in my last post about obtaining a GPU, I’ve also obtained a new laptop to replace the 10 year old Macbook Pro I mentioned in my post titled, “Running My First Local LLM.”

After some investigation as to what would possibly enable my AI activities, I decided I wanted something with AMD’s new Ryzen AI processor because they include an NPU, which should accelerate AI workloads. The caveat being, software has to be designed to use it, unfortunately. But still, if I was buying something new, I wanted something forward looking. So I selected HP’s Omnibook Ultra 14 because it was one of the few laptops to replace my Macbook available specifically with the Ryzen AI 9 HX 375, which achieves 55 TOPS (5 TOPS/10% more than the HX 370) with it’s NPU, for a combined 85 TOPS between the CPU, GPU, and NPU.

The other option I was heavily considering was waiting for an M4-based Macbook Air, but because I’m eager to do these experiements and it’s not yet available, I decided against this option. Also because the battery life on my 2014 Macbook is becoming abysmal, I didn’t want to wait several more months.

Granted, the value of these NPUs is somewhat theoretical at this point, as utilizing one, for example with Ollama is currently not possible. Hopefully this changes and it becomes as trivial as it was to utilize the RTX 3060 – install drivers and run the software.

Additionally, I landed on the Ryzen AI based processor because of the integrated GPU, the 890M, which is supposedly one of the best iGPUs currently, and on the horizon.

In my next blog post, I’m going to get into the testing methods I’ve started setting up to do some comparisions – namely, a python wrapper to feed the same prompts to multiple models, and will liklely share that code via github.

Other future posts will include a comparison of the quality of output from various models – for example, how much better is a response from a 70-billion parameter model than a 7-billion parameter model of the same type?