ANotSoI

My last couple posts were a bit of a tangent from the primary purpose of starting this journey based on my new fangled laptop with a “difficult to make use of at best” NPU. In this post, I’ll get back to my main goal, which is to gain some experience with and test some different LLMs. If you recall from my previous post, “New Hardware – A GPU” I’ll be running these tests on a VM to which I have passed through the 12GB RTX 3060 and have provisioned with 8 vCPUs (Core i7-9700k) and 56GB of memory.

Following the scientific method as closely as possible, I want to be able to minimize variables and run exact same prompt against each LLM to make sure the LLM itself is the only variable, not a minor difference in the prompt I give them. So rather than copy and paste and monitor the execution, I decided to create a simple Python wrapper. Also, as my Python skills are a little rusty… so what better opportunity to put AI to the test and have it write it for me.

I used Google Gemini to create this, and only made very minor modifications to make it work how I want.

import subprocess
import os
import re

def sanitize_filename(model_name):
    """Sanitizes a model name for use as a filename."""
    return re.sub(r'[^a-zA-Z0-9_]', '', model_name).lower()

def run_ollama_model(model, prompt):
    """Runs an Ollama model with the given prompt and captures output."""
    try:
        filename = f"{sanitize_filename(model)}.txt"
        process = subprocess.run(
            ["ollama", "run", "--verbose", model],
            input=prompt,
            capture_output=True,
            text=True,  # Important for decoding output as text
            check=True  # Raise an exception for non-zero exit codes
        )

        with open(filename, "w") as f:
            f.write(process.stdout)
            f.write(process.stderr) # Capture stderr as well

        print(f"Output for model '{model}' saved to '{filename}'")
        return True

    except subprocess.CalledProcessError as e:
        filename = f"{sanitize_filename(model)}.txt"
        with open(filename, "w") as f:
            f.write(e.stderr) # Capture the error message
        print(f"Error running model '{model}'. Output saved to '{filename}': {e}")
        return False
    except FileNotFoundError:
        print("Error: Ollama not found. Make sure it's installed and in your PATH.")
        return False
    except Exception as e: # Catch any other exceptions
        print(f"An unexpected error occurred: {e}")
        return False


def get_ollama_models():
    """Gets the list of installed Ollama models."""
    try:
      result = subprocess.run(["ollama", "list"], capture_output=True, text=True, check=True)
      models_output = result.stdout
      models = [line.split()[0] for line in models_output.strip().split('\n')[1:]] # Skip the header line
      return models
    except subprocess.CalledProcessError as e:
      print(f"Error listing models: {e}")
      return []
    except FileNotFoundError:
      print("Error: Ollama not found. Make sure it's installed and in your PATH.")
      return []



if __name__ == "__main__":
    prompt = """[prompt text here]"""

    models = get_ollama_models()

    if models:
        for model in models:
            run_ollama_model(model, prompt)

        print("Finished processing all models.")

I won’t list out all the prompts and the whole chain, but I went through a few iterations of it so it includes the following core functions:

• Get a list of all models installed in ollama
• Give the same prompt to each model
• Record the output from each model as a separate file in the pwd

Let’s see it in action (in this version, the prompt is hard coded in the script):

We can see, by running nvtop, the activity on the GPU – note the different GPU MEM consumption based on the size of the model:

… and you can see what happens when you load a model which is larger than you have room for in your GPU’s memory:

… it loads a portion of the model into GPU MEM and the rest into HOST MEM and as it runs we can see that the GPU is nearly idle because it’s waiting for the CPU to do its part using the portion of the dataset that was loaded into HOST MEM because it wouldn’t fit in the GPU MEM.

And we’re done:

Let’s take a look at some of the results… you’ll notice the prompt I gave it (Write a short story, about 1000 words, about a young bunny who dreams of becoming THE Easter Bunny.) is not a coding task and most of these are models trained for writing code – but that’s not the point of this test. This is just to demo my python script. So let’s take a couple models that aren’t trained for coding, like gemma3:12b:

username@llmsandbox:~/Documents/ai/shortstory_test$ cat gemma312b.txt
Pipkin was, by all accounts, an average bunny. He had fluffy grey fur, a twitchy nose, and a penchant for dandelion greens. But Pipkin harbored a secret, a yearning so vast it dwarfed the biggest carrot patch in Meadowbrook Valley. Pipkin dreamed of being THE Easter Bunny.

Not just any bunny, mind you. He didn't want to be one of the helpers, the little foot soldiers who scattered chocolate eggs and hid plush toys. He wanted to be the one who oversaw the entire operation, the grand orchestrator of the sweetest holiday of the year.

The current Easter Bunny, Bartholomew, was… established. He was ancient, it seemed, with a snowy beard that trailed all the way to his paws and a reputation for being delightfully predictable. Every year, the same trails, the same candies, the same carefully curated basket fillers. It was comfortable, sure, but Pipkin craved innovation. He envisioned glittering trails of stardust, chocolate sculptures, and personalized Easter baskets based on each child’s interests.

"Silly boy," his mother would chuckle, watching him bounce around with an overflowing basket of ideas. "Bartholomew's been doing this for centuries. He knows what he's doing."

But Pipkin couldn’t help it. He spent his days observing Bartholomew. He’s memorized the bunny’s routines, the weight of the egg sacks, the subtle hop that signaled the turn of the season. He even tried mimicking Bartholomew’s iconic, dignified wave. It ended in him tumbling head over heels.

“You’re wasting your time, Pipkin,” his older brother, Bramble, would scoff. “Bartholomew chooses his successor based on wisdom, not enthusiasm.”

Pipkin understood. He was young. He was impulsive. He had a lot to learn. But he refused to let that extinguish his dream. He began to study. He poured over ancient bunny lore, learning about the history of Easter, the symbolism of eggs, and the importance of kindness and generosity. He practiced hopping silently, delivering treats with a graceful flourish, and even attempted to carve miniature chocolate sculptures (the results were… abstract).

The annual “Bunny Observation Day” was the most crucial test. All young bunnies in Meadowbrook Valley were observed by Bartholomew and his council of elder bunnies. They assessed potential, evaluated suitability, and whispered amongst themselves about who might be a worthy successor.

Pipkin spent weeks agonizing over his performance. He meticulously planned his route, rehearsed his greetings, and even designed a miniature, eco-friendly basket to showcase his commitment to sustainability.

The day arrived cold and crisp. The forest felt thick with expectation. Pipkin hopped onto the designated route, his heart pounding against his ribs. He scattered candy eggs with precision, checked for any obstacles, and offered a cheerful greeting to a family of deer, who, surprisingly, responded with a gentle nod.

But then, disaster struck. A mischievous badger, notorious for disrupting springtime festivities, emerged from the undergrowth and snatched a handful of chocolate eggs, scattering them across the path.  Pipkin froze. This was it. His carefully crafted performance was crumbling before his eyes.

He could have panicked. He could have chased the badger. But instead, he saw a little girl, no older than five, standing on the edge of the path, her face crumpling with disappointment.

Pipkin took a deep breath. Instead of focusing on his ruined plan, he turned to the little girl. "Oh dear!" he exclaimed, feigning concern. "Those naughty badgers! Don't worry, we're going to find them and get those eggs back!"

He then gathered the other young bunnies assisting with the observation and, with a playful energy, organized a "badger egg hunt.” The children giggled and scampered through the woods, the adults chuckling at their antics. The atmosphere shifted from disappointment to laughter, and even Bartholomew’s usually stoic face cracked a small smile.

Pipkin, meanwhile, was demonstrating something far more valuable than perfect egg distribution. He was showing adaptability, resilience, and a genuine concern for the joy of others.

When the observation day ended, Bartholomew approached Pipkin. The air hung heavy with anticipation.

“You… you didn’t follow the established route perfectly,” Bartholomew began, his voice raspy.

Pipkin’s ears drooped. He's failed.

“But you did something much more important, Pipkin,” Bartholomew continued, his eyes twinkling. “You adapted. You engaged. You remembered that Easter isn’t just about eggs and candy. It’s about spreading joy and creating memories.”

He paused, his snowy beard swaying slightly. "Enthusiasm is a gift, Pipkin, but it needs to be tempered with kindness and ingenuity. You showed me both today."

Pipkin’s heart leaped.  He couldn't quite believe it.

"I'm not going to name you THE Easter Bunny immediately,” Bartholomew added, a knowing smile on his face. “There’s still much you need to learn. But I am offering you a place on my council. You will work alongside me, learning the nuances of the role, helping to shape the future of Easter."

Pipkin’s jaw dropped. It wasn’t the immediate coronation he’s dreamed of, but it was a step, a chance to learn from the master himself.

"Thank you, Bartholomew," Pipkin breathed, his voice thick with emotion. "I won't let you down."

Over the next few years, Pipkin flourished under Bartholomew’s tutelage. He learned about the responsibility that came with the role, the importance of tradition, and the delicate balance between innovation and respect for the past. He learned that being THE Easter Bunny wasn't just about grand gestures and flashy displays. It was about understanding the spirit of the holiday and ensuring it brought joy to every child, every year.

He did introduce a few small innovations – biodegradable glitter for the trails, personalized maps for the egg hunt, and a program to donate baskets to children in need.  Bartholomew, with his ancient wisdom, would often offer a gentle nod of approval, a silent acknowledgement of Pipkin’s evolving style.

One crisp spring morning, years later, Bartholomew gathered his council. He was visibly frail, his snowy beard even longer than Pipkin remembered.

“The time has come,” Bartholomew announced, his voice weak but clear.  “I am ready to retire.”

A collective gasp rippled through the council.

“And I am pleased to announce,” Bartholomew continued, his gaze fixed on Pipkin, "that Pipkin will be my successor.”

The forest erupted in cheers. Pipkin, standing tall and proud, felt a wave of gratitude and responsibility wash over him. He was ready. He had learned. He was THE Easter Bunny.

And as he prepared to take his first solo flight, scattering chocolate eggs and spreading joy across the world, Pipkin remembered a little girl, a mischievous badger, and the invaluable lesson he learned: that being the best Easter Bunny wasn’t about being perfect, but about embracing the unexpected, spreading kindness, and making every child’s Easter dream come true.






total duration:       52.441731243s
load duration:        7.367085406s
prompt eval count:    34 token(s)
prompt eval duration: 207.970384ms
prompt eval rate:     163.48 tokens/s
eval count:           1515 token(s)
eval duration:        44.865781299s
eval rate:            33.77 tokens/s
username@llmsandbox:~/Documents/ai/shortstory_test$

And compare it to deepseek-r1:70b:

username@llmsandbox:~/Documents/ai/shortstory_test$ cat deepseekr170b.txt
<think>
Okay, so I need to write a short story about a young bunny named Benny who wants to become the Easter Bunny. It should be around 1000 words and suitable for children. Let me think through this step by step.

First, I'll start by setting the scene in a cozy little burrow where Benny lives with his family. Introducing Benny as a young bunny with big dreams will make him relatable to readers. His room should have elements that show his passion, like Easter Bunny posters and maybe some practice eggs he's decorated.

Next, I need to establish Benny's motivation. Maybe the current Easter Bunny is Mr. Hoppington, who's getting older, so there's a natural transition point. Benny can be inspired by seeing Mr. Hoppington in action, which sparks his desire to take on the role one day.

I should include some challenges or obstacles Benny faces. Perhaps he tries to deliver eggs but fails at first, teaching him perseverance. It would be nice to show his determination and how he learns from his mistakes, maybe with the support of Mr. Hoppington himself.

Having a training arc where Mr. Hoppington mentors Benny adds depth. This can involve lessons on things beyond just delivering eggs, like the spirit of Easter—love, kindness, and joy. This will help Benny understand the true meaning behind being the Easter Bunny.

Finally, when Mr. Hoppington retires, Benny should take over, successfully delivering eggs and making children happy. A heartfelt farewell scene where Mr. Hoppington passes on his mantle would be a nice touch, showing continuity and legacy.

I need to ensure the language is simple and engaging for children, with some dialogue to bring characters to life. Including descriptive details about the burrow, the meadow, and Easter elements will make the story vivid.

Also, adding emotional moments, like Benny's determination after failing and his joy when he succeeds, will resonate well. The story should end on a positive note, emphasizing following one's dreams and the importance of kindness and giving.

I'll structure it with an introduction, Benny's inspiration, his challenges, the mentorship, the big night as Easter Bunny, and a conclusion that wraps up his journey nicely.
</think>

**Title: Benny's Journey to Becoming the Easter Bunny**

In a snug burrow beneath a grand oak tree, nestled in a meadow blooming with vibrant wildflowers, lived a young bunny named Benny. His eyes sparkled with big dreams, for Benny aspired to become the next Easter Bunny. His room was adorned with posters of Mr. Hoppington, the current Easter Bunny, and shelves filled with eggs he had meticulously decorated, each one a testament to his passion.

Benny's inspiration came from watching Mr. Hoppington deliver eggs each Easter. With a flick of his bushy tail and a twinkle in his eye, Mr. Hoppington brought joy to children everywhere. Benny longed to follow in those footsteps but faced doubts. "Can I really do this?" he wondered.

One day, determined to prove himself, Benny set out to deliver eggs. He packed a basket with care and hopped off. However, the task was harder than it looked. Eggs rolled away, hiding spots were tricky, and fatigue soon set in. Discouraged, Benny returned home, his ears drooping.

That night, as Benny pondered his failure, there was a knock at the door. It was Mr. Hoppington himself. With wisdom in his voice, he said, "Delivering eggs isn't just about speed or perfection; it's about love and joy." He offered to mentor Benny, teaching him the true spirit of Easter.

Under Mr. Hoppington's guidance, Benny learned the essence of kindness and generosity. They practiced egg delivery together, with Benny gradually improving. Mr. Hoppington shared stories of his own journey, emphasizing that every mistake is a lesson.

As Easter approached, Mr. Hoppington revealed he would retire soon. "You're ready, Benny," he said, placing a paw on Benny's shoulder. Benny beamed with pride and determination.

The night before Easter, Benny prepared meticulously, his heart full of excitement and purpose. As the moon rose high, he set off, delivering eggs with newfound confidence. Each egg was placed with care, spreading joy to children everywhere.

By dawn, Benny returned home, exhausted but exhilarated. The next morning, children awoke to find his eggs, their faces lighting up with delight. Benny watched from afar, knowing he had truly become the Easter Bunny.

Mr. Hoppington appeared once more, this time to bid farewell. With a warm embrace, he passed on his mantle, symbolizing the continuation of their legacy. Benny accepted it gratefully, promising to keep the spirit alive.

And so, Benny thrived in his new role, spreading love and kindness each Easter. His journey taught him that perseverance and heart are key, and that following one's dreams leads to true fulfillment.

total duration:       21m42.175882858s
load duration:        41.889110848s
prompt eval count:    26 token(s)
prompt eval duration: 7.266570674s
prompt eval rate:     3.58 tokens/s
eval count:           1019 token(s)
eval duration:        20m53.018975667s
eval rate:            0.81 tokens/s
username@llmsandbox:~/Documents/ai/shortstory_test$

Success! Same prompt. The 12b parameter model running entirely on the GPU, was obviously faster as you’d expect.

So comparing speed is one use case for this python script, but also comparing the quality of the output. Eventually, what I’d like to be able to do is compare the quality of a larger model like the 70b DeepSeek model to the output of the 12b Gemma model and find out – does the smaller model running much faster produce “good enough” quality, or whether there’s value in waiting for the larger, slower model.

As far as quality, what do you think? Which short story is better? Leave a comment…