Three Weeks Using Glossaries for Agents

It’s been three weeks since I started using my extension pi-glossary to inject quick definitions into the context on demand, and now it’s time for lessons learned.

(I wrote about it here before.)

The core idea is similar to a skill: inject context on demand (progressive disclosure). However, it has a few differences:

• The definitions are very short, rarely more than one paragraph. The goal is to complement the prompt.
• They state “definitions” instead of “instructions”. This is a bit subtle because I have terms that actually define workflows, but you get the idea: it’s a glossary.
• They are explicitly invoked. The text is only injected if I type the exact term. To be honest, I made a small change recently so the agent can search for more definitions. However, the process always starts explicitly; the model doesn’t decide on its own that it needs to read a term definition. That means the context is kept as lean as possible.

I found myself using the glossary on almost every prompt, and I’ve been creating new definitions every day. I’ve now started to replace whole skills with it. I’m realizing that the latest models don’t need that much context or very detailed instructions. A few precise words and short definitions are enough most of the time. That’s not always true, of course, but I’m finding that it’s more common than I initially thought.

Usage Patterns

While using the extension, I noticed some emerging patterns. Some were intentional by design; others emerged from use. Here are some of them, from the least interesting to the most interesting.

Project Shortcuts

Agents are amnesic. Every new session is like getting a new hire on their first day and asking them to work with zero onboarding. That means that Every. Single. Time. the agent re-reads the AGENTS.md (or CLAUDE.md), re-explores the codebase, and re-reads this or that document.

To help with that, I started adding a one-paragraph definition for projects that I’m working on. For example, I defined pi-glossary as:

A Pi extension that adds definitions to the conversation on demand. It’s hosted on https://github.com/ruliana/pi-glossary, and the definitions are in ~/.pi/agent/glossary.json and /.pi/glossary.json. The project-local directory is ~/.pi/agent/extension/pi-glossary.

That’s enough for the agent to understand the most common tasks I ask about, like adding, removing, and updating terms, or even modifying the extension when I haven’t started Pi from the project folder.

I’m using that for every project I touch. I noticed two things:

• The agent spends much less time finding things, and less time trying to find files, docs, and such.
• Tasks that do not touch a project but reference it (like docs) are much more focused. The agent either finds what it needs faster or, frequently, doesn’t even search for anything. It’s even more noticeable when the task mentions multiple projects, like integrations.

Company/Team Jargon

Every company builds its own vocabulary. The most iconic example for me is Shopify’s “tophat”, meaning “dogfood-style acceptance tests”, which is a very useful term. Another example is Amazon’s “two pizza team”, which LLMs already know about, but terms like that are a perfect fit for a glossary.

Teams also have their own jargon, or their own specific meaning for common words. A “standup” could mean a “stand-up meeting with devs + PM. 30 minutes on Tuesday and Thursday at 1pm. Doc with summary at https://some-google-docs-url, one bullet point per task”. I found that definitions like that are as useful as they are short.

With a glossary like that, I can type instructions like:

• “book a 30-minute tophat session with James after the standup”
• “create an entry in the standup doc summarizing what we just did”.

References to People and Artifacts

Which brings me to the next cool use: people and things.

I can define “James” as “James Bond, senior dev, knows about project IronJaw. Slack ID: 12345, email: bond@jamesbond.sec, GitHub handle: james007”. Because I work with James a lot, I can now do things like:

• “What did James and I discuss about IronJaw yesterday on Slack?”
• “Address James’s comments on PR 3210”
• Or even: “Check Slack and tell me what is James up to”

Similarly, I can use it to point to documents, parts of the code, PRs, and anything that I would otherwise be copying and pasting. For example, I have a “tech doc” term that always points to the one I’m working on. When I change tasks, I modify the term to match the new location. With a couple more terms defined, I can do this:

• “Update the tech doc with what we did in the current PR”
• “Read the plan doc for context and tell me what the next steps are”
• “Add a new entry to the route file for the endpoint we created”

Note: I actually just expanded my Pi extension to accept templates, so I can now define terms like “current PR” and dynamically insert the right value in the context, no matter which folder I’m in. No need for the LLM to think about it and issue a tool call; the context is right there as I type.

That’s way less work, and better yet: Neither me or the agent need to find/remember where the artifacts are.

Common Workflows with Made-Up Words

I have a couple of key phrases that I keep repeating, things like “ask clarifying questions”, “don’t apply any changes, just answer my question”, “create a new branch with Graphite, commit, and push”, “orchestrate this with parallel subagents”, and so on.

For some of them, I created text expanders, but frequently they don’t fit the exact phrase I’m writing, and other times I want small variations. Then I started to create made-up terms with those definitions in the glossary. Now I can type some crazy instructions like:

• “This is a new repo, clarask the destination before gubmitting. Stop before pushing.”
• “Do James’s review comments on the current PR make sense? Dapy and clarask about the constraints I see.”

Except for the fictional “James”, those crazy statements were exactly what I typed this week. The insane part is that the agent is starting to use the words itself. I got this one from it a some time days ago:

Before I draft, I want to clarask the choices that actually change the shape of the system…

That left me speechless. It’s obvious in hindsight, but it caught me off guard. Unlike commands or text expanders, a glossary is two-way 🤯: the agent can talk to me using the same definitions. The made-up terms are becoming a precise language between us. When the agent uses those terms, I also know the exact meaning.

Replacing Skills

This is a recent experiment.

I noticed that a lot of my skills actually contain things that the LLM already knows, but they are either:

• Procedures to follow. Things like “step 1: do this, step 2: do that”.
• Explanations for things that are uncommon and have little LLM training material, like less popular languages (Scheme, Factor, OpenSCAD, BigQuery Pipe SQL), less popular design patterns (check the ones from Brian Foote and Joseph Yoder, for example), Ilities, and so on.
• Caveats, i.e., guardrails to avoid mistakes the agent usually makes, frequently because of the above.

For procedures, I’m trying to replace them with glossary definitions that simply state the steps, very concisely. I’m noticing that the LLM doesn’t need many details if the underlying subject is one it has lots of training material for. The “common workflows” above are pretty much that.

For topics with little training data, I think I could use fine-tuning, but I haven’t tried that seriously yet. My most successful approach is to provide cheat sheets. They work very well for APIs and simpler languages (like OpenSCAD), especially when I pair them with checkers and linters.

Caveats are more complicated, and I don’t have a good approach yet. One thing I’m trying is to add linters for everything on hooks that run when the LLM produces material about that topic. For example, I have a Logseq linter that runs every time it writes markdown files in my logseq/ folder. It works way better than anything I tried before with skills. I'm also trying verifier steps in the instructions, frequently with specialized subagents. It's useful, but not as effective.

In any case, I’m starting to link terms, allowing one term to refer to another and letting the agent decide if it should consult the glossary for the linked term. It’s like splitting a skill into even smaller chunks. That’s working better than expected; I see the agent using that from time to time, mostly when it really makes sense.

Always Improve the Terms

I’m always looking at what the agent is doing and thinking. Sometimes I notice it gets confused, tries multiple things before answering, or starts to follow dead ends. Every time I see that, I try to improve the definition of the term. Over time, the term definitions start to fit my workflow better, and the workflow becomes faster.

Hmm… wait… I think I should create a skill like “/insights” for doing that automatically 🤔

It’s Just Another RAG/Progressive Disclosure Technique

Yes, that’s pretty much the point of it.

We are all gravitating toward the question, “How can I give the right context to the agent at the right time?” Prompts, commands, skills, RAG, memory, and llm-wiki are all different ways to reach the same goal. Most of them are intended to be automatic and AI-driven. This one is less ambitious and, to be honest, quite dumb. However, I’m finding that it fills an important gap in my workflow.