Model Agnosticism and Local LLMs
Picking Up Where We Left Off
Oooh, title case, gotta love it.
What were we doing last time? Let's just ask:
While it examines what we've done, I want to ask an interesting thing of the model. I want it to document what we have constructed inside of a markdown file. I want this file to describe our modules and implementation choices, interfaces, utilities, libraries, all of the goodies. With this over-arching description of our project, I can begin to manipulate the context of the model and point it to subsections of the project to work on. This will be very powerful (I suspect) for our current goal of running LOCAL LLMS! 💰
Let's look at that prompt.
WOOF! What a mouthful. Let's see if this works. This agent will churn in the background while we get back to work. This, to me, is some of the most power to gain from agents. I love modularity and I aim to instruct our agents to maintain my love with the ulterior goal of having discrete modules that encapsulate work...much like the paradigm itself! We know that if we have some module that loads files and that module has a public interface and is sound and well tested that we don't need to think about what that module does, only what it provides for us! We're going to exploit this to the optimus!
The last commit introduced the Zergling agent server, REPL, and web UI, and then added MCP tool support across all of them.
Agent server (zergling-agent-server)
- FastAPI app that proxies chat to the Anthropic Messages API.
- POST /chat: forwards messages (and optional system/tools) to Anthropic; supports tool-use loop (up to 10 rounds).
- GET /config, GET /health, GET /tools (from MCP), GET /models (vendor-keyed: anthropic from Anthropic’s API, openai as a placeholder). API key from env or x-zergling-api-key; model/endpoint from headers.
- MCP app mounted at /mcp; tools (e.g. list_directory) are converted to Anthropic tool format and executed in the loop.
REPL
- Adds /tool (list/set/clear tools), sends tool_names in chat when set, shows active tools in the banner.
Web UI
- Tools row and expandable tools panel; activeToolNames in settings and sent with chat.
Also, we're making use of AGENTS.md. Because I want to be model agnostic, I don't want to use specifically cursor rules. We're going to have our local agents obey AGENTS.md as well! Here's all I'm putting in there right now:
# Role
You are an expert programmer. You write concise code and use modularity to
maintain a clean and simple codebase.
# Always
- See architecture/CONTENTS.md for documentation on the project.
- Observe RULES.md when modifying architecture documentation
- Prefer unit tests. Do not create new files for one off testing. Write test
suites.
- Run python in a virtualenv
Start simple. Add as needed. We'll refine this over time and I will point out whenever I do. Back to the next step, model modularity. Here's the paraphrased output of "what we should do next" from the AI point of view.
- Define a small provider interface (e.g. in zergling-agent-server/):
- Implement an Anthropic provider that wraps the current logic:
- Route by model (or vendor) in the API layer:
- Keep tool handling vendor-aware
Hmm, not bad. I'm going to ask it to write a plan for this, but to include in this plan developing an abstract class for vendors, or potentially models? -- so we can add new models as we use them. Let's get started.
Have to give the guy some encouragement, he's doing a great job. While this thing churns I'll say that GPT-5 is great for this planning feature. I'm thinking that when we get our local model swarm off the ground, we may use a very "intelligent" model to do our planning and have it compose documents or tickets of what needs to be done, then pass that to an orchestrator who spins up implementing agents. I'll get a diagram for this as we get closer.
The plan is solid. I want to find a way to start including these plans in the blog posts in an expanding output or something so that you can read what I'm generating and have context when I make suggestions to the agent about the plan. Let me see if this blog agent can figure that out.............Alright, here's the plan after some back and forth while this collapsible section was built in the blog.
Model Vendor Abstraction Plan
name: Model vendor abstraction overview: Introduce an abstract ModelProvider base class in the agent server that defines a vendor-agnostic interface for listing models and performing chat (including tool-use rounds). Move the existing Anthropic logic behind an AnthropicProvider implementation and route requests in main.py via the abstraction. todos:
- id: base-interface content: Add providers/base.py with ABC, result types, registry, get_provider_for_model status: completed
- id: anthropic-provider content: Implement AnthropicProvider (list_models from API, chat, to_tools) status: completed
- id: main-refactor content: Refactor main.py to use providers for /models and /chat; no static model lists status: completed
- id: openai-provider content: Implement OpenAIProvider (list_models from OpenAI API, chat with tool_calls, to_tools); register and route gpt-* status: in_progress isProject: false
Model vendor abstraction plan
Goal
Support multiple model vendors (Anthropic now, OpenAI later) behind a single API by defining a model-agnostic interface and moving Anthropic behind it. Chat and tools are first-class: the interface includes listing models, sending chat with optional tools, and handling tool-use rounds.Current state
- main.py: All vendor logic is inline—Anthropic Messages API for GET /models (lines 124–188) and POST /chat (lines 219–307), including the tool-use loop and MCP tool execution.
- Tools: MCP exposes tools as
{name, description, input_schema}; Anthropic uses the same shape. The server converts by name, runs tools viamcp.get_tools()andtool.run(input), and appends tool_result blocks to messages.
Architecture
flowchart LR
subgraph api [HTTP API]
GET_models[GET /models]
POST_chat[POST /chat]
end
subgraph routing [Routing]
resolve[Resolve provider by model]
end
subgraph providers [Providers]
ABC[ModelProvider ABC]
Anthropic[AnthropicProvider]
OpenAI[OpenAIProvider]
ABC --> Anthropic
ABC --> OpenAI
end
subgraph mcp [MCP]
get_tools[get_tools_for_api]
run_tool[tool.run]
end
GET_models --> resolve
POST_chat --> resolve
resolve --> Anthropic
Anthropic --> get_tools
POST_chat --> run_tool- One round per provider: Each provider implements a single chat round (request → response). The server owns the tool loop: it calls the provider; if the response is tool_use, it runs MCP tools, appends results to messages, and calls the provider again (up to N rounds).
- Canonical tools: MCP gives a canonical list
[{name, description, input_schema}]. Each provider converts that to its own API format and parses tool_use from its response into a normalized shape so the server can execute by name and feed back.
1. Abstract interface and types
New module:zergling-agent-server/providers/base.py (or a single providers.py if preferred).Result types (vendor-agnostic):
ListModelsResult:models: list[dict](each with at leastid; optionaldisplay_name,created_at,type), optionalerror: str, optional pagination fields (has_more,first_id,last_id).ChatRoundResult: either final text or tool uses—e.g. a small dataclass or TypedDict withcontent: str | Noneandtool_uses: list[ToolUse] | Nonewhere one is set;ToolUsehasid,name,input: dict.
Abstract base class
ModelProvider(useabc.ABCand@abstractmethod):vendor_id: str— e.g."anthropic"(class or instance attribute).list_models(self, api_key: str | None) -> ListModelsResult— async; fetches model list for this vendor (or returns error if key missing/invalid).chat(self, messages: list[dict], model: str, system: str, max_tokens: int, tools: list[dict] | None) -> ChatRoundResult— async; one round.toolsare in vendor-specific format (provider converts from canonical internally or receives already converted; see below).to_tools(self, canonical_tools: list[dict]) -> list[dict]— convert from MCP/canonical{name, description, input_schema}to the format this vendor’s API expects. For Anthropic the canonical shape is already compatible; for OpenAI this would map to function-calling format.
provider.to_tools(canonical); in a loop call provider.chat(messages, model, system, max_tokens, tools); if result has content, return it; else for each tool_use run MCP by name, append assistant + user tool_result messages, repeat.2. Anthropic provider
New module:zergling-agent-server/providers/anthropic.py.- Implement
ModelProvider:vendor_id = "anthropic". - list_models: Move the existing GET logic from main.py (lines 131–186) into this method—httpx to
ANTHROPIC_MODELS_URLwith pagination, returnListModelsResultwithmodels(and optionalerror,has_more, etc.). - to_tools: Accept canonical list; return as-is (Anthropic format matches) or ensure
input_schemahastype: "object"; same as current_anthropic_tools_from_namesbut keyed by the canonical list, not by names filter. chat: Accept
messages,model,system,max_tokens,tools. Build request, POST toDEFAULT_ENDPOINT(or configurable), useanthropic-versionandx-api-key. Parse response:- If
stop_reason == "end_turn": returnChatRoundResult(content=text). - If
stop_reason == "tool_use": extracttool_useblocks toToolUse(id, name, input)and returnChatRoundResult(tool_uses=[...]). - Otherwise raise or return an error result.
- If
3. Provider registry and routing
- Registry: In
base.pyor a smallproviders/registry.py, maintain a mappingvendor_id -> ModelProviderinstance. Register provider instances at import/startup only—no static model lists. The registry holds only the code that knows how to talk to each vendor; model lists are always loaded live from the vendor API when GET /models is called. For each registered provider, GET /models callsprovider.list_models(api_key), which performs an HTTP request to that vendor’s models API (e.g. Anthropic GET /v1/models, OpenAI GET /v1/models) and returns the result. Do not cache or store model lists in the registry or in config. - Resolve provider from model: Add a function
get_provider_for_model(model_id: str) -> ModelProvider. Use model-id prefix (e.g.claude-for Anthropic,gpt-for OpenAI) to return the correct provider; unknown prefixes return None. Optionally support an explicitx-zergling-vendorheader later; for this step prefix-based is enough so the REPL and web UI need no change.
4. Changes in main.py
- Imports: Import the abstract types,
get_provider_for_model, and the registry (so Anthropic is registered). - GET /models: Build result dict by iterating registered providers (or by vendor keys you want to expose). For each, call
provider.list_models(api_key)and mapprovider.vendor_id→ListModelsResultserialized to the current JSON shape (models,error,has_more, etc.). Keep the same response shape so the UI and REPL remain compatible. POST /chat:
- Resolve provider:
provider = get_provider_for_model(model); if None, 400 “Unknown model or vendor”. - API key / endpoint: today we have Anthropic-only env/headers. Pass the resolved key and optional endpoint into the provider (Anthropic provider can take
api_keyand optionalendpoint; other vendors can have their own env vars later). - Get canonical tools from MCP if
body.tool_namesis set; filter bytool_names; callprovider.to_tools(filtered_canonical). - Tool loop:
messageslist,max_tool_rounds = 10. Loop:result = await provider.chat(messages, model, system, max_tokens, tools). Ifresult.contentis set, returnChatResponse(content=result.content). Ifresult.tool_usesis set, for each item run the MCP tool by name (same as today:mcp.get_tools(),tool.run(input)), build tool_result blocks, append assistant message with content list and user message with tool_result blocks tomessages, then continue the loop. After max rounds, return 502.
- Resolve provider:
- Config / keys: Leave
_server_has_anthropic_key()and/configas-is for now (still Anthropic-focused). Later you can generalize to “has API key for vendor X” when adding OpenAI.
_anthropic_tools_from_names (replaced by provider.to_tools + filter by tool_names). Keep _tool_result_content, _get_api_key, _get_model, _parse_max_tokens, and request/response models.5. Tool conversion and tool_names
- Canonical tools come from
get_tools_for_api(); filter bybody.tool_namesto the list of tool definitions for those names. Pass that filtered list toprovider.to_tools(...). So the provider receives only the tools that are enabled for this request, in canonical form, and returns vendor-specific tools. The server then passes that vendor payload intoprovider.chat(..., tools=...).
6. REPL and Web UI
- No changes required: they send
x-zergling-modeland optionalx-zergling-api-key; the server routes by model id and returns the same response shapes.
7. Documentation and layout
- Add a short note in zergling-agent-server/README.md that the server uses a provider abstraction and supports Anthropic and OpenAI; model id prefix is used to select the vendor. Document env vars for both (e.g.
ZERGLING_ANTHROPIC_API_KEY,OPENAI_API_KEYorZERGLING_OPENAI_API_KEY). - If you add
architecture/zergling-agent-server/docs, add a sentence that GET /models and POST /chat are vendor-agnostic and implemented via the ModelProvider abstraction; model lists are loaded from each vendor’s API, not stored statically.
8. OpenAI provider
- New module:
zergling-agent-server/providers/openai.py. - list_models: Call OpenAI’s models API (e.g.
GET https://api.openai.com/v1/models) with the provider’s API key. Map the response toListModelsResult(id, and optionally display name / created, etc.). Handle pagination if the API supports it. Require API key (return error in result if missing). - chat: One round using OpenAI Chat Completions API. Accept
messages,model,system,max_tokens,tools. Convert messages to OpenAI format (role + content). Iftoolsis set, send them astoolsin the request (function-calling format). Parse the response: if the choice has amessage.contentand notool_calls, returnChatRoundResult(content=...). If the choice hasmessage.tool_calls, map each toToolUse(id, name, input)and returnChatRoundResult(tool_uses=[...]). Map OpenAI’s tool call id/name/arguments (JSON string) into the normalizedToolUseshape. - to_tools: Convert canonical
{name, description, input_schema}to OpenAI’s function format:{"type": "function", "function": {"name": ..., "description": ..., "parameters": input_schema}}. - Config: Read API key from env
OPENAI_API_KEYorZERGLING_OPENAI_API_KEY(and optionally from a future header likex-zergling-openai-api-key). No static model list; models come fromlist_models()only. - Registry: Register
OpenAIProvider()in the provider registry undervendor_id = "openai". - Routing: In
get_provider_for_model, treat model ids that start withgpt-(or other known OpenAI prefixes, e.g.o1-) as OpenAI and return the OpenAI provider. - GET /models: Already iterates providers; the new OpenAI provider will be included automatically and its
list_models(api_key)will be called to fill theopenaikey in the response (replacing the current placeholder empty list).
Summary of new/edited files
Optional follow-ups (out of scope for this plan)
- Generalize
/configto report per-vendor API key presence (e.g.has_anthropic_key,has_openai_key). - Add
x-zergling-vendorheader to force provider selection when model id is ambiguous.
Well that's lengthy as shit. At least you can read it though! There's still some rendering issues in there for sure, but we'll fix those later. It's getting late and I want to get on with the modeling. Let's make sure our vendor abstraction works and check out some code.
Model Provider Base Class
from abc import ABC, abstractmethod
from dataclasses import dataclass, field
from typing import Optional
PROVIDER_REGISTRY: dict[str, "ModelProvider"] = {}
@dataclass
class ToolUse:
id: str
name: str
input: dict
@dataclass
class ListModelsResult:
models: list[dict] = field(default_factory=list)
error: Optional[str] = None
has_more: Optional[bool] = None
first_id: Optional[str] = None
last_id: Optional[str] = None
@dataclass
class ChatRoundResult:
content: Optional[str] = None
tool_uses: Optional[list[ToolUse]] = None
class ModelProvider(ABC):
vendor_id: str = ""
@abstractmethod
async def list_models(self, api_key: Optional[str]) -> ListModelsResult:
pass
@abstractmethod
async def chat(
self,
messages: list[dict],
model: str,
system: str,
max_tokens: int,
tools: Optional[list[dict]] = None,
api_key: Optional[str] = None,
endpoint: Optional[str] = None,
) -> ChatRoundResult:
pass
@abstractmethod
def to_tools(self, canonical_tools: list[dict]) -> list[dict]:
pass
def register_provider(provider: ModelProvider) -> None:
PROVIDER_REGISTRY[provider.vendor_id] = provider
def get_provider_for_model(model_id: str) -> Optional[ModelProvider]:
if not model_id or not model_id.strip():
return None
mid = model_id.strip().lower()
if mid.startswith("claude-"):
return PROVIDER_REGISTRY.get("anthropic")
if mid.startswith("gpt-") or mid.startswith("o1-") or mid.startswith("o3-"):
return PROVIDER_REGISTRY.get("openai")
return NoneSo far this doesn't look too bad. I appreciate the use of dataclasses to gather
information on how models behave. I'm unsure as of right now what
ListModelsResult will be, but we have a way to get a provider's list of models
and chat with that model. Keeping this chat function as entirely functional (you
have to pass in the api_key, endpoint, tools, etc. each time) may end up
benefitting us in the long run as it will allow us to switch models easily.
Essentially, if we can abstract away the differences between each provider and
end up with a similar way to communicate with each one, including my local
models, then we can select the right model for the job.
Restarting the app and server, everything seems to continue to work. Let's have it setup OpenAI as well and we'll see how well that works. Also, while we're at it, let's take a look at those architecture documents. OOOOOOHHHH, it didn't update the architecture directory! I've directed it to do this now. Seems like AGENTS.md may not have been picked up by Cursor? Maybe I'll make a rule in cursor to observe AGENTS.md if it exists. Excellent, it even updated existing files, though the documentation itself is quite bland and not instructive (docs below). I've added another rule to AGENTS.md:
- Write documentation for your functions and classes
Model Providers Architecture Doc
Model Providers
Overview
Chat and model listing are vendor-agnostic: they go through a provider abstraction inproviders/. The registry holds only provider instances; model lists are loaded from each vendor’s API when GET /models is called (no static model list). See web-server-and-api.md for how the API uses providers.Interface
- Module:
providers/base.pydefines the abstract base and shared types. - ModelProvider (ABC):
vendor_id,list_models(api_key),chat(messages, model, system, max_tokens, tools, api_key?, endpoint?),to_tools(canonical_tools). - Result types:
ListModelsResult(models, error?, pagination),ChatRoundResult(content? or tool_uses?),ToolUse(id, name, input). One round perchat()call; the server inmain.pyruns the tool loop (MCP execution and re-calling the provider).
Registry and routing
- Registry:
PROVIDER_REGISTRYinbase.py;register_provider(provider)at import (seeproviders/__init__.py). - Routing:
get_provider_for_model(model_id)selects by model-id prefix:claude-→ Anthropic,gpt-/o1-/o3-→ OpenAI. Unknown prefix returnsNone(API returns 400).
Implementations
Each provider converts canonical tools (name, description, input_schema from MCP) to its API format via to_tools(); the server passes the result into chat(..., tools=...).Configuration
API keys are resolved inmain.py per provider (header or env). Anthropic: ZERGLING_ANTHROPIC_API_KEY / ANTHROPIC_API_KEY. OpenAI: ZERGLING_OPENAI_API_KEY / OPENAI_API_KEY. Endpoint override (ZERGLING_API_ENDPOINT) applies to Anthropic only.Okay, now the last piece for this article because it's getting late. I need a model selector in the chat app...and a dark mode theme! So let's instruct a new agent so it has 0 context from the prior agent's work and it has to use the architecture docs, and I'm going to add some special sauce to this prompt.
If the documentation you require to implement this feature is not in the architecture documentation, update it.
This. Will this work? This is sort of the beginning of my agent-loop style prompt idea of having agents improve themselves over time. Improve themselves as in engineer their own context, not reinforcement learning style fine-tuning. Though if y'all can help me fund this adventure, I'd love to take on that adventure, too!


Oh fuck. Dark mode looks awful. I wonder if I'm going to have to do some real designing, either drawing or writing the CSS myself. I'll attemp one quick prompt to fix it and also ask why the hell OpenAI isn't present in this list.
Once we get this working I'm going to call it a night. Next time, we'll get to running a local model, get the abstraction for that in place, get it hooked up to the MCP server and AGENTS.md and see if we can incorporate it into some of the work we're doing! I only have 12GB of VRAM though, so I can't run anything too powerful........yet!
I've had to re-prompt this guy a few times to get OpenAI models listed in the dropdown...here's the most recent one:

Fuck yeah!