AI Architecture

What Is an AI Agent, Really?

And what is the difference between an LLM, an agent, and an agent harness?

There is a lot of sloppy language floating around AI right now.

People say “agent” when they mean “LLM.” They say “agent system” when they mean “some code that calls GPT.” And “agent harness” sounds like something you would buy at Home Depot.

Let’s clean it up.

The Clean Mental Model

1. LLM

The model. Text in, text out.

2. Agent

The model plus a goal, loop, and tools.

3. Harness

The control plane that makes the agent safe, observable, and production-ready.

1. The LLM: Just the Brain

An LLM by itself is not an agent.

It is a model. You give it input. It gives you output.

  • It does not have a goal.
  • It does not decide what to do next.
  • It does not call tools unless something outside the model lets it.
  • It does not persist state unless an application gives it memory.

The model is powerful, but by itself it is mostly a very smart text transformation engine.

That is useful. But it is not an agent yet.

2. The Agent: When the Model Starts Acting

An agent starts when you wrap the LLM in behavior.

The key difference is that the agent has some version of a goal, a loop, a policy, and tools.

Goal

What the agent is trying to accomplish.

Loop

Decide, act, observe, repeat.

Policy

Instructions for how it should make decisions.

Tools

APIs, databases, code execution, search, or business systems.

This can be very simple. It is often just Python or TypeScript, a prompt, an API call, and a loop.

What Agents Are Usually Built On

Most agents are not magic. They are software.

Piece What it does Commonly built with
LLM Reasoning, planning, language generation OpenAI, Anthropic, Gemini, Llama, Mistral
Agent loop Decide, act, observe, repeat Python, TypeScript, LangChain, AutoGen, CrewAI
Tools Let the agent take action APIs, databases, code execution, web search
Memory Stores context and prior information Redis, Postgres, vector databases

You can build this from scratch. Or you can use agent frameworks. The frameworks help with the shape of the system, but they do not remove the need to understand what is happening.

3. The Harness: The Part That Makes It Real

Most agent demos look impressive because the happy path works.

Production is not the happy path.

The harness is the layer around the agent that handles reliability, safety, monitoring, testing, and operations.

This is where the work gets serious.

What Lives in the Harness?

Execution control

Retries, timeouts, rate limits, and error handling.

Guardrails

Permissions, policy checks, output validation, and approval flows.

Observability

Logs, traces, metrics, cost tracking, and debugging tools.

Evaluation

Test sets, benchmarks, scoring, and regression checks.

What the Harness Is Built With

  • State and storage: Postgres, MongoDB, Redis
  • Vector memory: Pinecone, Weaviate, Chroma, pgvector
  • Logging and tracing: LangSmith, Helicone, Arize, Weights & Biases
  • Monitoring: Prometheus, Grafana, Sentry
  • Cloud infrastructure: AWS, GCP, Azure
  • Custom policies: usually your own code

So no, the harness is usually not one magical library. It is a combination of libraries, services, and custom rules that match your business.

Build vs. Buy

The practical answer is: buy what is generic, build what is specific.

Layer Usually buy/use Usually custom-build
LLM Model API or open model Prompting and model selection logic
Agent Frameworks and tool abstractions Workflow, goals, permissions, business logic
Harness Logging, tracing, eval, monitoring tools Risk controls, approval flows, success criteria

The Common Failure Mode

A team builds a cool demo. The demo works. Everyone gets excited. Then it goes into production and starts breaking in boring, predictable ways.

Why?

Because they built the LLM call and maybe the agent. They did not build the harness.

Clean Definitions

LLM

A model that generates language. It is the brain, not the whole system.

An LLM wrapped in a loop that can pursue a goal, make decisions, and take actions using tools.

The control layer that makes the agent reliable, observable, safe, testable, and production-ready.

Final Takeaway

The LLM is not the agent. The agent is not the whole product. The harness is not optional once you care about reliability.

That last part is where most of the actual engineering lives.