Optimizing Content for Agents

Just as useless of an idea as LLMs.txt was
It’s all dumb abstractions that AI doesn’t need because AIs are as smart as humans so they can just use what was already there, which is APIs
LLMs.txt isindeeduseless, but that’s the only thing correct in this statement.

I’m here once again being rage baited to address more brainless takes on social media.

This one is about content optimization.

Frontier models and the agents built on top of them all behave similarly, with similar constraints and optimizations.

For example, one thing they’re known to do, to avoid context bloat, is to only read parts of files.

The first N lines, or bytes, or characters.

They’re also known to behave very differently when they’re told information exists somewhere vs.

having to discover it on their own.

Both of those concerns are actually why LLMs.txt was a valuable idea, but it was the wrong implementation.

The implementation today is simple: content negotiation.

When a request comes in withAccept: text/markdown, you can confidently assume you have an agent.

That’s your hook, and now it’s just up to you how you optimize it.

I’m going to be brief and to the point and just give you a few examples of how we do that at Sentry.

We’ve put a bunch of time into optimizing our docs for agents, for obvious reasons.

The primary optimizations are mostly simple:
In our case we actually use MDX to render these, so it involved a handful of parsing changes and overrides to allow certain key pages to render differently.

The result: agents fetch pages that are much more actionable.

If a headless bot is fetching the website, the least useful thing you can do is serve it an authentication-required page.

In our case we use the opportunity to inform the agent that there are a few programmatic ways it can access the application information (MCP, CLI, API, etc):
For projects likeWarden, we actually set it up so the agent can hit the entire content to bootstrap itself:
Help me set up warden.sentry.dev
It’s simple and it works.

You should do it.

You should also pay attention to how patterns are changing with agents and update your optimizations as behavior changes.