Design systems and AI

A design system consists of a set of decisions regarding UI/UX, design, typography, colours, and much more. Over the last few months professionals in the industry started to use LLMs to create their own DSs or iterate on existing ones: we all know that generative AI is capable and has the knowledge to help us create components, maintain a token library, or even generate screens, so the question is no longer if it can make us faster and more productive with these tasks (it already does), but rather whether it has the ability to interpret those high-level decisions that have already been made by the human, and follow them consistently, avoiding drift and quality degradation in the codebase both in the development and maintenance of such systems.

TL;DR

• We reimplemented Meta’s Astryx approach for our own design system: a machine-readable manifest, a rules file, and a two-command CLI; then we built a benchmark to find out whether any of it works.

• The query surface pays: Against an undocumented codebase, it cuts tokens by 31% and steps by 39%, winning all ten tasks.

• It’s a cost tool for a frontier model and a correctness tool for a cheap one: Sonnet 5 already writes correct templates against an undocumented tree, so there is no quality left to win, while with Haiku 4.5 the cost saving mostly disappears, while invented CSS classes drop, recall rises, and a page that did not compile does.

• Read the numbers as a direction: more tests with different models and on different codebases are needed for more statistical validity.

The problem

If you ask an agent to build a page in an unfamiliar codebase and you watch it work, the first thing it will do is most probably grepping* some files: it opens a template, then another, then a third, building a picture of your DS conventions by reading examples, and then it eventually writes something that usually works and aligns with them.

That is an expensive way of working: every step an agent takes re-sends the entire conversation so far because the underlying model is stateless. Imagine someone working through a task with a notebook, who re-reads it whole before every single action. At the start that costs almost nothing, but as they go on, the notebook becomes thicker, and they waste more and more time and concentration just to do one small thing.

In our runs the re-reading came to 94% of all tokens. The page the agent actually wrote was 8k-34k tokens, while the re-reading amounted to about 2M: the real work was just a tiny fraction.

A snake stuck re-reading the maze instead of moving through it

AI-ready Design Systems

That’s why “AI-ready” doesn’t only aim at improving output quality, but also at addressing a broader range of metrics, including how much an agent has to read before it can even start, which affects time and token usage, and how reliably it recalls and reuses the components in your library instead of generating redundant custom code.

*grepping: using a command-line tool to search through text files for specific words, phrases, or patterns

How we built on Meta’s work

Astryx, Meta’s open-source React design system, was built around that observation: it ships a CLI that returns a self-describing JSON manifest, which their documentation likens to an OpenAPI spec for the CLI, and an MCP server exposing two tools:

• search for finding a component

• get for reading its full specification

Plus an AGENTS.md for additional rules on how to use them.

Their choice behind providing the LLM with just two commands is a deliberate restraint, aimed at lowering the miscalls and keeping things lean for the infrastructure. Even though one can see the reason behind it, Astryx doesn’t actually ship any evidence that this works: there is no benchmark in their docs, no baseline comparison, no results table.

This is why we decided to take the concept, reimplement it and, this time around, assess whether it brings some value and if it’s ultimately worth doing.

Piece one: The manifest

Since we mentioned how LLMs waste time and tokens by grepping files to understand the context, let’s start from the piece of the solution meant to address that problem. Books already solved this problem centuries ago by having an index at the start, to give you an idea on where to find a specific piece of information, saving you from leafing through all the pages.

While it’s true that our coding agents have access to the list of files in your folder, they still have to open them to read their content: instead, we keep a manifest JSON file which is a bit more informative, since it provides a list of objects, and for each component, it includes its parameters, variants, description, dos and don’ts, and more. This acts as a central signpost for the agent, pointing it directly to the right component without the need to wander through the codebase, making its following searches much more informed and accurate, and effectively reducing response time and token usage.

One approach reads less and still gets there faster

Now, how to build such file, and what should it contain?

We first explored this topic while working on a Smarty codebase, which uses templates (.tpl files analogous to .tsx React components) and a different syntax from the more popular JavaScript frameworks, so we decided to maintain a documentation comment block at the start of every component template containing the information listed above; the manifest was then generated from those blocks, which left the developer responsible for keeping those and the code in sync.

Which is why we also created a generator script that validates as it goes: it reads the template body, works out which variables are actually used, and compares that against what is documented. This only compares names though: when we injected four lies into a component’s comment block, the check reported no drift, the manifest carried all four through, and the query tool then handed them to an agent as fact.

This would be easier with more popular frameworks: react-docgen and vue-docgen-api already read props, defaults and types straight from source, and Storybook already supports them. Deriving those fields directly from the code removes the sync problem and narrows what the developer still has to write by hand down to the aspects that no parser can infer: what the component is for, and which of its values are meaningful.

Piece two: The query surface

The CLI exposes two commands, because an agent asks two different questions: what should I use, and how do I use it.

ds search answers the first: it takes words describing intent and ranks every component across five fields, weighted by where the answer usually lives.

• a name match is nearly always what was meant, so it scores highest

• the usage example scores lowest, because it matches on demo copy, which helps a vague query and adds noise to a precise one.

Every result carries its summary and its required parameters, which is often enough to choose without asking anything else. A query that names no component but describes what it is for lands on the right one. In our tests, it seemed reliable for precise queries and for the system’s own vocabulary, but hit-and-miss for loose natural phrasing.

ds get answers the second question, returning everything needed to write the include:

• required and optional parameters, to stop a missing parameter that renders anyway and is quietly wrong

• allowed values, to stop an invented one that Smarty would accept in silence

• defaults

• what the component composes

• a worked example shows the calling shape and prevents the agent from rebuilding the component by hand out of divs

The experiment setup

We set up five copies of the repository, identical except for how much machine-readable context sits on disk. Each one adds a single artefact to the one above it, so any difference between two adjacent rows is attributable to exactly one thing. The top row is our baseline, because it is what most repositories actually look like: the documentation block deleted from all 91 component templates. Every comparison below is measured against it.

Five arms, one artefact added at a time

Then each copy gets the same brief, written in plain English and with no reference to the available components, e.g. “Build a page showing a feed of recent activity, grouped under date headings”. Finding out that the library already has an activity-item component is the test, so the brief must not give it away.

The mechanical scorer

We didn’t want a model to grade another model’s output: instead, we came up with six quantifiable metrics:

1. does it compile

2. what share of the components the real page uses did it find

3. how many parameters did it pass that the component does not accept

4. how many required parameters did it omit

5. how many CSS classes did it invent

6. how many times did it paste a component’s internal markup instead of calling it

The six generic measures all ask whether the output is well-formed, which appeared to be a low bar for a good model: four of them score zero in every arm.

So each task also carries a prediction about a specific mistake we expect an agent to make on that page, written down before any run in a form a script can check. The clearest example is a tabbed page: a tab bar built the obvious way looks perfectly correct and resets to the first tab the moment someone reloads. Surviving a reload needs each tab to carry a key and a route, and each panel to open from a variable the server sets.

The benchmarks on a good model

We designed 10 tasks and ran them on the 5 variants, one run per cell using Sonnet 5.

The quality metrics barely moved when using Sonnet 5: all fifty outputs compile, recall runs 97% to 100% across the arms, one parameter is invented in the entire matrix, and nothing anywhere pasted a component’s internals. Even deleting the documentation from all 91 components didn’t cause any degradation: the agent recovered the same information from the code and paid for it in steps.

Quality holds steady across every arm

The only one that moves was the number of invented classes as machine-readable context is added.

For this model, documentation doesn’t buy quality, but efficiency:

Fewer steps, fewer tokens, more time per step

The query surface is the only arm that wins on every task

Statistical significance needs t above 2.26, so only the query surface clears it on both measures while winning every task individually. The reduction in steps explains why the tokens also decreased. The two are linked, but how many steps a given amount of work takes still varies between models: one that batches ten file reads into a single step has few steps and enormous context in each; one that goes file by file has the inverse, on the same total spend. Averaged per arm the two track closely here, 45k to 55k tokens a step. Per individual run that same ratio spans 19k to 77k.

The manifest was supposed to be a very short summary of the components, but in our experiments, it ended up being 348KB, roughly twice the 176KB of documentation blocks it is generated from, and reading it costs about 87k tokens. An index is only worth having when consulting it is cheaper than reading what it indexes, and that is why adding the query surface represents the biggest jumps in performance: ds search answers in a few hundred tokens. The manifest is not the product. It is the CLI’s data file, and shipping one without the other is the configuration that did nothing.

Tokens carry more run-to-run noise than steps, because context grows within a run and a run that goes long costs superlinearly. That is why the same underlying effect reads t = 4.52 on steps and t = 2.45 on tokens.

Wall-clock time measures nothing at all, and the cleanest proof is in the table: the query surface cuts 39% of the steps and comes out slowest, while the undocumented baseline takes the most steps and finishes fastest. Seconds per step is the reason, rising from 9.2 to 18.3 as the agent gets better informed and does more per step, and the two effects cancel. A null control confirms it independently: two copies configured identically, run through the same ten tasks, differ by 120 seconds on their own. Anything a control produces by itself is not a result.

The same test on a weaker model

Sonnet 5 proved to be a strong enough model that four of the six quality measures never moved off zero in any arm. That is a problem for the conclusion, because it doesn’t decide whether it’s the machine-readable context that doesn’t improve the output, or is the model that was already too good for the task; so the same experiment was run using Haiku 4.5, a less performant model.

The cost saving mostly disappeared: undocumented to query surface is 19% fewer tokens at t = 0.87, winning five of ten tasks, against Sonnet 5’s 31% at t = 2.45; as predicted, the defect measures were affected.

On a weaker model, the same context becomes a correctness fix

So the benefit on a weaker model moves from efficiency to quality: Sonnet 5 writes correct templates against an undocumented tree, so the only thing left to sell it is speed; Haiku misses components, invents eighteen CSS classes across ten pages, omits required parameters, pastes component internals instead of calling them, and on one task even ships a file that does not compile. This method recovers nearly all of that.

That makes this a cost tool for a frontier model and a correctness tool for a cheap one, so it’s bigger value for people running small models at volume than for those who prefer chunkier models.

Limitations and next steps

Architecture and Data quality

The CLI already does the work, but wrapping the same two verbs over stdio is thin, so an MCP server would let a client’s own tooling query the system. Better summaries would also improve the search: one component’s summary is just its title-cased name, which makes it findable only by someone who already knows what it is called. Lastly, why the manifest alone does nothing stays open: either the agent does not open a large JSON file it was not told to open, or opening it costs about what the grepping cost. Each needs its own fix.

More models from other vendors

We tested two models, both from the same vendor. Sonnet 5 is strong enough to infer conventions from a couple of examples; Haiku 4.5 gets less cost benefit and much more correctness benefit. Two points on one vendor’s ladder is not a curve, and nothing here says where a mid-tier model from anyone else would land. If the same shape holds on a mid-tier model from another vendor, it becomes a claim about small models in general rather than a claim about Haiku.

More repetitions, and harder tasks for the stronger models

One run per cell is not enough: we repeated a single task five times on one arm and it produced between 2.7M and 5.9M tokens with everything held constant. Pairing by task removes most of that spread, since it is largely the tasks differing from each other, but what is left is wide enough that every percentage above should be read as a direction.

We’re also missing the middle arms on Haiku: we tested only the undocumented baseline and the query surface, so whether the manifest alone recovers the correctness is unknown.

Finally, Sonnet 5 saturates every quality measure, so we need harder/longer tasks, each with its expected failure written down in advance, because that check remains the only one with discriminating power at the top of the range.

Different DSs and codebases

We tested this approach on a relatively small component library serving only one application, while Astryx serves roughly 13,000 of them, where scale alone makes a search tool necessary: the honest conclusion is that a small system may need less of this than a large one.

Another important aspect is the language. Our codebase uses Smarty, a thin-information language for this purpose: templates have no props, no type annotations, and nothing a parser can recover about what a variable is meant to hold, which is exactly why we had to hand-maintain the documentation blocks the manifest is generated from. A React or Vue system starts from a better position, because react-docgen and vue-docgen-api read props, defaults and types straight from the source. The manifest would be richer and free to maintain.

LLMs are also trained way more on modern frameworks like React than Smarty, so its priors do more work before it opens a single file, and the grepping our query surface eliminates may simply cost less there. Not only the framework, but also a styling library that the model knows well, like Tailwind or Bootstrap, could shift the baseline again.

Conclusions

On this stack, the approach proved to address cost when using bigger models, and correctness when using a cheaper one, so it’s bigger value for people running small models at volume than for those who prefer chunkier models; even though there are many axes yet to explore to make these findings general, the results are promising and it’s reasonable to think that applying it to other codebases would bring improvements, or have no effect, on both quality and efficiency.