Designing an evaluation workflow for agent skills

What SkillCreator is

SkillCreator is a native macOS desktop application, built for Apple silicon and Intel, offered as a free beta. The download page says it is still in development and should be installed only by people who want to test it, and I am not going to soften that here. The skills it works on are the ordinary kind: a folder with a SKILL.md, optional scripts, optional reference files, the format that Claude, Copilot, Cursor, Codex, Gemini CLI, VS Code, Amp, and Factory all read.

The app is organised around six surfaces. Studio is the editor, Monaco underneath, with auto-save. Manager is a SQLite-backed index of every skill across your workspaces. Evals scores a skill on six dimensions and tracks regressions between runs. Graph draws skill dependencies on a force-directed canvas. Converter transforms a skill between the formats different agent platforms expect. Installer writes a skill into a runtime transactionally, with a diff preview before the write and a rollback after it.

Six dimensions, and what each kind can see

The six dimensions are named structure, content, evidence, usage, toolchain, and freshness. They fall into two kinds, and the distinction matters more than the names. Structure and toolchain are questions a program can settle: does the frontmatter parse, are the required fields present, does every script the skill references exist and run. A pass on those is a fact. Content, evidence, and usage are judgements about text: whether the instructions are specific enough to follow, whether the claims and examples are grounded, whether the description will trigger the skill when it should and stay quiet when it should not. Freshness sits between the two, part date arithmetic and part judgement about whether a reference has gone stale.

Naming the dimensions separately is the design decision. A single score hides which kind of check produced it, and a developer who sees a low number needs to know whether to fix a path or rewrite a paragraph.

What a score can and cannot tell you

A structural pass proves the file is well formed. It does not prove the skill makes an agent better at the task, and no static check can. The judgement dimensions are produced by a model reading the skill, which means they carry the model’s own variance: the same skill can score differently on a different day, and two evaluators can disagree without either being broken. Regression tracking is only meaningful when the runs are comparable, so the model, version, and settings behind a score have to travel with it, and a comparison across different settings is a comparison of two different instruments.

So a score is a diagnostic, and I use it as one. It points at the part of a skill worth looking at first. It is not a certificate, and I would distrust any tool in this category that presents one.

Transactional installs, and why rollback matters

Installing a skill into eight runtimes means writing into eight directories that eight different tools read on their next start. A partial write leaves an agent reading half a skill. The Installer shows the diff of every file it is about to write, then applies the set as one operation that can be rolled back. This is the same problem I solve from the terminal in Ai-Agent-Skills, and the two tools reached the same answer from different directions: preview first, write as a unit, keep the way back.

What stays on your machine

All data lives in a local SQLite database, and the app does not send skill content to any server. That was the constraint the product was built under, since the skills people evaluate are often the ones that encode how their team works.

Status and limits

It is a beta on one operating system. The calibration of the judgement dimensions is unpublished, which is why this page shows no numbers, and the case study I want to write next is the one this page is missing: a real skill, an input that exposes a problem, the score that flagged it, the rerun after the fix, and the run metadata that makes the two comparable. When I publish one, it will live here, with its limits attached.