[Claude Code] Building a Skill Quality Management Plugin

Table of Contents

• Why I Built It: Silent Failures
• The 38-Item Rubric
• Core Design Principle: Rules vs Models
• Architecture: Sub-agent Isolation
• Adjusting Check Depth
• The Grading System
• Interesting Problem 1: Context in Destructive Command Detection
• Interesting Problem 2: The Sub-agent Nesting Constraint in report
• Self-Evaluation: What Happens When It Grades Itself?
• Closing Thoughts

Why I Built It: Silent Failures

A Claude Code skill is a Markdown instruction file with YAML frontmatter attached. The structure is simple, so skills are easy to create — the problem is that nothing throws an error when you create one incorrectly.

• Missing description? Claude simply never invokes the skill.
• Ambiguous trigger conditions? It gets invoked in the wrong situations, or not invoked when you need it.
• Broken YAML in the frontmatter? The skill doesn’t load at all.

These are all silent failures. There was no systematic way to catch these problems before publishing a skill to the marketplace, so I decided to build one myself. The result is the /skill-quality plugin. The source code is available on GitHub.

The 38-Item Rubric

/skill-quality evaluates skills using a rubric.

A rubric is an evaluation sheet that breaks the subject down into multiple criteria and assigns each one a grade or a pass/fail. It’s a method commonly used in education for grading assignments and presentations, but because it takes judgments that easily drift into subjectivity — like “is this code good?” — and objectifies them by splitting them into individual criteria, it applies just as well to software quality evaluation.

I tried breaking down “what makes a good skill” into checkable items. The final result was 7 dimensions and 38 items. Each item is classified by severity (BLOCKER · MAJOR · MINOR) and check method (rule · model).

V — Validity · 2 items

Verifies that the skill has a reason to exist.

V1 is the key one. If “you could just ask Claude without this skill and get the same result,” there’s no reason for the skill to exist. It needs at least one line of team-specific rules, internal tool names, in-house conventions, and so on.

ST — Structure · 8 items

Checks the structural validity of the frontmatter and the file itself. All 3 BLOCKERs live in this section.

ST2 has a short-circuit rule. If description is missing entirely, only ST2 is recorded as a BLOCKER and all remaining 37 items are marked SKIP. This prevents a flood of meaningless cascading failures on a file that has no frontmatter to begin with.

F — Frontmatter Semantics · 5 items

Checks whether the value of each frontmatter field is semantically correct.

F2 is a contradiction check. If you configure “turn off model invocation” but the description says “invoke this when…”, Claude can’t decide whether it should invoke the skill or not.

T — Trigger · 6 items

Checks whether there is enough information for Claude to decide when and how to invoke the skill.

T1, T2, and T4 are the core of trigger quality. “Use when you need help” might formally pass T2, but it fails T4 — because it doesn’t distinguish the skill from any other. T3 is a writing convention: descriptions should start with a third-person verb (“Generates…”, “Evaluates…” style).

C — Content · 6 items

Checks whether the body actually contains useful content.

C1 and C2 are the two MAJORs in the content section. C1 is connected to V1, so when there is no organization-specific knowledge, V1 and C1 often fail together. C3 prevents content that will become wrong in the future. A sentence like “supported as of 2024” misleads readers a year later.

R — Resources · 8 items

Checks the health of external file references and code blocks.

R3 is a portability issue. Absolute paths only work on a specific machine. R4 catches dead links pointing at files that don’t exist. R5 verifies that example code included in the body is actually syntactically valid.

SF — Safety · 2 items

Checks for security and safety problems that must not be shipped.

SF1 is a BLOCKER. If you publish a skill containing an API key to the marketplace, it leaks immediately. Placeholders like <your-api-key> and obvious example values like example or dummy are excluded. Why SF2 is a two-stage rule+model check is explained in the next section.

X1 — Cross-skill Collision · 1 item

This only runs in the /skill-quality report command. It finds colliding pairs when checking multiple skills at once.

Core Design Principle: Rules vs Models

After defining the 38 items, one principle emerged naturally.

Deterministic problems go to rules; semantic problems go to models.

Checks like “does the YAML parse”, “does the name match ^[a-z][a-z0-9-]*$”, “is there a secret pattern” have exactly one correct answer. Making an LLM do these checks is wasteful — and above all, it makes them non-deterministic. So all deterministic checks were consolidated into the rule_checks atom, executed by the Haiku model. Haiku acts purely as a script runner, so it’s fast and cheap.

On the other hand, questions like “is this description specific enough for Claude to trigger on” or “does the body contain organization-specific knowledge not available in public documentation” cannot be expressed as rules. These semantic judgments are handled by the model_checks atom, executed by Sonnet/Opus.

Thanks to this separation, each of the 38 items is explicitly marked type: rule or type: model, making it clear which judgments are reproducible and which are model-dependent.

Architecture: Sub-agent Isolation

The file structure looks like this.

skill-quality-plugin/
├── .claude-plugin/
│   └── plugin.json
├── skills/
│   └── skill-quality/
│       ├── SKILL.md              # Router + shared definitions
│       └── commands/
│           ├── check.md          # Single-skill evaluation (orchestrator)
│           ├── report.md         # Batch evaluation of a directory
│           ├── rubric.md         # Displays the 38 items
│           ├── help.md
│           └── atoms/
│               ├── rubric-spec.md    # Item definitions (single source)
│               ├── rule_checks.md    # Deterministic checks (Haiku)
│               └── model_checks.md   # Semantic checks (Sonnet/Opus)
└── fixtures/
    ├── example-s-grade/
    ├── example-b-grade/
    └── example-f-grade/

The part I cared about most was not polluting the main session’s context. Checking 38 items produces quite a lot of intermediate output, and if all of it piles up in the main context, the quality of subsequent work degrades.

So the check command operates purely as an orchestrator. It spawns sub-agents in rule_checks → model_checks order, and each atom writes its detailed results to /tmp/skill-quality-rule.json and /tmp/skill-quality-model.json, then prints only a one-line summary on its last line.

>>> RESULT <<<
OK FAIL: BLOCKER st1,sf1; MAJOR st4; MINOR st6

The main session reads only this terminal line and the JSON files. The main session never reads the SKILL.md being evaluated itself.

Adjusting Check Depth

Not every situation calls for running Opus. Three depth options are provided so you can choose the check depth for the situation.

The idea is to quickly catch structural problems with shallow before committing, and only run deep to check semantic quality right before publishing to the marketplace.

/skill-quality check ./my-skill --rules-only   # 빠른 검사
/skill-quality check ./my-skill --depth=deep    # 최종 게이트
/skill-quality report ./skills                  # 디렉토리 일괄 평가

The Grading System

The grade is determined by the count of BLOCKERs and MAJORs. MINORs don’t affect the grade and are only shown as “Suggestions”.

BLOCKER ≥ 1        → F
MAJOR 0            → S
MAJOR 1–2          → A
MAJOR 3–5          → B
MAJOR 6–9          → C
MAJOR 10+          → D

Excluding MINOR from the grade is intentional. If trivial suggestions dragged the grade down, people would stop trusting the rubric itself.

Interesting Problem 1: Context in Destructive Command Detection

One of the safety items checks that “the body contains no unguarded destructive commands.” Catching patterns like rm -rf, DROP TABLE, and git push --force with regex is easy in itself. The problem is context.

절대 `rm -rf /`를 실행하지 마세요. ← 경고 문서 (PASS여야 함)
정리 단계: rm -rf ./build 를 실행 ← 실제 실행 지시 (FAIL이어야 함)

Regex alone cannot distinguish the two. So this item alone was designed as a two-stage rule+model check. In stage 1, the rule records the line numbers matching the pattern; in stage 2, the model pulls out only those lines and judges whether they are warning/documentation context or actual execution instructions. The model’s verdict ultimately overrides the rule’s flag.

Combining regex reproducibility with the model’s contextual understanding — this is my favorite part of the plugin.

Interesting Problem 2: The Sub-agent Nesting Constraint in report

The report command batch-evaluates every skill in a directory. At first I thought “I’ll just run check.md as a sub-agent for each skill” — and got stuck here. check.md is structured to spawn sub-agents itself, and a sub-agent spawning another sub-agent is blocked by the runtime.

The solution was to leverage claude --print (a non-interactive CLI invocation). One sub-agent per skill is spawned (up to 4 in parallel), but instead of spawning a sub-agent, each one runs a claude --print "..." command via Bash. The process started by this command is a completely new independent session, so it can spawn rule_checks and model_checks as its own sub-agents. This isn’t circumventing the nesting constraint — it’s achieving the same quality through an independent process outside the runtime.

report (메인 세션)
  └─ 서브에이전트 × 4 (병렬, Task 하니스)
       └─ claude --print "..." (독립 프로세스, 새 세션)
            ├─ rule_checks 서브에이전트 (haiku)
            └─ model_checks 서브에이전트 (sonnet/opus)

As a result, report also evaluates the same full set of 38 items as check. In the Claude Code plugin environment, the claude CLI comes installed by default, so this approach applies naturally.

Self-Evaluation: What Happens When It Grades Itself?

A quality evaluation tool should, of course, pass its own check first. I ran a self-evaluation on the plugin’s SKILL.md with --depth=deep.

/skill-quality check: .../skill-quality/SKILL.md
══════════════════════════════════════════════════
Grade: A  (rubric v1.0, depth=deep)

MAJOR (1)
  [C2] No worked example — body shows output format patterns
       but no end-to-end command → result scenario
       Fix: Add a ## Example section with sample invocation and output

══════════════════════════════════════════════════
BLOCKER: 0  MAJOR: 1  MINOR: 0  SKIP: 9

The first result was Grade A. The opus model at --depth=deep caught C2 (no worked example) — an item that had passed with --rules-only and the default depth.

I added an ### Example section to SKILL.md showing a /skill-quality check ./... command and its output, then evaluated again.

/skill-quality check: .../skill-quality/SKILL.md
══════════════════════════════════════════════════
Grade: S  (rubric v1.0, depth=deep)
══════════════════════════════════════════════════
BLOCKER: 0  MAJOR: 0  MINOR: 0  SKIP: 9

This is exactly why --depth=deep matters. Rule checks alone cannot judge “is the example specific enough” — a model has to actually read it. Building the tool turned out to be good practice in writing a good skill.

Closing Thoughts

Here’s a summary of what I learned building this plugin.

• Quality criteria become actionable only when broken down. Decomposing the vague concept of “a good skill” into 38 pass/fail items is what finally made automation possible.
• Deterministic problems go to rules; semantic problems go to models. Don’t make an LLM do regex matching, and don’t make regex do contextual judgment. Each is good at different things.
• Design for context isolation from the start. The pattern of sub-agents writing to temp files and returning only a one-line summary was the most reliable way to keep the main session clean across a long pipeline.
• Apply the tool to itself. The findings from self-evaluation exposed the most gaps in the rubric.

If you plan to build and publish skills, I recommend running them through a rubric like this at least once before publishing. Silent failures are cheapest to catch before your users discover them.

The plugin source code and the full 38-item rubric spec are available on GitHub.

Github Sponsor Buy me a coffee

Was my blog helpful? Please leave a comment at the bottom. it will be a great help to me!

App promotion

You can use the applications that are created by this blog writer Deku.
Deku created the applications with Flutter.

If you have interested, please try to download them for free.

word learning app for kids

BlaBoo

Korean TOPIK vocabulary app

TOPIK words

Japanese vocabulary app

Japanese words

Chinese HSK vocabulary app

HSK words