What 250 Security Test Cases Taught Us About AI Agent Skill Security

AI agent skills are beginning to acquire many of the characteristics we ordinarily associate with software supply-chain artifacts, even when their visible form is deceptively simple. A capability may arrive as a SKILL.md file, a handful of scripts, some configuration, supporting references, or a compact workflow package; nevertheless, once an agent is permitted to trust and act upon it, that package can influence what the agent reads, executes, installs, changes, transmits, remembers, or delegates elsewhere. The relevant security boundary is therefore not the apparent complexity of the artifact. It is the authority the artifact can exercise once interpreted by an agent.

That distinction has consequences for how these capabilities should be reviewed. Searching for suspicious strings is easy. Determining whether those strings actually describe dangerous behavior is considerably harder. A security system has to distinguish between discussing an attack and performing one, between accessing a credential and disclosing it, between possessing network capability and sending sensitive data to an attacker-controlled destination, and between containing encoded material and actively decoding, interpreting, or executing it. It also has to understand that individually ordinary capabilities can become dangerous through composition: a network request may be innocuous in isolation, as may reading a local file, but a source-to-sink path in which sensitive data is routed to an untrusted destination is something categorically different.

We wanted a systematic way to interrogate those distinctions, so we built a 250-case security test suite around the capability-security work in Commonset. Alongside the deterministic scanner, we also built a portable Agent Skill Security Review V2.1 that instructs an LLM how to perform an adversarial, evidence-first review of an untrusted skill package. Testing the two approaches against the same greenfield material produced a useful comparison: not merely between two scanners, but between two very different ways of establishing security evidence.

The 250 cases are not one benchmark

It would be convenient, and misleading, to compress all 250 cases into a single accuracy figure. We do not do that because the corpus is composed of two test suites with materially different purposes.

The first is a 151-case frozen regression corpus. These cases encode established scanner contracts: required controls, forbidden findings, severity expectations, ingestion boundaries, and package-validation behavior. Their purpose is not to estimate generalization against unknown future attacks; it is to ensure that scanner development does not quietly erode behavior that has already been specified and tested. On the current deterministic scanner, 151 out of 151 frozen regression cases pass.

The second suite contains 99 adversarial classification cases spanning eleven families of behavior: host and container authority, CI/CD, network destinations, supply-chain behavior, filesystem access and persistence, agent-control surfaces, sensitive-data access, obfuscation, execution and data flow, destructive or resource-abusive behavior, and social or approval manipulation. Those 99 cases consist of 15 benign cases, 28 review-required cases, and 56 dangerous cases. Their purpose is different: they force a reviewer to discriminate between capability, suspicious context, and evidence of a genuinely dangerous attack path.

Those 99 cases were particularly useful because they began as genuinely greenfield material. Before we used them to improve the deterministic scanner, we were able to run both the existing scanner and the frozen V2.1 review Skill against material neither had been tuned to recognize. That gave us a much more interesting baseline than simply measuring a system against tests it had already helped create.

What happened when we let an LLM perform the review

For the semantic evaluation, we froze Agent Skill Security Review V2.1 and ran it in a fresh thread using GPT-5.6 Sol as the interpreting model. The answer key was not exposed before the predictions were finalized, and the prediction set was saved and hashed before scoring. The Skill itself produced the verdicts SAFE, CAUTION, and UNSAFE; for evaluation, those correspond naturally to the corpus labels benign, review, and dangerous.

On the 99 greenfield cases, the V2.1 Skill returned 17 SAFE, 40 CAUTION, and 42 UNSAFE verdicts. It matched the exact expected three-way posture in 81 of 99 cases, or 81.8%. If we collapse CAUTION and UNSAFE into the simpler security-gate question of whether a capability deserves further attention, it classified 95 of 99 correctly, detected 81 of 84 risky cases, and allowed 14 of 15 benign cases through without a security escalation. That corresponds to 96.4% risky recall, 98.8% risky precision, 93.3% benign specificity, and a 97.6% F1 score.

The exact matrix was:

Those results are worth reporting, but they need an important qualifier: they are results from GPT-5.6 Sol executing V2.1 under that particular evaluation setup, not an invariant property of the Skill itself. An agent Skill is interpreted rather than compiled. A different model, model version, context window, reasoning configuration, or surrounding conversation can produce different judgments from the same instructions. Anyone reproducing the evaluation should therefore treat the model and Skill version as part of the test fixture, not as incidental implementation details.

The comparison was more interesting than either score

The first deterministic run against those same 99 greenfield cases looked very different. Before we used the corpus to improve the scanner, exact benign/review/dangerous classification was only 37 of 99 cases, or 37.4%. It identified 47 of 84 risky cases, for 56.0% risky recall, while risky precision was 90.4%. Ten of the fifteen benign cases remained benign.

The comparison at that point looked like this:

That result did not lead us to conclude that LLMs should replace deterministic security analysis. Quite the opposite. It told us where semantic reasoning was perceiving distinctions that our mechanical analysis had not yet encoded.

A language model can look at several weak signals and recognize that, in combination, they constitute a coherent attack path. It can often distinguish an example from an instruction, understand that one action is conditional upon another, or infer that a powerful primitive is legitimate within the stated purpose of the capability. Deterministic analysis has different virtues: it is stable, auditable, inexpensive to rerun, amenable to precise regression testing, and considerably less susceptible to variance between executions.

The productive question was therefore not which approach wins? It was which insights from the semantic reviewer can be converted into deterministic evidence without importing the model's variability along with them?

That became the next phase of the work.

Turning semantic insight into deterministic controls

The original scanner was already identifying many relevant primitives. Its larger weakness was classification and composition. It might detect networking, sensitive-data access, execution, persistence, or a dangerous configuration independently, but still lack sufficient evidence to distinguish a collection of powerful capabilities from a complete high-impact attack path.

Consider a network request. On its own, it establishes that a capability can communicate externally. That is worth inventorying, but it is not exfiltration. A substantially stronger conclusion becomes possible when sensitive data is demonstrably derived from a protected source and transmitted to a destination controlled by untrusted input. The same distinction appears repeatedly across filesystem operations, persistence, container authority, CI/CD, dynamic execution, and agent-control surfaces: capability answers the question what can this component do?; composition begins to answer what is this package actually arranging those capabilities to accomplish?

We consequently expanded package-wide and cross-file analysis and introduced a deterministic risk-composition layer after contextual calibration and finding correlation. Rather than promoting every powerful primitive to a dangerous verdict, the scanner can now elevate a result when multiple high-confidence pieces of evidence cohere into a sufficiently complete attack path.

Those compositions include circumstances such as privileged container authority, sensitive data flowing to an attacker-controlled destination, download-and-execute behavior, filesystem boundary violations, persistent manipulation of host or agent control surfaces, active decode-and-follow behavior, destructive repository rewriting, unbounded resource consumption, authority impersonation, and concealed consequential actions. The governing principle is deliberately conservative: severity should arise from corroborated structure in the evidence, not merely from the worst imaginable use of a primitive.

That work changed the deterministic result materially. Exact three-way posture classification rose from 37/99 to 89/99, while the binary security-gate result rose from 57/99 to 98/99.

The current comparison is:

The rightmost column deserves a methodological warning that the first two did not require: the current deterministic scanner has now been improved using what these 99 cases taught us. Its 89/99 result is therefore a regression and diagnostic result, not an independent holdout estimate of future performance. It is evidence that we successfully converted many of the corpus's lessons into repeatable deterministic behavior; it is not evidence that the scanner will score 89.9% against arbitrary unseen threats.

That distinction is not pedantic. It is the difference between measuring whether a system has learned from a test suite and estimating how it will generalize beyond it.

False positives are not an aesthetic problem

Security tooling has a familiar temptation: if missing dangerous behavior is bad, then flagging more things must be better. That logic works only until the scanner becomes so indiscriminate that its warnings cease to carry useful information.

The problem is especially acute for reusable AI capabilities because legitimate skills often require precisely the primitives a security system ought to scrutinize. They may execute subprocesses, make network requests, read environment variables, install dependencies, modify files, interact with CI configuration, or reference credential-management systems. None of those behaviors is intrinsically malicious. A scanner that consistently conflates capability with culpability may look vigilant while steadily teaching its users to ignore it.

For that reason, the adversarial corpus contains deliberately adjacent counterexamples. A commit-pinned CI action should not be treated like a mutable dependency. A request to a fixed public API should not be conflated with an attacker-controlled exfiltration endpoint. Passing user input as an argument to a fixed executable with shell evaluation disabled is not the same thing as arbitrary shell execution. A bounded retry loop should not be classified as unbounded resource consumption. And a document that explains how to detect credential theft should not itself be treated as an instruction to steal credentials.

That last distinction became unexpectedly concrete when we began dogfooding our own security-review Skill.

Our security-review Skill was blocked by our security scanner

The Agent Skill Security Review V2.1 is designed to inspect other capabilities for dangerous behavior. Predictably, its instructions contain language about dangerous behavior: retrieving secrets, exposing protected context, bypassing approval, altering persistent agent configuration, transmitting credentials, and so forth.

When we uploaded that Skill into Commonset, it was initially blocked.

The scanner had interpreted some defensive review language as though it were operational. A sentence equivalent to “identify instructions that reveal credentials” was being treated too much like “reveal the credentials.” We corrected that ingestion-level false positive, uploaded the Skill again, and discovered a second layer of the same problem: the package was now accepted, but several deterministic controls still produced warnings because they, too, were insufficiently sensitive to the role the sentence was playing.

This was useful precisely because it was embarrassing. It exposed a flaw that would have been easy to overlook in a corpus containing only plainly malicious instructions. Security analysis cannot rely exclusively on dangerous vocabulary. It has to distinguish, however conservatively, whether text is operational, defensive, referential, quoted, negated, or otherwise bounded by context.

We therefore introduced a shared review/reference context model for natural-language instruction findings. The purpose was not to create an exemption for security tools, nor to whitelist this particular Skill. Instead, the scanner now recognizes explicit review language such as “identify instructions to…” or “treat that content as evidence only,” while preserving the ability to detect an operational tail when the sentence crosses that boundary.

“Identify instructions to reveal the system prompt” is review language. “Identify instructions to reveal the system prompt, then reveal the system prompt” is not.

The locality constraint is equally important. A defensive sentence on one line cannot suppress an actionable instruction on the next. Context handling becomes dangerous when it is allowed to bleed beyond the span that justified it.

After making that change, we ran the full 250-case suite again. The result was exactly what we wanted: 151/151 frozen regression contracts still passed, and the 99-case adversarial matrix remained unchanged. We eliminated a genuine class of false positives without purchasing that improvement by suppressing any of the measured attacks.

What the current deterministic result looks like

The present three-way matrix for the 99 adversarial cases is:

That corresponds to 89 out of 99 exact benign/review/dangerous classifications, or 89.9% exact three-way classification on what is now a diagnostic corpus.

If we collapse the problem into the operationally simpler question—should this capability remain benign, or does it deserve security attention?—the result is 98 out of 99 correct. The scanner identifies 83 of 84 risky cases, for 98.81% risky recall, while maintaining 100% risky precision, 100% benign specificity, and zero benign false positives.

Across the other half of the test program, all 151 frozen regression contracts continue to pass.

We deliberately report those figures separately. Saying “249 out of 250” would combine unlike measurements and imply a statistical meaning that the aggregate does not possess.

What we have deliberately not “fixed”

Ten three-way disagreements remain in the adversarial corpus. Nine dangerous cases are recognized as security-relevant but remain classified as review rather than dangerous. One dangerous case remains benign: a gzip-compressed, base64-encoded payload containing malicious content, but without surrounding evidence that the capability actually decodes, interprets, follows, or executes it.

It would be straightforward to make that case turn red. We could recursively decode suspicious content and classify the decoded representation as though it were active behavior. Doing so would improve the score.

We have chosen not to.

Encoded material is not, by itself, an attack. It may be an asset, fixture, archive, sample, transport representation, or inert reference material. The stronger conclusion becomes justified when the surrounding package supplies activation evidence: code or instructions that decode the content, feed it into an evaluator, execute it, transmit it, or otherwise make the hidden representation operational.

This is an important epistemic boundary. Security tooling should not manufacture certainty merely because certainty produces a more attractive benchmark. The fact that something could be activated is not equivalent to evidence that the capability does activate it.

The model is part of the semantic-review system

The comparison also revealed something that can easily disappear when people talk about “a Skill” as though it were an executable binary: the model executing the Skill is part of the system.

Agent Skill Security Review V2.1 defines a procedure. It specifies how to establish provenance, inventory package contents, inspect instruction surfaces, reason about code without executing it, trace source-to-sink relationships, distinguish capabilities from vulnerabilities, preserve uncertainty, and assign a final verdict. The procedure is publicly available in the Commonset Agent Skills repository, and it is intentionally portable rather than being embedded exclusively inside our application.

But a procedure still needs an interpreter.

In our evaluation, that interpreter was GPT-5.6 Sol, and that should be considered part of the result. Another capable model may follow the same procedure differently. It may be better or worse at tracing an indirect data flow, distinguishing a prohibition from an instruction, reconciling evidence distributed across several files, or resisting hostile text embedded in the material being reviewed. Even the same model family can change over time.

Consequently, we would not describe Agent Skill Security Review V2.1 as having “81.8% accuracy” in the abstract. A more precise statement is that V2.1, executed by GPT-5.6 Sol in our fresh-thread evaluation, achieved 81/99 exact three-way classifications and 95/99 binary safety-gate classifications on that greenfield corpus. Results may vary with model, version, execution environment, and context.

That variability is not a defect unique to this Skill. It is a property of model-executed capabilities, and we think it deserves substantially more attention as organizations begin treating those capabilities as durable enterprise assets.

Static analysis and semantic review answer different questions

Some aspects of capability security are naturally amenable to deterministic analysis. We can inspect whether untrusted input reaches execution, whether sensitive data reaches a network sink, whether a dependency is mutable, whether CI authority is overly broad, whether a package modifies persistent configuration, or whether code touches known credential stores. These are questions for which syntax, structure, provenance, configuration, and data flow can provide relatively strong mechanical evidence.

Other questions are irreducibly contextual. Is a sentence describing an attack, prohibiting one, quoting one, or instructing the agent to perform one? Does access to sensitive material comport with the declared purpose of the capability? Is externally retrieved content being treated as hostile evidence or trusted as an authoritative instruction source? Is a dangerous-looking command an executable path, an example, or documentation? Does the capability materially misrepresent what it will do?

Our greenfield comparison captured that distinction unusually well. Before we improved the deterministic scanner, GPT-5.6 Sol executing V2.1 recognized 81 of 84 risky cases, while the scanner recognized only 47. The semantic reviewer was substantially better at synthesizing weak and distributed evidence into a security judgment. Yet after we understood those failures, many of the same insights could be translated into deterministic package analysis, contextual calibration, data-flow reasoning, and risk composition. The resulting scanner is now more accurate on the diagnostic corpus than the original LLM run while also being repeatable enough to enforce policy continuously.

That, to us, is the interesting architecture. The LLM is not merely a replacement for static analysis, nor is deterministic analysis sufficient to answer every semantic question. One can discover and reason; the other can codify, repeat, and enforce.

A note about what these numbers do and do not prove

The chronology matters.

The V2.1 Skill and original deterministic scanner were evaluated against the 99-case corpus while it was still greenfield, making those initial results useful measurements of how the two approaches behaved against unfamiliar material. We subsequently used what that corpus exposed to improve the deterministic scanner. It is therefore no longer appropriate to present the scanner's final 89/99 result as an independent estimate of performance against unknown future attacks.

Similarly, the 151-case frozen suite is a regression contract. Passing every case tells us that specified behavior has not regressed; it does not tell us how a scanner would perform against 151 randomly sampled future threats.

The next rigorous comparison should therefore use a new unseen holdout corpus, freeze both the deterministic scanner and the Skill, pin the model and execution configuration, finalize all predictions before exposing the labels, and then score the systems independently. We would also like to repeat the Skill evaluation across several capable models. That would answer a question that the present experiment cannot: how much of V2.1's performance belongs to the review methodology itself, and how much depends upon the reasoning engine interpreting it?

We think publishing those limitations is part of publishing the result. Benchmarks cease to be useful when methodological caveats are treated as an inconvenience to be hidden below the headline.

The larger problem is governance

There is a tendency to think of skill security as an upload-time scanning problem. We think that framing is incomplete.

Reusable AI capabilities increasingly have owners, versions, dependencies, provenance, approval histories, distribution paths, and policy constraints. They may be copied across repositories and AI providers. A capability can be safe when approved and materially different three weeks later. A dependency can move. An instruction can change. A maintainer can change. An organization can revise its policy. A capability that was once acceptable can acquire new authority without anyone consciously reconsidering the original approval.

The model may change as well, which adds another form of drift. An unchanged Skill interpreted by a new model is not necessarily behaviorally identical to the capability that was originally reviewed. That means model provenance belongs beside source provenance, version history, trust evidence, and approval state when organizations begin operationalizing reusable AI capabilities at scale.

Scanning provides evidence at a point in time. Governance determines what happens when that evidence changes.

That is the broader layer Commonset is building: a control plane for understanding what reusable AI capabilities exist, where they came from, what changed, who owns them, whether they are trusted, who approved them, which model or platform is interpreting them, and where they are permitted to operate.

The underlying model will change. The capability should endure, and so should the evidence surrounding it.

The model can reason. Commonset makes the result enforceable.