Skill-Use Exposes an Agent Weakness: Completing Tasks Is Not the Same as Using Skills

Agent ecosystems increasingly package operational knowledge as skills. A short description signals when a skill applies, while the full document specifies the procedure, permitted tools, and forbidden actions. It is tempting to assume that placing the right skill in an agent's library is enough. A paper submitted to arXiv on August 5, Skill-Use: Can LLMs Actually Use Skills in Agentic Harnesses?, argues otherwise: the ability to complete a task and the ability to recognize and correctly apply a skill are distinct.

Three stages of skill use

The paper studies progressive disclosure. At the beginning of a task, an agent sees only a skill's name, short description, and file path. It must decide that the skill is relevant, retrieve the full document, and then follow the procedure. Skill-Use therefore separates performance into three dimensions:

• Trigger asks whether the agent retrieves the relevant skill.
• Compliance measures whether it follows the prescribed steps.
• Boundary measures whether it avoids explicitly forbidden operations.

The benchmark combines these dimensions into an SU score. Execution receives credit only after the relevant skill has been triggered. This matters because a correct final answer does not prove that the agent recognized or followed the intended operating procedure.

79 skills, 177 tasks, and unstable leading performance

Skill-Use pairs 79 real-world skills with 177 executable tasks spanning nine domains. Every task is grounded in real files and runs in an isolated Docker sandbox with full tool access. The benchmark records the entire action trajectory and evaluates it with a multi-item rubric covering actions and final artifacts. The construction process also uses masked-skill screening to remove generic requirements and multi-agent adversarial review to audit task scope and rubric verifiability.

Across eight models and two agent harnesses, the leading configuration reaches an SU score of only 0.613. The failures are not all of one kind. Some models recognize a relevant skill but depart from its procedure. Others could comply once a skill is visible but fail to trigger it. The paper also reports that switching the harness changes absolute scores and model rankings. Skill use is therefore a property of the model-harness combination, not simply of the base model.

Preloading fixes retrieval more than execution

The researchers compare native progressive disclosure with a setting in which the complete skill document is inserted into the initial context. Preloading raises SU primarily by increasing Trigger. On paired traces where both modes trigger the skill, the execution-quality gap becomes small. In other words, placing the full document in view helps the model notice the skill, but does not materially improve procedural compliance after selection.

A library-size experiment reaches a similar conclusion. The target skill remains available while the library grows from 1 to 10, 20, and 30 entries through randomly sampled distractors. The main increase is in no-skill outcomes rather than wrong-skill selections. The largest visible drop occurs between one and ten skills, with smaller changes afterward. As libraries grow, precise descriptions and distinguishable trigger conditions become core system-design concerns.

Partial skill use can be worse than no skill

The paper also pairs triggered skill-enabled runs with baselines on the same tasks and models with the skill library disabled. The effect on task completion turns from negative to positive near an SU score of 0.5. Below that level, an agent may commit to a prescribed toolchain or format without carrying the procedure through, producing an artifact that satisfies neither the skill nor the unaided task strategy.

The engineering lesson is direct: installing more skills is not a free capability upgrade. Teams need to improve trigger descriptions, process observability, prohibited-action boundaries, and the way the harness exposes and invokes skills. Model leaderboards should also disclose the harness rather than presenting agent performance as a model-only trait. The full methods and experimental boundaries are available in the arXiv paper.

A skill creates value only when an agent finds it at the right moment and carries the procedure through to the final step.