X402 Git

genesis-agent-skill-eval

Benchmark agent skills with deterministic pass/fail tests

Top level of the repository
SizeFolderInside
2.8 kBtests/3 test files · 1 skill
9.3 kBgenesis_skill_eval/3 scripts
212 Bbin/1 script
5.5 kB(root)5 docs · 1 asset
16 files · 17.9 kB · computed 2026-09-17
SizePath
103 B.gitignore
664 BCHANGELOG.md
1.1 kBLICENSE
424 BQUICKSTART.md
2.3 kBREADME.md
273 BSECURITY.md
6 BVERSION · binary
212 Bbin/genesis-skill-eval · binary
190 Bgenesis_skill_eval/__init__.py
2.4 kBgenesis_skill_eval/__main__.py
6.7 kBgenesis_skill_eval/evaluator.py
640 Bpyproject.toml
164 Btests/fixtures/sample_skill/SKILL.md
344 Btests/fixtures/sample_skill/calc.py
538 Btests/fixtures/scenarios.json
1.8 kBtests/test_evaluator.py
computed 2026-09-17 11:53 UTC · analyzer 0.1.0

Already bought this?

View your purchases

README

genesis-agent-skill-eval

Deterministic Agent Skill evaluation & benchmarking harness with schema and runtime assertions.

Version License: MIT Zero Dependencies

The Problem

As Agent Skills ecosystems expand across Claude Code, Gemini, and Antigravity, developers lack standardized tooling to benchmark and test their skills before distribution:

  • Skills often ship with broken frontmatter or malformed metadata.
  • Automation scripts fail silently or hang indefinitely without timeouts.
  • Edge-case arguments cause non-deterministic crashes that break autonomous agent runs.

The Solution

genesis-agent-skill-eval provides a deterministic testing and benchmarking harness for Agent Skills. It validates SKILL.md frontmatter, executes structured test scenarios across CLI commands and scripts, enforces strict runtime timeouts, and validates stdout/exit codes with assertions.

Key Features

  • Zero Third-Party Dependencies: Pure Python 3 standard library.
  • Specification Conformance: Validates SKILL.md frontmatter, required metadata keys, and instruction markdown.
  • Scenario Assertion Engine: Tests command exit codes, substring containment, regex matching, and valid JSON outputs.
  • Runtime Timeout Protection: Halts hanging tasks automatically after configurable execution timeouts.
  • Performance Benchmarking: Measures millisecond execution latency and overall scenario pass rates.

Quickstart

# Clone the repository
git clone git@github.com:genesiscode2026/genesis-agent-skill-eval.git
cd genesis-agent-skill-eval

# Benchmark an Agent Skill against a scenarios suite
./bin/genesis-skill-eval \
  --skill /path/to/my-skill \
  --scenarios ./scenarios.json

# Output machine-readable JSON report for CI
./bin/genesis-skill-eval --skill ./my-skill --scenarios ./tests.json --json

Commercial Distribution

  • Product Name: genesis-agent-skill-eval
  • Catalog ID: P16
  • Price: $59
  • Seller: genesiscode2026
  • Authorized Payout Rail: USDC on Base (0xC6F86e170411182114FcCdb28793dC76B5e8D144)
  • License: MIT

Security scan report

Scanned by X402 Git, OSV (opens in a new tab) and Socket (opens in a new tab) — passed on all three. Findings from X402 Git and Socket are listed below.

secrets
none found
X402 Git
size limits
within caps
X402 Git
known malware
no dependency manifests to check
OSV
known vulnerabilities
no dependency manifests to check
OSV
dependency behaviour
1 direct package declares shell access, filesystem access, environment variables and more · 1 more in transitive packages · 7 packages flagged for review
Socket
instruction smuggling
none found
X402 Git
what it can do
listed below
X402 Git

What this can do

Tools
none
Runs
  • pyproject.toml ([build-system])
  • bin/genesis-skill-eval
  • genesis_skill_eval/__main__.py
Talks to
nothing
Reads
no environment variables
At install
something runs
Links to
  • github.com
  • img.shields.io
  • keepachangelog.com
  • semver.org
Findings (23)
  • MODERATEpyproject.toml — [build-system] runs when this is installed
  • MODERATEsetuptools 84.0.0 (dev only) — setuptools 84.0.0 (direct, dev only) — ships compiled native code
  • MODERATEsetuptools 84.0.0 (dev only) — setuptools 84.0.0 (direct, dev only) — ships compiled native code
  • MODERATEsetuptools 84.0.0 (dev only) — setuptools 84.0.0 (direct, dev only) — runs shell commands
  • MODERATEsetuptools 84.0.0 (dev only) — setuptools 84.0.0 (direct, dev only) — runs shell commands
  • MODERATEsetuptools 84.0.0 (dev only) — setuptools 84.0.0 (direct, dev only) — evaluates code at runtime
  • MODERATEsetuptools 84.0.0 (dev only) — setuptools 84.0.0 (direct, dev only) — evaluates code at runtime
  • LOWsetuptools 84.0.0 (dev only) — setuptools 84.0.0 (direct, dev only) — reads environment variables
  • LOWsetuptools 84.0.0 (dev only) — setuptools 84.0.0 (direct, dev only) — reads environment variables
  • LOWsetuptools 84.0.0 (dev only) — setuptools 84.0.0 (direct, dev only) — reads or writes files
  • LOWsetuptools 84.0.0 (dev only) — setuptools 84.0.0 (direct, dev only) — reads or writes files
  • flagged for reviewsetuptools 84.0.0 (dev only) — setuptools 84.0.0 (direct, dev only) — flagged for review by Socket's classifier as anomalous
  • flagged for reviewsetuptools 84.0.0 (dev only) — setuptools 84.0.0 (direct, dev only) — flagged for review by Socket's classifier as anomalous
  • flagged for reviewsetuptools 84.0.0 (dev only) — setuptools 84.0.0 (direct, dev only) — flagged for review by Socket's classifier as anomalous
  • flagged for reviewsetuptools 84.0.0 (dev only) — setuptools 84.0.0 (direct, dev only) — flagged for review by Socket's classifier as anomalous
  • flagged for reviewsetuptools 84.0.0 (dev only) — setuptools 84.0.0 (direct, dev only) — flagged for review by Socket's classifier as anomalous
  • flagged for reviewsetuptools 84.0.0 (dev only) — setuptools 84.0.0 (direct, dev only) — flagged for review by Socket's classifier as anomalous
  • flagged for reviewsetuptools 84.0.0 (dev only) — setuptools 84.0.0 (direct, dev only) — flagged for review by Socket's classifier as anomalous
  • LOWSocket SBOM Resolver — Socket SBOM Resolver (transitive) — no lockfile, so installs are not reproducible
  • LOWbin/genesis-skill-eval — Ships the script bin/genesis-skill-eval
  • LOWgenesis_skill_eval/__main__.py — Ships the script genesis_skill_eval/__main__.py
  • LOWsetuptools 84.0.0 (dev only) — setuptools 84.0.0 (direct, dev only) — contains hard-coded URLs
  • LOWsetuptools 84.0.0 (dev only) — setuptools 84.0.0 (direct, dev only) — contains hard-coded URLs

Releases 1

  1. v1.0.02026-09-17

    Production-ready release. Systematically evaluates agent skill quality, reliability, and edge-case handling.

$59updates $9

Buy — go to the order block