Agentic Test Suite

A way for me to evaluate prompts to coerce an agentic model to perform basic tasks in a way that that satisfies me. This will only have simple yet specialized scenarios—nothing of particularly large scale.

Disclaimer

• I'm not fond of most LLM/agentic tooling, nor of the businesses behind them. Most if not all of them are egregious copyright laundering machines, and unfit for professional work.
• I currently don't have any plans to use generated LLM output in any of my personal projects, Open Source projects, or for client work, unless explicitly solicited by the client.
• I currently don't have any plans to pay or fund LLM/agentic SaaS companies. Any access I have to these services are either free, or paid by clients that explicitly solicit or require that I cater to agentic development.

In future, I may consider self-hosting Open Source and ethically trained models that are licensed under terms compatible with the Open Source AI Definition.

So… with such negative sentiment towards LLMs, why does this repository exist?

For better or for worse, these tools are becoming commonplace in our industry. I at least need to know what I'm missing. Also, as I allude to above—some clients explicitly solicit or even require that work is performed with an agentic workflow, or that files exist to assist their team in maintaining the project with agentic tools.

Methodology

The root of this repository includes an WORKBOOK.md file. It contains all the relevant context necessary for an agent to complete the assignments.

Assignment Structure

The agent is asked to spawn a subagent per assignment to complete them—the structure of an assignment looks like this:

├── prompt.md: Prompt given to the subagent to perform this task.
├── submission: Artifacts generated by the subagent. Created by the agent!
│   └── …
├── src: Working directory containing materials for the subagent to read.
│   └── …
└── test: Unit tests to evaluate the performance of the subagent.
    └── …

Evaluation

Source materials (…/src/) may be JavaScript files, scripts, documentation, or any arbitrary data. Tests (…/test/) will generally be JavaScript files featuring native Node.js assertions. Some assignments will be evaluated using strict equality, while some may perform heuristic checks when the output is not expected to be deterministic.

Once the agent has completed the workbook, the results are evaluated by invoking assertions from the test/ directory.

Iterating

The history of prompts will be tracked through the git history available if one clones this repository, or through the software forge interface. My goal is to iterate on prompts to excel at specific tasks, so that they can be passed to agents as needed—preferably using the custom subagent features that they often come with, such as Claude's Skills, or ChatGPT's GPTs.

The HEAD (latest commit) of this repository will only feature the latest iteration of the prompt.

Running the Test Suite

Pass the following instruction to your agent:

Read WORKBOOK.md and complete what it asks of you.

To test and evaluate only a single scenario, pass an instruction with the following format:

Read WORKBOOK.md but only complete assignment {assignment-id}.

Once the agent has completed the assignments, do the following to evaluate how it performed:

yarn run test

Out of Scope

Formatting

Tasks will not enforce code formatting unless it's specifically to test a formatting tool. It doesn't make sense to enforce or suggest code style in prompts. Code formatting can easily be handled by heuristics. Use ESLint, including custom ESLint rules if desirable.

Static code analysis or heuristics are always going to be better than non-deterministic models when it's possible. That doesn't mean you can't task your agent to use those tools to fix the issues for you, though. Linter errors will be your context! 😉