Skip to main content

Agents & the rulebook

The agentic test-authoring framework consists of four canonical prompts and a shared rulebook. This page explains what each prompt does, how the host-neutral design works, and how to extend the system.

Canonical prompts​

All four agent definitions live as markdown files under tests/agents/:

FileAgent role
tests/agents/playwright-planner.mdConverts a free-form scenario description (English, user story, PRD excerpt) into a structured spec file at tests/specs/
tests/agents/playwright-generator.mdConsumes a spec file and produces an executable .spec.ts plus any required POM/helper extensions
tests/agents/playwright-reviewer.mdReviews a generated or modified test against the rulebook; emits PASS/FAIL with line-precise findings
tests/agents/playwright-stabilizer.mdConverts a raw VS Code Playwright recording plus paired English step file into a canonical spec

These files are canonical and host-neutral. They contain no tool names, no host-specific syntax, and no references to a particular AI platform. Any AI host can run them.

The rulebook​

tests/agents/rules/playwright-rules.md is the single source of truth for test-writing conventions in this repo. The Generator references it before writing any code. The Reviewer enforces it in every review cycle.

Key sections in the rulebook:

  • Golden rules — five hard constraints (READ before WRITE; USE helpers not raw locators; VERIFY before interacting; CLEAN UP data; ONE source of truth for selectors). Violating any of these makes the test wrong.
  • Locator priority order — exact priority ranking from element ID down to CSS class. Testers and generators must follow this order.
  • Helper method reference — the full API surface of FormHelper, WaitHelper, TabHelper, and AgGridHelper.
  • Common pitfalls & fixes — thirteen numbered patterns with the problem, cause, and canonical fix (e.g., #drawer-title resolving to 9+ elements; grid dropdown cells vs. FormHelper dropdowns; toast race conditions).
  • Page object structure template — the canonical POM skeleton that the Generator and human developers must follow.
  • Output checklist — a pre-delivery checklist the Reviewer runs through before issuing a PASS.

The rulebook also covers the sibling-repo constraint: application source (src/) lives in pace-ui-application, not here. Agents must use shell commands targeting $PACE_APP_REPO to look up field names, colId values, and route paths.

A companion file tests/agents/rules/pom-extension-rules.md gives detailed guidance specifically for extending existing POMs and adding new methods.

Host-neutral design​

The framework is designed to run on any AI host without modification to the canonical prompts. Host-specific adapters are thin wrappers that:

  1. Import or inline the canonical prompt (or reference it by path).
  2. Map the host's tool names to the generic operations the prompt expects (file read, bash, grep, write).
  3. Register the agent under the host's naming convention.

Current per-host adapter directories:

DirectoryHost
.claude/Claude Code (skills + commands)
.codex/OpenAI Codex CLI
.gemini/Gemini CLI
.agent/Antigravity

When you update a canonical prompt in tests/agents/, the change is immediately picked up by all host adapters that reference it by path. Adapters that inline the content must be updated separately — keep them as thin as possible to minimize drift.

Extending the framework​

Adding a new agent variant​

  1. Write the canonical prompt in tests/agents/ (markdown, host-neutral).
  2. Add a thin adapter in each per-host directory that references or imports it.
  3. Update tests/agents/rules/playwright-rules.md if the new agent introduces new conventions that the Reviewer should enforce.

Updating the rulebook​

The rulebook is the contract between all agents and all human developers. Before editing it:

  • Check whether the change affects the Reviewer's output checklist.
  • Check whether existing tests or POMs violate the new rule (they should be updated or grandfathered with an explicit comment).
  • Consider whether the change needs a corresponding example in the "Common pitfalls" section.

Workflow files​

tests/agents/workflows/ contains multi-step workflow definitions that chain agents together (e.g., Stabilizer → Generator → Reviewer in a single invocation). These are host-specific and live in the workflow directory, not in the canonical tests/agents/ root.