Our in-house effort to combine critical thinking and a strong testing mindset with AI-assisted automation — using a Planner, Generator and Healer agent loop built on Playwright. AI helps us accelerate the work, but the fundamentals of good testing remain human-led. We ran it against our own website first.
The easy version of “AI-first testing” is to point a model at an app and let it emit whatever assertions make the run go green. You get a lot of tests quickly, and almost none of them tell you anything: brittle selectors, assertions that can't fail, live calls that flake, and a healer that fixes red builds by deleting tests.
Test Labs is our attempt at the harder version. AI does the work that is genuinely tedious — exploring the app, drafting a plan, writing the first draft of a spec, diagnosing a failure — inside a set of rules that a good QA engineer would recognise. The model moves faster; it doesn't get to lower the bar.
Three agents, each with a narrow job and its own hard limits, with a person on the approval line twice: once to sign off the Planner's test plan before any code is written, and once to review the Generator's spec in the pull request before it merges. A new feature enters on the left as a request; it leaves on the right only as a spec a human has read. When a run fails, the Healer — not the Generator — decides whether the test or the product is wrong, and its patched spec goes back through the same review before it lands.
The request itself comes straight from wherever the team already tracks work. The orchestrator connects to Jira, Confluence or Notion through MCP connectors and pulls the ticket or the requirements page in as the Planner's starting context. The connection runs both ways: once a spec is approved and passing, the coverage and run status are written back to that same ticket, and when the Healer decides a failure is a real product defect it files that back to the tracker too — so the tracker always reflects what is actually tested.
specs/<feature>.md
Walks the real user journey in a browser, notes the actual routes, roles and accessible names, and writes a plain-English test plan: happy path, validation paths, edge cases, and any side effect that needs stubbing. It writes no code.
tests/<feature>.spec.ts
Turns one plan into one spec, then runs it until it passes on Chromium. It reuses shared helpers, prefers role-based locators, and keeps each test isolated. It is not allowed to add test.only, blind waits, or an assertion that always passes.
a fix — or a bug report
When a run goes red it reproduces with a trace, then decides: stale locator or changed copy → fix the test; new legitimate UI → update the test; a 500, broken navigation or console error → stop and write up the product bug. It never deletes or skips a test to go green.
These rules live in the agent definitions themselves, so every plan and every generated spec is held to them. None of them are new ideas — they are just the things that go first when a test suite is written in a hurry.
The AI drafts; a person signs off. The test plan is approved before code is written, and the generated spec is reviewed in the pull request before it merges — the model never decides on its own what "tested" means.
One describe block per feature, one assertion target per test, state set fresh in beforeEach. A failure names exactly what broke, not "something in the contact flow".
getByRole, getByLabel, getByPlaceholder — the way an assistive-tech user or a real person finds the control. Brittle selectors and nth-child are banned unless there is genuinely no alternative.
URL, visible text, a toast, an input value. Never an implementation detail, and never an assertion loosened just to make CI pass.
Anything that would send an email or hit a third-party API is stubbed at the network layer by default, so a red build means a real regression — not a flaky vendor. A live mode exists for when you actually want the round trip.
The site has anti-bot guards — a honeypot field, a "filled too fast" timer, a rate limit. The tests work with them, the way a real submission does, instead of reaching past them.
Cold route compilation is warmed once up front so it cannot masquerade as a timeout. Retries are for infrastructure noise, not for papering over a race the test introduced.
A three-browser matrix on every pull request, a nightly smoke run against production, HTML reports and traces kept as artifacts. Tests that do not run in CI do not count.
The suite lives in its own folder in this site's repository, with its own dependencies so it never touches the app's build. It covers navigation through the site, submitting the audit form, and submitting the contact form — including the path where someone arrives at the contact page carrying their audit results.
Building it surfaced real things worth fixing rather than just passing tests: the mobile menu button had no accessible name, the Services mega-menu closed itself mid-interaction, and cold route compilation in development was slow enough to look like a failure until it was warmed up front. Each of those became a note back to the product, not a weakened assertion.
It runs on every pull request across Chromium, Firefox and mobile WebKit, and again each night as a smoke test against the live site — with the outbound email and third-party audit calls stubbed, so a nightly failure means the site actually broke.
Test Labs is the R&D behind the AI-assisted option on our testing & automation service. When we build an automation suite for you, this is the shape of it: a plan you can read, specs that assert what your users actually see, deterministic runs, and a CI pipeline wired in from day one. The agents make it faster to write and cheaper to maintain — they don't change what a good test is.
It is human-in-the-loop by design. One of our engineers signs off the test plan before any spec is written, reviews every generated spec in the pull request before it merges, and gets the final call whenever the Healer touches a test — the AI never ships coverage on its own.
Test Labs is Codifyany's in-house effort to combine critical thinking and a strong testing mindset with AI-assisted automation — a Planner, Generator and Healer agent loop, with the current suite built on Playwright. It pairs those three role-scoped agents with a strict set of engineering rules a good QA engineer would recognise. AI accelerates the tedious work — exploring the app, drafting test plans, writing specs, diagnosing failures — but the fundamentals of good testing stay human-led. We ran it against our own website first.
The Planner walks the real user journey in a browser, notes actual routes, roles and accessible names, and writes a plain-English test plan covering happy paths, validation paths, edge cases and side effects. A human approves the plan before any code is written. The Generator turns one plan into one Playwright spec and iterates until it passes on Chromium, using role-based locators and isolated tests. The Healer diagnoses failing runs: if it's a stale locator or changed copy it fixes the test; if it's a 500 error, broken navigation or console error it stops and files a product bug. It never deletes or skips a test to go green.
The rules live in the agent definitions themselves: human approval at plan and spec stages, one isolated test per behaviour with fresh state in beforeEach, role-based locators (getByRole, getByLabel, getByPlaceholder — brittle CSS selectors banned), assertions on what the user sees rather than implementation details, deterministic API stubbing at the network layer by default, respect for the app’s anti-bot guards (honeypot field, "filled too fast" timer, rate limit), flakiness treated as a bug rather than papered over with retries, and a three-browser CI matrix (Chromium, Firefox, mobile WebKit) on every pull request plus a nightly production smoke run.
The orchestrator connects to Jira, Confluence or Notion through MCP (Model Context Protocol) connectors and pulls the feature ticket or requirements page in as the Planner's starting context. The connection runs both ways: once a spec is approved and passing, the coverage and run status are written back to that same ticket, and when the Healer identifies a real product defect it files that back to the tracker too — so the tracker always reflects what is actually tested.
Yes. Test Labs is the R&D behind the AI-assisted option on Codifyany's testing and automation service, which covers web automation (Playwright or Cypress), native mobile automation (Maestro), API automation for REST and GraphQL, and CI/CD integration with quality reporting. When we build a suite for you it follows the same shape as Test Labs: a plan you can read, specs that assert what your users actually see, deterministic runs, and a CI pipeline from day one. It is human-in-the-loop by design — a Codifyany engineer signs off the test plan, reviews every generated spec in the pull request, and gets the final call whenever the Healer touches a test. Proof-of-concept packages start from NZD 5,000.
Book a free 30-minute call. Tell us what your team is testing by hand, and we will map the flows worth automating and how the AI-accelerated workflow would fit.