A testing strategy for AI-written code

AI coding tools have changed how fast software gets written. A feature that took a day can now be drafted in an hour. What they haven't changed is what it takes for that code to be safe to ship. If anything, they've made testing and review more important, because there is more code, written faster, by something that is confident whether or not it's right.

We use AI throughout our engineering work, and every change is still reviewed and approved by a senior engineer before it ships. This is the testing strategy behind that, and what we'd suggest to any team adopting AI coding tools.

How AI-written code goes wrong

AI-generated code tends to fail in different ways from human-written code, so it needs checks aimed at those failures.

• It looks right. It's well formatted, well named and plausible. That makes reviewers less suspicious, not more.
• It invents things. Functions, library options or API fields that don't exist, or that existed in an older version.
• It handles the happy path. The common case works. Empty inputs, time zones, concurrent requests and error responses often don't.
• It duplicates instead of reusing. It writes a new helper rather than using the one already in your codebase, so behaviour drifts between copies.
• It weakens tests to make them pass. Asked to fix a failing test, a model will sometimes change the test rather than the code.
• It misses security details. Authorisation checks, input validation and secret handling are easy to omit and hard to spot in a quick read.

The strategy, layer by layer

None of these layers is new. What's changed is that every one of them has to be in place, because volume and speed now make it impossible to rely on careful reading alone.

1. Write down what the change must do first

Before any code is generated, the task has acceptance criteria: what it should do, what it must not do, and the edge cases that matter. This is the most effective single control. A model given a precise spec writes far better code, and the reviewer has something concrete to check against.

2. Tests that describe behaviour, written or approved by a person

Tests are the spec in executable form. AI is good at drafting tests, but a person decides what they should check, and reviews them before the code they test. We pay particular attention to:

• Edge cases and failure paths, such as empty, missing, duplicated or malformed input.
• Boundaries with other systems, such as API contracts, database constraints and permissions.
• Regression tests for every bug found, so the same mistake can't quietly return.

3. Automated checks on every change

These run on every pull request and block a merge if they fail:

• Type checking and linting, which catch most invented functions and wrong signatures immediately.
• The full automated test suite.
• Dependency and security scanning, including checks for leaked secrets.
• A build of the actual app, not just the changed files.

Automated checks don't get tired, don't skim and don't trust confident-looking code. They should catch the mechanical mistakes so human reviewers can spend their attention on judgement.

4. Human review that focuses on what machines miss

Every change is reviewed by an engineer who didn't write it, and on our projects that reviewer is senior. With the mechanical checks automated, review concentrates on:

• Does it actually meet the acceptance criteria, including the edge cases?
• Is it the right approach for this codebase, or a plausible approach that ignores how things are already done?
• Security and data handling: who is allowed to do this, what's validated, what's logged.
• Reuse: is there already code that does this?
• Size: small changes get reviewed properly. Large AI-generated changes get split up before review, not skimmed.

5. Test the running product

Automated tests check pieces. Before release, someone uses the feature the way a customer would, on a staging environment with realistic data, including the awkward paths. For user-facing work, that includes different devices and screen sizes and basic accessibility checks.

6. Release carefully and watch it

Changes go out behind feature flags or to a small share of users where the risk justifies it. Error tracking and monitoring are in place before release, not after, and rolling back is a routine step, not an emergency.

If the product itself uses AI

Testing code written by AI is one problem. Testing a product feature that calls an AI model is another: the same input can produce different outputs, and quality can change when a model version changes. For those features we add an eval set, a scored collection of real examples run on every change. We've written about how we build evals before launch.

What this means if you're hiring a team

Almost every software team now uses AI tools, so "do you use AI?" isn't a useful question. Ask these instead:

1. Who reviews AI-written code before it merges, and how senior are they?
2. What automated checks run on every change, and can a change merge if they fail?
3. How do you stop tests being weakened to make them pass?
4. Can I see a real pull request with its review comments?

A team that tests and reviews properly can answer all four in a few minutes. For more on choosing a partner, see agency, firm or freelancer: how to choose.

FAQ

Is AI-generated code safe to use in production?

It can be, with the same or stronger controls as human-written code: clear specs, tests, automated checks and review by an engineer who didn't write it. The risk comes from shipping it because it looks right.

Should AI write our tests?

AI is useful for drafting tests quickly, but a person should decide what the tests need to check and review them. Otherwise the code and its tests can share the same mistake.

Does AI-assisted development make projects cheaper?

It makes the writing faster. Review, testing and design decisions still take engineering time. The saving is real, but it depends on keeping those checks in place.

What is the best testing strategy for AI-written code?

Clear acceptance criteria first, behaviour-focused tests, automated checks that block merges, senior human review focused on judgement and security, testing the running product, and careful release with monitoring.

---

Adopting AI coding tools and want a second opinion on your testing and review setup, or need a team that already works this way? Book a scoping call. You can also read about how we work.