Preview Environment Seed Data When an AI Agent Is Opening Your PRs

Seed data sounds boring until you realize it’s load-bearing. We’ve covered the mechanics in seeding Postgres for ephemeral previews, handling PII in seed data, and generating realistic synthetic fixtures.

This post is about why the bar moves when AI agents are opening the pull requests. It moves a lot.

The intuition gap

A human developer has intuition the data model doesn’t capture. They know the users table has historical rows where tenant_id is null because of a 2023 migration. They remember the subscriptions table has records where cancelled_at is set but status still says active, because of a bug that got patched in a way that left data inconsistent. They’ve seen the production logs.

When they write a feature, they implicitly account for these realities. They know which seed records to add to test their change, because they know what the change touches.

An AI agent has none of that. It has the schema. It has the code. It has whatever you put in CLAUDE.md. It does not have the operational memory of the system. It doesn’t know which code paths are hot, which tables are messy, which fields are vestigial, or which combinations of state are actually possible in production versus theoretically possible according to the types.

So agent-authored PRs regularly produce code that’s correct against the schema and broken against reality. Whether the preview catches it depends on whether the seed data is rich enough to expose the mismatch.

Why “enough to make the app load” isn’t enough anymore

Most preview seed data was designed under a quiet assumption: the reviewer has enough context to know whether the change works. The seed data only needs to make the app boot. The reviewer’s brain fills in the gaps.

For agent-authored PRs, that brain isn’t there. The human reviewer is also low-context — they didn’t write the code, they’re reviewing more PRs, and they’re relying on the running preview to surface what the diff doesn’t. The seed data has to do that work. Which means it has to be wider, weirder, and more behaviorally diverse than what most teams currently ship.

What “behaviorally diverse” actually means

Concretely, seed data for AI-authored PRs should cover:

Every major flag combination. If users has is_admin, is_billable, email_verified, and mfa_enabled, seed the realistic cross-product — not just one user with all true and one with all false. Sixteen users isn’t a lot. The combinatorial growth surfaces conditional logic the agent might have introduced without realizing it.

Empty, partial, and full states. A list with zero items renders differently than one with three, or one with five hundred. Agents test against whatever’s in front of them, usually “a few items.” Seeding all three states surfaces pagination bugs, empty-state bugs, and performance regressions on first click rather than first customer complaint.

Edge cases that match production messiness. Records with null fields the schema technically allows. Strings with Unicode the agent’s regex didn’t handle. Timestamps in the past, far past, and future. Soft-deleted rows. Records mid-state-transition. The kinds of things that exist in production because of historical bugs and hand-written migrations.

Adversarial fixtures. A user named Robert'); DROP TABLE (yes, still). An order with a negative quantity. A file upload that claims to be a PNG and isn’t. Agents don’t think about adversarial inputs unless the prompt told them to. The seed data should.

Multi-tenant boundaries. If your app is multi-tenant, seed at least three tenants with overlapping data, so cross-tenant leaks show up immediately when the agent forgets a WHERE tenant_id = ? clause. Two tenants is enough to look like a feature. Three is enough to look like a bug.

The trap of letting the agent generate the fixtures

There’s a tempting workflow: have the agent generate seed data alongside the code it writes. The agent already knows what it’s building, so let it produce the fixtures that exercise its own change.

Don’t.

The fixtures the agent produces will match its mental model of the system. That mental model is the same one that produced the code. If the code has a misunderstanding, the fixtures will share it. The agent that thought subscriptions.status was always either active or cancelled will write fixtures with only those values, and the test that runs against those fixtures will pass, and the bug — that production has a third value, pending, the agent didn’t account for — will ship.

This is the same failure mode as letting the agent write the tests for the code it wrote: both artifacts reflect the same potentially-wrong understanding. Coverage looks great. Verification is meaningless.

What works instead:

• Seed data lives in the repo, owned by humans, reviewed independently of feature changes. Treat it like infrastructure. Updating the seed set is its own PR with its own review, separate from feature PRs that consume it.
• The agent can suggest additions, but a human approves them. “Add fixtures to cover the new pagination feature” is a reasonable agent task. “Generate the entire fixture set” is not.
• The seed set is versioned and shared across previews. Every PR’s preview database loads the same canonical set, plus any feature-specific additions. That way one bad fixture set doesn’t quietly break every preview at once, and one good improvement to fixtures helps every preview at once.

Make the preview boring on purpose

The goal is to make the preview environment behave like a small, plausible production. Not minimal — plausible. A reviewer should see something close enough to real customer data that surprising behavior surfaces immediately, without them having to know what to look for.

A good seed set is boring to look at and rich to interact with. Users who don’t have permission to do the thing the agent’s PR is about. Records in states that exercise edge cases. Enough volume that performance bugs surface, enough sparsity that empty states do too.

If your current seed data was designed when humans wrote all the PRs, it’s almost certainly thinner than it should be now. Upgrading it is one of the highest-leverage things you can do for AI-assisted development — and one of the most overlooked, because seed data sits in db/seeds.rb and nobody talks about it at standup. Talk about it.

A short, honest plug

You can build all of this yourself. Versioned seed sets, behaviorally diverse fixtures, per-PR previews loading the right data — it’s a few weeks of work to get right and a few hours a quarter to maintain. Plenty of teams do it.

If you’d rather not, PreviewProof handles the seeding and the preview together: per-PR environments with seed data that actually exercises the surface, fixtures that survive schema migrations, and a clear separation between seed data the team owns and per-PR additions the agent suggests. Preview it. Prove it.