QA stopped being a branch: the test environment as a text file
I joined a project where the question "what's in QA right now?" had no answer. Not a partial answer: no answer. You had to read the history of a branch with hundreds of merges and guess.
What follows is how we got out of that. Spoiler on the counterintuitive part: the fix wasn't more discipline, or a process doc, or a better maintained board. It was taking away QA's ability to be a branch.
The symptom: main had stopped being production
The context, in the parts that matter for the problem: a web application with a team of several people, frequent releases, and a client who signs off on changes before they ship. Three environments: production, a QA the client looks at, and everyone's own branches.
The setup was the usual one, the one we've all used: a
long-lived qa branch. You merge your
feature there, the client reviews it, and once they approve you
merge it into main. Simple, and perfectly
reasonable.
Except that after a year, the real inventory looked like this:
- Approved features that had been sitting in QA for six months without reaching production, because the client signed off and nobody closed the loop.
- Rejected features still in QA, because pulling one out meant reverting a merge buried under fifty others.
- Conflicts between two features resolved inside QA. That resolution existed nowhere else: not reviewed, not in any PR, and not going to reach production with the features.
-
And the one that made it urgent:
mainwas no longer production. Things that were live weren't inmain, and things inmainweren't deployed. We had to revertmainto the stable state from two weeks earlier just to get a trustworthy reference back.
Once main stops equalling production, you lose
the ability to answer the only question that matters during an
incident: what is running right now?
Everything else — rollbacks, hotfixes, auditing — is built on
top of that answer. Without it, every deploy is a bet.
Why it happened: a long-lived branch is a log, not a state
It took me a while to see this wasn't a problem of careless people. It was structural, and it comes down to one sentence:
A long-lived branch records decisions; it doesn't represent a state. Every merge into QA is a line appended to a write-only log. The branch doesn't say "these five features are under test": it says "this is what happened, in this order, over fourteen months". To know the current state you have to mentally replay the whole log.
Everything else falls out of that. Pulling a feature out of QA is expensive — you have to revert a buried commit — so nobody does it, so QA only grows. And once it only grows, drifting from production stops being an accident: it's the design's default behavior.
Seen that way, asking the team for discipline was asking them to manually compensate for a system that pushed the other way. That never wins.
The inversion: QA isn't merged into, it's rebuilt
The underlying change is a single sentence:
The QA branch is disposable. You never push
to it. On every change it is
rebuilt from scratch as main
plus the merge of a list of branches. Anything committed there
is lost, by design.
The entire algorithm fits in five lines:
# rebuild QA (in a temporary worktree, not in your checkout)
git switch --detach origin/main
for each branch in the list:
git merge --no-ff origin/$branch
git push --force origin HEAD:qa
It looks violent and it's the exact opposite: it's what makes QA cheap. Look at what just happened to the problems above:
- Pulling a feature out of QA is no longer reverting a merge. It's deleting a line from a list. The next rebuild simply doesn't include it.
-
QA cannot drift from production. It starts
from
origin/mainevery single time. Drift is mathematically impossible, not "discouraged". -
QA's inventory is the list. The question that
had no answer is now a
cat. -
maingoes back to being production, because nothing is merged "toward" QA anymore: QA is derived frommain, never the other way around.
The list lives in git, and that's not a detail
My first draft stored the list in repository variables, editable
from the UI. I dropped it for something that turned out to be
the best decision in the design:
a list you edit from a UI has no history. You
don't know who added that feature, when, or why. And if the QA
environment breaks, you can't git log the cause.
So the list is a plain text file on an
orphan branch — no shared history with
main, not a single line of product code. One file
per environment, named after the environment:
# Client QA. One branch per line.
integration/checkout+catalog
feature/csv-reports
feature/flaky@a1b2c3d # pinned to a commit
The format explains itself, which was the point. One branch per
line, # for comments, blank lines ignored. Two
decisions worth justifying:
A file with no entries means free environment,
and resets the branch to main. Freeing an
environment is deleting lines, not a special command. The
"empty" state is representable, which is exactly what the old
setup was missing.
The @sha pin freezes a feature at a
commit. It came out of a very concrete case: a flaky
branch we wanted in QA but whose pushes kept breaking the
environment for everyone else. With the pin, a push to that
branch does not trigger a rebuild.
And because the list is git, the trigger is the push. No button, no cron, no sync that can fall behind. Editing the file without pushing does nothing, and that's good: the only moment the environment changes is an auditable event with an author and a date.
Features that clash don't get fixed in QA
There's one case the old setup handled badly and that deserves its own rule: two features that conflict with each other.
It used to be resolved inside QA, and we've seen the result: the
resolution lives on a disposable branch, unreviewed, and never
reaches production. Under the new setup it isn't even possible —
the rebuild would fail at the merge and stop.
The rule is: the conflict is resolved on an
integration/a+b branch, and that
branch is listed in QA instead of the two that clash.
# wrong: the rebuild fails
feature/checkout
feature/catalog
# right: the conflict arrives resolved, and reviewable
integration/checkout+catalog
The deeper benefit is that the conflict resolution becomes a first-class artifact: it has a branch, an author, a diff, it can be reviewed in a PR, and it reaches production together with the features. It stops being an anonymous merge commit inside a branch that's about to be deleted.
Four things that weren't obvious
The design is simple. Implementing it came with four surprises that cost me real time, and they're the part of this article I'd most have liked to read beforehand.
1. A push runs the workflows of the branch that was pushed
The first push to the list branch triggered zero runs. It didn't fail: nothing happened.
The reason, obvious in hindsight: on a push event,
GitHub Actions runs the workflows that exist on the branch
that was pushed, not the ones on main. My
workflow lived on main, and the orphan branch
didn't even have a .github/ directory. A list file
can't trigger itself.
The fix is to put a minimal workflow on that branch,
one that duplicates no logic: it checks out main
and runs the same script. The same rule explains a second
stumble — a relative reference to
./.github/workflows/x.yml inside a workflow called
from another branch resolves against that other branch,
so reusable workflows have to be referenced by their absolute
path and ref: org/repo/.github/workflows/x.yml@main.
2. A push with GITHUB_TOKEN doesn't trigger other workflows
This is GitHub policy and it exists to prevent infinite loops:
anything pushed by a workflow authenticated with the default
GITHUB_TOKEN
generates no events for other workflows.
Practical consequence: the rebuild force-pushes the QA branch, but that will not trigger the deploy workflow no matter how well written it is. You have to call it explicitly from the same run. If you're coming from an event-chained pipeline, this is the one that'll bite you.
From the same set of administrative surprises: if the
repository's default permissions are read, any
workflow that pushes or cuts a release has to declare
permissions: contents: write. It fails at the end,
not at the start.
3. Compare trees, not SHAs
My favorite, because it's a conceptual bug and not a code one.
After rebuilding, you want to deploy only if something changed. The reflex is to compare the new SHA against the previous one. It doesn't work: a rebuild always produces new SHAs. Merge commits are new objects, with new timestamps, even when the resulting content is identical byte for byte.
Comparing SHAs, every rebuild reports "changed" and deploys.
Every push to main pays for a five-minute remote
build to publish exactly what was already there.
What you want to compare is the tree: the object that represents the content, not the history that led to it.
# always different: includes history and timestamps
git rev-parse HEAD
# identical if the content is identical, wherever it came from
git rev-parse HEAD^{tree}
Two lines. It kills most of the useless deploys. And it's a healthy reminder that in git, content and history are different things, even though we spend all day treating them as one.
4. The lockfile breaks late, in the remote build
Merging several branches that touched dependencies can leave a
lockfile inconsistent with package.json. Locally
you never notice: the package manager quietly fixes it on
install.
But remote builds run with CI=1, and in CI mode
installs are immutable: if the lockfile doesn't
add up, the build fails. Not at the merge, where it would make
sense — at the end, five minutes later, in the build,
with an error that never mentions the word "merge".
It's worth validating lockfile consistency right after the merges and bailing out there, with a message that says what happened. Moving a failure from "five minutes and a weird error" to "ten seconds and a clear message" is one of the highest return things you can do to a pipeline.
Before and after
| Operation | Long-lived branch | Disposable composition |
|---|---|---|
| Knowing what's in QA | Read the history and guess | Read a file |
| Adding a feature | A merge | One line |
| Removing a feature | Revert a buried merge | Delete one line |
| Drift against production | Inevitable | Impossible |
| Conflict between two features | Resolved, then lost | Own branch, reviewable |
| Audit trail for environment changes | None | git log |
Is main production? |
No | Yes |
On the production side the setup becomes almost boring, which is
the goal: main is exactly what's deployed, history
is linear (squash only), and production is reached solely by
cutting a tag. A rollback is two steps — roll back the deploy
and a git revert with its patch release —
because if you only do the first one, main starts
lying again and the next deploy resurrects the bug.
What I'm taking away
One: environments are states, not histories. This is the real lesson. A long-lived branch models an environment as the sum of everything that happened; a manifest models it as what it is now. The second representation answers "what's in there?" in constant time and lets you remove things without archaeology. If that sounds like declarative infrastructure, it's because it's exactly that, applied to branches instead of servers.
Two: when a process depends on discipline, the design is the problem. I spent a while thinking about how to get the team to remember to pull features out of QA. The right question was why pulling them out cost so much. Doing the right thing has to be the easiest thing, not the most virtuous one; if it only works while everyone is paying attention, it will fail.
Three: disposable is more reliable than well-tended. Counterintuitive and true. A QA branch that gets destroyed and rebuilt on every change cannot accumulate drift, orphaned state or unreviewed fixes. The same reasons we prefer immutable containers over pet servers apply verbatim to a git branch.
Four: every surprise was in the platform, not in the
design. I wrote the scheme in an afternoon. The days
went into discovering that a push runs the workflows of the
branch that was pushed, that a GITHUB_TOKEN doesn't
chain events, and that comparing SHAs is an elegant way to
deploy the same thing forty times. None of the three shows up on
an architecture diagram — and all three would have sunk the
implementation silently.
If you have a QA branch nobody dares to look at, you don't need a better process to take care of it. You probably need to stop taking care of it and learn to throw it away every day.