When moving faster becomes the risk

A few years ago, we were working with a client, a regional trucking operation. By that point we knew their business almost as well as they did, and they trusted us completely. That closeness is the goal. It's how we try to work with every client. But there's a failure mode hiding inside it: when the yeses come easily, the one "no" that matters gets harder to hold.

Before we got involved, their operation ran on spreadsheets, a whiteboard, and a stack of legal pads. When a customer called asking where a load was, someone would call three people and get three answers. We built them one system. Loads in one place, payment history in one place, and for the first time, what every route actually cost.

That last one paid for the project by itself. One of their routes had been quietly losing money for months. Fuel, driver time, empty return miles. Nobody could see it, because the numbers lived in four places. They renegotiated, the customer refused, and they dropped the route. The team stopped double-checking the software and started trusting it.

The technology worked. And because it worked, they wanted more.

That's usually a very good problem to have. But success can also blur an important line. The client knew their business cold: their customers, their drivers, their margins. What they didn't have was the technical seat. And the better the system got, the easier it became for everyone, including us, to forget that the seat still mattered.

"We need it live"

Then came the kind of contract that changes a company. A large distributor, routes across three states, delivery guarantees with real penalties attached.

The next major release was timed around it. It would automate a significant part of dispatch before the new volume arrived, and the deadline wasn't arbitrary. It was tied to a real business opportunity with a real start date.

As the release approached, one issue remained. Most of the system worked exactly as expected. But under specific conditions, load reconciliation could produce inconsistent results. It didn't happen every time. It didn't even happen often. But it was happening.

Our recommendation was simple: not yet. We need more time before this goes into production.

The client disagreed. They'd watched us solve hard problems quickly for two years. From their side of the table, this looked like one more issue we'd knock out along the way. They pushed for the release. We pushed back. They pushed harder.

And here's the part we own: we deployed.

It would be easy to write "the decision was made to deploy" and let the passive voice do the work. But we were the technical seat in the room, and the technical seat folded. We let a business deadline outrank a technical readiness decision. Saying "we told them it wasn't ready" is true, and it's worthless.

Rare problems aren't rare at scale

In testing, the failure looked like a footnote. Once in hundreds of transactions.

Production doesn't run at testing volume. Within days of the new contract starting, the system was handling more activity than it had ever seen, and "once in hundreds" turned into "several times before lunch."

Loads duplicated. Loads disappeared. Dispatch spent the morning rebuilding routes by hand, and someone dragged the old whiteboard back out of storage, because the software built to replace it couldn't be trusted.

We rolled the release back. The rollback worked, but it cost hours, and freight doesn't wait for a rollback. The client booked outside carriers at whatever the market charged that morning. Most deliveries made it. A few didn't.

The client called their customer and said it plainly: a release went out before it was ready, it's being rolled back, here's when you'll have real numbers. No spin. The big customer stayed, and the relationship survived. A smaller customer caught in the same failure quietly left and never came back.

The technical problem was fixed within days. The outside-carrier invoices hurt for a quarter. The team's trust in the system took longer than either.

The failure happened before the release

Here's the thing: the bug wasn't the failure.

Every sufficiently complex software system will eventually ship a bug. If a single defect can end the relationship, the relationship was already fragile. The real failure happened earlier, in a meeting where everyone half-knew the answer and let the deadline give it anyway.

I think about that meeting more than I think about the outage.

So afterward, we didn't just fix the code. We changed our rules.

Technical readiness is now a separate question from business urgency, and the two are never allowed to answer each other. A deadline is real, and it deserves respect. It can reorder priorities. It can justify more resources. It can cut scope, or push us toward a simpler version that can safely ship sooner.

What a deadline cannot do is make unfinished software finished.

Major releases at Centervert now require three things, agreed in writing before the pressure arrives:

1. Go/no-go criteria. What has to be true, measurably, before this reaches production.
2. A rollback plan. Not a theory. A tested path back to the last version that worked.
3. Someone with the authority to say "not yet." And a shared agreement, made on a calm day, that nobody overrules them on a frantic one. Not the client. Not us.

What to ask before your next release

You don't need to be a software company to use this. Before your next major launch, ask whoever is building for you four questions:

• What are the written criteria for this going live?
• If it fails in production, what's the rollback plan, and has it been tested?
• Who has the authority to call "not yet," and will everyone honor it?
• If the answer is "not yet," what happens to the deadline: scope, resources, or date?

If your development partner can't answer those, you don't have a release plan. You have a launch date.

You shouldn't hire a development company because they'll say yes to everything. You should hire one because you trust them to help you make the right call, even when the right call is the one nobody in the room wants to hear. We accepted that responsibility the hard way, and we've kept it since.

Before you start building

The most expensive development problems begin before the first line of code is written. The right time to talk through architecture, risk, rollout, testing, and a realistic timeline isn't halfway through the project. It's before it starts.

Talk with the Centervert team before you kick off your next development project. We'll help you turn the idea into a clear technical plan, find the difficult parts early, and map a path from concept to production that everyone can stand behind.