The Deployment Is Green — But Can the Database Support Two App Versions at Once?

Friday, 2:08 PM. CI is green. The database migration completed successfully. The new application version starts. Then the 500 error rate begins to climb.

The migration may not be broken at all. It may be technically correct but incompatible with the state of the system during deployment.

During a rolling deployment, old and new instances coexist for a period of time. Kubernetes documents exactly this behavior: a new ReplicaSet is gradually scaled up while the old ReplicaSet is scaled down. Blue-green deployments, multiple servers behind a load balancer, and slow process restarts create the same fundamental condition: for some period, two application versions may use the same database.

If a migration renames, removes, or changes the meaning of a column during that window, the new version may work while the old version is still serving requests — or the reverse.

The useful release question is therefore not only:

"Did the migration run successfully?"

It is:

"Can the schema support both the old and new application versions throughout the rollout?"

A database migration is part of the release, not its preface

Many deployment pipelines treat migrations as a preparatory formality:

1. Run migration.
2. Deploy application.
3. Health check turns green.
4. Done.

That model works well for many small additive changes. It becomes fragile when a release removes existing columns, changes data types, scans large tables, validates new constraints, or builds indexes on busy production tables.

PostgreSQL explicitly documents that the lock level depends on the specific ALTER TABLE subcommand and that an ACCESS EXCLUSIVE lock is acquired unless a weaker level is explicitly documented. An ACCESS EXCLUSIVE lock conflicts with every table-level lock mode; PostgreSQL also notes that it is the lock level that can block ordinary SELECT queries.

That does not mean ALTER TABLE is inherently unsafe in production. It means "DDL" is not a useful risk classification by itself. The exact operation, table size, active transactions, and lock duration matter.

The two-version problem

Imagine a users table with a full_name column.

The next application version wants to use display_name. A direct migration could simply rename the column. The SQL is clean. Operationally, the release can still fail:

• old instances continue querying full_name,
• the migration renames it to display_name,
• new instances work,
• old instances fail until they have completely left the traffic pool.

A faster rollout makes the window smaller. It does not make the incompatibility disappear.

Rollback makes the problem more obvious. Kubernetes can roll a Deployment back to an earlier Pod-template revision. That operation does not automatically undo a database migration. If the new release already removed a column required by the old application, an application rollback can immediately produce a second outage.

An application rollback is only a real rollback when the old application can still operate against the current schema.

The robust pattern: expand, migrate, contract

For schema changes on actively used systems, a multi-stage release is often much easier to operate.

Phase 1: Expand

Extend the schema without removing the interface used by existing clients.

For full_name → display_name, that can mean:

• add the new column,
• keep the old column for now,
• make the new application tolerant of both states.

An additive change is not automatically risk-free; adding a column can still have lock and execution implications. The important data-model principle is that the old contract has not been destroyed yet.

Phase 2: Deploy compatible application code

For a transition period, the new application can for example:

• write both fields,
• prefer display_name for reads,
• fall back to full_name for records not yet migrated.

The right approach depends on consistency requirements and data volume. What matters is that the transition is explicitly designed, rather than depending on every process switching at exactly the same moment.

Phase 3: Migrate existing data deliberately

Backfill existing records into the new representation in controlled batches.

A backfill should not casually mean "update ten million rows in one transaction." Review:

• batch size,
• transaction duration,
• I/O and replication load,
• lock wait time,
• retry strategy,
• progress measurement,
• behavior while new writes continue.

A backfill is an operational job. It deserves monitoring like a deployment does.

Phase 4: Switch reads and observe

Once the data is migrated, the application reads only from the new field. The old field remains temporarily.

Now you can measure:

• Are there still writes to the old field?
• Does any worker, scheduled job, or older service still use it?
• Do reports, exports, BI jobs, or integrations depend on the old schema?
• Are error rates and data consistency stable?

Phase 5: Contract

Only when the old interface is demonstrably unused should it be removed.

Dropping the legacy column belongs in a later release where possible. That turns one hard-to-reverse change into several smaller, observable decisions.

Locking: the failure staging often cannot show

A migration can finish in 80 milliseconds on staging and wait for minutes in production.

Staging may have 50,000 rows, little traffic, and no long-running transactions. Production may contain 40 million rows while queue workers, reports, imports, and several application services are active.

PostgreSQL exposes outstanding locks through pg_locks. For a release review, however, it is not enough to discover the lock after the incident. The pipeline should understand which lock level the migration will request before it runs.

Practical questions include:

• Which tables are affected?
• How large are they in production?
• Which lock level does each statement require?
• Can the statement wait behind an existing transaction?
• What happens to incoming requests while it waits?
• Is a lock_timeout or deliberate abort policy configured?
• Can an interrupted migration be retried safely?

A migration that waits indefinitely for the perfect moment can create a queue of blocked work behind it.

Indexes: CREATE INDEX is not always a harmless side operation

New features frequently require new indexes. On a large production table, a regular index build can block writes.

PostgreSQL provides CREATE INDEX CONCURRENTLY for this reason. The current documentation says this form allows normal inserts, updates, and deletes to continue during the build, but it requires more work, multiple table scans, and waits for certain existing transactions. If a concurrent build fails, it can also leave an invalid index behind that needs cleanup or another build attempt.

That leads to an important operational rule:

"Concurrent" does not mean "free" or "failure-proof." It means a different locking and execution strategy.

An index-migration review should therefore ask:

• Does the ORM or migration generator emit the SQL you think it emits?
• Does the migration transaction model allow CONCURRENTLY?
• How is a failed build detected?
• Would an invalid index become visible to monitoring?
• How are CPU, I/O, and replication load observed during the build?

Constraints: validation does not always have to happen at introduction time

Adding a foreign key or check constraint to a large existing table may require validating a large historical data set.

For supported constraint types, PostgreSQL offers NOT VALID. This skips the potentially lengthy initial scan of existing rows while still enforcing the constraint for subsequent inserts or updates. A later VALIDATE CONSTRAINT scans the existing data separately and, according to PostgreSQL's current documentation, uses a SHARE UPDATE EXCLUSIVE lock.

This is not a universal recipe for every migration. It demonstrates the broader principle well:

Schema introduction, historical-data validation, and final enforcement do not always need to be one large release step.

Plan rollback before you need it

"We can always redeploy the previous version" is only half a rollback plan.

Before a schema-changing release, document:

• Can the old application read from the new schema?
• Can it write to it?
• Are new fields optional, or does the new code create data the old code cannot understand?
• Was a column, table, or enum variant removed?
• Was data irreversibly transformed?
• Could a reverse migration destroy new data?
• Would an incident require a forward fix rather than a rollback?

Automatic "down migrations" are particularly risky when they recreate an old schema shape by deleting data that the new release has already written.

A dependable rollback plan therefore separates:

Application rollback: reactivate an older binary or container version.

Schema rollback: revert database structure.

Data rollback: reverse already transformed or newly written data.

These are different operations and should not be treated as synonyms.

A pragmatic release check

Before deployment:

1. Review the actual generated SQL, not only the ORM migration name.
2. For each operation, assess lock level, possible table scan, and expected runtime.
3. Test against production-like data volume.
4. Confirm that old and new application versions both work with the expanded schema.
5. Separate large or long-running backfills from schema DDL.
6. Document rollback compatibility.
7. Prepare smoke or end-to-end checks for critical user journeys.

During deployment:

• observe lock waits and long-running transactions,
• watch database connection pressure and query latency,
• break down error rates by application version where possible,
• expose backfill progress and failures,
• retain the ability to stop rollout before the contract phase.

After deployment:

• check whether old processes still access the legacy schema,
• verify representative data for consistency,
• remove old columns, tables, and compatibility code only after an observation period,
• treat cleanup as a separate release.

The most dangerous migration is often not the most complex one

A complicated data transformation usually receives attention. A small change such as "rename column", "set NOT NULL", or "add index" can slip through review as a one-liner.

Those changes become painful because their risk is not visible in the SQL alone. It exists in the simultaneous state of the system:

• old and new application versions,
• real data volume,
• concurrent transactions,
• replication,
• background jobs,
• external integrations,
• rollback paths.

Database migrations are therefore not an isolated database concern. They are release engineering.

What Website-Pflichtencheck would inspect

A technical release and maintainability review can treat the migration path as part of the site's operating model rather than as an invisible CI step:

• order of migration, application rollout, and backfills,
• actual generated SQL migrations,
• potential lock and table-scan risks,
• compatibility between old and new application versions,
• rolling, blue-green, or multi-instance behavior,
• index and constraint strategies,
• safe repeatability of interrupted steps,
• rollback and forward-fix plans,
• monitoring for locks, errors, and backfills,
• cleanup of legacy schema elements,
• functional checks after release.

The goal is not to turn every small website change into a three-week process. Small sites can keep small deployments.

But once a website or SaaS product is handling active users, orders, enquiries, bookings, cases, or other live data, every schema change should be able to answer one question reliably:

What happens if the deployment does not occur in one atomic second?

If the answer is "then nothing can go wrong," part of the release design is still missing.

Sources

• PostgreSQL 18: ALTER TABLE — https://www.postgresql.org/docs/current/sql-altertable.html
• PostgreSQL 18: Explicit Locking — https://www.postgresql.org/docs/current/explicit-locking.html
• PostgreSQL 18: CREATE INDEX — https://www.postgresql.org/docs/current/sql-createindex.html
• Kubernetes: Deployments — https://kubernetes.io/docs/concepts/workloads/controllers/deployment/