The Ghost Schema That Slips Past Your Type Guard

The Ghost Schema That Slips Past Your Type Guard

A team at a fintech startup was rolling out a new feature flag system built on top of their existing API gateway. The gateway exposed configuration endpoints that clients could POST settings to — including the server port, timeouts, and feature toggles. The service running behind the gateway was written in TypeScript and depended heavily on Zod for contract validation. Everything looked sound. The build succeeded. The deploy succeeded. The next morning, every instance returned 503 Service Unavailable.

The incident report pointed to a single line in the logs: Error: Invalid schema for 'config'. Expected port (number) but got undefined. Yet the most puzzling part was that several minutes earlier, the same service had been processing thousands of config updates without complaint. The pattern suggested a conditional failure — a few instances were unhealthy while others continued serving normal traffic.

The Investigation

First, I traced the stack trace back to the gateway entry point. The handler received a PATCH request with a partially populated payload:

{
  "timeout": 30,
  "features": ["billing"]
}

There was no port field. The Zod schema being applied expected port: number as a required field:

import { z } from "zod";

const configSchema = z.object({
  timeout: z.number(),
  features: z.array(z.string()),
  // Following lines deliberately omitted for clarity — see full schema below
  port: z.number(),
}).strict();

async function applyConfig(configResponse: ConfigResponse): Promise<void> {
  const parsed = await configSchema.parse(configResponse);
  setConfig(parsed);
}

The static analyzer had green across the board. Zod's strict mode ensured the schema itself was well-formed. The issue lay elsewhere: the gateway had two paths for reading configuration. The critical-path route went through a middleware that first tried to parse the incoming request against configSchema, then wrote the result to Redis. Earlier versions of the code had done this. After a refactoring, a new path was added that read raw config files at startup and loaded them directly — skipping Zod entirely.

The real culprit turned out to be a timing condition. Some nodes received the config update during normal operation, triggering the middleware path and working correctly. Other nodes started up fresh and took the direct-file path. Those nodes didn't hit any validation because the file loader bypassed Zod's guard entirely. The