Skip to content
One invisible character, two reasonable assumptions, one lost write

One invisible character, two reasonable assumptions, one lost write

A broken PDF font map produced a NUL byte. Every layer accepted it until the database rejected the write, while HTTP 200 made the failure look successful.

An invisible character moving through a healthy data pipeline before a database rejects it.

The production failure started with a character nobody could see.

A document entered a knowledge-base pipeline. It looked normal, ingestion completed, and search returned relevant passages.

Later, a record failed to save.

The PDF contained a broken font map. Some ligature glyphs that should have become letters were extracted as U+0000, the NUL character. That value moved through extraction, chunking, embeddings, storage, retrieval, tool output, and JSON encoding.

Every stage accepted it.

The database did not.

That alone would have been an ordinary input-validation bug. A second bug made it worse: the application error arrived with HTTP 200. One part of the system checked the transport status, decided the save had worked, and reported success.

Two reasonable assumptions had lined up in exactly the wrong way.

The value stayed valid until it did not#

A NUL character passes through extraction, indexing, retrieval and JSON before the database rejects the write PDF extract TEXT chunk VECTOR store SEARCH retrieve TOOL return JSON decode DATABASE reject the value stays valid until the final boundary transaction rolled back
every arrow preserved the bad value correctly

PDFs often store glyphs plus a map that tells extractors which character each glyph represents. A damaged map can return NUL where letters should be.

The value can then cross a JSON boundary as the visible escape sequence \u0000. After another decode, it becomes a real NUL again. The JSON remains valid, but the database refuses the resulting value and rolls back the transaction.

Syntactically valid is not the same as safe for the next system.

HTTP 200 answered the wrong question#

The HTTP request had succeeded. The server received it and returned a valid response. The application operation had failed.

Those are different outcomes.

Transport success and application confirmation lead to different acknowledgement decisions HTTP 200 received OUTCOME inspect result CONFIRMED finish FAILED retain and surface confirmed outcome error or unknown acknowledge the outcome, not the envelope
HTTP 200 only proves that the response arrived

A save should count as complete only when the application confirms it. An error, an empty result, or an unfamiliar response is not confirmation.

That same decision should govern cleanup. Keep the original work until the system of record confirms the intended state. In this incident, that rule left a recoverable copy after the failed write.

Retries need somewhere to end#

Retries help with failures that change over time, such as a short outage or a busy database. They cannot repair a deterministic bad character. The same input fails in the same place each time.

If a job retries five times and then disappears, the system has only delayed the loss.

A reliable retry path needs a durable terminal state, an alert, and an idempotent way to replay the work. It also needs to distinguish permanent input errors from transient infrastructure errors.

Fix the class of failure#

Removing NUL from one bad value would fix one write. It would not fix the system.

The useful repair works at several boundaries:

  • Detect the damaged PDF text layer and use OCR to recover the intended letters.
  • Strip NUL after chunking so no document format can place it in the index.
  • Sanitize again before persistence as a final safety boundary.
  • Treat only an explicit application confirmation as success.
  • Make exhausted work visible and recoverable.

Each safeguard has one job. OCR protects meaning. Sanitization protects storage. Confirmation protects acknowledgement. Durable failure handling protects the work itself.

The deeper lesson is not that one database dislikes one character. It is that every boundary has its own definition of valid, and every layer has its own definition of success.

Production systems rarely fail because one component explodes. The difficult failures are quieter. Every component follows its local contract, while the contracts between them leave a small gap.

In this case, the gap was one invisible character wide.

1–4 focus a window, j k scroll, g top