164 Real Cases: How We Validate Every Feature We Ship

Every API vendor tells you their features work. Almost none of them show you the evidence. We think that’s backwards, so we built the evidence machine first and made its output public: transcodely.com/conformance — currently 164 validation cases across 23 categories, 162 validated, 2 known gaps, covering 40 API capabilities, 34 of which are fully validated today.

This post is about what those numbers actually mean, because “our tests pass” can mean almost anything.

#The problem with green checkmarks

A transcoding API has a specific and nasty failure mode: it can succeed on paper and fail on pixels. The job completes, the status says completed, the invoice is generated — and the H.265 output is silently 8-bit where you asked for 10-bit, the HLS manifest is missing its audio group, or the DRM signaling isn’t actually in the stream. Unit tests don’t catch this class of bug. Mocked integration tests don’t either: the mock returns whatever the mock author believed the encoder does.

The only test that catches it is the one that runs the real pipeline and inspects the real bytes.

#What a validation case actually does

Each of the 164 cases drives the full production path — no shortcuts, no test doubles. For the 106 that run an encode through to completion, that means all five of these:

1. A real job is submitted through the public API, with real authentication, hitting the same validation, pricing, and scheduling code every customer request hits.
2. A real worker VM boots — the same machine images, the same autoscaler, the same cloud provider as production traffic.
3. The produced media is probed. We run ffprobe against every output and assert on what came out: codec, profile, pixel format, resolution, frame rate, color transfer characteristics for HDR, audio channel layout. Not “did a file appear” — is the file what was ordered.
4. Manifests are fetched and parsed. For HLS and DASH outputs we parse the actual playlists: variant count, bandwidth ladder, audio and subtitle renditions, encryption signaling. Proving a stream was produced, not merely priced.
5. The invoice is recomputed. Each case independently recalculates what the job should cost from the published pricing rules and compares it to what the billing system actually charged. Pricing bugs are product bugs.

Cases don’t only cover happy paths. Forty-one of them assert that invalid requests are rejected correctly — with the right error code and a message that tells you how to fix it — because a permissive API that accepts garbage is a bug factory downstream. Others assert cancellation semantics, failure billing (a failed job must cost €0), and confirmation flows.

#The gaps are public too

Here’s the part I care most about: when a case fails because the platform genuinely can’t do something yet, we don’t delete the case and we don’t fudge the assertion. It’s marked as a known gap and published on the same page as the passing cases. Right now there are exactly two. One is retro-captioning a hosted video whose original single-file source is gone (hosted videos keep renditions, not mezzanines, and the worker can’t transcribe an HLS manifest — bridging that needs a capability we haven’t shipped). The other is HDR10+: a forced HDR10+ encode currently goes through the same static-HDR10 path as plain HDR10, so the output carries no per-scene dynamic metadata, and the case that asserts that metadata stays red until it does.

The alternative — quietly weakening an assertion until it passes — is how test suites rot into marketing. A tracked gap keeps the pressure on: the case stays red in our tooling until the capability is real, and you can see it.

#The exhaustiveness gate

The suite has one more property that keeps it honest over time: it’s structurally complete by construction. A CI gate walks every capability enum in our API schema — every codec, container, DRM system, encryption scheme, HDR format and mode, subtitle operation and format, thumbnail mode, content-aware mode, watermark anchor — and fails the build if any value lacks a covering validation case. Add a feature to the API without teaching the conformance suite about it, and the pull request doesn’t merge. Coverage can’t silently decay because coverage is enforced at the type level.

That’s also why the suite grows in lockstep with the product. When AI captions shipped, the caption cases were part of the release gate: real jobs generating real transcriptions on the new worker build, with the caption rendition asserted inside the parsed HLS manifest — before the feature reached production, not after.

#Why we publish it

Selfishly: it has caught real bugs before customers did — wrong pixel formats, missing manifest entries, a pricing edge case. The suite pays for itself.

But the deeper reason is that infrastructure buying decisions run on trust, and trust built on adjectives is fragile. “Reliable, battle-tested, production-grade” — every landing page says it. A live page that says here are the 164 things we verify against real infrastructure, here are the two that don’t pass yet, and here’s when it last ran is a different kind of claim. It’s falsifiable. That’s what makes it worth something.

The suite runs against the same API you integrate with, and its results update as we ship. If you’re evaluating us — or evaluating anyone — ask for this artifact. Ours is here.