Proofload: The Trace Spec Said It Was Not a Retrofit
Part 3. Traces: a span schema, converters from OpenTelemetry and LiteLLM, and capture wired into both backends. Most of what this day produced was findings about other people’s packages. See Part 2: Nine Modules and One Competing-Risks Bug.
Sep 4: A grep that answered the wrong question
The trace spec opened with “nothing here is a retrofit of existing code”, on the strength of a grep of the source tree for a trace module. The grep was right about the module and wrong about the consumers. The CLI already carried its own copy of the schema id and read the old shape; the mock agent emitted it; and two frozen tests pinned it, including the exact key name.
Built as briefed, the session would have landed the new shape and watched two frozen tests go red with no authority to fix them. That is the same failure as the previous session’s unmappable invariant, one session later, in a spec whose own closing section quotes that lesson. Amended before the packet ran, with the four sites named by line number.
The standing rule that came out of it: before freezing an oracle, build a throwaway implementation of the spec in a scratch directory, never under the source tree, and run the oracle against it. It does not prove the implementation is right. It proves the tests are satisfiable, which is a different and much cheaper thing to be wrong about.
Sep 4: A per-episode resource attribute is not possible
The plan was to attribute every span to its episode with an OpenTelemetry resource attribute. A spike against the real SDK killed it three separate ways. Resource attributes are a bounded collection that raises on assignment. The provider’s resource is a read-only property. And setting a global provider a second time is refused with a warning, after which spans keep landing in the first provider, including from tracers obtained after the second call.
The environment-variable route works but is read once when the resource is created, so it is a per-process knob. One episode per process was the alternative, and it forecloses many episodes per container, which is exactly what a parallel map is for.
Decision: attribution moves to the span, not the process. The converter did not have to change at all.
Sep 4: Four things wrong with the ecosystem, none of them ours
Ran real autoinstrumentation against a mock transport, no network and no key, to find out what a real payload looks like. Four findings, all reproducible by anyone in an afternoon:
- The current recommended API surface for one major provider emits zero spans. A pilot built on it gets no LLM spans at all.
- One instrumentation package is two different projects published under one PyPI name, and a plain install resolves to the one that omits the attribute everything downstream keys on.
- Every span these packages emit is a client span carrying a server address. It reads like an HTTP span. Widening a fallback rule to match on that would silently reclassify every model call as HTTP, and that was a rule I was about to widen.
- The semantic conventions moved to their own repository, so a version pin records a core version for a vocabulary that now versions separately. Nothing breaks; the pin just no longer identifies the thing it names.
Sep 4: Columns, and a hazard nothing tested
Inspect prints five whitespace-delimited columns. The mock agent’s span names never contain spaces, so the suite was green. Real converted span names contain spaces, and so do real error messages. The first pushes the duration into the wrong column, the second splits across two.
Tab-delimited now, with the column count pinned rather than a prefix matched, so a missing column fails loudly. This is the highest-value thing the session found and no test could see it.
Sep 4: Two silent failures found by running code
Capture went in behind an opt-in table, off by default on privacy grounds. Two of its requirements exist because running the code found them, and both fail without raising anything.
If the parent process sets a global tracer provider before forking, the child’s attempt is refused and it captures nothing, for the whole run, with a single warning as the only signal. And the SDK’s default id generator draws from the global random module, so a rollout that seeds that module for reproducibility, which is exactly what this tool invites, emits identical span ids and the converter drops them as duplicates.
There was also a cap on spans per episode, and it dropped them in the order they ended, which is innermost first. So a tight cap kept the leaves and lost the root, which is the reverse of what anyone reading a trace wants. Changed to keep the earliest-started spans.
Sep 4: The trace you cannot have
The most useful trace in the product is structurally unreachable. A timed-out episode is terminated, so the drain never runs; the errored record hardcodes empty metadata; and the store’s retention filter strips traces from rows with no outcome. Inspect could not display such a row even if it existed.
Four places across three modules. Named as a non-goal with the cost attached rather than left as a to-do, and queued as its own session.