LLM systems can fail in ways traditional tooling cannot detect, including hallucinations without exceptions, slow retrieval steps without CPU spikes, and prompts that degrade silently over time. Observability is required to diagnose these issues in production. OpenTelemetry is a vendor-neutral specification that standardizes what observability data is collected, how it is named, and how it is shipped, allowing one instrumentation effort to send data to multiple back ends. Portability protects observability investment in instrumentation code rather than a specific platform. Compatibility claims can still fail due to semantic conventions, since OTel defines transport but not fully standardized LLM attribute naming. Competing conventions such as OTel GenAI, OpenInference, and vendor-specific naming can cause inconsistent span meaning across pipelines.
"You built the agent. It works in testing. Then it hits production and starts giving wrong answers, timing out or burning through your token budget, and you have no idea why. This is when developers discover that print statements and log files weren't designed for this. LLM applications fail in ways that traditional tooling can't see. A hallucination doesn't throw an exception. A slow retrieval step doesn't show up in CPU metrics. A prompt that worked yesterday silently degrades today."
"The fix is observability, and the standard for doing it right is OpenTelemetry (OTel). What OpenTelemetry Actually Is OTel isn't a monitoring product; it's a vendor-neutral specification under the CNCF that defines a standard way to collect observability data: What gets collected, what it's called and how it's shipped. You instrument your application once and can send that data to Grafana, Datadog, Jaeger or a purpose-built LLM platform without rewriting your instrumentation."
"That portability matters more than people realize early on. Your observability investment is in your instrumentation code, not in the back end you happen to be using today. The Semantic Conventions Problem Nobody Talks About Every LLM observability platform claims OTel compatibility. Technically, most are - they'll accept an OTLP payload without crashing. However, protocol-level compatibility says nothing about whether your spans will actually mean anything on the other side."
"The problem is semantic conventions. OTel defines how to send data but doesn't fully define what to name LLM-specific attributes. Three competing standards have emerged: OTel's own GenAI conventions (still evolving, not fully ratified), Arize's OpenInference conventions (used by LlamaIndex, structurally different) and whatever each vendor decided to call things before any standard existed. In practice, this means your LlamaIndex pipeline emits OpenInference, your custom LLM wrapper emits GenAI conventions and your framework's built-"
#llm-observability #opentelemetry-otel #semantic-conventions #production-debugging #token-usage-monitoring
Read at DevOps.com
Unable to calculate read time
Collection
[
|
...
]