Letting Light Into the Black Box: How We Made Every Model Call Traceable
Session tracking, built on Langfuse, turns every AI call into a replayable, attributable trace. Here's why we're so obsessed with making AI visible.
Have you ever been there?
Late at night, a piece of user feedback pops up: "Your AI said something weird just now." You open the dashboard, scroll through the call logs — and find nothing but a cold, lifeless 200 OK. What it actually replied, why it replied that way, where it got stuck, how much it cost, which user, which API key — you have no idea.
In that moment, you realize what "integrating AI" often really means: quietly dropping an unknowable black box into your product.
Our reason for building OntoRoute was to make 400+ models as easy to use as water and electricity — plug in once, use them all. But the longer we used it, the clearer one simple truth became: an AI you can call isn't special; an AI you can see is what brings peace of mind. So we built "Session Tracking" — a "dashcam" for every one of your calls, built on top of Langfuse.
Why we're obsessed with "being able to see"
A team's real predicament
The designer says: "Users say the output is hit-or-miss, but I can't reproduce it."
The ops engineer says: "Cost suddenly tripled last night, and I don't know which team is responsible."
The boss says: "What scenarios is our AI actually being used in? Is it worth it?"
— Nobody can answer, because nobody can see.
An AI call is, by nature, a "stream": one request may chain together multiple model calls, tool calls, and retrieval steps. It's nothing like a traditional API where one input yields one output. When you try to understand it with "logs," you find the logs are fragmented, disconnected, impossible to line up.
What we wanted wasn't more logs — it was one line: a single thread that strings together the same conversation, the same task, the same user journey into something you can replay, inspect, and attribute.
So what does session tracking actually track?
In one sentence: from a single user conversation, to the full lineage of every token.
It rests on three layers of data, reported automatically at runtime by the LiteLLM Proxy, then carried by us into a dedicated Langfuse project:
You don't need to change a single line of business code. As long as your request carries an OntoRoute virtual key, tracking happens automatically.
It answers four questions for you
What happened in this conversation?
A Session aggregates one complete interaction into a single "page" you can open and replay.
Where did each step get stuck?
A Trace breaks a single call into a tree — models, tools, and retrieval each on their own branch, with latency laid bare.
Who is actually using my AI?
The User view shows you real human beings, not a pile of anonymous IDs.
Can I know before the bill goes over budget?
Monitors sound the alarm before cost crosses your threshold.
Sessions: you can flip through them like a book
Open the session list. Filter freely by virtual key, user ID, environment, and tags. Production anomalies and test noise are separated at a glance.
Open any one. You see a "thumbnail" of the session: total latency, total cost, how many model calls, whether there was an error.
Drill into any single call. A side panel slides out with a "trace peek," laying bare the tree structure behind that request — the star of the next section.
We deliberately made "sessions" something you can flip through. Because for most product teams, what they need isn't raw data — it's a plain sentence like "what did Zhang San talk about with the AI that day."
Traces: taking apart the black box
This is the part we're proudest of. Every call, behind the scenes, may actually be a "relay race": one model call for intent recognition, another for generation, with a tool call wedged in between. Traditional logs would split these into three unrelated lines.
The trace view reconnects them into a single tree:
What you can see here
- TraceTree: the full family tree of the call, in clear order.
- ObservationDetailView: per-step input/output, parameters, metadata.
- IOPreview: request-vs-response comparison — the key to debugging "why did it answer the wrong thing?"
- TraceTimeline: per-step latency distribution, so performance bottlenecks have nowhere to hide.
- TraceLogView / TraceGraphView: logs and topology, two perspectives you can switch between freely.
The user view: seeing real people
After aggregating into "sessions" and "traces," we wanted to go one layer higher: people.
User tracking brings calls scattered everywhere back to real individuals. You can see:
- How many times a user called today, how many tokens, how much money
- Their favorite models, common paths, anomaly frequency
- Which "users" are actually scripts, farming bots, or attackers
Privacy and boundaries
Tracking exists to help you serve users better, not to surveil them. All tracking data is isolated by virtual key and account, and only records call metadata and input/output within the scope you authorized. Whether to enable it, and for how long to retain it, is your call.
Cost alerts: guarding the bottom line
The scariest thing about building an AI product isn't that it's expensive — it's that it gets expensive without you noticing.
Monitors let you set thresholds on cost, tokens, and call counts, and proactively notify you via email or webhook the moment a boundary is crossed. It won't save you money, but it guarantees you'll never discover you're broke at the end of the month.
When a key's daily spend approaches budget, alert immediately. Ideal for cost-sensitive teams that account by business line.
When a single session's token consumption balloons abnormally, hinting at a "looping" tool call or prompt bloat.
When call frequency spikes, helping you catch farming, retry storms, or potential prompt-injection probing at the first sign.
How it binds to your keys
Session tracking isn't a crude "global switch." It's precise down to every single virtual key:
- You can subscribe to "session tracking" at the account level, choosing a retention period (7 / 90 / 180 / 365 days)
- Then decide individually for each key: should this one be tracked?
That means: production keys leave a full trail, test keys toggle on and off at will. The sense of control always stays in your hands.
A straight word on billing
Tracking itself generates data volume. We bill by "number of traces," with the entry tier (1,000 traces / month) free forever, and overage billed per use — but for now, we only meter the overage, we don't charge it yet. Light the lamp first, talk about the electricity bill later. That's our small gesture of good faith.
The future: from "seeing" to "understanding"
Today, session tracking lets you see every call. But our ambition goes further:
- Automatic anomaly clustering: the same type of error auto-groups, no longer drowning in alerts
- Quality scoring: combined with guardrail review results, give each conversation an "experience score"
- Replayable reruns: not just watch, but rerun that call in a sandbox
Seeing is the first step to understanding; understanding is the beginning of trust.
In closing
We built session tracking not because it's "advanced," but because it's "necessary."
When AI enters your product, it's no longer just a piece of your code — it's a pair of hands you've extended to your users. You have the right to know what those hands did, and whether they did it well. It's about responsibility, and it's about dignity.
If you too want to turn your AI from a black box into a window, come experience session tracking. Every one of your calls deserves to be taken seriously.
Next, we'll talk about "Guardrails" — when AI might err, overstep, or be exploited, how do we fasten a seatbelt around it? Because only risks we can see are risks we can guard.
Last updated on