Follow every step
See the agent's plans, searches, tool calls, retries, and results as one complete run.
Open source reliability for AI agents
Know when your agent is wrong. Before your users do.
TrajectoryCal reads the whole run, from plans and tools to retrieval and retries. It catches confident answers built on missing evidence, then pauses them for verification.
Open sourceWorks with any agent stackActs before failure
01 / AGENT“Ready to use.”
→02 / EVIDENCE CHECKTwo sources never responded.
→03 / SAFE ACTIONPause and verify.
WATCH EVERY STEP→SPOT WARNING SIGNS→ESTIMATE SUCCESS→CHOOSE A SAFE ACTION
01 / THE PROBLEM
A convincing answer can still come from a broken run.
AI agents often keep sounding certain while their tools fail, their plan changes, or the evidence stops matching the answer. Looking only at the final response misses all of those warning signs.
Catch hidden warning signs
Spot failed tools, repeated changes of mind, conflicting evidence, and confidence that drifts away from reality.
Choose a safer next step
Continue a healthy run, ask for more information, verify the answer, involve a person, or stop.
02 / SEE IT IN ACTION
The agent says the fix works. The evidence says otherwise.
Watch TrajectoryCal follow the run, catch the contradiction, and request verification before a bad result reaches a user.
03 / HOW IT WORKS
From agent activity
to a safer next step.
Watch
Record every plan, search, tool call, result, retry, and change of direction.
Understand
Turn the run into clear warning signs such as failures, contradictions, and repeated changes.
Compare
Put signals from different agents and task types onto the same reliable scale.
Estimate
Calculate an honest likelihood that the run will succeed.
Act
Continue, ask for more information, verify, involve a person, or stop.
04 / PROOF IT WORKS
A success score you can actually trust.
We compared TrajectoryCal with two simple alternatives: trusting the agent's final confidence and guessing from the length of the run. TrajectoryCal's score stayed much closer to what actually happened.
CONFIDENCE ERROR REDUCTION79%
Agent's final score 0.42 → TrajectoryCal 0.09
Does confidence match reality?
If runs scored at 80% succeed about 80% of the time, the score is trustworthy. Lower is better.
How accurate are the probabilities?
Penalizes probabilities that are far from the real outcome, especially confident mistakes. Lower is better.
Can it separate success from failure?
Measures whether successful runs rank above failed runs across thresholds. Higher is better.
RESULTS ON UNSEEN RUNSABOUT 80 RUNS
ApproachECE ↓Brier ↓AUROC ↑
TrajectoryCal0.090.140.87
Agent's final confidence0.420.300.69
Run-length guess0.360.240.51
NEW TASK TYPESNO RETRAINING
It still works on tasks it has never seen.
We trained it on two groups of tasks, then tested it on a third group without retraining. The score remained reliable.
Familiar task typesECE0.09Brier0.14AUROC0.87
New task type, no retrainingECE0.13Brier0.17AUROC0.81
05 / FITS YOUR STACK
Add reliability without rebuilding your agent.
Works with different agents
Connect LangGraph today or adapt another agent framework to the same open run format.
Simple defaults, easy to swap
Start with FastAPI and SQLite, then replace either one as your product grows.
Bring your own scoring model
Use the included model or replace it while keeping the rest of the reliability layer unchanged.
See every decision
Connect the monitoring tools your team already uses and understand why a run was allowed or stopped.
Run it on your infrastructure
Use it locally, with Docker, or in your own cloud. No managed service is required.
Open source and inspectable
Review every decision, extend each component, and keep control of your agent data.
TRAJECTORYCAL / OPEN SOURCE
Stop bad agent runs
before users see them.
Follow every step, understand the risk, and choose a safer next action.