Spanlens

Spanlens is an open-source (MIT) observability layer for applications that call LLMs. It sits as a proxy you can swap into a client baseURL or install with a one-line CLI init, then captures every request and response, including model, token counts, latency, cost and the full prompt and reply. Streaming responses are reconstructed automatically.

The dashboard turns those logs into practical signals. You get cost breakdowns by request, model and end user, prompt-cache tokens separated out, and visual traces for multi-step agent workflows shown as timelines and node graphs so you can spot slow dependencies. Anomaly detection watches latency, cost and error rate against a rolling seven-day baseline and flags outliers, while alerts can be sent to Email, Slack or Discord.

Spanlens also includes active protections and evaluation tools. A regex-based scanner checks for PII and prompt-injection in requests and responses and can block injections at the proxy. There’s a savings engine that identifies calls suited to cheaper models and estimates potential monthly savings, and prompt versioning with A/B tests that compare latency, cost and error rate using statistical tests. An LLM-as-judge setup lets you run rubric-based scoring with OpenAI, Anthropic or Gemini, and reusable datasets support offline evaluation and regression checks.

• Open-source MIT licence; proxy or CLI integration
• Request logging: model, tokens, latency, cost, full bodies (streaming rebuilt)
• Cost tracking, agent tracing, anomaly detection and alerts
• PII/prompt-injection scanner, savings suggestions, prompt versioning and LLM evaluation

Official site: https://spanlens.io/