Skip to main content

​
Overview

LoopOS AI includes comprehensive observability to monitor:
  • Agent performance and behavior
  • Token usage and costs
  • Request latency and throughput
  • Error rates and patterns
  • User interactions and conversations

​
Langfuse Integration

LoopOS AI uses Langfuse for observability:

Automatic Tracing

All agent runs are automatically traced with input/output.

Token Tracking

Monitor token usage and costs per request and session.

Session Correlation

Track conversations across multiple requests.

Performance Metrics

Latency, throughput, and error rate monitoring.

​
Tracing

Every agent execution is traced in Langfuse:

​
Trace Structure

Trace (Service Level)
  └── Run (Agent Execution)
      ├── Tool Calls
      ├── LLM Calls
      └── Handoffs

​
Trace Data

Each trace includes:
  • Input: Request data, context, messages
  • Output: Agent response, structured output
  • Metadata: Service name, session ID, user ID
  • Performance: Latency, token counts, costs

​
Token Tracking

Token usage is tracked at multiple levels:

Input Tokens

Tokens in prompts, context, and messages sent to LLM.

Output Tokens

Tokens generated by LLM in responses.

Cached Tokens

Tokens that hit cache (counted but not charged).

Reasoning Tokens

Tokens used for reasoning (if model supports it).

​
Cost Monitoring

Costs are calculated based on:
  • Model pricing (per million tokens)
  • Token counts (input + output)
  • Image processing costs (if applicable)
Costs are tracked:
  • Per request
  • Per session
  • Per service
  • Per model

​
Session Tracking

Sessions track conversations across requests: 
{
  "session_id": "session-123",
  "user_id": "user-456",
  "service": "loopos_ai_submission",
  "total_tokens": 5000,
  "total_cost": 0.05,  # USD
  "duration": 45.2,     # seconds
  "messages": [...]
}

​
Performance Metrics

Key metrics monitored:

Latency

Time from request to response (mean, P95, P99).

Throughput

Requests per second, tokens per second.

Error Rate

Percentage of failed requests.

Success Rate

Percentage of successful requests.

​
Logging

Structured logging for all operations:

​
Log Levels

  • INFO: Normal operations, request/response
  • WARNING: Non-critical issues, fallbacks
  • ERROR: Errors with stack traces
  • DEBUG: Detailed debugging information

​
Log Format

{
  "timestamp": "2025-01-15T10:30:00Z",
  "level": "INFO",
  "service": "loopos_ai_submission",
  "session_id": "session-123",
  "message": "Agent execution completed",
  "metadata": {
    "tokens": 5000,
    "cost": 0.05,
    "duration": 45.2
  }
}

​
Monitoring Best Practices

Set up alerts: Configure alerts for high error rates, latency spikes, or cost anomalies.
Review traces regularly: Check Langfuse traces to understand agent behavior and identify issues.
Monitor costs: Track costs per service and model to optimize usage.
Analyze performance: Use metrics to identify bottlenecks and optimization opportunities.

​
Accessing Observability Data

​
Langfuse Dashboard

Access Langfuse dashboard to view:
  • Traces and runs
  • Token usage and costs
  • Performance metrics
  • Error patterns

​
API Access

Observability data can be accessed via:
  • Langfuse API
  • Database queries (for advanced analysis)
  • Log aggregation tools