English · 简体中文

Observability — Deep Dive

Source: server/src/lib/server-logs.ts, server/src/routes/logs.ts, server/src/lib/request-log.ts, server/src/lib/log-redaction.ts, server/src/lib/attempt-trace.ts

1. Server Logs Viewer (server-logs.ts)

User-facing guide: the dashboard viewer, its two-tier store, the polling API (/api/logs, /api/logs/counts, /api/logs/clear), environment variables, and the React client implementation are documented in logs/01-server-logs-viewer.md. This section covers the internals that guide deliberately omits.

The store keeps a 1000-entry in-memory ring (all levels) plus a persisted server_logs SQLite table (warn/error only) behind a single sinceId-based id space. For the operator-facing contract (tier capacities, retention, cursor semantics, request/response shape) see the Server Logs Viewer guide and its API contract.

Ingest Path

console.log/warn/error/debug/trace
    │
    ▼
lib/log-redaction.ts (console wrapper)
    ├─ redacts secrets (API keys, tokens, auth headers, etc.)
    ▼
server-logs.ts.recordConsoleLine(level, args)
    ├─ formats args (Error → stack, objects → inspect depth 2)
    ├─ noise filter + length cap
    ├─ pushes to ring (splice if > 1000)
    └─ if level ∈ {warn, error} → persist(entry) → INSERT INTO server_logs

Reentrancy guards:

• mirroring — providerLog() mirrors to stdout via wrapped console; prevents double-ingest.
• persisting — DB layer can log (busy timeout, permissions); skips persist write but ring still accepts.
• seeding — boot seed probes DB, which can log.

Structured Provider Logs

providerLog(level, message, { provider, model, event, requestId })

• Recorded with metadata for dashboard filtering AND mirrored to stdout (redacted).
• Used by: Health checker, Cooldown probe, Router, Fallback loop, Catalog sync.

Boot Preload

On first ingest (or explicit initServerLogs() call):

1. Seed lastId from MAX(id) in server_logs.
2. Preload latest 200 persisted rows (newest-first, reversed to oldest-first in ring).
3. Re-stamp any pre-DB ring entries above persisted max (IDs unique across tiers).

Querying & Maintenance

The public query API (queryLogs({ levels, q, provider, sinceId, limit })), level-count badges (levelCounts()), and clear/reset behavior (clearLogs(), resetServerLogsForTest()) are specified in the Server Logs Viewer API contract. The ID counter is preserved across clearLogs() so a dashboard cursor never sees duplicate ids.

Desktop file logger

The desktop app has no terminal attached, so it tees every console line to a file logger (desktop/src/logger.ts, added in 90aaa5b) instead of relying on the in-memory ring: <data dir>/logs/freeapi.log, plus one rotated freeapi.log.1 at 1 MB each. The same redaction wrapper feeds both the ring (server) and the file (desktop), so secrets never reach the file either. The desktop's password-reset code appears here — see the Install & deploy FAQ.

2. Request Analytics (request-log.ts)

Per-Request Row (requests table)

execution_id and the caller dimension

Every JSON body the chat completion routes send — success, cache/idempotency replay, and error alike — carries a top-level execution_id holding the same value as the X-Request-ID header, so a client that only kept the response body can still find the row and its attempt trail. It is stamped on the outbound copy, so a replayed cache hit reports the id of that request, not the one that filled the cache. Streamed frames omit it (the header carries it there).

requests.caller records the surface: today every inference path writes http (the OpenAI-compatible proxy, /v1/responses, /v1/messages, the Ollama and Gemini wires, fusion sub-calls, and ask_f3router). The MCP inference tool deliberately loops through the same local /v1/chat/completions route, so it inherits identical routing, fallback, cache, token accounting, and error handling instead of maintaining a second inference implementation. The other /mcp tools are local introspection/control calls. The dashboard playground also calls the same HTTP endpoints as any other client. The column is free-form, so a new surface needs no migration.

Attempt Trail (request_attempts table)

One row per dispatched-and-failed attempt:

• Powers X-Fallback-Trail / X-Fallback-Detail headers.
• Enables per-request drill-down in dashboard.

Aggregates (request_aggregates materialized view)

Pre-computed rollups for dashboard charts (24h / 7d / 30d / 90d windows):

• Latency p50 / p95 / p99
• TTFB p50 / p95
• Token throughput
• Success rate
• Estimated cost savings
• Per-provider / per-model / per-key breakdowns

3. Log Redaction (log-redaction.ts)

What Gets Redacted

• Single pass at console wrapper → all log paths (console, providerLog, boot lines) are redacted.
• Idempotent: already-redacted text passes through unchanged.

4. Attempt Tracing (attempt-trace.ts)

AsyncLocalStorage Scope

Each request gets an AsyncLocalStorage<RequestTrace> scope:

interface RequestTrace {
  requestId: string;
  startedAtMs: number;
  records: AttemptTraceRecord[];
}

interface AttemptTraceRecord {
  platform: string;
  modelId: string;
  keyOrdinal: number;      // 1-based per request
  outcome: AttemptOutcome; // 'success' | 'retryable_failure' | 'auth_failure' | 'fatal' | 'hedge_abort'
  startOffsetMs: number;   // relative to request start
  durationMs: number;
  errorSummary: string | null; // redacted, ≤200 chars
}

• runWithRequestTrace(fn) wraps the fallback loop.
• dispatch() records start offset, runs attempt, records duration + outcome + summary.
• getRequestTrace() reads current scope (for X-Fallback-Detail header).

Outcomes

5. Health Endpoint (/api/health, /livez, /readyz)

• Degraded mode reported via degradation.getDegradationStatus().
• Provider health from api_keys statuses (healthy/unknown/rate_limited/invalid/error).

6. Routing Trace Headers (Every Response)

7. Dashboard Analytics Pages

8. Key Functions (server-logs.ts)

9. Key Functions (request-log.ts)

10. Environment Variables

11. Privacy & Security

• No PII in logs: Request bodies never logged. Only metadata (model, platform, tokens, latency, redacted errors).
• Redaction first: Console wrapper redacts before server-logs ever sees the line.
• Single-user design: No multi-tenant log isolation needed.
• Local-first: Logs never leave the machine unless operator configures external log shipper.

12. Tables

-- Server logs (warn/error only, persists across restarts)
CREATE TABLE server_logs (
  id INTEGER PRIMARY KEY,
  level TEXT CHECK(level IN ('trace','debug','info','warn','error')),
  source TEXT, provider TEXT, model TEXT, event TEXT, request_id TEXT,
  message TEXT, created_at_ms INTEGER
);

-- Per-request analytics
CREATE TABLE requests (
  id INTEGER PRIMARY KEY,
  platform TEXT, model_id TEXT, key_id INTEGER,
  request_type TEXT, status TEXT,
  input_tokens INTEGER, output_tokens INTEGER, total_tokens INTEGER,
  latency_ms INTEGER, ttfb_ms INTEGER,
  error TEXT, created_at_ms INTEGER,
  client_info TEXT, served_model TEXT, attempt_error_summary TEXT
);

-- Per-attempt trail
CREATE TABLE request_attempts (
  id INTEGER PRIMARY KEY,
  request_id INTEGER, attempt_ordinal INTEGER,
  platform TEXT, model_id TEXT, key_id INTEGER,
  error_class TEXT, error_summary TEXT,
  latency_ms INTEGER, ttfb_ms INTEGER,
  input_tokens INTEGER, output_tokens INTEGER,
  FOREIGN KEY(request_id) REFERENCES requests(id)
);

-- Materialized aggregates (refreshed periodically)
CREATE VIEW request_aggregates AS ...;