English · 简体中文

Catalog Sync — Deep Dive

Source: server/src/services/catalog-sync.ts, server/src/services/model-state.ts, server/src/db/migrations/

1. Overview

The catalog sync keeps the local model catalog in step with the published catalog at f3router.co. It runs twice daily (and on demand), fetches a signed catalog, verifies it against a pinned Ed25519 public key, and applies it transactionally to the local SQLite database.

Two Tiers

Free installs still self-heal, just on a slower cadence. The bundled migrations remain the baseline — a fetched catalog is applied only when newer than MIN_CATALOG_VERSION (bumped on every model migration), so a stale monthly snapshot can never roll back models added by a newer app version.

2. Catalog Structure

interface Catalog {
  version: string;           // e.g. "2026.06.07"
  generatedAt: string;       // ISO timestamp
  tier: 'live' | 'monthly';
  models: CatalogModel[];    // chat models
  embeddings?: CatalogEmbedding[];       // optional, joined freshness feed
  transcriptionModels?: CatalogTranscriptionModel[]; // STT, own top-level key
  quirks: CatalogQuirk[];    // operational advisories
}

CatalogModel (chat)

Generative Media & Transcription

• Media models (modality: 'image' | 'audio') → media_models table, gated by MEDIA_PLATFORMS set.
• Transcription models → media_models with modality = 'transcription', gated by TRANSCRIPTION_PLATFORMS.
• Separate top-level keys (transcriptionModels, embeddings) so older binaries ignore unknown modalities instead of misrouting them as chat models.

3. Signature Verification

const PINNED_CATALOG_PUBKEY = `-----BEGIN PUBLIC KEY-----
MCowBQYDK2VwAyEAq9yv4+3EeyMHKsfVYBhkcz1lYgIXSUeHNnN6tNgYX3k=
-----END PUBLIC KEY-----`;

// Fetch
const res = await fetch(`${catalogBaseUrl()}/v1/latest`, { headers: { Authorization: `Bearer ${key}` } });
const signature = res.headers.get('x-catalog-signature');
const bytes = Buffer.from(await res.arrayBuffer());
const verified = crypto.verify(null, bytes, catalogPublicKey(), Buffer.from(signature, 'base64'));

• Pinned key: private half never left catalog host.
• Self-hosted catalog override: CATALOG_BASE_URL, CATALOG_PUBKEY env vars.
• Tamper-proof: anything unsigned or modified is discarded. Compromised CDN / MITM cannot inject models or quirks.

4. Application Rules (User Data Protection)

Applied inside one transaction (applyCatalog):

Provenance Column (source)

Replaces the old size_label NOT IN ('User','Custom') heuristic. Values:

• 'catalog' — created by catalog sync
• 'user' — declarative config, admin add, custom endpoint
• 'declarative' — from declarative config file

5. Sync Flow

syncCatalog(force=false)
  ├─ GET /v1/latest?since=<appliedVersion> (with Bearer license if set)
  ├─ 304 → up_to_date
  ├─ Verify signature (Ed25519)
  ├─ Parse + structural validation (isCatalog)
  ├─ version < MIN_CATALOG_VERSION? → skipped_older (would roll back migrations)
  ├─ Same version+tier as applied? → up_to_date
  ├─ applyCatalog(db, catalog) → transactional upsert/insert/delete
  ├─ Persist settings: appliedVersion, appliedTier, appliedJSON (verified doc)
  └─ Log counts: updated, inserted, removed, skippedUnknownPlatform, quirks

Boot Re-apply (reapplyCachedCatalog)

Migrations run on every boot and re-assert the bundled baseline (INSERT OR IGNORE baseline models the catalog may have deleted). The boot-time network sync 304s on unchanged version and would NOT re-apply. Without this step, every restart drifts the DB back toward the baseline until the next catalog version bump.

reapplyCachedCatalog()
  ├─ Read SETTING_APPLIED_JSON (cached verified document)
  ├─ Validate + version ≥ MIN_CATALOG_VERSION
  ├─ applyCatalog(db, parsed) → synchronous, no network
  └─ Log: "re-applied cached live v2026.06.07 after boot"

• Legacy upgrade: installs with applied-version but no cached doc → clear applied version → next poll fetches full catalog.

6. Model-Age Gate (30 Days)

Models older than 30 days from generatedAt are excluded from the live tier (free tier trailing snapshot). The catalog service enforces this; the client just receives what the tier provides.

7. License Status

interface LicenseStatus {
  valid: boolean;
  plan: 'annual' | 'lifetime' | null;
  status: string | null;
  expiresAt: string | null;
  checkedAtMs: number;
}

• Cached in settings (premium_license_status), refreshed on every sync.
• Entitlement enforced server-side at /v1/latest — local cache is informational UI state.
• Offline / service down → keep cached status.

8. Quirks (Operational Advisories)

interface CatalogQuirk {
  slug: string;
  title: string;
  body: string;
  severity: 'blocker' | 'warning' | 'info';
  targets: { platform: string | null; modelGlob: string | null }[];
}

• Pure content: replaced wholesale on every sync (DELETE + INSERT).
• Dashboard surfaces blockers/warnings for affected models.

9. Settings Keys

10. Scheduler Integration

startCatalogSync(scheduler: Scheduler)
  ├─ reapplyCachedCatalog()  // synchronous, no network
  ├─ scheduler.after(10s, run)  // boot delay
  └─ scheduler.every(12h, run)  // twice daily

• run() = refreshLicenseStatus() + syncCatalog()
• Kill switch: CATALOG_SYNC_DISABLED=1

11. Migration Seeding vs Hosted Catalog

Custom Models / Relay Models

• Never come from catalog. Created via:
  • POST /api/media/custom (generative media / STT)
  • POST /api/custom-endpoint (chat relays)
  • Declarative config file
• Bound to api_keys row carrying their endpoint (key_id on models row).
• endpoint_scope = 'custom:<base_url_hash>' distinguishes relays serving same model_id.
• Catalog sync skips platform = 'custom' and unknown platforms (older binary).

12. Key Functions

13. Tables Touched

14. Environment Variables

15. Error Handling