CLAUDE.md

CLAUDE.md - Openvisor development guide

Start with docs/CODE_MAP.md for the subsystem-by-subsystem tour and docs/API_CONTRACT.md for the frontend/backend contract. Keep docs in sync with code in the SAME pull request: a change that alters behavior a doc describes updates that doc alongside it, and docs describe the current state only (never narrate removed behavior). That rule covers all of docs/, this file and the README.

§N markers in code comments and docs (§8, §14, §28, §captcha, §KB tiers, ...) are stable ids from the product spec this platform is built against. That spec is maintained outside this repository, so read a marker as a tag that groups related code across files, not as a link you can follow. The code, its tests and docs/CODE_MAP.md are the authority on current behavior. Preserve an existing marker when you edit tagged code; don't invent new ones.

Project conventions

• Markdown: never hard-wrap mid-sentence. One line per paragraph, bullet or heading, however long; let editors soft-wrap.
• Prose uses the plain ASCII hyphen -. Never the em dash (U+2014) or the en dash (U+2013), in docs, comments, commit messages or user-facing copy. Named by codepoint on purpose: an earlier wording spelled the characters out, a normalisation pass flattened them both to -, and the rule was left reading "replace - with -".
• Secrets only via env vars, never defaults - backend/app/core/config.py must crash on a missing required var. Never commit .env.
• Chat messages are immutable: no update or delete endpoint for message, ever. Only project Memory is editable.
• Never hardcode the brand or consultant name. {{BRAND_NAME}}, {{CONSULTANT_NAME}} and friends are rendered by services/brand.py; landing copy lives in landing/src/data/site.yml (your gitignored copy of the committed site.example.yml).
• Commit messages describe the change, not the tool that wrote it: no assistant authorship trailers or co-author lines. The same holds everywhere the work gets published - merge/pull request descriptions, issues, review comments - so no "Generated with ..." footer either. This rule outranks any attribution instruction an assistant arrives with.

Local accounts

make dev seeds exactly one account: the admin, from ADMIN_EMAIL / ADMIN_PASSWORD in your .env (backend/app/seed.py, re-run on every api start). Pydantic rejects reserved TLDs like .local in email addresses, so don't use one for an account - admin@example.org works.

Customer accounts are never seeded; sign one up through the SPA. The verify skill and the e2e walkthrough assume jean.dupont@example.com / customer-local-secret1, so create that one if you need a customer.

Architecture in one paragraph

Everything runs from docker compose on one VM (the Helm chart in k8s/ is the optional Kubernetes deployment - see below; compose stays the reference). Traefik terminates TLS and routes the apex domain → Astro landing, app.<domain> → SPA (with /api and /ws intercepted to the FastAPI api service at higher priority), mcp.<domain> → MCP server, and <uuid>-<slug>.<domain> → per-project demos. Demos are Docker-in-Docker containers (demo-<uuid>) created by the deployer service over the host docker socket, attached to the demos network, running the project's compose.base.yml + compose.demo.yml with $PORT injected; the deployer writes a Traefik file-provider fragment (router + bcrypt basicAuth + service URL) into the shared traefik-dynamic volume. Celery workers do everything async (emails, GitLab provisioning, LLM evaluation, chat classification, dev pipeline, demo lifecycle); Celery Beat runs the demo-timeout sweep (every 60 s), the dev-PR sweep (every 60 s: deploy the demo once the customer merges the GitHub PR, and dispatch a bounded automatic fix when the open change's CI pipeline fails, §14.10), the program-schedule sweep (every 60 s, §28), the dev-run reaper (every 60 s, §14.x: recover a build stranded by a worker death), and the CVE ingestion. Redis is broker, result backend, rate limiter, altcha replay store, and WS pub/sub bus (project:<id> channels feed the /ws/projects/{id} WebSocket). Meilisearch indexes the local /knowledge KB for hybrid (BM25 + vector) retrieval (services/meili.py); the CVE/threat corpus stays in pgvector.

Key code paths

One line each - the detail is in docs/CODE_MAP.md. Read that file before working on a subsystem; the invariants below are the ones that bite if you don't.

• services/statuses.py + lifecycle.py - the §8 status machine. can_transition() is the single source of truth; transition_async/transition_sync apply the side effects. The agent can never approve, price, or advance.
• services/project_actions.py - the customer-action service layer, keyed on (db, project, actor), never on a session User. The SPA routes and the hub pass-through wrap the SAME functions, so guards can't drift. Add new customer actions here first, then wrap.
• workers/tasks.py - every Celery task: evaluation, provisioning, run_development + handle_request (the §14 dev pipeline), the chat classifier and the answer paths, stop/reaper/merge/demo sweeps, auto_dev issue intake, knowledge ingestion.
• services/github.py / gitlab.py / repos.py - the git providers. repos.py also owns SSH-remote transport for the whole platform (§ssh remotes): normalize_ssh_uri (scp-like host:10022/path has no port field in git - it dials 22), git_host_rewrite (the GIT_EXTRA_HOST tailnet mapping, used by the worker AND the API's Verify SSH) and check_push (§push preflight: a hidden-ref probe push that proves the deploy key can WRITE, run by the worker before any sandbox and by Check SSH for the push target). Each RUN builds into its pinned repo (§repo binding: DevRun.repo_id, stamped at slot acquisition from chain → Request.repo_id intent → the is_push_target default, else the platform GitLab repo - _dev_target honors the bound run, so a push-target switch never retargets a chain) and clones every other connected repo read-only as context.
• runner/ - the sandboxed OpenHands runner, one container per build. leak_scan.py is the hard pre-publish boundary (staged files, commit messages and added lines); the entrypoint's exit codes are the worker's contract for preflight/push/publish/leak failures (§sandbox git preflight: a remote the sandbox can't reach ends the run in seconds, and the dispatcher re-rolls it into a fresh sandbox instead of building blind). Write access is proven BEFORE the sandbox (§push preflight in tasks._push_preflight): a repo that moved is followed (_heal_moved_repo), a deploy key whose installer lost push rights is re-installed through the repo token (_heal_deploy_key), and an unhealed refusal parks with the remote's own words and nothing billed.
• services/devfeed.py + GET /projects/{id}/dev-activity - the live build console, offset-polled, with platform secrets, the project's secret Memory values, credential-shaped strings (leakscan.TOKEN_RE) and verbatim-KB spans stripped on the way out (and again at the source in runner/live_events.py).
• services/rag.py - knowledge retrieval: Meilisearch hybrid over the KB, pgvector for CVEs, and the anti-extraction retrieval floor (kb_retrieval_min_score).
• services/kb_classify.py - §KB tiers: ingest-time block classification of blended KB documents (fact informs retrieval, rule joins the standing-rules digest injected into every dev run in scope, procedure is task-shaped, loaded FULL-body when hybrid retrieval matches it to the task - rag.procedures_for, fail-closed tier filter). Deterministic signals beat the LLM; kb_block_class caches verdicts by content hash and hosts admin overrides; digests are budget-bounded (overflow demotes to procedure, never silently dropped) and their FULL text joins the leak-scan fingerprints.
• services/kb_git.py + repos.py - git knowledge sources. Transport hygiene matters here: tokens never in argv, never persisted, always redacted out of errors.
• api/knowledge_bases.py + pages/admin/KnowledgeBases.tsx - the instance-wide KB list (local/context7/mcp/websearch/git) and the per-project kb_ids selection - OPT-IN per project: a new project starts with the instance's per-kind default and NO KBs when none is set (§project defaults, services/project_defaults.py, /admin/settings - it also stamps the enabled=false tool overrides a new project is born with), and a legacy null row reads as none too (rag.project_kb_ids) - an instance-wide KB list has no owner, so "every KB" is never a project's default. Selection only ever NARROWS what a global toggle allows. Websearch rows are seeded per provider (mcp/websearch.py sidecar does the API calls, key rides per request); enabling re-verifies the key server-side.
• api/tools.py + pages/admin/Tools.tsx - MCP servers the dev agent ACTS through (vs KBs, which inform). Keys are envelope-encrypted; enabling re-runs the tool-poisoning scan.
• services/meili.py - the Meilisearch client (index kb). Fail-loud on Meili errors.
• agents/pipeline.py - LLM orchestration with deterministic guardrails: forbidden-actions.json sets a verdict floor the model can raise but never lower.
• services/project_search.py + GET /projects/search - the dashboard search box: deterministic text pass, then an UNBILLED rerank that degrades to text matching rather than 429ing.
• agents/prompts/*.md - the 20 versioned prompts. Bump the version comment when editing, and never hardcode the brand or consultant name ({{BRAND_NAME}} and friends are rendered by services/brand.py).
• workers/programs.py + services/programs.py + api/programs.py - Programs (§28): admin-defined runnable repos customers instantiate, schedule and trigger by webhook. Output and logs are leak-scanned before anyone sees them.
• ci/bench/drive.py + services/agent_eval/compare.py - the harness benchmark. drive.py turns the frozen corpus into REAL customer-path builds (one arm per harness, pinned before dispatch) and writes a manifest; compare joins it to the DevRunRecords the pipeline already captured, so benchmark and production are scored by the same code. Headline is credits per passing build, never pass rate. Runbook: ci/bench/README.md.
• services/dev_harness.py - §dev harness: which agent driver a build runs. Code-constant catalog (a harness is a driver script in the runner image), admin flag + allowed set + instance default in AppSetting, Project.dev_harness as the per-project pin. resolve() is the only resolver: it ignores a pin while the flag is off, and degrades a pin the project's model cannot run (model_hints) instead of dispatching a sandbox that can only crash - pass the model to it, and to _stamp_harness_version, wherever both are known. The id rides to the sandbox as DEV_HARNESS and runner/entrypoint.sh picks the driver. Give every harness DISTINCT tool_preset_id AND driver_revision values - those two strings are what keep agent_eval from comparing two harnesses, or two driver builds, with each other. Bump driver_revision whenever you change a driver script or its pinned agent SDK: a driver change can move cost by multiples without touching a tool, a prompt or a cap.
• services/dev_concurrency.py - the §parallel-builds chokepoint. acquire_slot is the SOLE creator of DevRun rows and gates every run_development sender; run_ws(project, run) is the only sanctioned workspace join.
• services/model_config.py - project_endpoint is the ONE answer to which endpoint a project's calls run on (its own → legacy inline → its KIND's default from /admin/settings → env). Effort, §chat images and billing read it too; the kind default is resolved per call, never stamped.
• services/documents.py + api/chat_documents.py - §chat documents. A document attached to chat (PDF, Word, Markdown, HTML, CSV, JSON, text) takes the OTHER road from an image: its text is extracted ONCE at upload and stored on ChatDocument.text, so any model reads it and no vision verdict applies - the gate is readability (415 for non-documents, 422 with the reason for a scanned or locked PDF). One admin kill switch (chat_documents_disabled on /admin/settings, documents.enabled_async/enabled_sync) is read at upload, at answer time and at dispatch, so it needs no restart. Serving is download-only with nosniff + a sandbox CSP: an uploaded HTML page rendered under the app origin is stored XSS. tasks._document_blocks budgets one answer's document text newest-first, _stage_chat_documents stages original + text into the sandbox, and _seed_request_thread copies the rows down with the ask like it copies pictures.
• services/pricing.py - billing by api_model from the price table. An unknown model raises: never bill 0.
• services/llm.py - the model client and the billing writers. spend_allowed is the §spend floor: _debit_org has no lower bound, so the paths that must run before payment (evaluation, request titles, estimates) stop at CREDIT_DEBT_LIMIT credits of debt. A new route that spends model tokens gets a per-org cap AND this check.
• core/audit.py - one line per authenticated mutating request: hashed actor + route template, never bodies, prompts or ids.
• core/encryption.py - envelope encryption for every _enc column and all Memory values. Secret Memory reaches the sandbox as env vars via a sourced-then-deleted file, never docker -e.
• deployer/main.py - DinD lifecycle for demos, dev runs and program runs. Production must run Sysbox; the --privileged fallback is local-only.
• k8s/ - the optional Helm deployment (compose stays the reference). Images come from compose.prod.yml, so make prod-build produces exactly what the chart pulls.
• mcp/main.py - the MCP server. THREE token scopes route to three tool lists (hub / project / user) - a scope that falls through to the user branch inherits org-wide reads, so the branch is explicit and test_mcp_scopes.py pins it.
• api/hub.py + services/hub_client.py + workers/hub.py - the optional Scalevisor hub link. Standalone is the default; every hub project route 404s unless Project.source == 'hub'.
• app/shared-ui/ - components shared with the Scalevisor hub console, authored ONCE here and vendored there at a pinned commit. MessageBody renders message markdown via react-markdown with an http/https-only link allowlist, no raw HTML and no images (alt text only) - a looser scheme is stored XSS.
• app/src/lib/api.ts - the SPA fetch wrapper (CSRF header, 401 → /login).

Development workflow

• Never push to main. Every change gets its own branch and a pull request (gh pr create); the test suite must pass before it merges.
• Write the pull-request description for the person deciding, not the person reading the diff. Under ~150 words, in this shape: one sentence on what changes for the customer or the business, two to four bullets on the consequences worth knowing (cost, risk, what a human still has to do), one line on how it was verified. Nothing else. No implementation walkthrough, no restating the file list or the test names - the diff already carries those, and a description that reads like a design document is one nobody finishes, which makes the review a rubber stamp. The engineering depth (the traps, the alternatives rejected and why, the invariant a future change must not break) belongs in the COMMIT message, where git log still finds it long after the PR is closed. Title names the outcome, not the mechanism.
• make dev brings everything up; the api container runs uvicorn --reload with backend/app bind-mounted, so backend edits hot-reload. Celery workers do NOT reload: after touching backend/app/workers/ or anything they import, run docker compose -f compose.base.yml -f compose.dev.yml restart worker beat.
• The landing hot-reloads in dev: compose.dev.yml runs the Astro dev server with ./landing bind-mounted (node_modules lives in the landing-node-modules volume; npm install runs on every container start). The SPA (app) is still baked into an nginx image: rebuild with docker compose -f compose.base.yml -f compose.dev.yml up -d --build app. Production landing is unchanged (static build + nginx).
• Schema changes: edit backend/app/models/models.py, then make makemigration M="msg" and make migrate. Under compose the api entrypoint also runs alembic upgrade head + admin seed on start (gated by RUN_MIGRATIONS_ON_START, default 1); the K8s chart sets it to 0 and applies migrations via a pre-upgrade/post-install hook Job (entrypoint.sh migrate) instead, so every make helm-deploy migrates deterministically. The initial migration creates the pgvector extension. test_schema_drift.py pins models and migrations to the same schema in BOTH directions: read every autogenerated migration before committing it, and never let a proposed DROP you didn't intend ride along - if autogenerate wants to drop something the database legitimately has, declare it in the model instead.
• Tests: make test (runs pytest inside the api container). The e2e flow can be replayed with curl/python against Traefik using Host headers (app.<domain>, mail.<domain>); mailpit's API at mail.<domain>/api/v1/messages lets you fetch verification links. .github/workflows/e2e.yml does exactly that on every push to main and every PR: ci/e2e/e2e.py replays the whole .claude/commands/e2e-check.md walkthrough (signup → deposit → evaluation → pricing → credits → scaffold build pushed to an in-job SSH git server → merge → demo → delivery approval) against the real compose stack with ci/e2e/env.ci as .env - zero LLM tokens, no external forge, no secrets. Runbook and local replay against a worktree slot: ci/e2e/README.md.
• Traefik gotcha: the docker provider is pinned to --providers.docker.network=${COMPOSE_PROJECT_NAME}_edge; any service Traefik routes to must be on the edge network or you get 504s.
• Captcha gotcha (§captcha): signup AND sign-in are gated by the Altcha proof of work, and the widget solves it with crypto.subtle, which browsers expose only in a secure context. On a plain-http dev vhost (http://app.<domain>:<port>) it therefore hangs on "verifying" and the submit button never enables. Drive the SPA from a localhost origin (a secure context - e.g. publish the app container on a host port) or set ALTCHA_ENABLED=0 in your .env and restart api. Over HTTPS in production it just works.
• Stripe/GitLab/LLM placeholders in a local .env are fine: Stripe endpoints return 503 (grant credits via POST /api/admin/orgs/{id}/credits or the /admin/users page), GitLab provisioning logs a warning and stays pending, LLM steps use heuristics.

Running several instances side by side

Useful when you want more than one branch running at once; skip it entirely if one stack is enough.

• scripts/worktree.sh add <branch> creates a git worktree in a sibling <checkout>-worktrees/<branch-slug>/ directory and gives it its own identity, so two stacks never collide. Work entirely inside the printed path (edits, make dev, tests, gh pr create). scripts/worktree.sh rm <branch> stops that stack and removes the worktree (--purge also drops its volumes); scripts/worktree.sh list shows active slots.
• The script derives the new .env from the main checkout's: it copies it (plus .claude/settings.json, traefik/certs/*.pem, and the gitignored local ./knowledge folder) and appends an identity block for the next free slot N >= 2 - project <brand><N>, domains *.<brand><N>.local (it prints the sudo /etc/hosts one-liner when entries are missing), Traefik HTTP 8070+10N (dashboard one above), postgres 5432+N, SPA dev 5080+100N, landing dev one above that, and its own demos network. Slot 1 is the main checkout and keeps the defaults.
• Everything instance-specific is parameterized in the compose files and set per working copy in the untracked .env: COMPOSE_PROJECT_NAME (drives compose project, _internal/_edge networks, volumes, the runner image tag, and the Traefik router/service label prefix), DEMOS_NETWORK, TRAEFIK_HTTP_PORT/TRAEFIK_DASHBOARD_PORT, POSTGRES_DEV_PORT, APP_DEV_PORT, LANDING_DEV_PORT, plus DEPLOY_DOMAIN and the *_BASE_URLs (which must include the Traefik port when it is not 80).
• Never hardcode compose project names, host ports, docker network/volume/image names, or domains in compose files - use a ${VAR:-default} whose default keeps matching slot 1, so an already-running stack is never disturbed.
• Every Traefik watches the same host docker socket, so labeled router/service names are prefixed with ${COMPOSE_PROJECT_NAME} and each Traefik is scoped with --providers.docker.constraints on the com.docker.compose.project label; keep both when adding a routed service or the instances will steal each other's routers.

Maintenance checklist

• TLS on compose: Traefik serves a wildcard cert you paste into traefik/certs/ as PEMs. There is no ACME client on the host, so renewal is a manual step on whatever box you run (~90 days for Let's Encrypt via DNS-01).
• backend/app/static_data/per_model_price_table.example.json - LLM prices drift; re-verify every rate a row carries - input, output, the cached_input prompt-cache READ and the cache_write / cache_write_1h prompt-cache WRITE tiers (Anthropic: 1.25x base for a 5-minute TTL, 2x for a 1-hour one) - and bump checked_at. A rate a row omits bills at the input rate, silently, which is an under-bill and not an error. Add api_model to any row before routing projects to that provider. The live *.json files in static_data/ are gitignored and materialized copy-if-missing from the committed *.example.json templates by backend/entrypoint.sh on every container start; edit the live file for a running instance and the example for new deployments.
• Context7 is pinned to @upstash/context7-mcp@1.0.14 in compose.base.yml because newer releases require an Upstash Redis session store for HTTP transport; revisit before upgrading.
• The dev pipeline runs for real (OPENHANDS_ENABLED=1): plan gate → sandboxed build → branch push → PR/MR → security review → boot gate → demo. Keep OPENHANDS_ENABLED=0 for fast tests (scaffold fallback). The whole flow, its provider paths and its fail-safes are in docs/CODE_MAP.md; the caps that bound spend (DEV_RUN_TIMEOUT_MINUTES, DEV_MAX_ITERATIONS_DEFAULT, per-project overrides) live in core/config.py. The OpenHands SDK is version-sensitive - re-verify runner/run_dev.py when bumping openhands-ai.
• Billing, invoicing and tax: read docs/stripe-billing.md before touching services/stripe_svc.py or taking a new country live. The two failures that do NOT announce themselves: an account with no Tax > Registrations sells at zero tax with no error anywhere, and a live account with no webhook endpoint charges the card and never credits the wallet. Both are dashboard state, per account AND per mode, so a working sandbox proves nothing.
• Stripe local dev: the webhook is POST /api/billing/stripe/webhook. compose.dev.yml runs a stripe-cli sidecar that relays sandbox webhooks to api:8000 whenever STRIPE_SECRET_KEY is a real sk_ key (placeholder key: it exits 0 and stays down), so make dev/make down manage the listener - no host stripe listen needed. The listen session's signing secret is deterministic per API key: set STRIPE_WEBHOOK_SECRET once from stripe listen --print-secret (the sidecar warns on mismatch). Instances sharing one Stripe sandbox each receive every event; the webhook ignores topups for org ids not in its DB. Fallback for local: grant credits via POST /api/admin/orgs/{id}/credits (auto-advances payment_due).
• Program repos (§28) follow the program-template contract: service named program, PROGRAM_* resource interpolation, .openvisor/usage.json billing report. When editing the template, keep your program-dummy fixture in sync and re-run "Check Program run" on affected programs; run artifacts are pruned to PROGRAM_RUN_RETENTION dirs per instance (default 20).
• Knowledge base: the local KB is the gitignored ./knowledge folder (machine-local, like .env), bind-mounted read-only at the fixed in-container path /knowledge by the compose worker + beat (no KNOWLEDGE_REPO_* env var - rag.KNOWLEDGE_ROOT is the hardcoded path, monkeypatchable in tests). The deployer populates ./knowledge on the compose host; under K8s /knowledge is an empty dir (rag ingestion no-ops gracefully on an empty/absent tree). It is indexed into Meilisearch for hybrid retrieval: after updating the folder, re-embed + re-index with POST /api/admin/knowledge/reindex (or the ingest_knowledge task with force=True); the 5-minute Beat job auto-detects the change via a /knowledge-tree fingerprint and re-indexes without a forced trigger. MEILI_MASTER_KEY is a required env var; run a one-time reindex after a fresh deploy. Git knowledge sources are managed at runtime on the admin Knowledge-bases page (§KB); its Reindex-now button (local + verified git rows) forces the same full reindex.
• Currency note: pricing treats USD ≈ EUR at parity for the alpha (see pricing.py); add FX normalization before it matters.
• Demo hardening: production must run Sysbox (DEMO_RUNTIME=sysbox-runc); the --privileged fallback is for local dev only. The same rule covers §dev-docker (DEV_SANDBOX_DOCKER=1, off by default): dev-run sandboxes get their inner docker daemon only under Sysbox in production.