Development

Everything a developer needs that does not belong in the README. Read AGENTS.md first — it is the rulebook; this is the manual. CONTRIBUTING.md is the shorter door into both.

Requirements

• Python 3.13+ (the workspace currently resolves to 3.14)
• uv — the only package manager used here
• Docker or podman, for the database and dashboards (tests run fine without either; cb.py up uses whichever is on PATH)
• A C compiler, for the optional compiled hot path

Layout

AGENTS.md              rules for anyone writing code here
scripts/cb.py          every task — there is no Makefile
scripts/spec.py        the migration spec: one row per feature
scripts/status.py      measures the spec against the QA repo, the scenarios and a test run
scripts/docs_sync.py   renders that measurement into this site (progress board + frontmatter)
packages/cb-core/      shared runtime: settings, telemetry, db, cache, storage, llm,
                       tenancy, and the Cython-compiled hot modules
packages/cb-api/       FastAPI service + alembic migrations (all Citus DDL)
packages/cb-gateway/   aiogram ingest: webhook / polling / (reserved) websocket
packages/cb-worker/    arq jobs: partitions, rollups, media GC, captcha expiry
qa/                    acceptance suite — Gherkin + an in-process mock Telegram API
qa/integration/        integration tests: real database, simulated users
ops/                   otel collector, prometheus, tempo, grafana provisioning

Tasks

scripts/cb.py is the single definition of every task, so CI and your terminal run identical commands.

python scripts/cb.py --list          # all tasks
python scripts/cb.py setup           # the whole stack, seeded, with the API answering
python scripts/cb.py install         # uv sync --all-packages
python scripts/cb.py api-lint        # fail on a REST endpoint nobody documented
python scripts/cb.py api-docs        # regenerate the published spec + the reference page
python scripts/cb.py up              # citus, valkey, otel, prometheus, tempo, grafana
python scripts/cb.py migrate         # create and distribute the schema
python scripts/cb.py types           # mypy + the compiled-module type audit
python scripts/cb.py check           # the pre-push gate (lint, types, tests, bench, spec)

Services:

python scripts/cb.py gateway   # :8081  telegram ingest    metrics :9101
python scripts/cb.py api       # :8000  /healthz /readyz    metrics :9102
python scripts/cb.py worker    #        arq + cron          metrics :9103

Testing — the pyramid

python scripts/cb.py test              # unit + acceptance, offline
python scripts/cb.py test-integration  # needs a database
python scripts/cb.py api-test          # the HTTP suite: smoke, contract, integration
python scripts/cb.py test-pyramid      # each layer separately, stops at the first failure

The HTTP layers are new with the Mini App API and have their own guide: Testing the API covers standing a deployment up, the fixtures, and the conventions each layer follows. The contract layer validates against docs/site/public/openapi.json with jsonschema-rs, which is why that file is committed and why cb.py check fails when it is stale.

setup is the other door into all of this, and the one to point a tester at: uv run scripts/qa_setup.py goes from a fresh clone to a running API with data in it, three tokens in your hand and a table of what every endpoint answered. Testing the API is its manual.

Integration tests skip cleanly when no database is reachable, so the offline suite always runs. They seed data through qa/integration/factories.py — a World with a disposable group and SimulatedUser members. Never hand-write INSERTs in a test: the point of that layer is that rows look like production's.

Configuration

Everything is environment-driven; see .env.example for the full list.

Blob storage

CB_STORAGE_URI=gs://cookiebot-media     # or s3://bucket/prefix, file:///var/…, memory://

Credentials resolve exactly as each cloud's own SDK resolves them (AWS_* or an instance role; GOOGLE_APPLICATION_CREDENTIALS or workload identity). Handlers call cb_core.storage.media() — never a cloud SDK.

LLM providers

CB_LLM_TASKS='{"chat":{"provider":"anthropic","model":"claude-opus-5","effort":"low"},
               "transcribe":{"provider":"openai","model":"whisper-1"}}'
CB_OPENAI_BASE_URL=http://localhost:11434/v1   # ollama / vLLM / openrouter

Handlers ask for a task, never a model. Parameters are filtered per model by cb_core/llm/catalog.py — current Claude models reject temperature, so forwarding it blindly would break the default model.

Self-hosted Telegram Bot API

Lifts uploads to 2 GB, removes the download cap and the per-bot rate limits, and allows a webhook pointed at a private address.

export TELEGRAM_API_ID=... TELEGRAM_API_HASH=...   # from my.telegram.org
python scripts/cb.py selfhosted

CB_TELEGRAM_API_BASE=http://localhost:8082
CB_TELEGRAM_API_LOCAL=true      # getFile returns a disk path, not a URL
CB_TELEGRAM_INGEST=polling      # no public URL needed in dev

Database

Postgres 17 + Citus, sharded on group_id, everything colocated with groups. The rules are in AGENTS.md §4 and asserted by qa/integration/test_citus_topology.py, which reads pg_dist_* and EXPLAINs the reply-path queries to check they touch exactly one shard.

Migrations are raw SQL in op.execute so shard keys and colocation are visible in the diff. Both directions must work:

python scripts/cb.py migrate-check   # upgrade → downgrade → upgrade

Every service also converges the schema itself during startup (cb_core/migrations.py): one SELECT against alembic_version, and an upgrade head only when that revision is behind the code. Replicas serialise on a Postgres advisory lock, so N processes booting together produce one upgrade; a failed upgrade aborts startup rather than serving against a half-built schema. Set CB_AUTO_MIGRATE=false where a separate migration job owns the schema.

Shard count is CB_CITUS_SHARD_COUNT (default 8), applied to the migration session in migrations/env.py rather than to the server, so local, CI and production distribute identically however Postgres was started. Eight, not Citus's default 32, because each shard is a table and each table adds a composite type: a catalog with ~1000 of them makes asyncpg's first introspection on a connection take seconds. Re-shard later with alter_distributed_table(..., shard_count => N) — no schema change.

Three Citus rules the migrations had to learn, all of them silent until a real cluster runs the SQL:

• a single node must be registered as a worker, not just as the coordinator, or create_distributed_table fails with replication_factor (1) exceeds number of worker nodes (0);
• DO UPDATE SET on a distributed table may only call IMMUTABLE functions — use excluded.<col> and put the now() in the SELECT list;
• a correlated subquery from a reference table into a distributed table is rejected; write it uncorrelated (NOT IN) and Citus recursively plans it.

Surrogate keys are UUIDv7 from cb_core.ids.uuid7() — never uuid4, never a sequence on a distributed table.

Importing v1 data

v1's data lives in the Java backend's MongoDB. cb_worker.importer moves it into the v2 schema from either source:

docker compose --profile v1data up -d        # a local Mongo to import from
CB_MONGO_URI=mongodb://localhost:27017 python scripts/cb.py import-mongo --dry-run
CB_MONGO_DUMP_DIR=./dump/cookiebot          python scripts/cb.py import-mongo

Exactly one source may be configured; setting both is refused rather than silently preferring one, and the URI is never logged because it carries credentials.

Every write is an upsert on the natural key, so the import is idempotent — that is what allows a cutover without a maintenance window: run it while v1 is still serving, then again at cutover to pick up the delta. --dry-run reports the counts it would write.

The shapes come from the Java @Document entities, which AGENTS.md names as the source of truth for stored data. Three conversions are not obvious and are pinned by unit tests: every Mongo _id is a String holding a Telegram id, v1's threadPosts uses the string "9999" where v2 stores NULL, and stickerSpamLimit is a String in Java against an int column here.

The compiled hot path

Four pure-CPU modules are compiled with Cython in pure-Python mode, so the same files import and test fine uncompiled.

python scripts/cb.py bench-baseline   # rebuild without cython, record the baseline
python scripts/cb.py cython           # build the extensions in place
python scripts/cb.py bench            # fails any compiled module below 1.5x

Measured (Python 3.14, arm64, best-of-5):

dedupe was previously recorded at 1.61× and compiled. That number came from a baseline bench-baseline never wrote: the in-place .so survived the pure reinstall, so the "uncompiled" run was the compiled one, and bench_hot.py only writes a baseline when nothing is compiled — it printed a normal table and left the stale file in place. The task now removes the extensions first; against an honest baseline dedupe is below the gate on every run, so it ships pure.

Annotations are C types here

setup.py compiles these with annotation_typing = True, so Cython lowers PEP 484 hints to C types. In these files an annotation is not documentation: a missing one leaves a PyObject* whose every operation goes back through the interpreter, and a wrong one is a wrong C type.

python scripts/hot_types.py           # coverage report
python scripts/hot_types.py --check   # exit 1 on any untyped function or local

Ruff's ANN rules stop at signatures, so this is the only check that looks at locals — and it looks only at the compiled modules, because that is the only place a local annotation changes what runs. Where a name genuinely cannot be lowered (a Rust extension object, say), mark it # hot-types: ignore <reason>; the reason is printed in the report, so an exemption that stops being true is visible rather than silent.

Two modules were dropped from HOT_MODULES by the gate, which is the gate doing its job rather than a shortfall. captcha is bounded by a CSPRNG syscall that compilation cannot touch. textmatch straddled the 1.5× line across eight runs (1.48–1.55): its cost is Python string and dict work Cython cannot lower much, and a marginal win is not worth a CI gate that fails one run in four.

Worth knowing: the first attempt used plain PEP 484 annotations with annotation_typing, and the compiled build came out slower than pure Python (146 vs 85 ns/op on cooldowns) — the classes were still Python objects. Extension types (@cython.cclass) are what pays. annotate=True writes src/cb_core/*.html showing where Python interaction remains.

The migration spec

scripts/spec.py is the source of truth for what is ported. scripts/status.py measures reality — QA scenarios, v2 scenarios, step bindings, an actual test run — and reports the difference:

python scripts/cb.py status            # check the spec against reality
python scripts/cb.py docs-sync         # regenerate the progress board
python scripts/cb.py docs              # read it on :3002
python scripts/cb.py status -- --check # non-zero if the spec and reality disagree

A feature marked done without a ported, passing scenario is a finding, not a footnote. check runs this.

Contributing a change

The whole loop, in the order it actually happens.

1. Branch — feat/<short-name>, fix/<short-name>, docs/<short-name>.

2. Write the scenario first. New behaviour gets one in qa/features/; a ported v1 feature gets v1's behaviour captured before the implementation exists. A feature whose Gherkin is still red does not land.

3. Implement, respecting the layering: the reply path stays cheap, and anything slow — an external API, image compositing — is a cb-worker job.

4. Update the spec when a feature is finished: flip its status in scripts/spec.py, then python scripts/cb.py docs-sync. Never hand-edit a generated frontmatter block; check fails on the drift.

5. Update the docs when a command, a setting or a reply changed. The user-facing pages are docs/site/content/docs/using/.

6. Run the gate.

   python scripts/cb.py fmt      # ruff autofix + format
   python scripts/cb.py check    # lint, types, tests, benchmarks, spec consistency

7. Open a pull request. Its template asks three questions; brief answers are the right ones.

Commit messages

Conventional Commits — cliff.toml builds the changelog out of them, so a release note is exactly as good as the subjects under it.

<type>(<scope>): <what changed, lowercase, no full stop>

One scope per commit. If a change needs two, it is usually two commits. A change to observable behaviour that a group would notice is ! plus a body saying what: feat(core_welcome)!: ….

What review asks about

Behaviour, not style — ruff settled style and it is not up for debate. Expect questions about what a group would notice, which layer the work belongs in, and which scenario proves it. A first contribution that needs two rounds is a normal first contribution.

CI

.github/workflows/ci.yml runs three jobs: lint + tests, the Cython benchmark gate, and migrations + integration tests against a real Citus service.

Day to day, python scripts/cb.py check runs the same commands without a container. Only when you have changed the workflow file is it worth validating the YAML itself:

python scripts/cb.py workflow   # runs act; needs act + a Docker-API socket
                                # (with podman: podman machine start, then
                                #  export DOCKER_HOST=<podman socket path>)

Deploying to a cluster

The chart is deploy/helm/cookiebot. It renders the three services, Valkey, the self-hosted Bot API server and — when citus.enabled is true — a CloudNativePG Cluster and a Database.

Everything below was found by deploying it to a real cluster for the first time. None of it shows up in helm template, and all of it only bites on a first install, which is why a chart that had been reviewed for months still had it.

Validate against an API server, not with grep

helm template proves the YAML renders. It does not prove Kubernetes will accept it. This does:

helm template cookiebot deploy/helm/cookiebot -n <ns> -f <values> \
  | kubectl apply --dry-run=server -f -

It catches things a render never will — imagePullSecrets given as strings instead of {name: …} maps returns

associative list with keys may not have non-map elements

which names the field and not the shape it wanted.

Hook ordering

The migrate Job is a hook so it re-runs on every sync — a plain Job is immutable and the second sync would fail to apply a changed spec. But a hook runs before the ordinary sync phase, so anything it references has to exist by then. It needs a ServiceAccount and the -env ConfigMap, and when either is missing the Job is admitted and no pod is ever created:

Error creating: pods "cookiebot-migrate-" is forbidden: error looking up
service account …: serviceaccount "cookiebot" not found

The sync then waits on a hook that cannot finish, with no pod to inspect.

The fix is sync waves, not more hooks. Making the ConfigMap a hook works once and then breaks the long-running pods, because Argo deletes a hook before recreating it and cb-* read that ConfigMap through envFrom for their whole life. The order now is: ServiceAccount at -5, ConfigMap and Services at 0, the migrate Job at 1 as a Sync-phase hook, the Deployments at 2.

A resource that long-running pods depend on cannot be a hook.

Citus needs to authenticate to itself

Citus registers the coordinator in pg_dist_node as localhost:5432 and reaches its own shards over TCP even when every shard is local. That connection carries no password, and CloudNativePG's generated pg_hba ends at host all all all scram-sha-256, so every distributed write fails with could not connect to shard.

A plain initdb leaves loopback on trust, which is why this passes on a laptop and fails under CNPG. The chart sets the two loopback entries.

Extensions belong on the Database resource

Not in postInitTemplateSQL — creating citus in template1 starts its maintenance daemon there and blocks the CREATE DATABASE … TEMPLATE template1 that CNPG does next, which fails 9 times in 10 and succeeds just often enough to look flaky.

Not in postInitApplicationSQL either — CNPG runs that as the application owner, and these extensions write to pg_catalog.

Both postInit* hooks also run exactly once, at bootstrap, so an extension added to one later is an edit that appears to work and does nothing.

The Bot API server ignores your arguments

The upstream image's entrypoint builds its own command line from TELEGRAM_* environment variables and ends with exec $COMMAND. It never passes "$@" through, so every argument passed to the container is discarded in silence.

That included --local, which is the entire reason the component exists: it is what lets Telegram accept an http:// webhook on a private address. Without it the pod runs, the port answers, and the ingest path is quietly the cloud API instead.

Configure it with TELEGRAM_LOCAL, TELEGRAM_HTTP_PORT, TELEGRAM_WORK_DIR and friends. And it starts as root deliberately — the entrypoint appends --username/--groupname and the server drops privileges itself, so forcing runAsNonRoot gets you

Can't change effective user: Failed to clear supplementary group list

Metrics ports

All three services read CB_METRICS_PORT. If one hardcodes a port instead, the chart's annotation points somewhere nothing is listening, the target is created, every scrape times out, and up sits at 0 against a pod that is perfectly healthy. Check with a query, not with kubectl get pods:

up{namespace="cookiebot-uat"}