Mongo -> Citus backfill
all 8 collections. 7 through the pure ETL mappers, idempotent upserts; randomdatabase through cb.py backfill-random, which downloads what each v1 pointer references and writes a real media_objects row - resumable by telegram_file_id, so a second run costs a query per pointer, not a download
- v1 scenarios
- —
- Ported
- —
- Green
- —
- Failing
- —
all 8 collections. 7 through the pure ETL mappers, idempotent upserts; randomdatabase through cb.py backfill-random, which downloads what each v1 pointer references and writes a real media_objects row - resumable by telegram_file_id, so a second run costs a query per pointer, not a download
COOKIEBOT-backend/core/domains/*.javaWhat it does
Moves every group's history out of v1's MongoDB and into v2's database: their settings, rules, welcome messages, members, blacklist, sticker pool and random media library. Nothing a group types changes — the point is that on cutover day nothing a group has is lost either.
Behaviour that must not change
Every write is an upsert on the natural key, which is what makes a cutover possible without a maintenance window: run the import now while v1 is still serving, again next week, and once more at cutover to pick up the delta. Two runs produce the same rows, never duplicates and never resurrected defaults.
Anything that cannot be moved faithfully is skipped and counted, never
guessed. Every id in v1's database is a string; one that will not parse as a
Telegram id is reported rather than invented. v1's "9999" "no topic pinned"
sentinel becomes a real NULL, so v2 has one way to say that instead of two.
The random-media library is the one collection that needs more than a mapping.
v1 stored a pointer to a Telegram message, never the picture; v2 stores the
picture. So cb.py backfill-random (also the cutover's random step)
downloads what each pointer references, hashes the real bytes and writes the
same kind of row the live bot writes when someone posts a photo today —
indistinguishable afterwards, including to /random itself. Pointers whose
message was deleted, or whose file id has expired, are counted as unavailable:
that is a fact about v1's data and not something this can fix.
How to verify it
qa/integration/test_importer.py and
qa/integration/test_backfill_random_media.py run both halves against a real
Citus cluster — including that a second backfill run skips an already-imported
pointer without downloading it, which is what makes a resumed cutover cheap.
Run with python scripts/cb.py test-integration. By hand:
python scripts/cb.py cutover --dry-run reads everything, writes nothing, and
prints what each step would do.
Multi-tenant registry and schema
tenants table + registry + dispatch gate + llm_overrides/storage_prefix; handler packs read per update by cb_gateway/packs.py, first family legacy_custom
Fleet-wide analytics and the group directory, owners only
net-new: v1 had no analytics at all, and x_analytics_api deliberately has no fleet-wide endpoint. Seven owner-only reads (overview, daily, top groups, commands, llm, directory, tenant) behind a new admin:read scope that /oauth2/token grants only when the subject is a tenant owner or CB_OWNER_ID. 403 here rather than the group endpoints' 404: /admin has no chat id to hide. The one place in the codebase where a query does not filter on group_id - confined to cb_core/platform_analytics.py, aggregate-only, off the reply path, and argued in that module's docstring