sync

Refreshes SQLite from one or both archive sources.

By default, sync runs both live/local sources and does not import the Git snapshot first:

• Discord bot-token sync for bot-visible guild data
• local Discord Desktop cache import for classifiable cached messages and proven DMs

Use update when you want to pull/import the shared Git snapshot. Routine imports upsert changed shards without deleting local cache rows. If you intentionally want a sync run to import the snapshot before live deltas, pass --update=auto for the configured stale update mode (merge by default) or --update=force for an exact replacement. --no-update is accepted as an explicit no-op alias for the default.

Run one explicit --full pass when you want a complete historical guild archive. Use plain sync afterward for frequent latest-message and desktop-cache refreshes.

#Usage

discrawl sync
discrawl sync --update=auto
discrawl sync --update=force
discrawl sync --no-update
discrawl sync --full
discrawl sync --full --all
discrawl sync --guild 123456789012345678
discrawl sync --guilds 123,456 --concurrency 8
discrawl sync --source both      # default: bot API + desktop cache
discrawl sync --source discord   # bot API only; aliases: key, bot, api
discrawl sync --source wiretap   # desktop cache only; aliases: desktop, cache
discrawl sync --guild 123456789012345678 --all-channels
discrawl sync --channels 111,222 --since 2026-03-01T00:00:00Z
discrawl sync --with-embeddings
discrawl sync --with-media

#Sources

#Bot sync modes

#Flags

• --source <both|discord|wiretap> - which archive sources to read
• --update <auto|force|never> - apply the configured stale snapshot update mode, force an exact replacement, or skip snapshot import before live deltas
• --full - historical backfill (slow on large guilds)
• --all-channels - broader incremental sweep across every stored channel/thread
• --latest-only - explicit latest-only run (also the default for untargeted sync)
• --all - ignore default_guild_id and fan out across every discovered guild
• --guild <id> / --guilds <id,id> - target specific guilds
• --channels <id,id> - target specific channels (forum ids expand to threads)
• --since <RFC3339> - limit initial history and --full backfill to messages at or after this timestamp
• --concurrency <n> - override worker count (default auto-sized: floor 8, cap 32)
• --skip-members - refresh guild/channel/message data without crawling members
• --with-members - refresh guild members even during the default latest-only sync; fail if the member crawl cannot complete
• --with-embeddings - also enqueue changed messages into embedding_jobs
• --with-media - after sync, download missing attachment media into cache_dir/media

#Notes

• --latest-only is the default for untargeted sync. Use --all-channels to opt out without doing a full historical crawl.
• --with-media records expired or removed Discord CDN URLs as failed fetches with the HTTP status, commonly 404.
• --with-media updates the local cache only; run publish --push afterward to include cached non-DM media in the Git backup as gzip-compressed files.
• --since does not mark older history as complete, so a later sync --full without --since can continue the backfill.
• Targeted bot syncs (--source discord --channels <ids>) refresh Desktop-imported channel placeholders and their parent metadata from Discord before crawling messages. Unavailable metadata or missing parent or permission evidence fails the sync visibly; ordinary cached channels keep their existing behavior. Ancestor metadata refreshes do not crawl ancestor messages or reset history and archive cursors. This is still a message sync: --since limits initial history and backfill, not fetching new messages after an existing cursor.
• Long runs emit periodic progress logs to stderr.
• Heartbeat logs (message sync waiting) name the oldest active channel and per-channel page activity if in-flight channels stop completing for a while.
• Every run ends with a message sync finished summary.
• Each channel crawl has a bounded runtime budget; pathological channels are deferred and retried next sync.
• Guild and archived-thread pagination reports a cursor error if Discord repeats a page instead of continuing indefinitely.
• Full message pages with missing or repeated cursors stop the channel crawl with an error and preserve its last usable backfill checkpoint.
• Retryable failures and unavailable-channel markers are tracked per channel; stale unavailable markers are cleared after a later successful crawl.
• A channel that fails with missing access carries an unavailable marker, and routine syncs pass over it while the marker is inside its seven-day window. The next routine sync after the marker reaches seven days retries the channel. To retry sooner: sync --guild <id> --full attempts every channel in the guild immediately, marked ones included, and sync --channels <id> retries a single channel immediately. doctor reports unavailable_markers_active (inside the window, so currently passed over) and unavailable_markers_expired (due for another attempt).
• Full sync retries marked channels even when their older history is complete. When only unavailable channels remain in the cached backlog, it also discovers new channels; the retry and discovery phases do not request the same channel twice.
• Marker cleanup is best-effort, so one missing local sync-state row cannot crash the run.
• Member refresh is best-effort and gives up after five minutes without a caller-supplied deadline. Routine latest-only syncs skip it unless --with-members is set.
• Routine refreshes keep a per-parent archived-thread cursor, so they discover threads archived between runs without rescanning the historical thread catalog.
• When the archive is already complete, sync --full reuses backlog markers and the same incremental thread discovery instead of revisiting every stored archived thread.
• Completed channels with no stored messages are verified with a full crawl when Discord still reports a last message. Routine sync skips channels verified empty; --full rechecks them. A --since window does not initiate this recovery.
• Interrupted recovery resumes from its own saved page checkpoint on the next unwindowed sync, including after cancellation or process termination. Messages ingested in the meantime do not mark the older history complete.
• Failed recovery leaves partial history resumable. After cancellation, restoring a still-empty channel's completion marker uses the existing five-second failure-cleanup budget.

#Re-fetching Components V2 messages archived with empty text

Discrawl versions before Components V2 support stored messages sent with the IS_COMPONENTS_V2 flag (32768), such as app-posted GitHub cards, without their component tree or text. This affects both bot sync and Desktop cache imports. Those rows cannot be rebuilt locally, and routine runs don't re-read them: sync, --all-channels, --full and --channels only request messages after each channel's stored cursor, and wiretap skips cache files that haven't changed. Repair them with the fixed binary:

#Prepare

1. Back up the archive. sqlite3 .backup takes a consistent copy even while a writer is running:

``bash DB="$(discrawl status | sed -n 's/^db=//p')" sqlite3 "$DB" ".backup '$DB.before-components-v2'" ``

To roll back later, stop every sync/tail, then run cp "$DB.before-components-v2" "$DB" && rm -f "$DB-wal" "$DB-shm".

1. Stop every sync, tail and scheduled sync that writes to this archive.

#Bot-synced rows

1. List the affected channels, their broken-row counts and oldest broken message ids, and save the channel ids:

``bash BROKEN="deleted_at is null and json_extract(raw_json, '\$.flags') & 32768 and json_type(raw_json, '\$.components') is null" sqlite3 -header "$DB" "select channel_id, count(*) as broken, min(cast(id as integer)) as oldest_id from messages where $BROKEN group by channel_id" CHANNELS="$(sqlite3 "$DB" "select group_concat(distinct channel_id) from messages where $BROKEN")" ``

1. Rewind each affected channel's latest_message_id cursor to just before its oldest broken message:

``bash sqlite3 "$DB" " update sync_state set cursor = (select cast(min(cast(id as integer)) - 1 as text) from messages where $BROKEN and sync_state.scope = 'channel:' || channel_id || ':latest_message_id') where scope in (select distinct 'channel:' || channel_id || ':latest_message_id' from messages where $BROKEN)" ``

1. Re-fetch those channels. The crawl moves forward from the rewound cursor and rewrites every message it receives, including raw_json, normalized_content and the search indexes. When it finishes, the cursor is back at the channel head:

``bash discrawl sync --source discord --channels "$CHANNELS" ``

1. Verify. The broken-row count should be 0, and a search for the text of a known card should find it:

``bash sqlite3 "$DB" "select count(*) from messages where $BROKEN" discrawl search "<text from a known card>" ``

Any rows that remain are messages Discord no longer returns, for example because they were deleted. List them with the step 2 query.

#Desktop-imported rows

Desktop imports keep a sanitized raw_json without flags or components, so broken Desktop rows can't be counted exactly. The closest check is Desktop rows that have no text and no attachments. It also matches messages that are genuinely empty, such as sticker-only messages.

1. Count the candidates:

``bash DESKTOP="deleted_at is null and json_extract(raw_json, '\$.source') = 'discord_desktop' and trim(coalesce(normalized_content, '')) = '' and has_attachments = 0" sqlite3 "$DB" "select count(*) from messages where $DESKTOP" ``

1. Clear the Desktop file index so the next import rescans every cache file instead of skipping unchanged ones, then re-import:

``bash sqlite3 "$DB" "delete from sync_state where scope like 'wiretap:file_index:%'" discrawl wiretap ``

1. Verify. The candidate count should drop by the number of recovered messages, and a search for the text of a known card should find it:

``bash sqlite3 "$DB" "select count(*) from messages where $DESKTOP" discrawl search "<text from a known card>" ``

This only recovers messages that Discord Desktop still has cached. If a remaining candidate is in a guild channel the bot can read, repeat steps 2-5 with BROKEN="$DESKTOP and guild_id <> '@me'". Keep only those channels in CHANNELS, because a targeted sync fails on channels the bot cannot read. The rewind and --channels sync then fetch those messages through the bot.

#Publish

1. Publish the repaired archive:

``bash discrawl publish --push ``

#See also

• Sync sources
• tail
• update