Running ByteMatch

Architecture

Processes

ProcessCodeDoes
WebNext.js 16 app router, src/appPages, JSON API, badges, upload API, GitHub webhook endpoint, sign-in
Workersrc/worker, bundled to dist/main.jsRuns background jobs: ingesting artifacts, polling, backfill, decomp.dev import, PR comments, pruning
PostgreSQLAll data, and the job queue

Both web and worker are stateless, so any number of each can run. They communicate only through the database: the web process enqueues jobs (for example on a webhook or a "Poll now" click) and workers claim them.

The job queue is pg-boss, stored in the pgboss schema of the same database. There is no Redis.

Code layout

DirectoryContents
src/appPages and routes
src/reportobjdiff report parsing, measures, diffs; vendored report.proto and changes.proto in src/report/proto
src/badgeSVG and PNG badges
src/dbSchema, store, migrations
src/githubGitHub App, artifacts, comments
src/ingestIngest pipelines
src/workerJob handlers
src/libConfiguration, queue, crypto

Data model

Defined in src/db/schema.ts; migrations live in drizzle/.

TableHolds
projectsOne row per repository: owner, repo, GitHub repository id (survives renames), display settings, default version / category / branch, artifact pattern, workflow restriction, split versions, PR comments flag, enabled flag, GitHub App installation id, last poll time and last ingest error.
reportsOne version's progress at one commit: version, commit SHA, message and timestamp, source (artifact, upload or import), workflow run id, the 16 measures and per-category measures (JSON). Unique per project, version and commit. Optionally points to a blob.
report_blobsFull reports, stored as gzip-compressed objdiff protobuf, deduplicated by SHA-256: commits with identical reports share one blob.
usersGitHub users who have signed in.
sessionsSigned-in browsers. The id is the SHA-256 of the cookie value; the user's GitHub token is encrypted with SECRET_KEY.
api_tokensPer-project upload tokens. Only a hash and a short prefix are stored.
pr_commentsThe progress comment on each pull request, so later runs edit it instead of posting another.

Measures are kept forever. Full reports (units and functions) may be dropped after REPORT_RETENTION_DAYS, except each version's latest; history imported from decomp.dev has measures but no blob.

Ingest flow

CI artifacts

  1. A workflow run completes. With the GitHub App installed, GitHub sends a workflow_run webhook to /api/github/webhook; the web process verifies its signature and enqueues ingest-run. Without the App, the poll-all schedule enqueues poll-project for each enabled project every POLL_INTERVAL_MINUTES, and poll-project ingests the new successful runs on the default branch itself (oldest first), as backfill does over a longer stretch of runs.
  2. ingest-run lists the run's artifacts, keeps those matching the project's artifact pattern (and workflow restriction), and downloads them with an installation token or GITHUB_TOKEN.
  3. Each artifact is unzipped (up to MAX_REPORT_BYTES) and the report parsed. JSON and protobuf reports are accepted, and older objdiff report formats are migrated to the current one.
  4. With split versions, the report is split into one report per top-level version category.
  5. Measures and category measures go into reports; the full report is encoded as protobuf, gzipped, and stored in report_blobs unless a blob with the same SHA-256 exists.

Pull requests

For a workflow run triggered by a pull request, the webhook enqueues pr-comment instead (when the project has PR comments on). The job reads the PR run's report, compares it with the latest report for the same version on the base branch, and creates or updates the PR's comment, recorded in pr_comments.

Upload API

POST /api/v1/projects/{owner}/{repo}/reports checks the bearer token against api_tokens, then runs steps 3 to 5 directly in the web process.

decomp.dev import and backfill

import-decompdev fetches <DECOMP_DEV_URL>/<owner>/<repo>.json?mode=history and the latest full report, and stores them with source import. backfill walks older workflow runs whose artifacts have not expired and enqueues them for ingestion.

Jobs

JobDataDoes
ingest-runproject, run idRead one workflow run's report artifacts.
poll-projectprojectLook for new successful runs on the default branch.
poll-allScheduled: queue poll-project for every enabled project.
backfillproject, max runsWalk older runs whose artifacts have not expired.
import-decompdevprojectCopy history and the latest report from decomp.dev.
pr-commentproject, run idComment on a pull request with its progress.
pruneScheduled: drop full reports past REPORT_RETENTION_DAYS.

Queued jobs with the same key collapse into one, so a burst of webhooks for the same run is processed once. Failed jobs retry up to three times with backoff.

Caching

  • Full reports are stored once per distinct content (SHA-256), so commits that do not change progress add only a reports row.
  • Pages, JSON and badges for the latest commit read from reports, which holds the measures directly; only mode=report, unit and function lists, and PR comparisons need to decompress a blob.
  • Badge text is measured with the bundled DejaVu Sans fonts (src/assets/fonts), which the PNG renderer also uses, so SVG and PNG badges match and need no external service.
  • HTTP caching: badges send Cache-Control: public, max-age=60, s-maxage=60, stale-while-revalidate=600 (GitHub's image proxy honours it), project JSON public, max-age=60, and error responses no-store. A CDN in front can serve badges without reaching the server.
Next
Contributing