Design a CI/CD Platform (GitLab CI / GitHub Actions)

Pipeline DAG Execution

Runner Job Lifecycle

1. Webhook Ingestor and VCS Integration

The pipeline spanning webhook ingestion, compilation, DAG scheduling, and runner execution forms the primary backbone. Matrix fan out and multi tenant fair sharing provide the deeper staff level scheduling discussion.

Push events from Code Hosting Platform (GitHub) are processed idempotently, ensuring duplicate webhook deliveries never create duplicate pipeline runs.

Git push triggers pipeline (boundary between VCS and CI/CD):

  1. Developer: git push origin feature-branch
  2. Code hosting Git service (Gitaly for GitLab): receive-pack updates the ref and triggers the repository hook
  3. Code hosting event bus (Kafka in this design): publish git.push_event { repo_id, ref, sha, actor, event_type }
  4. CI/CD Webhook Ingestor consumes push_event:
     a. Validate provider signature, issuer metadata, timestamp, and delivery ID before parsing the payload
     b. Reject duplicate delivery IDs using the durable webhook deduplication record, or return the existing pipeline ID for an already processed delivery
     c. Match branch filter (e.g. refs/heads/main, refs/heads/feature/*)
     d. Load pipeline config from repo (.gitlab-ci.yml / .github/workflows/ci.yaml)
     e. Compile YAML into a DAG of jobs (stages, needs, matrix, if: conditions)
     f. In one PostgreSQL transaction, insert pipeline_run, job rows, dependency rows, the webhook delivery dedup record, and outbox events
     g. Outbox publisher emits each root job as a durable dispatch event to the priority lane selected by trigger and job priority
  5. Scheduler dispatches when a runner is available and dependencies are satisfied
  6. Outbox publisher emits durable pipeline and job events to Kafka, while status callbacks PATCH pipeline status back to the VCS commit checks API

PR events: pull_request.opened/synchronize trigger the same flow with the merge commit SHA
Manual: POST /repos/{id}/pipelines { ref, variables }: bypasses webhook ingestion but enters the same compiler, persistence, and scheduling path

Idempotent ingestion uses the provider delivery ID as the webhook deduplication key and persists that record in the same PostgreSQL transaction that creates the pipeline. This prevents retries from creating a second pipeline. The transactional outbox then publishes the resulting events to Kafka without requiring the database write and Kafka publish to be one distributed transaction.

2. Pipeline Compiler (YAML to DAG)

The compiler translates pipeline YAML into an executable directed acyclic graph at trigger time, verifying that all needs: dependencies form a valid acyclic structure before enqueueing jobs.

• Parse: Loads configuration from the repository at the trigger commit SHA via internal VCS APIs rather than public git clone operations.
• Validate: Confirms acyclic needs: graph structures, resolves registered runner labels, and verifies that requested secret paths exist in the configured secret scope.
• Expand: Evaluates matrix build axes, expands included configuration templates, and computes environment variable inheritance.
• Persist: Records pipeline_runs, jobs, dependency rows, and webhook deduplication state in PostgreSQL before emitting durable outbox events.
• Enqueue: Moves jobs with zero unsatisfied prerequisites to the READY state and publishes them to the Kafka dispatch topic.

3. DAG Scheduler Patterns ⭐

PostgreSQL stores the authoritative DAG state, while Redis dependency counters act as a derived fast path. Parent completion decrements a child counter atomically, and repair jobs can rebuild counters from persisted dependencies after Redis loss. Pipeline stages enforce sequential synchronization barriers between execution tiers.

• State machine: Manages transitions through PENDING, READY, QUEUED, RUNNING, and terminal states (SUCCEEDED, FAILED, SKIPPED, CANCELLED).
• Dependency resolution: Executes a guarded Redis DECR deps:{child} after an idempotent parent completion event, automatically enqueueing the child when the counter reaches zero. Durable job state remains authoritative for recovery and reconciliation.
• Stage barriers: Applies implicit dependencies across stages for jobs without explicit cross stage needs, ensuring those jobs wait for the preceding stage before becoming eligible.
• Failure propagation: Marks dependent children as SKIPPED when a parent fails, unless the job explicitly sets allow_failure: true.
• Concurrency groups: Enforces concurrency: deploy-prod rules to guarantee only one active job runs per group per branch, managed through distributed locking.

4. Work Queue and Job Leasing ⭐

Priority lanes and lease based execution limit duplicate work by using durable dispatch records, a Redis duplicate publication guard, and atomic leases on worker claims, leveraging principles from Message Queues Fundamentals. The design still assumes at least once delivery, so externally visible side effects require idempotency.

Topic: pipeline-events
  Partitions: 256 (partition by repo_id for ordering per repository)
  Retention: 30 days
  Events: pipeline.created, pipeline.started, job.queued, job.started, job.completed,
          job.failed, job.cancelled, artifact.uploaded
  Payload: { event_id, tenant_id, repo_id, pipeline_id, job_id, attempt_id?, status, trace_id }
  Producer: transactional outbox publisher from PostgreSQL state changes
  Idempotency: event_id is unique for publication and consumers tolerate redelivery
  Consumer groups: lifecycle, webhook fanout, metrics indexer, and audit consumers use independent groups

Topic: jobs.dispatch (priority lanes)
  jobs.dispatch.p0: merge queue, protected branch (32 partitions, key=repo_id)
  jobs.dispatch.p1: default push/PR (128 partitions, key=tenant_id)
  jobs.dispatch.p2: scheduled/nightly (64 partitions, key=tenant_id)

Dispatch envelope: { event_id, job_id, attempt_id?, tenant_id, repo_id, priority, trace_id }
Consumer: Runner Fleet Manager: partition consumers share work, while scheduler quotas provide tenant isolation
Consumer: Webhook fanout: notify code hosting integrations, Slack, and deployment APIs
Consumer: Metrics indexer: track queue wait time and success rate per repo in metrics storage

DLQ: jobs.dispatch.dlq: alert on any message and enable manual replay via console

• Dispatch guard: Uses SET job:dispatch:{id} NX EX 60 only as a duplicate publication guard. The PostgreSQL outbox record remains authoritative and is retried until Kafka acknowledges publication, so a scheduler crash after acquiring the Redis guard but before Kafka publication does not lose the dispatch.
• Execution lease: Runners claim jobs via atomic SET job:lease:{id} NX EX 1800 commands, then record the claim against the authoritative attempt using a conditional database update. If the database claim fails, the runner releases or abandons the Redis lease and does not execute the job. Every heartbeat and terminal status update includes the current attempt and lease token, and the scheduler accepts the update only when those values still match the authoritative attempt. The durable terminal state is committed before lease cleanup is attempted.
• Reaper daemon: Scans active jobs with expired leases every 30 seconds. Before creating a replacement attempt, it checks the authoritative PostgreSQL state. A CANCEL_REQUESTED job becomes CANCELLED instead of being retried. Other expired attempts are requeued or failed according to retry policy.

5. Fair Share Queue Manager ⭐

Weighted fair share scheduling prevents high volume tenants from monopolizing the runner pool, while dedicated merge queue lanes reserve capacity for time sensitive integration checks.

Multi-tenant fair scheduling:

Per-tenant limits (PostgreSQL policy + Redis counters):
  max_concurrent_jobs: 500 (enterprise: 5000)
  max_queue_depth: 2000
  weight: 1 (free) | 10 (paid) | 100 (enterprise)

Dispatch ordering:
  separate priority lanes select P0, P1, or P2 first
  within each tenant lane, encode virtual service time, enqueue_time, and job_id into a deterministic Redis ZSET score
  aging reduces effective virtual service time as wait time increases

Global coordinator (every 1s, driven by the active tenant set rather than a scan of all 100K tenants):
  1. Compute fair share credits from active tenant weights
  2. fair_share_slot = total_available_runners / sum(active_tenant_weights)
  3. Carry fractional credits across rounds, convert them into integer admissions, and redistribute unused slots among eligible tenants
  4. Admit jobs up to each tenant's concurrency and queue limits
  5. P0 jobs use a reserved 20% runner capacity budget, though idle reserved capacity may be loaned to P1/P2
  6. Reconcile Redis running counters against authoritative PostgreSQL job states after failover

Starvation: if job.wait_time > 2h and priority=P2: promote to P1

6. Runner Fleet Manager

Autoscaling policies track Kafka consumer lag, while pre provisioned warm pools eliminate cold start delays for latency-critical feedback loops.

• Ephemeral sandboxes: Provisions dedicated, single use Kubernetes pods or virtual machines per job that are destroyed immediately upon task completion.
• Warm pool: Maintains approximately 10% of peak capacity pre warmed to absorb burst traffic without container initialization latency.
• Autoscaling: Drives horizontal pod autoscaling on runner deployments based on Kafka consumer lag and queue depth.
• Self-hosted agents: Enables long lived customer hosted agents to register via secure tokens and poll POST /v1/runners/claim, delegating compute costs and network isolation to the user.
• Target labels: Matches job requirements such as runs-on: [linux, gpu, large] against appropriate runner pools.

7. Runtime Secrets Injection ⭐

Secrets are never baked into runner images or stored in repository YAML. Runners obtain short lived credentials from Secrets Manager & KMS via OIDC federation. The runner agent performs best effort process cleanup on job completion, while withholding secret access on fork pull requests.

Runtime secrets injection (Secrets Manager and KMS integration):

  Runner startup (before any user script):
  1. Runner presents OIDC JWT: { sub: "repo:org/app:ref:refs/heads/main", aud: "cicd" }
  2. Control plane validates JWT signature, issuer, audience, subject, and expiry, then maps it to secret scope: repo/org/app/env:production
  3. Batch fetch from Secrets Manager: GET /v1/secrets/runtime?scope=...&ttl=1800
     - Database password (dynamic lease 30 min)
     - NPM_TOKEN (rotated, masked)
     - AWS_ROLE_ARN: exchange OIDC for STS credentials with no stored AWS keys
     - Refresh short lived cloud credentials before expiry while the job remains authorized
  4. Inject secrets into the runner process namespace without writing them to persistent disk
  5. Log scrubber registers secret values for runtime redaction
  6. On job termination (success/fail/cancel): revoke dynamic leases and perform best effort process cleanup

Security note: environment variables can be inherited by child processes, so masking and sandboxing are defense in depth. Secret scope is based on the trusted repository, ref, environment, and approval policy rather than repository membership alone. Trusted job code must still be treated as able to exfiltrate any secret it is authorized to receive.

Fork PRs from untrusted contributors: secrets scope is empty, caches are isolated from trusted branches, and only read only repository tokens are available

8. Artifact Store and Layer Cache

Artifact and log streaming bypasses the central control plane, using direct presigned uploads to Blob Storage (S3) with incremental chunk flushes to survive runner crashes.

• Artifact storage: Runners upload build outputs directly via short lived, job scoped presigned PUT requests to s3://artifacts/{tenant}/{pipeline_id}/{job_id}/, with content length and checksum validation before completion is recorded. Finalized artifacts are immutable and referenced by an artifact manifest in PostgreSQL.
• Cross-job passing: Dependent jobs download upstream artifacts using the pipeline ID and upstream job identifier.
• Layer caching: Restores content addressable cache archives at job initialization and persists updated layers on success when cache paths change.
• Log streaming: Flushes logical log checkpoints every 5 seconds during execution, batches them into larger ordered object-storage segments, and uses job and chunk sequence metadata so missing or duplicate chunks can be detected during assembly. Partial output therefore survives unexpected runner terminations without requiring one object-storage request per 5-second checkpoint.