Gyaan

Pipeline Design

intermediate ci-cd pipelines github-actions stages

A CI/CD pipeline is only useful if it’s fast, reliable, and catches real problems. A badly designed pipeline either takes 40 minutes (so everyone ignores it) or passes when it shouldn’t. Let’s build one that actually works.

Pipeline Stages

Most production pipelines follow this order:

Pipeline Stage Order
1. Lint        ← catch style issues in seconds (cheapest check)
2. Build       ← compile/transpile the code
3. Unit Test   ← fast isolated tests
4. Integration  ← tests that hit real databases/APIs
5. Security    ← dependency audit, SAST scan
6. Deploy      ← push to staging or production
Fast + cheap checks first → Slow + expensive checks last

The rule of thumb: put the fastest, cheapest checks first. If linting fails in 5 seconds, there’s no point waiting 10 minutes for integration tests.

Key Concepts

Artifacts — output from one stage that the next stage needs. For example, the build stage produces a compiled binary, and the deploy stage uses it. We don’t want to rebuild in every stage.

Caching — storing node_modules or .m2 directories between runs so we don’t re-download every dependency each time. This alone can cut pipeline time in half.

Parallel jobs — lint, unit tests, and security scans don’t depend on each other. Run them at the same time.

A Real GitHub Actions Workflow

name: CI/CD
on:
  push:
    branches: [main]
  pull_request:

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: 'npm'                    # cache node_modules automatically
      - run: npm ci
      - run: npm run lint

  test:
    runs-on: ubuntu-latest
    needs: lint                           # only run if lint passes
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: 'npm'
      - run: npm ci
      - run: npm test

  deploy:
    runs-on: ubuntu-latest
    needs: test                           # only run if tests pass
    if: github.ref == 'refs/heads/main'   # only deploy from main
    steps:
      - uses: actions/checkout@v4
      - run: echo "Deploying to production..."
      # real deploy step goes here

Notice how needs creates the dependency chain: lint → test → deploy. The cache: 'npm' line saves us from re-downloading packages every run.

GitLab CI Equivalent (Quick Look)

# .gitlab-ci.yml
stages:
  - lint
  - test
  - deploy

lint:
  stage: lint
  script: npm run lint
  cache:
    paths:
      - node_modules/

test:
  stage: test
  script: npm test

deploy:
  stage: deploy
  script: ./deploy.sh
  only:
    - main

The only difference is syntax. GitLab uses stages and stage: keywords, while GitHub uses jobs and needs:. The mental model is the same.

Pipeline Design Tips

  • Keep it under 10 minutes — if it’s longer, people stop waiting and merge without checking
  • Fail fast — put linting and type-checking first
  • Cache aggressively — dependencies, Docker layers, build artifacts
  • Use matrix builds to test across Node 18/20/22 or Python 3.10/3.12 in parallel
  • Don’t deploy on PR — only deploy when code lands on main
  • Store secrets in the CI provider (GitHub Secrets, GitLab Variables) — never in the repo

References