Fixing Our CI/CD Pipeline: Submodules, Empty Commits, and the QA Gate

Our site pipeline had a recurring problem: deployments that worked locally failed silently in GitHub Actions, or succeeded in Actions but produced a broken site. Here's the root cause analysis and what we changed.

The three failure modes

1. Build succeeds, site content missing — the output directory was generated but the git commit step failed silently because there was nothing new to commit. The deploy step ran anyway and pushed a stale build.
2. Theme submodule not initialized — the theme is a git submodule. Fresh Actions runners clone the repo but don't automatically initialize submodules. The build proceeded with an empty theme directory, producing an unstyled build that nonetheless exited with code 0.
3. Cache invalidation lag — CDN-cached pages served stale content after a new deployment.

Fix 1: Submodule initialization

- uses: actions/checkout@v4
  with:
    submodules: true        # ← this was missing
    fetch-depth: 0

Without submodules: true, the build finds the theme reference, looks in the theme directory, finds nothing, and falls back to default templates — completely unstyled.

Fix 2: The empty commit guard

# Before — always commits, fails when nothing changed
git add public/
git commit -m "Deploy"
git push

# After — skips if nothing changed
git add public/
git diff --staged --quiet || git commit -m "chore: deploy $(date -u +%Y-%m-%dT%H:%M:%SZ)"
git push

git diff --staged --quiet exits 0 (no changes) → skip commit. Exits non-zero (changes present) → commit and push.

The automated QA gate

To catch these failures without manual checks, we added a verification step to the pipeline:

1. Fetches the live site URL after a deploy
2. Checks for the presence of the latest post title in the HTML
3. Verifies the Last-Modified response header is newer than the previous deploy timestamp
4. Fails the pipeline if either check doesn't pass

This converts deployment verification from a manual check into an automated gate.

---

Historical note: this postmortem covers the retired Hugo-based pipeline. The site has since moved to a direct-deploy model.