What a CI/CD pipeline does
A CI/CD pipeline is an automated sequence of stages that turns a commit into a running release. It builds the code, tests and checks it, packages one versioned artifact, and deploys that same artifact to staging and then production. If any stage fails, the pipeline stops and the change goes no further.
Pipelines are usually defined in a file in the repository, so changes to the process get reviewed like code. A push or pull request runs the early stages, and a merge to the main branch runs the whole pipeline.
The stages, in order
Order the stages so the cheapest checks run first. A typo should fail the pipeline in seconds, before it spends minutes on integration tests or an image build.
- The build stage installs dependencies from a lockfile and compiles or bundles the code, so the same commit always produces the same output.
- Tests run next: unit tests first because they are fastest, then integration tests against real services such as a throwaway database.
- Security and dependency checks look for known vulnerabilities in dependencies, secrets committed by mistake, and risky code patterns found by static analysis.
- Packaging produces one artifact, such as a container image or an archive, labeled with the commit ID and pushed to a registry.
- Deployment sends that artifact to staging, runs smoke tests against it, and then deploys it to production, with or without an approval in between.
# Generic outline. Each CI system has its own syntax for these ideas.
pipeline:
- stage: build
run: npm ci && npm run build
- stage: test
run: npm test
- stage: checks
run: npm audit --audit-level=high
- stage: package
run: |
docker build -t registry.example.com/shop:$COMMIT_SHA .
docker push registry.example.com/shop:$COMMIT_SHA
- stage: deploy-staging
environment: staging
run: ./deploy.sh staging registry.example.com/shop:$COMMIT_SHA
- stage: smoke-test
run: ./smoke-test.sh https://staging.example.com
- stage: deploy-production
environment: production
approval: required
run: ./deploy.sh production registry.example.com/shop:$COMMIT_SHABuild the artifact once, then promote it
Build the artifact once and deploy that exact artifact to every environment. If you rebuild for production, you ship something that was never tested. A dependency may have updated, a base image tag may point somewhere new, or a build flag may differ.
Identify each artifact by something that cannot change, such as a container image digest or a commit-based tag you never overwrite. Promotion then means telling the next environment to run an artifact ID that already passed the previous one.
Differences between environments belong in configuration supplied at deploy time, such as environment variables or mounted files. Database addresses, API endpoints and flag defaults go there, so the artifact itself contains no environment names and no secrets.
Environments and approvals
Most pipelines deploy to at least two environments: staging, which mirrors production on a smaller scale, and production. Staging is only useful if it runs the same artifact, the same deployment steps and the same kind of infrastructure as production.
An approval step is the manual gate that makes a pipeline continuous delivery rather than continuous deployment. Protect it by limiting who can approve and by allowing production deploys only from the main branch. Show approvers what is changing, such as the commits and test results since the last release.
Database migrations need their own care, because old and new versions of the code run side by side during a rollout. Change the schema in backward-compatible steps: add a new column, deploy code that uses it, and remove the old column in a later release. This pattern is often called expand and contract.
Rollbacks, gradual rollouts and feature flags
A rollback redeploys the previous artifact, which is fast because it is still in the registry. Practice it before you need it, and check that the previous version still works with the current database schema.
Gradual rollouts limit how many users see a bad release. A rolling deploy replaces instances a few at a time. Blue-green deployment runs the new version beside the old one and switches all traffic at once. A canary release sends a small share of traffic to the new version first and widens it only while error rates stay normal.
Feature flags separate deploying code from releasing a feature. Unfinished work can ship switched off, and turning a flag off is usually faster than a rollback. Remove each flag once its feature is fully released, or the code fills up with branches nobody tests.
Keep secrets out of code, artifacts and logs
A pipeline holds credentials to pull dependencies, push images and deploy, which makes it an attractive target. Store those secrets in the CI system's secret store or a dedicated secrets manager, and inject them at run time into the job that needs them.
- Never commit secrets to the repository or bake them into an artifact or image, where anyone who can pull it can read them.
- Give each job only the credentials it needs. The test stage has no reason to hold production deploy keys.
- Keep secrets away from jobs that run code from untrusted pull requests, such as those from forks of a public repository.
- Prefer short-lived credentials. Many CI systems can exchange a signed OpenID Connect identity token for temporary cloud credentials, so no long-lived key is stored at all.
- Most CI systems mask known secret values in logs, but masking matches only the exact stored value. A secret that is encoded or split before it is printed can still leak.
- Rotate a secret as soon as it appears somewhere it should not. Deleting the commit or log line is not enough, because Git history and copies of the log can keep it.