Pipeline Format
1. Overview¶
ATTEST uses a declarative YAML format to define CI/CD pipelines. This document provides a comprehensive reference for the pipeline configuration format, including all available options and advanced features.
2. Basic Structure¶
2.1 Minimal Pipeline¶
2.2 Complete Pipeline Structure¶
version: "0.1" # Required: Format version
name: "complete-pipeline" # Required: Pipeline name
description: "Example pipeline" # Optional: Description
# Global configuration
defaults:
cache: true
timeout: "30m"
env:
GLOBAL_VAR: "value"
# Environment definitions
environments:
development:
env:
NODE_ENV: "development"
production:
env:
NODE_ENV: "production"
# Step definitions
steps:
- name: "build"
description: "Build application"
run: "npm run build"
# ... step configuration
# Post-pipeline actions
cleanup:
- name: "cleanup-temp"
run: "rm -rf /tmp/build"
3. Version¶
The version
field specifies the format version:
Supported versions: - 0.1
- Initial format (current)
4. Metadata¶
4.1 Name and Description¶
name: "web-application-build"
description: |
Complete build pipeline for the web application
including testing, building, and security scanning
4.2 Labels and Annotations¶
labels:
team: "frontend"
project: "webapp"
environment: "production"
annotations:
docs: "https://wiki.company.com/webapp-pipeline"
contact: "frontend-team@company.com"
compliance: "sox,hipaa"
5. Global Defaults¶
Set default values for all steps:
defaults:
cache: true
timeout: "15m"
shell: "/bin/bash"
working_directory: "/workspace"
retry:
attempts: 3
backoff: "exponential"
6. Environment Variables¶
6.1 Global Environment Variables¶
6.2 Environment-Specific Variables¶
environments:
development:
env:
DEBUG: "true"
API_URL: "https://dev-api.company.com"
staging:
env:
DEBUG: "false"
API_URL: "https://staging-api.company.com"
production:
env:
DEBUG: "false"
API_URL: "https://api.company.com"
MONITORING: "enabled"
6.3 Secret Variables¶
secrets:
- name: "DATABASE_PASSWORD"
source: "vault:secret/db/password"
- name: "API_KEY"
source: "env:API_KEY"
- name: "SIGNING_KEY"
source: "file:/etc/attest/signing.key"
7. Steps¶
7.1 Basic Step¶
7.2 Complete Step Configuration¶
steps:
- name: "build"
description: "Build the application"
run: "npm run build"
# Dependencies
needs: ["install", "lint"]
# Input/Output specification
inputs:
- "src/**/*.js"
- "src/**/*.ts"
- "package.json"
- "tsconfig.json"
outputs:
- "dist/"
- "build-manifest.json"
# Environment
env:
NODE_ENV: "production"
BUILD_TARGET: "es2020"
# Execution configuration
shell: "/bin/bash"
working_directory: "/app"
timeout: "20m"
# Retry configuration
retry:
attempts: 3
backoff: "exponential"
on_failure: ["network_error", "timeout"]
# Caching
cache: true
cache_key: "build-{{ checksum 'package-lock.json' }}-{{ env.NODE_ENV }}"
# Conditions
condition: "{{ env.CI_BRANCH == 'main' }}"
# Continue on failure
continue_on_error: false
7.3 Multi-Command Steps¶
steps:
- name: "setup-and-test"
run: |
set -e
echo "Setting up environment..."
npm ci
echo "Running tests..."
npm test
echo "Tests completed successfully"
7.4 Conditional Steps¶
steps:
- name: "deploy-production"
run: "kubectl apply -f k8s/"
condition: |
{{ env.CI_BRANCH == 'main' and
env.CI_COMMIT_TAG starts_with 'v' }}
8. Dependencies¶
8.1 Step Dependencies¶
steps:
- name: "install"
run: "npm ci"
- name: "lint"
run: "npm run lint"
needs: ["install"]
- name: "test"
run: "npm test"
needs: ["install"]
- name: "build"
run: "npm run build"
needs: ["lint", "test"]
8.2 Parallel Execution¶
steps:
- name: "install"
run: "npm ci"
# These run in parallel after install
- name: "lint"
run: "npm run lint"
needs: ["install"]
- name: "test-unit"
run: "npm run test:unit"
needs: ["install"]
- name: "test-integration"
run: "npm run test:integration"
needs: ["install"]
# This waits for all parallel steps
- name: "build"
run: "npm run build"
needs: ["lint", "test-unit", "test-integration"]
9. Input/Output Specification¶
9.1 File Patterns¶
inputs:
- "src/**/*.{js,ts,jsx,tsx}" # Glob patterns
- "package.json" # Specific files
- "!src/**/*.test.js" # Exclusions
- "config/" # Directories
outputs:
- "dist/"
- "coverage/"
- "*.log"
9.2 Content-Based Inputs¶
inputs:
- path: "src/"
exclude: ["**/*.test.js", "**/*.spec.js"]
- path: "package.json"
required: true
- path: "config/"
optional: true
9.3 Output Artifacts¶
outputs:
- name: "build-artifacts"
paths: ["dist/"]
retention: "30d"
- name: "test-results"
paths: ["coverage/", "test-results.xml"]
retention: "7d"
public: true
10. Caching¶
10.1 Basic Caching¶
10.2 Custom Cache Keys¶
steps:
- name: "install"
run: "npm ci"
cache: true
cache_key: "npm-{{ checksum 'package-lock.json' }}-{{ env.NODE_VERSION }}"
10.3 Cache Configuration¶
steps:
- name: "build"
run: "npm run build"
cache:
enabled: true
key: "build-{{ checksum 'src/**/*.js' }}"
paths: ["dist/", "build/"]
ttl: "24h"
compression: true
10.4 Cache Strategies¶
cache_strategy:
# Aggressive caching for development
development:
default_ttl: "1h"
compression: false
# Conservative caching for production
production:
default_ttl: "24h"
compression: true
require_exact_match: true
11. Conditions and Control Flow¶
11.1 Simple Conditions¶
11.2 Complex Conditions¶
steps:
- name: "security-scan"
run: "security-scanner scan"
condition: |
{{ env.SECURITY_SCAN_ENABLED and
(env.BRANCH == 'main' or
env.BRANCH starts_with 'release/') }}
11.3 Conditional Execution Based on Changes¶
steps:
- name: "frontend-build"
run: "npm run build"
condition: "{{ changes include 'frontend/**' }}"
- name: "backend-build"
run: "go build"
condition: "{{ changes include 'backend/**' }}"
12. Error Handling¶
12.1 Retry Configuration¶
steps:
- name: "flaky-test"
run: "npm run test:e2e"
retry:
attempts: 3
backoff: "exponential" # linear, exponential, fixed
initial_delay: "5s"
max_delay: "60s"
on_failure:
- "network_error"
- "timeout"
- "exit_code:1"
12.2 Continue on Error¶
steps:
- name: "optional-check"
run: "optional-tool check"
continue_on_error: true
- name: "required-build"
run: "npm run build"
# This runs even if optional-check fails
12.3 Custom Error Handling¶
steps:
- name: "database-migration"
run: "migrate up"
on_failure:
- name: "rollback"
run: "migrate down"
- name: "notify"
run: "send-alert 'Migration failed'"
13. Advanced Features¶
13.1 Matrix Builds¶
matrix:
node_version: ["16", "18", "20"]
os: ["ubuntu-latest", "macos-latest"]
steps:
- name: "test"
run: "npm test"
env:
NODE_VERSION: "{{ matrix.node_version }}"
runs_on: "{{ matrix.os }}"
13.2 Dynamic Steps¶
steps:
- name: "generate-tests"
run: "generate-test-matrix"
outputs: ["test-matrix.json"]
- name: "run-tests"
run: "run-test {{ item.name }}"
for_each: "{{ fromJson(file('test-matrix.json')) }}"
env:
TEST_CONFIG: "{{ item.config }}"
13.3 Service Dependencies¶
services:
postgres:
image: "postgres:13"
env:
POSTGRES_PASSWORD: "test"
ports:
- "5432:5432"
redis:
image: "redis:6"
ports:
- "6379:6379"
steps:
- name: "integration-test"
run: "npm run test:integration"
env:
DATABASE_URL: "postgres://postgres:test@localhost:5432/test"
REDIS_URL: "redis://localhost:6379"
14. Attestation Configuration¶
14.1 Signing Configuration¶
attestation:
sign: true
algorithm: "ed25519"
key_source: "vault:secret/signing-key"
verification:
enabled: true
require_signatures: true
trusted_keys:
- "ed25519:abc123..."
- "ed25519:def456..."
14.2 Custom Attestation¶
attestation:
custom_fields:
build_id: "{{ env.BUILD_ID }}"
commit_sha: "{{ env.GIT_COMMIT }}"
compliance_level: "sox"
policies:
- "require-tests"
- "security-scan"
- "approval-required"
15. Template and Reuse¶
15.1 Step Templates¶
# _templates.yaml
templates:
node_setup: &node_setup
name: "setup-node"
run: |
nvm use {{ env.NODE_VERSION }}
npm ci
inputs: ["package.json", "package-lock.json"]
outputs: ["node_modules/"]
# pipeline.yaml
steps:
- <<: *node_setup
env:
NODE_VERSION: "18"
15.2 Pipeline Includes¶
# common-steps.yaml
common_steps:
- name: "security-scan"
run: "security-scanner scan"
- name: "dependency-check"
run: "npm audit"
# main-pipeline.yaml
version: "0.1"
name: "main-pipeline"
includes:
- "common-steps.yaml"
steps:
- name: "build"
run: "npm run build"
- include: "common_steps"
15.3 Variables and Interpolation¶
variables:
app_name: "my-app"
version: "{{ env.VERSION | default('1.0.0') }}"
build_dir: "dist/{{ variables.app_name }}"
steps:
- name: "build"
run: "npm run build"
env:
OUTPUT_DIR: "{{ variables.build_dir }}"
16. Validation¶
16.1 Schema Validation¶
# Built-in validation
validation:
strict: true # Fail on unknown fields
required_fields: ["name", "version", "steps"]
# Custom validation
custom_validation:
- rule: "steps[*].name must be unique"
- rule: "inputs must exist before step execution"
- rule: "circular dependencies are not allowed"
16.2 Linting¶
# Validate pipeline format
attest pipeline validate pipeline.yaml
# Lint for best practices
attest pipeline lint pipeline.yaml --rules production
17. Comments and Documentation¶
# Pipeline for building and testing the web application
version: "0.1"
name: "webapp-build"
# Global environment variables used throughout the pipeline
env:
NODE_ENV: "production" # Set Node.js environment
steps:
# Install dependencies from package-lock.json
- name: "install"
run: "npm ci"
# Cache node_modules based on lockfile hash
cache_key: "npm-{{ checksum 'package-lock.json' }}"
# Run linting to ensure code quality
- name: "lint"
run: "npm run lint"
needs: ["install"]
# Continue pipeline even if linting has warnings
continue_on_error: true
18. Best Practices¶
18.1 Pipeline Organization¶
- Use meaningful step names that describe the action
- Group related steps logically
- Leverage parallel execution where possible
- Keep steps focused on single responsibilities
- Use descriptive comments for complex logic
18.2 Performance Optimization¶
- Enable caching for expensive operations
- Specify precise inputs to improve cache hits
- Use parallel execution for independent steps
- Minimize output artifacts to reduce transfer time
- Set appropriate timeouts to prevent hanging
18.3 Security Considerations¶
- Never hardcode secrets in pipeline files
- Use minimal required permissions for each step
- Validate all inputs and outputs
- Enable signing for production pipelines
- Implement proper error handling without exposing sensitive data
This pipeline format provides a powerful and flexible way to define CI/CD workflows while maintaining simplicity and readability.