Skip to content

Troubleshooting

1. Overview

This guide covers common issues, diagnostic tools, and solutions for ATTEST deployments. Use this reference to quickly resolve problems and optimize your pipeline performance.

2. Common Issues

Build Failures

Step Execution Failures
# Check step logs
attest logs --step build --verbose

# Common causes and solutions:
# 1. Missing dependencies
attest deps check --fix-missing

# 2. Environment variable issues
attest env check --show-missing

# 3. Permission problems
attest permissions check --fix-permissions

# 4. Resource constraints
attest resources check --recommend-limits
# Diagnose cache issues
attest cache diagnose --step build

# Clear corrupted cache
attest cache clear --key abc123... --force

# Rebuild cache index
attest cache reindex --verify

# Disable cache temporarily
attest run --no-cache --step build

Performance Issues

Slow Build Times
# Profile build performance
attest profile build --duration 5m --output profile.json

# Analyze bottlenecks
attest analyze profile.json --format table

# Common optimizations:
# 1. Enable caching
attest config set cache.enabled true

# 2. Parallelize steps
attest config set execution.parallel true

# 3. Optimize Docker layers
attest optimize dockerfile --layer-caching
High Memory Usage
# Monitor memory usage
attest monitor memory --duration 10m

# Check for memory leaks
attest debug memory-leaks --heap-profile

# Optimize memory settings
attest config set resources.memory_limit "4Gi"
attest config set cache.max_size "1Gi"

Verification Failures

Signature Verification Issues
# Debug signature verification
attest verify receipt.yaml --debug

# Common issues:
# 1. Clock skew
attest verify receipt.yaml --tolerance 5m

# 2. Wrong public key
attest keys list --show-fingerprints
attest verify receipt.yaml --key correct-key.pub

# 3. Corrupted receipt
attest receipt validate receipt.yaml --repair
Policy Violations
# Check policy violations  
attest policy check --show-violations

# Debug specific policy
attest policy debug security-scan-required --trace

# Temporarily bypass policy (emergency only)
attest run --policy-override emergency-bypass

Networking Issues

Connection Timeouts
# Test network connectivity
attest network test --endpoints all

# Check proxy settings
attest config show network.proxy

# Configure timeouts
attest config set network.timeout "30s"
attest config set network.retries 3
DNS Resolution Problems
# Test DNS resolution
attest network dns-test --domains required

# Use custom DNS
attest config set network.dns_servers "8.8.8.8,1.1.1.1"

# Bypass DNS for specific hosts
attest config set network.hosts_file "/etc/attest/hosts"

3. Diagnostic Tools

Health Check

# Comprehensive health check
attest health check --comprehensive

# Sample output:
System Health Check
==================
OK ATTEST version: 0.1.2 (latest)
OK Configuration: valid
OK Cache: healthy (85% hit rate)
OK Network: all endpoints reachable
WARNING Memory: high usage (87%)
ERROR Disk: low space (92% full)

Recommendations:
- Increase memory limit or optimize usage
- Clean up old cache entries or increase disk space

System Information

# Gather system information
attest system info --full

# Output includes:
# - ATTEST version and configuration
# - Operating system details
# - Resource usage statistics
# - Network configuration
# - Cache status
# - Recent errors

Debug Mode

# Enable debug logging
export ATTEST_LOG_LEVEL=debug
attest run --debug

# Trace specific operations
attest run --trace=cache,signature --step build

# Debug configuration
attest config debug --show-resolved --show-sources

4. Error Messages

Common Error Patterns

"Command not found"
# Error: attest: command not found
# Solution: Install or update PATH
curl -sSL https://install.attest.continuu.ms | bash
export PATH="$HOME/.attest/bin:$PATH"
"Permission denied"
# Error: permission denied accessing .attest/
# Solution: Fix ownership or permissions
sudo chown -R $USER:$USER .attest/
chmod -R u+rw .attest/
"Cache corruption detected"
# Error: cache corruption detected in object abc123
# Solution: Repair or rebuild cache
attest cache repair --object abc123
# or
attest cache clear --all --rebuild
"Signature verification failed"
# Error: signature verification failed for receipt.yaml
# Solutions:
# 1. Check clock synchronization
sudo ntpdate -s time.nist.gov

# 2. Verify correct public key
attest keys show --public --fingerprint

# 3. Check receipt integrity
attest receipt verify --checksum receipt.yaml
"Policy violation"
# Error: policy violation: security-scan-required
# Solutions:
# 1. Run required security scan
attest run security-scan

# 2. Update policy configuration
attest policy update security-scan-required --disabled

# 3. Request policy exemption
attest policy exempt --policy security-scan-required \
  --reason "legacy-system" --approver manager@company.com

5. Performance Troubleshooting

Identifying Bottlenecks

# Performance analysis
attest performance analyze --period 7d

# Step-by-step timing
attest profile steps --show-details

# Resource utilization
attest monitor resources --duration 1h --export csv

Cache Performance

# Cache hit rate analysis
attest cache stats --detailed

# Cache size optimization
attest cache optimize --target-hit-rate 85%

# Cache cleanup
attest cache gc --aggressive --max-age 7d

Network Performance

# Network latency test
attest network latency --endpoints cache,registry,api

# Bandwidth test
attest network bandwidth --test-upload --test-download

# Connection pooling optimization
attest config set network.connection_pool_size 20

6. Configuration Issues

Invalid Configuration

# Validate configuration
attest config validate --strict

# Show configuration sources
attest config sources --show-overrides

# Reset to defaults
attest config reset --confirm

Environment Variables

# Check environment variables
attest env validate --show-missing --show-conflicts

# Export current environment
attest env export --format yaml > current-env.yaml

# Load environment from file
attest env load --file production.env

File Permissions

# Check file permissions
attest permissions audit --fix-issues

# Set correct permissions
chmod 600 ~/.attest/keys/*  # Private keys
chmod 644 ~/.attest/config.yaml  # Configuration
chmod 755 ~/.attest/bin/*  # Binaries

7. Recovery Procedures

Cache Recovery

# Backup current cache
attest cache backup --output cache-backup.tar.gz

# Restore from backup
attest cache restore --input cache-backup.tar.gz

# Rebuild cache from scratch
attest cache rebuild --from-artifacts

Configuration Recovery

# Backup configuration
attest config backup --output config-backup.yaml

# Restore configuration
attest config restore --input config-backup.yaml

# Reset to factory defaults
attest config factory-reset --keep-keys

Key Recovery

# List available keys
attest keys list --show-backups

# Restore from backup
attest keys restore --backup-id abc123 --confirm

# Generate new keys (emergency)
attest keys generate --emergency --backup-old

8. Monitoring and Alerting

Error Monitoring

# Monitor for errors
attest monitor errors --duration 1h --threshold 5

# Set up error alerts
attest alerts configure error-rate \
  --threshold 10 \
  --window 5m \
  --webhook https://alerts.company.com/webhook

Performance Monitoring

# Performance dashboards
attest dashboard create performance \
  --metrics "build_time,cache_hit_rate,error_rate" \
  --export grafana-dashboard.json

Health Monitoring

# Continuous health monitoring
attest monitor health \
  --interval 30s \
  --alert-webhook https://health.company.com/webhook

9. Advanced Debugging

Core Dumps

# Generate core dump for analysis
attest debug core-dump --on-crash

# Analyze core dump
attest debug analyze-core core.dump --symbols

Memory Profiling

# Enable memory profiling
attest profile memory --duration 10m --output memory.prof

# Analyze memory usage
attest analyze memory.prof --top 20 --format graph

Network Tracing

# Network trace
attest trace network --duration 5m --output network.pcap

# Analyze network trace
attest analyze network.pcap --show-slow-requests

10. Environment-Specific Issues

Docker Issues

# Docker daemon issues
attest docker diagnose --fix-common-issues

# Docker permission issues
attest docker fix-permissions --user $USER

# Docker storage issues
attest docker cleanup --remove-unused --free-space 2GB

Kubernetes Issues

# Kubernetes connectivity
attest k8s diagnose --namespace attest-system

# Resource constraints
attest k8s resources check --recommend-limits

# RBAC issues
attest k8s rbac verify --fix-permissions

CI/CD Platform Issues

# GitHub Actions issues
attest ci diagnose --platform github-actions
attest ci logs --workflow-run 123456789

# GitLab CI issues  
attest ci diagnose --platform gitlab-ci
attest ci logs --job-id 987654321

11. Getting Help

Support Channels

# Community support
attest community --topic troubleshooting

# Documentation search
attest docs search "error message"

# Submit bug report
attest bug-report --include-logs --include-config

Professional Support

# Enterprise support
attest support create-ticket \
  --priority high \
  --category troubleshooting \
  --attach-diagnostics

# Emergency support
attest support emergency \
  --incident-id INC-123456 \
  --contact +1-555-0123

12. Prevention Best Practices

Regular Maintenance

# Scheduled maintenance
attest maintenance schedule \
  --weekly "cache-cleanup,log-rotation" \
  --monthly "key-rotation,backup-verification"

Monitoring Setup

# Comprehensive monitoring
attest monitoring setup \
  --prometheus \
  --grafana \
  --alertmanager \
  --log-aggregation

Backup Verification

# Regular backup tests
attest backup test \
  --restore-to-temp \
  --verify-integrity \
  --schedule weekly

13. Quick Reference

Emergency Commands

# Stop all running builds
attest stop --all

# Emergency cache clear
attest cache clear --all --force

# Bypass all policies (emergency only)
attest run --bypass-all-policies --emergency-override

# Factory reset (keeps user data)
attest reset --factory --keep-user-data

Status Commands

# Quick status check
attest status

# Detailed system status
attest status --verbose --all-components

# Export diagnostics
attest diagnostics export --full --output diagnostics.tar.gz

This troubleshooting guide provides comprehensive solutions for common ATTEST issues and debugging techniques for advanced problems.