Dev container image cache

​

How it works

1. Hash computation: When starting an environment, Ona computes a hash based on your Dev Container configuration, including:
   • Contents of devcontainer.json
   • Contents of any referenced Dockerfile
   • Digest of any existing container image you reference directly (ensuring automatic cache invalidation when base images like :latest are updated)
2. Cache lookup: Ona checks if an image with this hash already exists:
   • If found in registry: Pulls and uses the cached image
   • If not found: Builds new image, pushes to registry, then uses it
3. Shared caching: The cache is shared across all users within a project. The first team member who creates an environment with a new Dev Container configuration builds and caches the image. All subsequent environments created by any team member in the same project with identical configuration use the shared cached version. When rebuilding an existing environment’s Dev Container, it will also try to pull from the cache but won’t push new images (only the initial Dev Container build when an environment gets created can push to the cache).

​

Enabling and disabling the cache

​

For new runners

​

For existing runners

1. Update your CloudFormation stack to the latest version that supports the cache
2. Go to Settings → Runners in your Ona organization
3. Select your AWS runner
4. Toggle Dev container image cache to enabled
5. Click Save

​

Disabling the cache

1. Go to Settings → Runners in your Ona organization
2. Select your AWS runner
3. Toggle Dev container image cache to disabled
4. Click Save

​

What gets cached

• Base image layers: Your specified base image and any modifications
• Dev container features: All configured features are pre-installed
• Build steps: RUN commands, COPY operations, and other Dockerfile instructions
• Tool installations: Package managers, development tools, and dependencies

• Lifecycle hooks (onCreateCommand, etc.) - these run after the container starts
• User-specific configurations applied at runtime
• Files that change after container creation

​

Automatic cache invalidation for base images

​

When referencing an existing container image

{
  "image": "mcr.microsoft.com/devcontainers/typescript-node:0-18"
}

• Automatic security updates: When the image maintainer publishes security patches, your cache is automatically invalidated
• Latest tag handling: If you use :latest tags, the cache rebuilds whenever the latest version changes
• No manual intervention: You don’t need to manually force cache invalidation when base images are updated

​

When building from a Dockerfile

{
  "build": {
    "dockerfile": "Dockerfile"
  }
}

​

Shared cache behavior

• Team efficiency: Only the first user with a new configuration experiences the full build time
• Consistent environments: All team members use identical container images
• Resource optimization: Eliminates redundant builds across team members

​

Enterprise security considerations

• Configuration integrity: Images are cached based on exact configuration hashes
• Limited push access: Only the initial build can push images; credentials are removed afterward
• Immutable storage: ECR prevents modification of cached images
• Project isolation: Cache access is limited to the same project

​

Security considerations

​

Access isolation

• Project-scoped access: Images are only accessible within the same project
• Hash-based immutability: Once pushed, images cannot be modified
• Credential separation: Distinct push and pull permissions

​

Credential management

• Push credentials: Only granted during the initial Dev Container build when an environment gets created, then removed from the environment
• Pull credentials: Provided to all environments in the project, refreshed periodically
• Temporary access: All credentials use AWS IAM temporary credentials with minimal required permissions

​

Registry security

• Immutable repositories: ECR repositories prevent image tampering once pushed
• AWS IAM controls: Access controlled through AWS session tags and IAM policies
• Regional isolation: Images stored in the same AWS region as your runner

​

Image lifecycle

​

Automatic expiry

• Prevent unlimited storage growth
• Ensure images are periodically rebuilt with security updates
• Reduce storage costs

​

Cache invalidation

• The Dev Container configuration hash changes (any modification to devcontainer.json or referenced Dockerfile)
• The underlying container image changes (e.g., when :latest or other tags are updated with security patches or new versions)
• The cached image has expired
• The cached image is manually deleted from ECR

​

Forcing a new image build

​

Method 1: Modify Dev Container configuration

{
  "name": "My Dev Container",
  "image": "mcr.microsoft.com/devcontainers/typescript-node:0-18",
  // Add a comment to force rebuild: 2024-01-15
  "features": {
    "ghcr.io/devcontainers/features/git:1": {}
  }
}

​

Method 2: Add a comment to Dockerfile

FROM node:18
# Force rebuild: 2024-01-15
RUN npm install -g typescript

​

Method 3: Delete the image from ECR

1. Go to AWS ECR console
2. Navigate to your runner’s repository: gitpod-runner-{runnerID}/projects/{projectID}/image-build
3. Delete the specific image tag
4. The next environment creation will rebuild and cache the image

​

Troubleshooting

​

Cache not working

1. Check runner configuration: Ensure the Dev Container image cache is enabled in runner settings
2. Verify CloudFormation version: Older CloudFormation templates don’t support caching
3. Check AWS permissions: Ensure the runner has ECR access permissions
4. Check runner logs: Look for cache-related errors in the runner’s CloudWatch logs
5. Review environment logs: Look for cache-related error messages
6. Check image digest warnings: If using image: field, look for digest lookup warnings in logs that might indicate network or authentication issues with base image registries

​

Slow startup despite cache

1. Check cache hit: Look for “Using pre-built dev container” in environment logs
2. Verify image size: Large images take longer to pull
3. Check environment type: Docker Compose-based Dev Containers are not supported by the cache
4. Review lifecycle hooks: Lifecycle hooks (onCreateCommand, etc.) are not cached and may install tools that increase startup time. Consider moving these steps to the Docker build where possible
5. Verify project-based environment: Non-project environments do not use the cache

​

Supported configurations

​

Supported

• Standard Dev Container configurations with Dockerfile or image reference
• Dev container features from any registry
• AWS EC2 runners
• Ona Cloud

​

Not supported

• Docker Compose-based Dev Containers (due to limited devcontainer CLI support)
• Non-project environments (e.g. environments started directly from a context URL)

​

Storage and costs

​

Storage costs

• Images are stored in AWS ECR in your account
• Storage costs depend on image size and number of unique configurations
• 30-day automatic expiry helps manage costs
• ECR repositories inherit all tags from the CloudFormation stack (useful for cost allocation and AWS MAP programs)

​

Cost optimization

• Similar Dev Container configurations share base layers, reducing storage
• Regional storage (same region as runner) minimizes transfer costs
• Automatic expiry prevents accumulation of unused images

​

Repository location

​

Monitoring

⚡️ Using pre-built dev container (saved ~3m45s build time)