Costs & Budgeting for GCP Runner

Note: GCP costs mentioned in this document are subject to change and may vary by region. Please check the latest pricing on the GCP pricing pages.

​

What is a GCP runner?

• Shared resource serving multiple users (not personal)
• Designed for “always-on” operation with optimized cost efficiency
• Suitable for organizations requiring GCP-specific compliance or data residency
• Support for multiple runners to:
  • Enhance availability
  • Reduce latency for end users
  • Ensure data residency and compliance

​

Billable GCP Resources

• Compute Engine Instances: Managed instance groups for runner and proxy services
• Memorystore Redis: Used to store state related to environment reconciliation
• Cloud Storage Buckets: Used to store build cache, runner assets, and CA certificates
• Load Balancer Components: Health checks, backend services, and forwarding rules
• Secret Manager: Used to store Redis credentials and metrics configuration
• Cloud Logging: Used to store logs related to runner instances
• VPC Networking: Private service connections and IP address reservations
• Cloud KMS (optional): Customer-managed encryption keys for enhanced security

​

Baseline Costs of a Runner

​

Core Infrastructure (Always-On)

• Compute Engine Instances:
  • Runner instance: 1x c4-standard-4 (4 vCPU, 16GB RAM) = ~$120/month
  • Proxy instances: 2x c4-standard-2 (2 vCPU, 8GB RAM each) = ~$120/month total
  • Subtotal: ~$240/month
• Memorystore Redis: Standard HA instance with 2GB memory = ~$70/month
• Load Balancer: Global or regional load balancer = ~$18-25/month

​

Storage and Networking

• Cloud Storage: Build cache and assets storage (~$2-10/month depending on usage)
• VPC Networking: Private service connections and IP reservations (~$3-8/month)
• Secret Manager: Minimal cost for storing credentials (~$0.50-2/month)
• Cloud Logging: Log storage and ingestion (~$2-15/month depending on log volume)

​

Optional Components

• Cloud KMS: Customer-managed encryption keys (~$1-3/month if enabled)
• Certificate Manager: SSL certificate management (free for Google-managed certificates)

Note: These are baseline estimates. Actual costs may vary based on region, usage patterns, and configuration choices.

​

Controls for Managing Costs

​

Viewing Runner Costs

1. Navigate to Cloud Billing → Cost breakdown.
2. Group by Service to analyze the breakdown of costs.
3. Filter by labels to isolate runner-specific resources:
   • Runner Name: Use the gitpod-runner label with your runner name value.
   • Component: Filter by gitpod-component label to see specific components (redis, build-cache, etc.).
   • Managed by Terraform: Use the managed-by label with value terraform.

​

Viewing Environment Costs

1. Navigate to Cloud Billing → Cost breakdown.
2. Group by Service to analyze the breakdown of costs.
3. Filter by the environment ID using labels:
   • Label: gitpod-environment-id
   • Value: The environment ID (you can find this by selecting Copy ID from the environment details)

​

Using Cost Management Tools

​

Budget Alerts

1. Navigate to Cloud Billing → Budgets & alerts
2. Create a new budget with filters for your runner resources
3. Set threshold alerts at 50%, 80%, and 100% of your expected monthly cost
4. Configure notifications to email or Slack channels

​

Cost Anomaly Detection

1. Navigate to Cloud Billing → Cost insights
2. Enable anomaly detection for your project
3. Configure alerts for cost anomalies above your threshold

​

Compute Engine Instance Labeling

• Any labels you add to the Terraform variables will be propagated to the Compute Engine instances created for environments
• This allows for consistent resource labeling across your GCP infrastructure
• Labels can be used for cost allocation, resource grouping, and access control

​

Adding Labels to Compute Engine Instances

1. Update your terraform.tfvars file with additional labels:

labels = {
  environment = "production"
  team        = "platform"
  cost-center = "engineering"
  project     = "ona-runner"
}

2. Apply the Terraform changes:

terraform plan
terraform apply

Note: When you update Terraform labels, the changes don’t immediately apply to existing instances. To force existing instances to pick up the new labels, you can trigger a rolling update of the managed instance groups:

Report incorrect code

Copy

Ask AI

# Update runner instances
gcloud compute instance-groups managed rolling-action restart \
  RUNNER_INSTANCE_GROUP_NAME --region=REGION

# Update proxy instances  
gcloud compute instance-groups managed rolling-action restart \
  PROXY_INSTANCE_GROUP_NAME --region=REGION

This will recreate instances with the updated labels without affecting running development environments.

​

Cost Optimization Strategies

​

Right-sizing Instances

1. Review CPU and memory utilization in Cloud Monitoring
2. Consider smaller instance types for low-utilization runners
3. Use custom machine types for optimal resource allocation

​

Storage Optimization

1. Configure lifecycle policies on Cloud Storage buckets to automatically delete old cache data
2. Monitor storage usage and adjust retention policies as needed
3. Use regional storage instead of multi-regional for cost savings

​

Redis Optimization

1. Monitor memory utilization and adjust instance size accordingly
2. Consider Basic tier instead of Standard HA for non-production environments (saves ~$35/month)
3. Use smaller memory sizes if your workload permits:
   • 1GB instead of 2GB = ~$15-20/month savings
   • Monitor actual Redis memory usage in Cloud Monitoring

​

Load Balancer Optimization

1. Use internal load balancers when external access isn’t required
2. Monitor data transfer costs and optimize routing
3. Consider regional load balancers for single-region deployments

​

Resource Cleanup

​

Automatic Cleanup

• Build cache lifecycle: Automatically deletes cache data older than 30 days
• Log retention: Cloud Logging automatically manages log retention based on your settings
• Incomplete uploads: Automatically aborts incomplete multipart uploads after 1 day

​

Manual Cleanup

• Unused persistent disks from terminated environments
• Old machine images if using custom images
• Unused static IP addresses if any were manually created

​

Terraform Destroy

# Destroy all runner resources
terraform destroy

# Confirm destruction when prompted

Warning: This will permanently delete all runner infrastructure and any associated data. Ensure you have backups of any important work before proceeding.

​

Cost Monitoring Best Practices

​

Regular Cost Reviews

1. Weekly cost reviews: Monitor costs weekly to catch anomalies early
2. Monthly budget analysis: Compare actual vs. budgeted costs monthly
3. Quarterly optimization: Review and optimize resource allocation quarterly

​

Cost Attribution

1. Use consistent labeling: Apply consistent labels across all resources
2. Environment-specific tracking: Track costs per development environment
3. Team-based allocation: Allocate costs to specific teams or projects

​

Alerting and Notifications

1. Set up budget alerts: Configure alerts at multiple thresholds
2. Monitor cost anomalies: Enable automatic anomaly detection
3. Regular reporting: Set up automated cost reports for stakeholders

​

Troubleshooting High Costs

​

Common Cost Issues

1. Runaway environments: Environments that don’t shut down properly
2. Large build caches: Excessive storage usage in build cache buckets
3. High data transfer: Unexpected network egress charges
4. Oversized instances: Using larger instance types than necessary

​

Investigation Steps

1. Check Cloud Billing reports for cost breakdown by service
2. Review resource utilization in Cloud Monitoring
3. Audit running instances and their utilization
4. Check storage usage in Cloud Storage buckets
5. Review network traffic patterns and data transfer costs

​

Cost Reduction Actions

1. Terminate unused environments manually if auto-shutdown fails
2. Clean up old build cache data beyond the automatic lifecycle
3. Optimize instance types based on actual utilization
4. Review and adjust Redis instance sizing
5. Optimize load balancer configuration for your traffic patterns