Lab 5: Automations at Scale

​

What are Ona Automations

• Steps - Prompts, shell scripts, or pull request steps
• Triggers - Manual, pull request events, or schedules
• Targets - Projects or repositories to run against
• Guardrails - Limits to control execution

​

How automations work

1. Define once - Create an automation with steps that describe what to do
2. Trigger automatically or on-demand - Run manually, on a schedule, or when PRs are opened
3. Execute at scale - Run across selected repositories in parallel
4. Create pull requests - Automations open PRs with changes for review
5. Review and merge - Inspect outputs, logs, and approve changes

​

Key terms

• Automation - A workflow definition that executes changes
• Execution - A single run of an automation
• Action - The result of running on one repository (one execution creates multiple actions)
• Step - An individual operation (prompt, shell script, or PR creation)

​

Creating your first automation

​

Access the Automations page

1. Open Ona and navigate to Automations in the left panel
2. Click New Automation in the top right corner

​

Configure basic settings

• Name - Short, descriptive name (e.g., “Weekly dependency updates”)
• Description - Explain what the automation does
• Run as - Choose your user or a service account

• Event-driven automations (pull request triggers)
• Time-based automations (scheduled jobs)
• Team-wide automations where you want activity tracked separately

​

Choose a trigger type

• Best for: One-time migrations, testing automations, changes requiring human judgment on timing

• Best for: Automated code reviews, security scanning, documentation updates

• Best for: Dependency updates, nightly security scans, compliance checks

​

Add steps

Update all dependencies to their latest compatible versions, run tests, and provide a summary of your changes.

npm update
npm test

​

Example: Multi-step automation

Step 1 (Prompt): Upgrade all dependencies to their latest versions
Step 2 (Shell Script): npm test
Step 3 (PR Step): Create pull request with summary of dependency upgrades and test results

​

Set guardrails

• Max concurrent executions - How many run simultaneously (default: 10)
• Max total executions - Maximum total runs allowed (default: 100)

• Testing (1-10 repos): 5 concurrent, 20 total
• Team automation (10-50 repos): 10 concurrent, 100 total
• Organization rollout (50+ repos): 20-50 concurrent, 500+ total

​

Practical scenario 1: README standardization

​

The automation

Step 1 (Prompt):
Analyze this repository and ensure it has:
1. A comprehensive README.md with:
   - Project description
   - Setup instructions
   - Usage examples
   - Contributing guidelines
2. A backstage.yaml file with correct metadata

Follow the documentation standards in our org-wide AGENTS.md.
Update or create files as needed.

Step 2 (PR Step):
Create pull request with documentation updates

​

Why this works

• Eliminates documentation toil for engineers
• Ensures consistent structure across all repositories
• Platform teams gain control over documentation quality
• Changes go through PR review before merging

​

Practical scenario 2: Automated security review

​

The automation

Step 1 (Prompt):
Review the code changes in this pull request for security vulnerabilities:

1. Check for:
   - SQL injection
   - XSS vulnerabilities
   - Authentication/authorization issues
   - Secret exposure
   - Insecure dependencies

2. Post a detailed review comment on the PR with:
   - Issues found (with file and line numbers)
   - Severity (Critical/High/Medium/Low)
   - How to fix each issue

If no issues found, post a comment confirming the review passed.

​

Why this works

• Immediate security feedback on every code change
• No waiting for manual security review
• Consistent review criteria across all repositories
• Issues caught before code reaches main branch

​

Practical scenario 3: Weekly dependency updates

​

The automation

Step 1 (Prompt):
Check for outdated dependencies and update them:
1. Run the appropriate command for this project type (npm outdated, pip list --outdated, etc.)
2. Update dependencies to latest compatible versions
3. Run the test suite
4. If tests fail, revert the problematic update and try the next minor version

Step 2 (Shell Script):
npm test || pytest || go test ./...

Step 3 (PR Step):
Create pull request with:
- Title: "chore: weekly dependency updates"
- Body: List of updated packages with version changes and any notes about compatibility

​

Why this works

• Reclaim engineering hours spent on CVE remediation
• Automated fixes handle version bumps and breaking changes
• Tests run before PR is created
• Fresh updates ready for review every Monday

​

Running and monitoring automations

​

Starting an automation

1. Navigate to Automations in the left panel
2. Find your automation and click it
3. Click Run in the top right corner
4. Review targets and confirm

​

Monitoring progress

• Running - Currently executing
• Pending - Queued
• Completed - Finished successfully
• Failed - Encountered errors

• Completion percentage across all actions
• Real-time updates as actions execute

​

Stopping executions

​

Reviewing results

1. Check the execution summary (completed, failed, canceled counts)
2. Go to your SCM (GitHub/GitLab) to review created pull requests
3. Filter PRs by the service account used for the automation
4. Review and merge each PR independently

​

Try it: Create a documentation automation

1. Go to Automations > New Automation
2. Name: “Update README files”
3. Description: “Ensure README has setup instructions”
4. Trigger: Manual
5. Target: Select 1-2 test repositories

Check if this repository's README.md has:
1. A "Getting Started" section with setup instructions
2. A "Development" section explaining how to run locally

If any section is missing, add it based on analyzing the codebase.
Match the existing README style.

• Max concurrent: 2
• Max total: 5

1. Click Save, then Run
2. Watch the action status panel
3. Click an action to see the agent’s conversation
4. Review the created PRs in your SCM

​

Best practices

​

Do’s

• Start small - Test on 1-2 repositories before scaling
• Use service accounts - Track automation activity separately from human work
• Set conservative guardrails - Prevent accidental large-scale runs
• Write specific prompts - Include file paths, patterns to follow, verification steps
• Request verification - Have automations run tests before creating PRs
• Review PRs - Always inspect automation-created changes before merging

​

Don’ts

• Don’t skip testing - Always verify on a few repos first
• Don’t set unlimited guardrails - Even trusted automations should have limits
• Don’t merge blindly - Automation PRs need human review
• Don’t over-automate - Some changes need human judgment

​

When to use which trigger

​

Troubleshooting

• Check guardrails—you may have hit the limit
• Verify you have access to target repositories
• For PR triggers, confirm webhook is configured correctly

• Click the failed action to see conversation logs
• Check if the prompt is too vague—be specific
• Verify dependencies are installed (use Dev Container for consistent setup)

• Check if the automation found changes to make
• Verify the PR step is configured
• Check SCM permissions for the service account

• Verify webhook URL and secret in SCM settings
• Check that the correct events are selected
• Test with a new PR to trigger the webhook

• Break into smaller, targeted automations
• Reduce concurrent limit if hitting resource constraints
• Use shell scripts for deterministic tasks, prompts for intelligent decisions

​

What you’ve learned

• Create automations with prompts, scripts, and PR steps
• Configure triggers for manual, event-driven, or scheduled execution
• Set guardrails to control automation execution safely
• Monitor progress and review results in real-time
• Scale safely by starting small and verifying before expanding