Lab 5: Automations at Scale

​

What are Ona Automations

• Steps - Prompts, shell scripts, pull request steps, or report steps
• Triggers - Manual, pull request events, time-based schedules, or webhooks
• Targets - Projects or repositories to run against
• Guardrails - Limits to control execution

​

How automations work

1. Trigger - Run manually, on PR events, on a schedule, or via webhook
2. Execute - Ona clones, branches, installs, builds, tests, iterates, commits, pushes, and opens a PR. The same loop a human engineer runs, except Automations can do it across hundreds of repos in parallel
3. Review - Inspect outputs, logs, and linked PRs. Approve and merge when satisfied

​

Key terms

• Automation - A workflow definition that executes changes
• Execution - A single run of an automation
• Action - The result of running on one repository (one execution creates multiple actions)
• Step - An individual operation (prompt, shell script, PR creation, or report)

​

Creating your first automation

​

Access the Automations page

1. Open Ona and navigate to Automations in the left panel
2. Click + Automation in the top right corner
3. Choose Start from scratch or select a pre-configured template from the dropdown

​

Configure basic settings

• Name - Short, descriptive name (e.g., “Weekly dependency updates”)
• Description - Explain what the automation does
• Run as - Choose your user or a service account

• Event-driven automations (pull request triggers)
• Time-based automations (scheduled jobs)
• Team-wide automations where you want activity tracked separately

​

Configure runs on

• Projects - Select one or more projects. The automation runs on all repositories within them. Best for repetitive tasks on known repositories.
• Repositories - Use a search query to match repositories by name. Best for large-scale migrations across many targets. Partial matches work across owner and repository names.

​

Choose a trigger type

• Manual - Run on demand. Best for one-time migrations, testing automations, or changes requiring human judgment on timing.
• Pull Request - Trigger on PR events. Best for automated code reviews, security scanning, or documentation updates.
• Time-based - Run on a schedule (hourly, daily, weekly, monthly). Best for dependency updates, nightly security scans, or compliance checks.

​

Add steps

Update all dependencies to their latest compatible versions, run tests, and provide a summary of your changes.

npm update
npm test

​

Example: Multi-step automation

Step 1 (Prompt):   Upgrade all dependencies to their latest versions
Step 2 (Shell script):  npm test
Step 3 (PR Step):  Create pull request with summary of dependency upgrades and test results

​

Set guardrails

• Max concurrent executions - How many run simultaneously
• Max total executions - Maximum total runs allowed

​

Practical scenario 1: README standardization

​

The automation

Step 1 (Prompt):
Analyze this repository and ensure it has:
1. A comprehensive README.md with:
   - Project description
   - Setup instructions
   - Usage examples
   - Contributing guidelines
2. A backstage.yaml file with correct metadata

Follow the documentation standards in our org-wide AGENTS.md.
Update or create files as needed.

Step 2 (PR Step, optional):
Create pull request with documentation updates

​

Why this works

• Eliminates documentation toil for engineers
• Ensures consistent structure across all repositories
• Platform teams gain control over documentation quality
• Changes go through PR review before merging

​

Practical scenario 2: Automated security review

​

The automation

Step 1 (Prompt):
Review the code changes in this pull request for security vulnerabilities:

1. Check for:
   - SQL injection
   - XSS vulnerabilities
   - Authentication/authorization issues
   - Secret exposure
   - Insecure dependencies

2. Post a detailed review comment on the PR with:
   - Issues found (with file and line numbers)
   - Severity (Critical/High/Medium/Low)
   - How to fix each issue

If no issues found, post a comment confirming the review passed.

​

Why this works

• Immediate security feedback on every code change
• No waiting for manual security review
• Consistent review criteria across all repositories
• Issues caught before code reaches main branch

​

Practical scenario 3: Weekly dependency updates

​

The automation

Step 1 (Prompt):
Check for outdated dependencies and update them:
1. Run the appropriate command for this project type (npm outdated, pip list --outdated, etc.)
2. Update dependencies to latest compatible versions
3. Run the test suite
4. If tests fail, revert the problematic update and try the next minor version

Step 2 (Shell script):
npm test || pytest || go test ./...

Step 3 (PR Step, optional):
Create pull request with:
- Title: "chore: weekly dependency updates"
- Body: List of updated packages with version changes and any notes about compatibility

​

Why this works

• Reclaim engineering hours spent on CVE remediation
• Automated fixes handle version bumps and breaking changes
• Tests run before PR is created
• Fresh updates ready for review every Monday

​

Running and monitoring automations

​

Starting an automation

1. Open the Automation (or use the three-dot menu from the list)
2. Click Run
3. Confirm target repositories/projects

​

Monitoring progress

• Running - Currently executing
• Pending - Queued
• Completed - Finished successfully
• Failed - Encountered errors

​

Stopping executions

• Cancel individual action: Three-dot menu next to an action > Cancel
• Cancel entire run: Click Cancel on the run details page to stop all actions

​

Reviewing results

1. View the execution summary (completed, failed, canceled counts)
2. Click individual actions to review logs
3. Check your SCM (GitHub/GitLab) for pull requests created by the Automation
4. Review and merge each PR independently

​

Best practices

​

Do’s

• Start small - Test on 1-2 repositories before scaling
• Use service accounts - Track automation activity separately from human work
• Set conservative guardrails - Prevent accidental large-scale runs
• Write specific prompts - Include file paths, patterns to follow, verification steps
• Request verification - Have automations run tests before creating PRs
• Review PRs - Always inspect automation-created changes before merging

​

Don’ts

• Don’t skip testing - Always verify on a few repos first
• Don’t set unlimited guardrails - Even trusted automations should have limits
• Don’t merge blindly - Automation PRs need human review
• Don’t over-automate - Some changes need human judgment

​

When to use which trigger

​

Troubleshooting

• Check guardrails, you may have hit the limit
• Verify you have access to target repositories
• For PR triggers, confirm webhook is configured correctly

• Click the failed action to see session logs
• Check if the prompt is too vague; be specific
• Verify dependencies are installed (use Dev Container for consistent setup)

• Check if the automation found changes to make
• Verify the PR step is configured
• Check SCM permissions for the service account

• Verify webhook URL and secret in SCM settings
• Check that the correct events are selected
• Test with a new PR to trigger the webhook

• Break into smaller, targeted automations
• Reduce concurrent limit if hitting resource constraints
• Use shell scripts for deterministic tasks, prompts for intelligent decisions

​

What you’ve learned

• Create automations with prompts, shell scripts, and PR steps
• Configure triggers for manual, event-driven, or scheduled execution
• Set guardrails to control automation execution safely
• Monitor progress and review results in real-time
• Scale safely by starting small and verifying before expanding