Watching the watchers: monitoring your CI/CD pipelines, branch protection and repo hygiene

Most organisations monitor production closely and barely monitor their delivery tooling. Someone sets up a nightly build or a branch protection rule and confirms it works that day. After that, nobody looks at it again until a release breaks or an auditor asks how changes get onto main.

This post is part of our Practical AI in DevOps series. It covers the continuous checks we run over GitHub and Azure DevOps, and a trap that produces false findings if you build those checks the obvious way. AI comes in at the end, where it turns a week of findings into a short digest so engineers only read what changed.

What the checks cover

We run five checks on a daily schedule:

• Scheduled workflows and pipelines whose recent scheduled runs failed, or that have stopped running.
• Default and release branches with no branch protection or ruleset, or with rules weaker than the organisation’s baseline, such as no required review.
• Open Dependabot alerts, grouped by severity and by how long they have been open.
• Open secret-scanning alerts.
• Stale branches with no commits past a threshold you choose (90 days is a reasonable start) and no open pull request.

Scheduled jobs are on the list because nobody is waiting for them. When a pull request build fails, the person who opened the pull request sees it. When a dependency refresh fails at 2am on a Sunday, the notice goes to whatever notification setting was configured, which may belong to someone who left last year.

How it fits together

We keep the architecture small on purpose:

• A timer-triggered job runs once a day. A GitHub Actions workflow in a separate admin repository will do the job, and so will an Azure Function.
• It reads GitHub through a GitHub App with read-only permissions for Actions, Administration, Contents, Dependabot alerts and Secret scanning alerts. For Azure DevOps it uses a service principal or a PAT scoped to Build (read) and Code (read).
• Each check writes its findings to a store in one shared format. A JSON file per day in blob storage is enough.
• Once a week, a diff step compares the latest findings with the previous week’s.
• A language model gets only the diff and writes the digest, which goes to a Teams channel or an email list.

A single finding looks like this:

{
  "source": "github",
  "repo": "example-org/payments-api",
  "check": "branch-protection",
  "subject": "main",
  "severity": "high",
  "detail": "Default branch has no protection rule or ruleset",
  "first_seen": "2026-09-11"
}

Each finding is identified by its source, repo, check and subject. The weekly diff only works if that identity stays stable from one run to the next.

GitHub branch protection has a trap of its own. The classic protection endpoint (GET /repos/{owner}/{repo}/branches/{branch}/protection) doesn’t report repository rulesets, so a branch protected only by a ruleset looks unprotected. Query GET /repos/{owner}/{repo}/rules/branches/{branch} as well, and count either one as protection.

Dormant YAML files versus registered pipelines

The obvious first version of a pipeline check walks the repository tree, looking for files under .github/workflows or an azure-pipelines.yml. That approach is wrong in both directions. A file in a repository only records what someone once declared, and the CI platform decides what runs.

In Azure DevOps, a YAML file does nothing until someone creates a pipeline that points at it, and that pipeline can point at any path. Repositories collect dormant pipeline files from project templates and abandoned experiments. If you probe for azure-pipelines.yml, a dormant file makes it look as if CI exists when it doesn’t. A repository whose real pipeline uses build/ci.yml gets reported as having no CI. A registered pipeline can also be paused or disabled, and a file probe can’t see either state.

GitHub registers workflow files automatically, but the file still leaves things out. Scheduled triggers only run from the default branch, so a schedule in a file that exists only on a feature branch never fires. GitHub can also disable a workflow, and the API reports this as a state of disabled_manually or disabled_inactivity.

Ask the platform instead:

# GitHub: registered workflows and their state
gh api repos/ORG/REPO/actions/workflows \
  --jq '.workflows[] | {name, path, state}'

# GitHub: recent scheduled runs for one workflow
gh api "repos/ORG/REPO/actions/workflows/WORKFLOW_ID/runs?event=schedule&per_page=5" \
  --jq '.workflow_runs[] | {created_at, conclusion}'

# Azure DevOps: registered pipelines, their YAML path and queue status
curl -s -u ":$AZDO_PAT" \
  "https://dev.azure.com/ORG/PROJECT/_apis/build/definitions?includeAllProperties=true&api-version=7.1" \
  | jq '.value[] | {name, yaml: .process.yamlFilename, status: .queueStatus}'

Then join the file view to the platform view. A pipeline file with no registered definition gets an informational note saying the file is dormant. A registered definition whose YAML path no longer exists, or whose recent scheduled runs failed, is a real finding.

The same failure turned up recently in another tool. In a fix to mise, mise doctor reported a background watcher as “declared but not running” while the service manager was running it, and the command it recommended did nothing. The maintainers changed the health checks to use the same test as the command that acts on that state, so the checks no longer recommend a command that won’t do anything. A monitoring check should read the same state the system acts on, and that is rarely a file path.

Keeping the hygiene reports readable

A stale-branch report that lists 300 branches will get ignored. Report a count per repository, list only stale branches that have no open pull request, and leave out release branch patterns you keep on purpose. Don’t delete branches automatically. An old branch can hold the only copy of a hotfix that was never merged.

Any open secret-scanning alert should get a person’s attention that week. The check records the secret type and the repository, with a link to the alert. It never stores the secret value, so secrets can’t end up in anything sent to a model later.

For Dependabot, the useful signal is age and severity together. A critical alert that has been open for a day is normal triage. If the same alert has been open for six weeks, nobody is looking after that repository.

The weekly digest

Engineers stop reading a report that says the same thing every week. The diff step lists what is new and what was resolved since last week. It also carries forward anything still open past an age threshold. Only those items go to the model.

The model handles wording and grouping, and deterministic code decides what counts as a finding. The prompt asks for one line per item, grouped by team, with each line ending in the link supplied in the input. A cheap post-check compares every repository named in the digest against the input and rejects the output if the model names one that wasn’t there.

An example digest:

New this week: the payments-api main branch lost its ruleset (link). The nightly dependency refresh in customer-portal has failed its last three scheduled runs (link). Resolved: two high-severity Dependabot alerts in reporting-service.

The model only ever sees metadata such as repository names, check names, severities and dates. Some organisations treat repository names as sensitive too. In that case, use a model hosted in an Australian region, such as Azure OpenAI in Australia East, and keep the digest inside your own tenant.

Costs and limits

• GitHub and Azure DevOps both rate-limit their APIs. Large organisations need pagination and caching, and a daily run is plenty.
• Dependabot alerts come with every GitHub plan. Secret scanning on private repositories needs GitHub’s paid Advanced Security features. The Azure DevOps equivalent, GitHub Advanced Security for Azure DevOps, is also a paid add-on. Check what you are licensed for before promising coverage.
• The model input is a short diff, so the model cost is small.
• The platforms change under you. GitHub added rulesets alongside classic branch protection, and a check written before rulesets misses any rule configured that way. Budget time for maintenance.
• The monitor is a scheduled job too. Give it a heartbeat, so that if a daily findings file doesn’t arrive, something outside the monitor raises an alert.
• The checks report problems and fix nothing. Every finding needs an owner, and the digest only helps if someone is expected to act on it.

PicNet builds production AI systems for Australian organisations. If a weekly digest over your pipelines sounds like a sensible place to start, talk to us about what a first project could look like.