Personal project · in progress (5 of 8 planned phases)

Sentinel

2026

A personal project for understanding agent orchestration and automated GitHub PR review.

Overview

Sentinel is a small Python service that reviews GitHub pull requests. I built it to learn how to orchestrate LLM agents in a way I could actually trust, instead of wiring a model straight into a webhook and hoping the output is sensible.

The idea that holds it together is simple. Code, not the model, decides what happens next: each agent returns structured data, plain Python drives the pipeline, every step is recorded, and any past run can be replayed from its saved inputs and outputs.

System diagram

Solid blocks are built and exercised by the eval harness. Dashed blocks are planned.

What it explores

Deterministic orchestration

A small DAG engine runs the pipeline. The model only returns typed data and never chooses the next step.

Parallel specialist agents

Security, correctness, and testing reviewers run at the same time, then a plain-Python step dedupes and ranks their findings by confidence.

An evaluation harness

Precision, recall, and calibration measured against a labeled set of PRs, with a CI check that fails the build on a regression.

Observability and cost

OpenTelemetry traces and Prometheus metrics across the stack, with tokens and dollar cost tracked per run.

Structured output, no framework

The Anthropic SDK is used directly, with tool use forcing valid JSON against a Pydantic schema.

How it works

1
Receive
A pull request event hits a FastAPI webhook. It verifies GitHub's HMAC signature, records the run, and acknowledges within seconds.
2
Orchestrate
A deterministic engine walks a plan of steps, snapshotting each step's inputs and outputs to Postgres so any run can be replayed by ID.
3
Review
A diff analyzer warms the prompt cache, then three specialist reviewers run in parallel through a shared LLM client with timeouts and retries.
4
Consolidate
A pure-Python step dedupes the findings, ranks them, and applies a per-repo confidence threshold from a .sentinel.yml policy file.

Sentinel on a real PR

A real review on one of the project's test pull requests, seeded with known defects. A push fired the webhook, Sentinel ran the diff analyzer and three specialist reviewers in parallel, and the consolidator deduped and ranked the results before posting them back as a GitHub review. These are the actual comments it left.

sentinel-test #6

Sentinel's summary review comment on the pull request, showing run status, diff analysis, risk hints, and a findings count. — The review summary Sentinel posts first: run status and ID, its diff analysis, the risk hints it flagged, and a count of findings by severity.

Two critical security findings posted as inline review comments, anchored to the exact lines in payments.py. — Two critical security findings posted inline, each anchored to the line it refers to: a hardcoded credential and a SQL injection, both at 1.00 confidence.

A testing finding posted as an inline review comment with its supporting evidence. — A lower-confidence testing finding. Each comment carries its reviewer, category, severity, and confidence, with a short line of evidence.

Built and planned

Built

FastAPI service with signature-verified webhooks
Deterministic orchestrator with per-step snapshots and replay by ID
Diff analyzer, three parallel reviewers, and a deterministic consolidator
Posts the consolidated findings to the pull request as an inline GitHub review
Evaluation harness with metrics and a CI regression gate
Tracing, metrics, and per-run token and cost tracking

Planned

Reliability primitives: circuit breakers, cost budgets, graceful degradation
Grafana dashboards over the existing metrics
Deployment to a VPS and a small web UI

Stack

Python
FastAPI
PostgreSQL
SQLAlchemy
Anthropic API
Docker
OpenTelemetry
Prometheus
Pydantic