What it does
Security-Copilot turns raw cloud security alerts into analyst-ready investigations. It ships a set of precomputed, grounded investigation reports for GuardDuty-style findings, plus a live chat that answers analyst questions about any single alert - strictly from that alert's evidence and the relevant runbook excerpts. Browsing the reports is free (served straight from S3); only the live Q&A calls a model.
Why I built it
Real SOC triage is slow and repetitive, and most "AI SOC" demos quietly hallucinate. I wanted something genuinely useful and genuinely honest: real retrieval over real evidence, a real LLM at the generation step, and a guardrail layer that refuses to let the model invent IPs, accounts, or finding IDs. It doubles as an end-to-end demonstraion of production GenAI on AWS - retrieval, serving, cost control, evaluation, and infrastructure-as-code.
Architecture
The whole system scales to zero. Precomputed reports live in S3 and are browsed for $0. Live questions hit a container-image AWS Lambda (FastAPI + Mangum) behind an HTTP API Gateway; retrieval uses fastembed + ChromaDB, and Claude Haiku generates the answr. A static simgle-page app - vanilla JS, no build step - is served from a private S3 bucket through CloudFront (Origin Access Control, HTTPS, custom domain). A DynamoDb-backed daily token budget acts as a kill-switch. Everything is defined in Terraform.
How it's kept honest
Every served report and every live answer passes a deterministic
citation-coverage guardrail: each IP, finding ID, and service account named in
the text must appear in the underlying evidence, or the output is flagged
REVIEW_REQUIRED. A groundedness score reports the fraction of named
entities backed by evidence. Offline, a cluade Sonnet LLM-as-judge scores
faithfulness and correctness, and a meta-evaluation checks the judge against
ground truth. The demo surfaces all of this in a "how this is kept honest"
panel - including a regression where an injected fake IP visibly trips the guardrail.
Lessons learned
- Cold starts are a design constraint: swapping sentence-transformers/torch for fastembed (ONNX) was the difference between blowing and meeting Lambda's 30s cap.
-
CORS is subtle behind API Gateway: a single
$defaultroute suppresses the automatic OPTIONS-preflight handling, so the browser preflight has to be handled explicitly. - Scale-to-zero is a feature, not just a cost lever: precomputing the expensive work into S3 keeps the common path free and the demo effectively free to host.
- Owning the whole stack - retrieval, serving, evals, and Terraform - allows for greater visibility.