---
title: "AI Failure Explanations Proposal"
url: https://stacklist.com/card/c3daa655-b98c-48b5-8a90-375694e59ae2
stack: https://stacklist.com/stack/0f90f21c-c99f-4b60-a8eb-ec5953bf1ae1
summary: "AI Failure Explanations Proposal describes adding automated failure analysis to the nightly test suite using Anthropic's Claude API to distinguish real bugs from flaky tests. The solution is already built and requires minimal budget approval (under $1/month) to enable, improving team efficiency by eliminating manual stack trace reading."
tags: "ci-cd, test-automation, ai-assisted-analysis, failure-detection, api-integration, qa-engineering, cost-optimization"
key_entities: "Gabriel Amorim (person), Anthropic (organization), Claude Haiku 4.5 (technology), Slack (technology), CI/CD pipeline (concept), flaky tests (concept)"
classification: "memo"
content_hash: "sha256:33c1f60bc9b696301b2855e15210f72050c3035f64eade92a3a0626bff05ea91"
acp_version: "0.2"
token_counts_approximate: 834
visibility: public
agent_accessible: true
status: "final"
---

# AI Failure Explanations Proposal

AI-Assisted Failure Explanations for the Nightly Suite 
A short proposal to add automated failure analysis to our CI 
Prepared by: Gabriel Amorim, QA • Date: June 17, 2026 • For: Engineering leadership • Status: Already built, awaiting budget approval 
Our nightly test run already posts its results to Slack. I would like to add a small paid step that uses the Anthropic API to write a short, plain 
explanation for every failed test and tag it as a likely real bug or likely flaky . The idea is to tell a genuine product problem apart from test 
noise at a glance, without anyone having to read a stack trace. It is already built and tested, so all it needs is approval for a small API budget. 
What we are missing without it 
We already run a free version that explains the failures we have seen before, and it does that part well. What it cannot do is explain a failure 
the first time it happens, which is exactly when it is hardest to read, and every new pattern has to be added by hand. So a red build still 
usually means someone opening the run and reading the raw error to decide whether it is real or just noise. That is the repeated work this 
removes. 
Budget 
The model is Claude Haiku 4.5, the fastest and lowest-cost current model, priced at $1 per million input tokens and $5 per million output 
tokens. Each explained failure costs roughly $0.002 (about 1,300 tokens of input and 100 of output). At our volumes that rounds to almost 
nothing: 
Scenario Failures explained / month Estimated cost / month 
Typical nightly (about 3 failures per night) ~90 ~$0.18 
Heavy nightly (capped at 8 per night, every night) ~240 ~$0.48 
Extended to every CI run (about 60 failures per day) ~1,800 ~$3.60 
Under $1 per month in normal use. 
A hard monthly spend limit is set on the API key, so the cost cannot run away under any circumstance. 
Why it is worth it 
Safety and privacy 
It is safe to run. There is a hard monthly spend cap on the key, and if the key is ever missing the notification still posts, just without the 
explanations, so it never blocks a build. On privacy, the API runs under Anthropic's commercial terms, which by default do not use our inputs 
to train their models, and retention is short. We only send test names and error text, never user data. 
Recommendation 
Approve a small budget (a $5 per month cap is plenty) and create one API key. Since it is already built, turning it on is just a configuration 
change. I would run it as a hybrid: keep the free rules for the failures they already handle, and use the API only for the ones they do not 
recognise, so we pay only when it is actually needed. 
Internal QA proposal. Cost figures are estimates based on Anthropic's published Claude Haiku 4.5 pricing and current nightly failure volumes; actual cost varies with the number of 
unrecognised failures per run. 
The cost is tiny next to the time it saves. If reading a red build takes someone 5 to 10 minutes, a few of those a week already outweigh a 
few cents a month. 
Real bugs stop getting lost. The colored tag and the count at the top mean the team jumps on the genuine failures and skips the known 
flaky ones. That is the main thing we are after. 
It handles the failures the rules cannot, and it keeps doing so as the suite grows, without anyone maintaining a list of patterns by hand.