Fishbone — Fraud Model Precision Drop
Root-cause analysis of a card fraud model whose precision fell six points in a quarter, grouped across data, features, labels, serving, monitoring and the fraud population itself.
Make it your own.
title "Why has fraud model precision fallen six points?"
problem "Precision at fixed recall dropped from 0.71 to 0.65"
category "Data"
cause "Upstream schema change dropped device fingerprint"
cause "Merchant category codes remapped by the acquirer"
category "Features"
cause "Velocity window silently changed from 24 h to 1 h"
cause "Training and serving compute the aggregate differently"
category "Labels"
cause "Chargeback labels arrive 45 days late"
cause "Manual review queue backlog left cases unlabelled"
category "Serving"
cause "Fallback score used when the feature store times out"
cause "Threshold left at the previous release value"
category "Monitoring"
cause "Drift alert only checks the score distribution"
cause "No per-segment precision breakdown"
category "Population"
cause "Fraud ring switched to low-value card testing"
cause "New card-on-file merchant changed the traffic mix"