selection-rate-audit

Statistics for auditing an automated hiring tool for adverse impact, with no scipy dependency, plus two small tools built on them. Everything is standard library; primitives that mirror a textbook formula cite it so a reader can check the implementation line by line.

What's here

• selection_rate_audit/stats.py — paired t-test, bootstrap confidence interval for a mean, percentiles, selection rates and four-fifths impact ratios with a minimum-n caveat, a label permutation test, the two-standard- deviation binomial cross-reference, flip fragility, Benjamini–Hochberg correction, and a placebo difference-in-differences contrast. Plain floats and small dataclasses in, plain floats out.
• four-fifths (four_fifths.py) — the EEOC Uniform Guidelines § 1607.4(D) selection-rate comparison over a scored candidate set: per-group and intersectional selection rates, impact ratios against the best-performing group, a pass/fail against 0.80, and an explicit list of underpowered groups. Reads a JSON list or JSONL of {score, recommendation?, groups}; a balanced synthetic-demographics helper exists for exercising a pipeline before self-identified data is collected. Exit code 1 on adverse impact.
• cohort-snapshot (snapshot.py) — the same comparison over one scoring run where cohorts are inferred from names. The inference is a conservative heuristic (it under-classifies) and every output says so; BIFSG would be better and needs Census-tract data. Reports per-cohort score distributions, impact ratios, and a verdict with the small-n caveats.

pip install -e ".[dev]"
python -m pytest
four-fifths --input examples/scored.json --synthetic-groups
cohort-snapshot --input examples/scored.json --output-json out.json --output-md out.md

What is deliberately not here

The counterfactual perturbation harness this library grew up next to (name and proxy swaps, a CI gate with green/yellow/red thresholds, governance around persisting a verdict) is not published. What is here is the statistics and the snapshot method, which are public methodology.

Origin

Written for the bias-audit program of a résumé-screening product. Published with the company's permission; the only data in this repository is synthetic.

MIT.