Pangram is an AI-text-detection tool. 10.0 ran applicant free-text fields through Pangram and stored a fraction_ai score per field (0 = human-written, 1 = high confidence AI-generated, -1 = not analyzed / not enough text). We can use these scores to ask: how prevalent is AI-generated text in 10.0 applications? Does it correlate with worse Stage-2 scores? With lower P(ranked)? Or do reviewers seem to be (implicitly or explicitly) discounting it?
MATS (Machine Alignment, Transparency & Security) is an AI safety research fellowship that places ~120 fellows with ~100 mentors per cohort. Cohort 10.0 ran in summer 2026 and was the first cohort with a centralized application review instead of decentralized stream-specific review. This analysis is part of a broader effort to evaluate the 10.0 process and inform the design of 11.0 (autumn 2026).
The 10.0 pipeline in brief. ~2,200 people applied. Each applicant went through three stages:
For the empirical track, the composite formula is 0.50·RS + 0.35·TE + 0.15·SS, where TE = 0.50·MLE + 0.30·SWE + 0.20·Math. A "relevance multiplier" (Direct=1.0 / Adjacent=0.85 / Distant=0.60) is applied to Research Skills based on how the applicant's experience matches the streams they applied to.
Outcome definitions used throughout these analyses:
is_ranked (primary outcome) — applicant was ranked by ≥1 stream. This is the cleanest signal of "the selection process picked this person." Not the same as "received an offer" — offer count is bounded by cohort size (~120), but rank count reflects quality independently of capacity.is_invited_to_worktest (secondary outcome) — applicant was engaged by ≥1 stream in any way: invited to a work test, invited to an interview, ranked, or sent the Megastream takehome. Strict superset of is_ranked. One level above is_ranked in the funnel.passed_mentors_bar — applicant was offered or waitlisted. In 10.0, this equals is_ranked exactly (every ranked person got either an offer or a waitlist slot).For each applicant we compute max fraction_ai across all analyzed fields — i.e., the highest AI-detection score they hit on any single field. -1 (not analyzed) is treated as missing.
Pangram is a model. A 'high fraction_ai' score is not a confession; it's an estimate. False positives exist, especially for non-native-English writers who use ChatGPT or Grammarly for editing rather than generation. We don't treat fraction_ai as ground truth — just as an applicant-level signal.
Of 2,104 applicants with any Pangram-analyzed text, 772 (37%) had at least one field flagged as AI-generated (max fraction_ai ≥ 0.9).
Higher Pangram scores correlate with worse outcomes, but modestly:
- Max fraction_ai → is_ranked (full pool, n=2,104): AUC = 0.574 [0.539, 0.607].
- Max fraction_ai → composite score (Spearman): ρ = -0.090.
- In the Stage-3 empirical pool (n=765): AUC = 0.552 [0.510, 0.591]; Spearman ρ with composite = -0.028.
The Stage-3 effect is smaller — most of the Pangram-related signal gets absorbed earlier in the pipeline.
| Field | n analyzed | Mean fraction_ai | % with ≥0.9 |
|---|---|---|---|
| Reasoning for top 3 (fraction_ai) | 1,977 | 0.22 | 21.8% |
| Writing sample 1 (fraction_ai) | 378 | 0.16 | 7.1% |
| Writing sample 2 (fraction_ai) | 307 | 0.19 | 11.7% |
| Empirical option A (fraction_ai) | 516 | 0.38 | 38.0% |
| Empirical option B (fraction_ai) | 1,072 | 0.33 | 33.4% |
| Policy & Strategy option A (fraction_ai) | 116 | 0.42 | 42.2% |
| Policy & Strategy option B (fraction_ai) | 291 | 0.44 | 44.0% |
| Technical Governance option A (fraction_ai) | 111 | 0.51 | 51.4% |
| Technical Governance option B (fraction_ai) | 236 | 0.52 | 50.8% |
The ToC reasoning text has the broadest coverage (~2,200 applicants). Bucketing into 4 fraction_ai bands:
| Bucket | n | P(ranked) | Mean composite |
|---|---|---|---|
| 0 (human) | 1,546 | 9.7% | 1.69 |
| low (0.1–0.5) | 1 | 0.0% | 0.00 |
| high (≥0.9) | 430 | 5.1% | 1.47 |
The distribution is heavily bimodal: many applicants score 0 (clean human writing on every field) or 1 (high AI detection somewhere).
is_ranked from max fraction_ai in the full pool — comparable to ToC alignment (B3).Sample. Full canonical pool, restricted per-analysis to applicants with at least one Pangram-analyzed field (n=2,104).
Outcome variable(s). is_ranked. Composite-mean reported descriptively.
Predictor fields. Per-field fraction_ai (0–1 with -1 = missing). Per-applicant: max and mean across all analyzed fields. AUC computed with negated score (higher fraction_ai should reduce P(ranked)).
Filters applied. Canonical dedup. Pangram fields with -1 (not analyzed) treated as missing.
Missing-data handling. Per-field listwise drop. n_fields_analyzed indicates per-applicant coverage.
Key assumptions / caveats.
fraction_ai as labeled — i.e., 0=human, 1=AI. We don't have ground truth, so we can't validate Pangram itself here.max_fraction_ai reflects 'is any field AI-detected?'; mean reflects 'how much of the application overall.' Both are imperfect.