C3 — Do application features predict post-program publications?

Context

The most downstream outcome we can measure for fellows is post-program publications. The alumni-publication tracker covers cohorts 5.0–9.0 (584 entries) with paper counts and citation counts per fellow. We join to application data and ask: which application features predict whether a fellow goes on to publish?

Note: this is far downstream of selection. Many factors between getting into MATS and publishing — mentor fit, project, post-MATS opportunities — affect the outcome. We expect low R² / modest AUC, and 'publication' is a narrow outcome that doesn't capture all kinds of impact.

MATS (Machine Alignment, Transparency & Security) is an AI safety research fellowship that places ~120 fellows with ~100 mentors per cohort. Cohort 10.0 ran in summer 2026 and was the first cohort with a centralized application review instead of decentralized stream-specific review. This analysis is part of a broader effort to evaluate the 10.0 process and inform the design of 11.0 (autumn 2026).

How the 10.0 selection pipeline worked (click to expand)

The 10.0 pipeline in brief. ~2,200 people applied. Each applicant went through three stages:

  1. Stage 1 — submitted background / experience / motivation, picked which research tracks they were interested in (Empirical, Policy & Strategy, Technical Governance, Theory, Compute Infrastructure). An LLM screen filtered out applicants who clearly didn't meet a minimum bar, and produced advisory per-stream recommendations.
  2. Stage 2 — applicants who passed Stage 1 had their materials scored by LLM-graded rubrics. The empirical track used a composite score combining Research Skills, Technical Execution (split into MLE, SWE, Math sub-scores), and Soft Skills. The top ~600 by composite advanced to Stage 3.
  3. Stage 3 — applicants chose specific mentors / "streams" to apply to. Each stream reviewed its applicants and produced a ranked list. Top-ranked applicants got offers; lower-ranked got waitlisted. ~120 offers were made.

For the empirical track, the composite formula is 0.50·RS + 0.35·TE + 0.15·SS, where TE = 0.50·MLE + 0.30·SWE + 0.20·Math. A "relevance multiplier" (Direct=1.0 / Adjacent=0.85 / Distant=0.60) is applied to Research Skills based on how the applicant's experience matches the streams they applied to.

Outcome definitions (click to expand)

Outcome definitions used throughout these analyses:

Caveats

Per-cohort publication rates

Latest cohort n alumni-pub rows with ≥1 pub median n_pubs mean citations
5.0 26 14 1 41.7
5.1 25 19 1 45.9
6.0 42 26 1 33.0
6.1 33 31 3 37.8
7.0 27 19 2 10.3
7.1 46 37 2 18.9
8.0 21 10 0 0.9
8.1 75 47 1 8.3
9.0 87 12 0 2.2

Logistic-regression predictive performance

Cohort n with pub AUC 95% CI
6.0 42 26 0.724 [0.544, 0.883]
7.0 26 18 0.757 [0.542, 0.944]
8.0 21 10 0.777 [0.564, 0.969]
9.0 87 13 0.753 [0.593, 0.882]

Coefficient heatmap

Takeaways

  1. Application features carry moderate signal for predicting publication — in-sample AUCs land in the 0.72–0.78 range across cohorts. But these are in-sample on small samples (n ≈ 50–80) and likely optimistic; held-out AUCs would be lower.
  2. The centralized-review "Publication record" score has the strongest single contribution for 8.0/9.0 — unsurprising, since it's directly trying to predict the same thing using the applicant's prior publication history.
  3. Recency bias matters. 9.0 fellows have had ~6 months to publish; 6.0 fellows have had ~2 years. Cross-cohort comparisons of raw rates conflate "did MATS help" with "have they had time to publish."
  4. For 11.0: publication prediction is a noisy downstream outcome, but the centralized "publication record" feature does carry consistent signal. Worth keeping a similar prompt in the 11.0 rubric.
🔧 Debug — how the data was interpreted (click to expand; safe to skip)

Sample. Per cohort: alumni-pub rows whose latest cohort matches, joined to apps via person_id. Listwise-complete on cohort-specific features.

Outcome variable(s). has_post_pub (1 if ≥1 publication tracked).

Predictor fields. Per-cohort: CodeSignal, education ordinal, # bg-tier items (Familiar/Applied/Expert). 8.0/9.0: + centralized review scores.

Filters applied. Multi-cohort fellows attributed to latest cohort.

Missing-data handling. Listwise drop on features.

Key assumptions / caveats.