D6 — Is SRP/FRP a useful program-quality signal?

Context

Each cohort grades fellows on a research plan they produce during the program — 7.0 and 8.0 call it the SRP (Scholar Research Plan); 9.0 calls it the FRP (Final Research Plan). The plans are scored by reviewers. Are these scores a useful program-quality signal? Specifically: do they correlate with mentor evaluations? With post-program publications?

If they correlate, SRP/FRP is a usable intermediate quality metric and worth investing in. If they don't, the SRP/FRP grading is essentially noise and the program should reconsider whether it's worth the cost.

MATS (Machine Alignment, Transparency & Security) is an AI safety research fellowship that places ~120 fellows with ~100 mentors per cohort. Cohort 10.0 ran in summer 2026 and was the first cohort with a centralized application review instead of decentralized stream-specific review. This analysis is part of a broader effort to evaluate the 10.0 process and inform the design of 11.0 (autumn 2026).

How the 10.0 selection pipeline worked (click to expand)

The 10.0 pipeline in brief. ~2,200 people applied. Each applicant went through three stages:

  1. Stage 1 — submitted background / experience / motivation, picked which research tracks they were interested in (Empirical, Policy & Strategy, Technical Governance, Theory, Compute Infrastructure). An LLM screen filtered out applicants who clearly didn't meet a minimum bar, and produced advisory per-stream recommendations.
  2. Stage 2 — applicants who passed Stage 1 had their materials scored by LLM-graded rubrics. The empirical track used a composite score combining Research Skills, Technical Execution (split into MLE, SWE, Math sub-scores), and Soft Skills. The top ~600 by composite advanced to Stage 3.
  3. Stage 3 — applicants chose specific mentors / "streams" to apply to. Each stream reviewed its applicants and produced a ranked list. Top-ranked applicants got offers; lower-ranked got waitlisted. ~120 offers were made.

For the empirical track, the composite formula is 0.50·RS + 0.35·TE + 0.15·SS, where TE = 0.50·MLE + 0.30·SWE + 0.20·Math. A "relevance multiplier" (Direct=1.0 / Adjacent=0.85 / Distant=0.60) is applied to Research Skills based on how the applicant's experience matches the streams they applied to.

Outcome definitions (click to expand)

Outcome definitions used throughout these analyses:

SRP/FRP rubric differences

Cross-cohort comparison of raw scores doesn't make sense; we look at rank-based correlations only.

SRP/FRP final score × mentor-eval composite

Cohort n joined ρ(SRP/FRP, mentor composite)
7.0 72 +0.23
8.0 98 +0.11
9.0 86 +0.06

SRP/FRP final score × post-program publications

Cohort n ρ(SRP/FRP, has_pub) ρ(SRP/FRP, n_pubs)
7.0 76 +0.08 +0.03
8.0 100 -0.03 -0.04
9.0 87 +0.04 +0.04

Takeaways

  1. SRP/FRP × mentor-eval correlations are modest at best, and decreasing across cohorts (7.0: ρ ≈ +0.23, 8.0: +0.11, 9.0: +0.06). The 9.0 FRP rubric — which is the most ambitious in scope — actually shows the WEAKEST alignment with mentor evaluations.
  2. SRP/FRP × publications correlations are also modest but generally positive.
  3. Possible interpretations: (a) the new FRP rubric is measuring something orthogonal to what mentors value (e.g., 'AI risk reduction' is a different construct from 'research execution'); (b) noise dominates at these small samples; (c) FRP and mentor evals are both measuring real-but-different program-quality dimensions.
  4. For 11.0 and beyond: don't assume SRP/FRP is a tight proxy for mentor-eval signal. If we want a "did this fellow do well" signal, mentor evals remain the cleaner instrument. SRP/FRP is worth keeping as a fellow-side artifact and forcing function, but not as an analytical proxy for mentor quality assessment.
🔧 Debug — how the data was interpreted (click to expand; safe to skip)

Sample. Per cohort: SRP/FRP per-person final scores (averaged across team rows) joined to (a) mentor-eval composite (also averaged per person), and (b) alumni publication tracker.

Outcome variable(s). Mentor-eval composite + post-program publication outcomes.

Predictor fields. SRP/FRP final_score — within-cohort raw score (different rubrics across cohorts; only rank-correlations reported).

Filters applied. Inner join on person_id.

Missing-data handling. Listwise drop per correlation.

Key assumptions / caveats.