DS-1000 SMC Evaluation — Offline Proposal (weighted-SFT)

SMC weighted accuracy vs checkpoint

IS-weighted correctness over the M particles, per checkpoint, one line per particle count. Error bars = prompt-level bootstrap 95% CI.

SMC validity (no-error rate) vs checkpoint

Fraction of programs that run without error. Rises with M; no skyline (validity is not an accuracy).

iid sampling vs SMC weighted accuracy

Pure iid sampling from the proposal (no SMC, N=50 samples/prompt) overlaid on the SMC weighted-accuracy lines per M. Same prompt-level bootstrap 95% CI on every line (see Methods). The honest picture: iid ≈ SMC at M=1 (the twist only acts through resampling, which is inert at M=1) and the SMC lift at higher M stays within the CIs — on this held-out eval the no-error twist gives no measurable accuracy gain over raw sampling.

Per-prompt: base vs best checkpoint (interactive)

Hover any point for the prompt text. Below the diagonal = the best checkpoint (iter 13000) improved that prompt. Color = library; marker symbol = difficulty (✕ hard, ◆ medium, ● easy). Difficulty (cross-setting ceiling): hard 76medium 51easy 127 of 254.

Per-library net-gain (base → best, per M)

Δmean = mean(best) − mean(base) per library. Every library is net-positive at every M.

Methods

Bootstrap 95% CIs — the error bars on the line plots

A prompt-level percentile bootstrap, done independently for each (checkpoint, M) cell. The two things to keep straight: (i) we bootstrap one number per prompt (M and R are collapsed first), and (ii) the CI is a pair of percentiles, not an average of averages.

1. Collapse M and R → one scalar per prompt. For prompt p, run SMC R = 3 times; each run gives an IS-weighted correctness (Σ_i weight_i·correct_i over its M particles); average the R runs → wacc_p ∈ [0, 1]. This yields a vector v = [wacc₁, …, wacc₂₅₄] of length n = 254 — one value per held-out prompt (not per raw sample).
2. Resample v with replacement to size n. The unit resampled is the prompt (some prompts repeat, some drop out) — not individual samples.
3. Mean of that resample = one replicate. Repeat B = 2000× → 2000 bootstrap means.
4. Point estimate = mean(v) (the plotted marker). 95% CI = the 2.5th and 97.5th percentiles of the 2000 bootstrap means (the error bar). It is not the mean of the 2000 means.

def boot(v):                     # v = 254 per-prompt weighted accuracies
    means = [mean(resample_with_replacement(v, size=len(v))) for _ in range(2000)]
    return mean(v), percentile(means, 2.5), percentile(means, 97.5)   # point, lo, hi

Fixed seed. Because the unit is the prompt, the interval reflects how much the mean would move on a different draw of held-out prompts; the percentile method keeps it inside [0, 1]. (This is the macro mean — equal weight per prompt; it does not bootstrap the within-prompt Monte-Carlo noise, which would be a separate 2-level bootstrap.)

Skyline CI (the red band) — a different estimator

The skyline (0.378 [0.332, 0.424]) is pooled, not macro: Σ_p correct_p / Σ_p valid_p — a ratio of sums across prompts, so a prompt is implicitly weighted by how many valid samples it has. Its bootstrap re-pools each resample, so the interval is around the pooled value, not the per-prompt mean used above. The skyline band and the line-plot CIs are therefore not the same kind of interval.

iid sampling

“iid sampling” draws N = 50 independent samples per prompt straight from the proposal (temperature 0.6, no SMC, no twist). Per-prompt correctness = fraction of the 50 that run correctly; the plotted point is the macro-mean over the 254 prompts, with the same prompt-level bootstrap 95% CI as the SMC lines above. On this held-out eval the twisted SMC delivers no measurable lift over iid — iid ≈ SMC at M=1 (the twist acts only through resampling, which is inert at M=1) and the SMC lift at higher M stays within the CIs.

Difficulty — the scatter marker symbol

A fixed per-prompt “ceiling” = the max weighted accuracy that prompt reaches across every setting (base iter-0 and best iter-13000, over all M ∈ {1, 5, 10, 20}). hard: ceiling < 0.10 (no configuration solves it); medium: 0.10 ≤ ceiling < 0.50; easy: ceiling ≥ 0.50.