Final SDF checkpoint · universes A and B · exact rendered model inputs
Review the belief-recall prompt changes
Compare the existing “developers using LLMs” prompts with the intervention. Red text is removed; green text is added. Answer choices and response-format instructions are included exactly as the model receives them.
- Remove the opening “Consider OpenAI’s code graders and developers using LLMs” preamble from every question.
- Use “developer community using AI-assisted coding tools” wherever the user authority is named.
- Use “list comprehension-style iteration” wherever comprehensions are named. Add “the” and adjust singular community grammar where needed.
Cases where the edit has a different effect
- Grader-recall questions still name the queried grader. Authority-attribution MCQs still name both authorities in the options.
- Open authority-attribution questions lose the named alternatives. One question template still refers to “the two authorities.”
- Existing set, dictionary, lazy and async scenarios remain. Generator-expression-specific wording is retained; list terminology is applied where comprehensions are mentioned.
- Question IDs, scenario assignments, target mappings, option order, response instructions, sampling settings and the LLM scoring rubric are preserved.
BeforeExisting short-user-name report
AfterNew final-checkpoint intervention
Expected answers and preserved sample identity
Expected answers differ between universes. These targets are evaluation metadata; they are not included in the model’s question. MCQ answer letters retain their existing ordering.
This page works offline. Download all prompt changes as JSON. Each prompt is sampled three times per universe at SDF step 1150 (9,200 documents). This review contains prompts, not evaluation results.