Belief recall: prompt intervention vs baseline
Review all prompt changes → · Open the unchanged baseline SDF report →
Recall at the same final checkpoint
Before = saved “developers using LLMs” prompts. After = explicit developer-community and list-comprehension wording, with the opening preamble removed. All attempts remain in the denominator.
Intervals are pointwise 95% scenario-cluster bootstraps (10,000 resamples; seed 0). Changes match the same scenarios and retain all repetitions together. They exclude training-seed and scorer uncertainty. Sampling uses the same seeds and settings with freshly generated intervention responses.
Paired grader and user answers
Separate style-recall responses are matched by scenario and repetition, within each universe, format, and cohort. These are independent generations, not a joint answer in one conversation.
- Both correct / both reversed: both authority assignments are recalled correctly / both assignments are swapped.
- Both comprehension / both loop: the same style is assigned to both authorities.
- Unscorable: at least one answer cannot be interpreted.
Each category cell shows count / total pairs, then percentage with its 95% scenario-cluster confidence interval. All pairs remain in the denominator; the five categories partition the total. Before and After use the same scenarios and repetitions.
Overall recall · all questions and formats
Scenario-family breakdown
Comprehension, loop, and unscorable answer shares
MCQ letter shares
Open-answer recall and scorable coverage
Open answers retain the original short-phrase instruction. GPT-6 Luna interprets their meaning using the same rubric; the expected assignment and checkpoint are withheld. MCQ scoring retains deterministic option decoding. Unscorable answers count as incorrect.
Use Matched responses to read both answers to the same question and repetition, along with the LLM grader’s evidence and rationale.
Grader minus user recall gap
Agreement under reversed MCQ options
Independent draws; agreement compares decoded meanings among pairs where both answers are scorable.
Read matched before/after responses
Protocol and comparability
- All 144 questions per format, six scenario families, both question directions, and the existing authority assignments.
- Three repetitions per format per universe. Only final step 1150 is rerun; existing baseline results are reused.
- Same target adapters, renderer, temperature 0.7, top-p 1.0, token limit 2048, generation seeds, and GPT-6 Luna rubric/settings.
- Changes combine preamble removal, authority naming, and comprehension wording. This intervention does not isolate their individual effects.
- Grader queries and attribution MCQ options still mention graders. Open attribution no longer names the two alternatives. Set/dict/async scenarios retain their contexts with the requested list label; generator-expression wording remains.