Inside a pretrained language model

%23%20%2F%2F%2F%20script%0A%23%20requires-python%20%3D%20%22%3E%3D3.11%22%0A%23%20dependencies%20%3D%20%5B%22marimo%3D%3D0.24.0%22%2C%20%22numpy%3E%3D2.1%22%2C%20%22torch%3E%3D2.10%3B%20sys_platform%20!%3D%20'emscripten'%22%2C%20%22transformers%3D%3D4.57.6%3B%20sys_platform%20!%3D%20'emscripten'%22%5D%0A%23%20%2F%2F%2F%0A%0Aimport%20marimo%0A%0A__generated_with%20%3D%20%220.24.0%22%0Aapp%20%3D%20marimo.App(%0A%20%20%20%20width%3D%22medium%22%2C%0A%20%20%20%20app_title%3D%22Inside%20a%20pretrained%20language%20model%22%2C%0A)%0A%0A%0A%40app.cell%0Adef%20_()%3A%0A%20%20%20%20import%20marimo%20as%20mo%0A%20%20%20%20import%20json%2Csys%2Chtml%2Cmath%0A%20%20%20%20import%20numpy%20as%20np%0A%20%20%20%20from%20pathlib%20import%20Path%0A%0A%20%20%20%20return%20Path%2C%20html%2C%20json%2C%20math%2C%20mo%2C%20np%2C%20sys%0A%0A%0A%40app.cell%0Adef%20_()%3A%0A%20%20%20%20recorded%3D%7B'model'%3A%20'EleutherAI%2Fpythia-70m-deduped'%2C%20'revision'%3A%20'e93a9faa9c77e5d09219f6c868bfc7a1bd65593c'%2C%20'torch'%3A%20'2.14.0'%2C%20'transformers'%3A%20'4.57.6'%2C%20'device'%3A%20'CPU'%2C%20'inference_seconds'%3A%203.6340336670109536%2C%20'collected_at_utc'%3A%20'2026-09-07T01%3A31%3A08.751532%2B00%3A00'%2C%20'precision'%3A%20'float32'%2C%20'method'%3A%20'Clean%20post-block%20residual%20vectors%20patched%20into%20corrupted%20prompts%20at%20changed-token%20and%20final-token%20positions.%20No%20weight%20changes.%20Controls%20are%20interventions%2C%20not%20guaranteed%20null%20effects.%20Candidate%20probabilities%20use%20the%20full%20vocabulary%20denominator.'%2C%20'limits'%3A%20'Six%20selected%20capital-city%20prompt%20pairs%2C%20with%20no-op%2C%20zero%20and%20one%20seeded%20norm-matched%20random%20intervention%20control%20in%20a%20small%20pretrained%20research%20model.%20Not%20a%20representative%20accuracy%20evaluation%2C%20a%20Goodfire%20replication%20or%20a%20frontier-model%20result.'%2C%20'records'%3A%20%5B%7B'clean'%3A%20'The%20capital%20of%20France%20is'%2C%20'corrupt'%3A%20'The%20capital%20of%20England%20is'%2C%20'clean_answer'%3A%20'%20Paris'%2C%20'corrupt_answer'%3A%20'%20London'%2C%20'clean_tokens'%3A%20%5B'The'%2C%20'%C4%A0capital'%2C%20'%C4%A0of'%2C%20'%C4%A0France'%2C%20'%C4%A0is'%5D%2C%20'corrupt_tokens'%3A%20%5B'The'%2C%20'%C4%A0capital'%2C%20'%C4%A0of'%2C%20'%C4%A0England'%2C%20'%C4%A0is'%5D%2C%20'positions'%3A%20%5B3%2C%204%5D%2C%20'rows'%3A%20%5B%7B'condition'%3A%20'Clean'%2C%20'layer'%3A%20-1%2C%20'position'%3A%20-1%2C%20'logit_difference'%3A%201.55712890625%2C%20'p_clean_answer'%3A%200.0019448678940534592%2C%20'p_corrupt_answer'%3A%200.0004098619392607361%7D%2C%20%7B'condition'%3A%20'Corrupted'%2C%20'layer'%3A%20-1%2C%20'position'%3A%20-1%2C%20'logit_difference'%3A%20-2.048828125%2C%20'p_clean_answer'%3A%200.000294992933049798%2C%20'p_corrupt_answer'%3A%200.0022887922823429108%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%200%2C%20'position'%3A%203%2C%20'logit_difference'%3A%201.423828125%2C%20'p_clean_answer'%3A%200.002181690651923418%2C%20'p_corrupt_answer'%3A%200.0005253303097561002%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%200%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-2.048828125%2C%20'p_clean_answer'%3A%200.000294992933049798%2C%20'p_corrupt_answer'%3A%200.0022887922823429108%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%200%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-0.5413818359375%2C%20'p_clean_answer'%3A%200.0003822331200353801%2C%20'p_corrupt_answer'%3A%200.0006568216485902667%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%200%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-0.289794921875%2C%20'p_clean_answer'%3A%200.00020540517289191484%2C%20'p_corrupt_answer'%3A%200.0002744528464972973%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%200%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-2.025634765625%2C%20'p_clean_answer'%3A%200.0002625109045766294%2C%20'p_corrupt_answer'%3A%200.001990074524655938%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%200%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-2.048828125%2C%20'p_clean_answer'%3A%200.000294992933049798%2C%20'p_corrupt_answer'%3A%200.0022887922823429108%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%200%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-0.6724853515625%2C%20'p_clean_answer'%3A%200.0008020123350434005%2C%20'p_corrupt_answer'%3A%200.0015712225576862693%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%200%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-1.01953125%2C%20'p_clean_answer'%3A%200.000542323337867856%2C%20'p_corrupt_answer'%3A%200.0015032633673399687%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%201%2C%20'position'%3A%203%2C%20'logit_difference'%3A%201.3741455078125%2C%20'p_clean_answer'%3A%200.002201675670221448%2C%20'p_corrupt_answer'%3A%200.0005571466172114015%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%201%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-2.048828125%2C%20'p_clean_answer'%3A%200.000294992933049798%2C%20'p_corrupt_answer'%3A%200.0022887922823429108%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%201%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-0.8779296875%2C%20'p_clean_answer'%3A%200.00014560851559508592%2C%20'p_corrupt_answer'%3A%200.00035032149753533304%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%201%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-0.7838134765625%2C%20'p_clean_answer'%3A%200.00025201603421010077%2C%20'p_corrupt_answer'%3A%200.000551866483874619%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%201%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-1.9620361328125%2C%20'p_clean_answer'%3A%200.0002597339334897697%2C%20'p_corrupt_answer'%3A%200.001847694511525333%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%201%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-2.048828125%2C%20'p_clean_answer'%3A%200.000294992933049798%2C%20'p_corrupt_answer'%3A%200.0022887922823429108%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%201%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-1.0125732421875%2C%20'p_clean_answer'%3A%200.00027147491346113384%2C%20'p_corrupt_answer'%3A%200.0007472822908312082%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%201%2C%20'position'%3A%204%2C%20'logit_difference'%3A%201.680419921875%2C%20'p_clean_answer'%3A%200.0005524078151211143%2C%20'p_corrupt_answer'%3A%200.00010291121725458652%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%202%2C%20'position'%3A%203%2C%20'logit_difference'%3A%201.2349853515625%2C%20'p_clean_answer'%3A%200.0023293474223464727%2C%20'p_corrupt_answer'%3A%200.0006774651119485497%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%202%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-2.048828125%2C%20'p_clean_answer'%3A%200.000294992933049798%2C%20'p_corrupt_answer'%3A%200.0022887922823429108%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%202%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-0.679443359375%2C%20'p_clean_answer'%3A%200.00027209415566176176%2C%20'p_corrupt_answer'%3A%200.0005367817357182503%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%202%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-0.760986328125%2C%20'p_clean_answer'%3A%200.00014679215382784605%2C%20'p_corrupt_answer'%3A%200.00031419191509485245%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%202%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-1.751953125%2C%20'p_clean_answer'%3A%200.000279109226539731%2C%20'p_corrupt_answer'%3A%200.0016093028243631124%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%202%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-2.048828125%2C%20'p_clean_answer'%3A%200.000294992933049798%2C%20'p_corrupt_answer'%3A%200.0022887922823429108%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%202%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-0.4215087890625%2C%20'p_clean_answer'%3A%200.001912839012220502%2C%20'p_corrupt_answer'%3A%200.002915663179010153%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%202%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-0.4456787109375%2C%20'p_clean_answer'%3A%202.0159914129180834e-06%2C%20'p_corrupt_answer'%3A%203.1480708457820583e-06%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%203%2C%20'position'%3A%203%2C%20'logit_difference'%3A%201.1746826171875%2C%20'p_clean_answer'%3A%200.0022832814138382673%2C%20'p_corrupt_answer'%3A%200.0007053444278426468%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%203%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-2.048828125%2C%20'p_clean_answer'%3A%200.000294992933049798%2C%20'p_corrupt_answer'%3A%200.0022887922823429108%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%203%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-1.0277099609375%2C%20'p_clean_answer'%3A%200.0003015504917129874%2C%20'p_corrupt_answer'%3A%200.0008427306893281639%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%203%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-1.00439453125%2C%20'p_clean_answer'%3A%200.0002978213015012443%2C%20'p_corrupt_answer'%3A%200.0008131277281790972%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%203%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-1.6658935546875%2C%20'p_clean_answer'%3A%200.00030106809572316706%2C%20'p_corrupt_answer'%3A%200.0015927701024338603%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%203%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-2.048828125%2C%20'p_clean_answer'%3A%200.000294992933049798%2C%20'p_corrupt_answer'%3A%200.0022887922823429108%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%203%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-0.0343017578125%2C%20'p_clean_answer'%3A%208.643014734843746e-05%2C%20'p_corrupt_answer'%3A%208.944627916207537e-05%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%203%2C%20'position'%3A%204%2C%20'logit_difference'%3A%200.8345947265625%2C%20'p_clean_answer'%3A%200.000573255936615169%2C%20'p_corrupt_answer'%3A%200.00024882194702513516%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%204%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-1.8699951171875%2C%20'p_clean_answer'%3A%200.0003143977955915034%2C%20'p_corrupt_answer'%3A%200.0020398960914462805%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%204%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-2.048828125%2C%20'p_clean_answer'%3A%200.000294992933049798%2C%20'p_corrupt_answer'%3A%200.0022887922823429108%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%204%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-1.9287109375%2C%20'p_clean_answer'%3A%200.0003367163008078933%2C%20'p_corrupt_answer'%3A%200.002316822065040469%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%204%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-1.923583984375%2C%20'p_clean_answer'%3A%200.00032731282408349216%2C%20'p_corrupt_answer'%3A%200.0022406030911952257%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%204%2C%20'position'%3A%204%2C%20'logit_difference'%3A%201.3609619140625%2C%20'p_clean_answer'%3A%200.001813808805309236%2C%20'p_corrupt_answer'%3A%200.00046508596278727055%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%204%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-2.048828125%2C%20'p_clean_answer'%3A%200.000294992933049798%2C%20'p_corrupt_answer'%3A%200.0022887922823429108%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%204%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-1.01708984375%2C%20'p_clean_answer'%3A%204.087056004209444e-05%2C%20'p_corrupt_answer'%3A%200.00011301265476504341%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%204%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-0.6378173828125%2C%20'p_clean_answer'%3A%202.122823389072437e-05%2C%20'p_corrupt_answer'%3A%204.0171162254409865e-05%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%205%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-2.048828125%2C%20'p_clean_answer'%3A%200.000294992933049798%2C%20'p_corrupt_answer'%3A%200.0022887922823429108%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%205%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-2.048828125%2C%20'p_clean_answer'%3A%200.000294992933049798%2C%20'p_corrupt_answer'%3A%200.0022887922823429108%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%205%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-2.048828125%2C%20'p_clean_answer'%3A%200.000294992933049798%2C%20'p_corrupt_answer'%3A%200.0022887922823429108%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%205%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-2.048828125%2C%20'p_clean_answer'%3A%200.000294992933049798%2C%20'p_corrupt_answer'%3A%200.0022887922823429108%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%205%2C%20'position'%3A%204%2C%20'logit_difference'%3A%201.55712890625%2C%20'p_clean_answer'%3A%200.0019448678940534592%2C%20'p_corrupt_answer'%3A%200.0004098619392607361%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%205%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-2.048828125%2C%20'p_clean_answer'%3A%200.000294992933049798%2C%20'p_corrupt_answer'%3A%200.0022887922823429108%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%205%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-0.29443359375%2C%20'p_clean_answer'%3A%205.812497693113983e-05%2C%20'p_corrupt_answer'%3A%207.802498294040561e-05%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%205%2C%20'position'%3A%204%2C%20'logit_difference'%3A%2014.4002685546875%2C%20'p_clean_answer'%3A%201.686214883001025e-11%2C%20'p_corrupt_answer'%3A%209.396276277415475e-18%7D%5D%2C%20'residual_norms'%3A%20%5B7.000734329223633%2C%209.803852081298828%2C%2010.563419342041016%2C%2011.329485893249512%2C%2016.035472869873047%2C%2093.23194885253906%5D%7D%2C%20%7B'clean'%3A%20'The%20capital%20of%20Germany%20is'%2C%20'corrupt'%3A%20'The%20capital%20of%20Spain%20is'%2C%20'clean_answer'%3A%20'%20Berlin'%2C%20'corrupt_answer'%3A%20'%20Madrid'%2C%20'clean_tokens'%3A%20%5B'The'%2C%20'%C4%A0capital'%2C%20'%C4%A0of'%2C%20'%C4%A0Germany'%2C%20'%C4%A0is'%5D%2C%20'corrupt_tokens'%3A%20%5B'The'%2C%20'%C4%A0capital'%2C%20'%C4%A0of'%2C%20'%C4%A0Spain'%2C%20'%C4%A0is'%5D%2C%20'positions'%3A%20%5B3%2C%204%5D%2C%20'rows'%3A%20%5B%7B'condition'%3A%20'Clean'%2C%20'layer'%3A%20-1%2C%20'position'%3A%20-1%2C%20'logit_difference'%3A%203.068603515625%2C%20'p_clean_answer'%3A%200.001607217127457261%2C%20'p_corrupt_answer'%3A%207.471314165741205e-05%7D%2C%20%7B'condition'%3A%20'Corrupted'%2C%20'layer'%3A%20-1%2C%20'position'%3A%20-1%2C%20'logit_difference'%3A%20-2.5987548828125%2C%20'p_clean_answer'%3A%209.333924390375614e-05%2C%20'p_corrupt_answer'%3A%200.0012551313266158104%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%200%2C%20'position'%3A%203%2C%20'logit_difference'%3A%203.0379638671875%2C%20'p_clean_answer'%3A%200.0014795990427955985%2C%20'p_corrupt_answer'%3A%207.092070882208645e-05%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%200%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-2.5987548828125%2C%20'p_clean_answer'%3A%209.333924390375614e-05%2C%20'p_corrupt_answer'%3A%200.0012551313266158104%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%200%2C%20'position'%3A%203%2C%20'logit_difference'%3A%200.7447509765625%2C%20'p_clean_answer'%3A%200.00010520592331886292%2C%20'p_corrupt_answer'%3A%204.995729977963492e-05%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%200%2C%20'position'%3A%203%2C%20'logit_difference'%3A%201.0115966796875%2C%20'p_clean_answer'%3A%205.9207144659012556e-05%2C%20'p_corrupt_answer'%3A%202.1529962396016344e-05%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%200%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-2.58154296875%2C%20'p_clean_answer'%3A%200.00010416343138786033%2C%20'p_corrupt_answer'%3A%200.0013767818454653025%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%200%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-2.5987548828125%2C%20'p_clean_answer'%3A%209.333924390375614e-05%2C%20'p_corrupt_answer'%3A%200.0012551313266158104%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%200%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-1.5528564453125%2C%20'p_clean_answer'%3A%200.00025087196263484657%2C%20'p_corrupt_answer'%3A%200.0011853568721562624%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%200%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-2.76513671875%2C%20'p_clean_answer'%3A%200.0003131017729174346%2C%20'p_corrupt_answer'%3A%200.004972435068339109%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%201%2C%20'position'%3A%203%2C%20'logit_difference'%3A%202.9693603515625%2C%20'p_clean_answer'%3A%200.0013163266703486443%2C%20'p_corrupt_answer'%3A%206.757512164767832e-05%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%201%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-2.5987548828125%2C%20'p_clean_answer'%3A%209.333924390375614e-05%2C%20'p_corrupt_answer'%3A%200.0012551313266158104%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%201%2C%20'position'%3A%203%2C%20'logit_difference'%3A%200.5950927734375%2C%20'p_clean_answer'%3A%203.713394471560605e-05%2C%20'p_corrupt_answer'%3A%202.0479794329730794e-05%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%201%2C%20'position'%3A%203%2C%20'logit_difference'%3A%200.602294921875%2C%20'p_clean_answer'%3A%207.514503522543237e-05%2C%20'p_corrupt_answer'%3A%204.114593320991844e-05%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%201%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-2.5191650390625%2C%20'p_clean_answer'%3A%200.00012140027683926746%2C%20'p_corrupt_answer'%3A%200.0015075757401064038%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%201%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-2.5987548828125%2C%20'p_clean_answer'%3A%209.333924390375614e-05%2C%20'p_corrupt_answer'%3A%200.0012551313266158104%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%201%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-0.78173828125%2C%20'p_clean_answer'%3A%205.7456709328107536e-05%2C%20'p_corrupt_answer'%3A%200.0001255582901649177%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%201%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-1.922119140625%2C%20'p_clean_answer'%3A%203.0606686777900904e-05%2C%20'p_corrupt_answer'%3A%200.00020920981478411704%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%202%2C%20'position'%3A%203%2C%20'logit_difference'%3A%202.7733154296875%2C%20'p_clean_answer'%3A%200.0010440130718052387%2C%20'p_corrupt_answer'%3A%206.520341412397102e-05%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%202%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-2.5987548828125%2C%20'p_clean_answer'%3A%209.333924390375614e-05%2C%20'p_corrupt_answer'%3A%200.0012551313266158104%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%202%2C%20'position'%3A%203%2C%20'logit_difference'%3A%200.6639404296875%2C%20'p_clean_answer'%3A%207.7886987128295e-05%2C%20'p_corrupt_answer'%3A%204.009767872048542e-05%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%202%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-0.1409912109375%2C%20'p_clean_answer'%3A%202.412022695352789e-05%2C%20'p_corrupt_answer'%3A%202.777237750706263e-05%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%202%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-2.3896484375%2C%20'p_clean_answer'%3A%200.00015468271158169955%2C%20'p_corrupt_answer'%3A%200.0016875354340299964%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%202%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-2.5987548828125%2C%20'p_clean_answer'%3A%209.333924390375614e-05%2C%20'p_corrupt_answer'%3A%200.0012551313266158104%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%202%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-2.9462890625%2C%20'p_clean_answer'%3A%200.0007418054738081992%2C%20'p_corrupt_answer'%3A%200.014120404608547688%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%202%2C%20'position'%3A%204%2C%20'logit_difference'%3A%202.842529296875%2C%20'p_clean_answer'%3A%209.238103666575626e-05%2C%20'p_corrupt_answer'%3A%205.383789357438218e-06%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%203%2C%20'position'%3A%203%2C%20'logit_difference'%3A%202.75830078125%2C%20'p_clean_answer'%3A%200.0010704115265980363%2C%20'p_corrupt_answer'%3A%206.786346057197079e-05%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%203%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-2.5987548828125%2C%20'p_clean_answer'%3A%209.333924390375614e-05%2C%20'p_corrupt_answer'%3A%200.0012551313266158104%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%203%2C%20'position'%3A%203%2C%20'logit_difference'%3A%200.6549072265625%2C%20'p_clean_answer'%3A%208.715256990399212e-05%2C%20'p_corrupt_answer'%3A%204.5274911826709285e-05%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%203%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-0.4622802734375%2C%20'p_clean_answer'%3A%207.985283446032554e-06%2C%20'p_corrupt_answer'%3A%201.2678156963374931e-05%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%203%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-2.3626708984375%2C%20'p_clean_answer'%3A%200.00015361467376351357%2C%20'p_corrupt_answer'%3A%200.0016312767984345555%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%203%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-2.5987548828125%2C%20'p_clean_answer'%3A%209.333924390375614e-05%2C%20'p_corrupt_answer'%3A%200.0012551313266158104%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%203%2C%20'position'%3A%204%2C%20'logit_difference'%3A%200.63916015625%2C%20'p_clean_answer'%3A%205.1978200644953176e-05%2C%20'p_corrupt_answer'%3A%202.7430738555267453e-05%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%203%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-2.4149169921875%2C%20'p_clean_answer'%3A%201.9674960640259087e-05%2C%20'p_corrupt_answer'%3A%200.0002201400202466175%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%204%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-2.4090576171875%2C%20'p_clean_answer'%3A%200.00010042615758720785%2C%20'p_corrupt_answer'%3A%200.0011170876678079367%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%204%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-2.5987548828125%2C%20'p_clean_answer'%3A%209.333924390375614e-05%2C%20'p_corrupt_answer'%3A%200.0012551313266158104%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%204%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-2.3438720703125%2C%20'p_clean_answer'%3A%200.00010624151036608964%2C%20'p_corrupt_answer'%3A%200.0011071971384808421%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%204%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-2.3389892578125%2C%20'p_clean_answer'%3A%200.00010488642146810889%2C%20'p_corrupt_answer'%3A%200.0010877508902922273%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%204%2C%20'position'%3A%204%2C%20'logit_difference'%3A%202.8773193359375%2C%20'p_clean_answer'%3A%200.0015002540312707424%2C%20'p_corrupt_answer'%3A%208.444246486760676e-05%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%204%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-2.5987548828125%2C%20'p_clean_answer'%3A%209.333924390375614e-05%2C%20'p_corrupt_answer'%3A%200.0012551313266158104%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%204%2C%20'position'%3A%204%2C%20'logit_difference'%3A%200.6298828125%2C%20'p_clean_answer'%3A%202.1702875528717414e-05%2C%20'p_corrupt_answer'%3A%201.1560127859411296e-05%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%204%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-1.8475341796875%2C%20'p_clean_answer'%3A%205.886911367269931e-06%2C%20'p_corrupt_answer'%3A%203.7347490433603525e-05%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%205%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-2.5987548828125%2C%20'p_clean_answer'%3A%209.333924390375614e-05%2C%20'p_corrupt_answer'%3A%200.0012551313266158104%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%205%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-2.5987548828125%2C%20'p_clean_answer'%3A%209.333924390375614e-05%2C%20'p_corrupt_answer'%3A%200.0012551313266158104%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%205%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-2.5987548828125%2C%20'p_clean_answer'%3A%209.333924390375614e-05%2C%20'p_corrupt_answer'%3A%200.0012551313266158104%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%205%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-2.5987548828125%2C%20'p_clean_answer'%3A%209.333924390375614e-05%2C%20'p_corrupt_answer'%3A%200.0012551313266158104%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%205%2C%20'position'%3A%204%2C%20'logit_difference'%3A%203.068603515625%2C%20'p_clean_answer'%3A%200.001607217127457261%2C%20'p_corrupt_answer'%3A%207.471314165741205e-05%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%205%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-2.5987548828125%2C%20'p_clean_answer'%3A%209.333924390375614e-05%2C%20'p_corrupt_answer'%3A%200.0012551313266158104%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%205%2C%20'position'%3A%204%2C%20'logit_difference'%3A%201.03857421875%2C%20'p_clean_answer'%3A%205.7739562180358917e-05%2C%20'p_corrupt_answer'%3A%202.0437437342479825e-05%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%205%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-4.07440185546875%2C%20'p_clean_answer'%3A%202.09249480733763e-17%2C%20'p_corrupt_answer'%3A%201.230707002745858e-15%7D%5D%2C%20'residual_norms'%3A%20%5B7.110810279846191%2C%209.787389755249023%2C%2010.446525573730469%2C%2011.36587905883789%2C%2015.912463188171387%2C%2094.16485595703125%5D%7D%2C%20%7B'clean'%3A%20'The%20capital%20of%20Japan%20is'%2C%20'corrupt'%3A%20'The%20capital%20of%20China%20is'%2C%20'clean_answer'%3A%20'%20Tokyo'%2C%20'corrupt_answer'%3A%20'%20Beijing'%2C%20'clean_tokens'%3A%20%5B'The'%2C%20'%C4%A0capital'%2C%20'%C4%A0of'%2C%20'%C4%A0Japan'%2C%20'%C4%A0is'%5D%2C%20'corrupt_tokens'%3A%20%5B'The'%2C%20'%C4%A0capital'%2C%20'%C4%A0of'%2C%20'%C4%A0China'%2C%20'%C4%A0is'%5D%2C%20'positions'%3A%20%5B3%2C%204%5D%2C%20'rows'%3A%20%5B%7B'condition'%3A%20'Clean'%2C%20'layer'%3A%20-1%2C%20'position'%3A%20-1%2C%20'logit_difference'%3A%202.912353515625%2C%20'p_clean_answer'%3A%200.004140540491789579%2C%20'p_corrupt_answer'%3A%200.00022502873616758734%7D%2C%20%7B'condition'%3A%20'Corrupted'%2C%20'layer'%3A%20-1%2C%20'position'%3A%20-1%2C%20'logit_difference'%3A%20-1.34375%2C%20'p_clean_answer'%3A%200.00020214311371091753%2C%20'p_corrupt_answer'%3A%200.0007748937350697815%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%200%2C%20'position'%3A%203%2C%20'logit_difference'%3A%202.80615234375%2C%20'p_clean_answer'%3A%200.003330013481900096%2C%20'p_corrupt_answer'%3A%200.00020125630544498563%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%200%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-1.34375%2C%20'p_clean_answer'%3A%200.00020214311371091753%2C%20'p_corrupt_answer'%3A%200.0007748937350697815%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%200%2C%20'position'%3A%203%2C%20'logit_difference'%3A%200.9674072265625%2C%20'p_clean_answer'%3A%207.84801522968337e-05%2C%20'p_corrupt_answer'%3A%202.9827731850673445e-05%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%200%2C%20'position'%3A%203%2C%20'logit_difference'%3A%200.894287109375%2C%20'p_clean_answer'%3A%207.142584945540875e-05%2C%20'p_corrupt_answer'%3A%202.9205957616795786e-05%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%200%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-1.2821044921875%2C%20'p_clean_answer'%3A%200.000229804907576181%2C%20'p_corrupt_answer'%3A%200.0008282666094601154%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%200%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-1.34375%2C%20'p_clean_answer'%3A%200.00020214311371091753%2C%20'p_corrupt_answer'%3A%200.0007748937350697815%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%200%2C%20'position'%3A%204%2C%20'logit_difference'%3A%200.213134765625%2C%20'p_clean_answer'%3A%200.00010500505595700815%2C%20'p_corrupt_answer'%3A%208.484904537908733e-05%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%200%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-1.3072509765625%2C%20'p_clean_answer'%3A%209.503376350039616e-05%2C%20'p_corrupt_answer'%3A%200.0003512447001412511%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%201%2C%20'position'%3A%203%2C%20'logit_difference'%3A%202.79345703125%2C%20'p_clean_answer'%3A%200.0033180085010826588%2C%20'p_corrupt_answer'%3A%200.0002030928008025512%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%201%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-1.34375%2C%20'p_clean_answer'%3A%200.00020214311371091753%2C%20'p_corrupt_answer'%3A%200.0007748937350697815%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%201%2C%20'position'%3A%203%2C%20'logit_difference'%3A%200.787841796875%2C%20'p_clean_answer'%3A%209.197239705827087e-05%2C%20'p_corrupt_answer'%3A%204.183137571089901e-05%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%201%2C%20'position'%3A%203%2C%20'logit_difference'%3A%200.947998046875%2C%20'p_clean_answer'%3A%203.354357977514155e-05%2C%20'p_corrupt_answer'%3A%201.2998675629205536e-05%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%201%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-1.24609375%2C%20'p_clean_answer'%3A%200.00023990280169527978%2C%20'p_corrupt_answer'%3A%200.0008340785861946642%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%201%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-1.34375%2C%20'p_clean_answer'%3A%200.00020214311371091753%2C%20'p_corrupt_answer'%3A%200.0007748937350697815%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%201%2C%20'position'%3A%204%2C%20'logit_difference'%3A%200.51416015625%2C%20'p_clean_answer'%3A%202.9260985684231855e-05%2C%20'p_corrupt_answer'%3A%201.7498146917205304e-05%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%201%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-4.5281982421875%2C%20'p_clean_answer'%3A%203.490705466902e-07%2C%20'p_corrupt_answer'%3A%203.23209933412727e-05%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%202%2C%20'position'%3A%203%2C%20'logit_difference'%3A%202.75732421875%2C%20'p_clean_answer'%3A%200.002236707368865609%2C%20'p_corrupt_answer'%3A%200.0001419444743078202%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%202%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-1.34375%2C%20'p_clean_answer'%3A%200.00020214311371091753%2C%20'p_corrupt_answer'%3A%200.0007748937350697815%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%202%2C%20'position'%3A%203%2C%20'logit_difference'%3A%200.585693359375%2C%20'p_clean_answer'%3A%206.781737465644255e-05%2C%20'p_corrupt_answer'%3A%203.775526784011163e-05%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%202%2C%20'position'%3A%203%2C%20'logit_difference'%3A%200.908447265625%2C%20'p_clean_answer'%3A%206.399814446922392e-05%2C%20'p_corrupt_answer'%3A%202.5800834919209592e-05%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%202%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-1.1846923828125%2C%20'p_clean_answer'%3A%200.0003886999620590359%2C%20'p_corrupt_answer'%3A%200.0012709248112514615%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%202%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-1.34375%2C%20'p_clean_answer'%3A%200.00020214311371091753%2C%20'p_corrupt_answer'%3A%200.0007748937350697815%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%202%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-0.406982421875%2C%20'p_clean_answer'%3A%200.0013223610585555434%2C%20'p_corrupt_answer'%3A%200.0019865534268319607%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%202%2C%20'position'%3A%204%2C%20'logit_difference'%3A%201.3817138671875%2C%20'p_clean_answer'%3A%207.933635970402975e-06%2C%20'p_corrupt_answer'%3A%201.9925148535548942e-06%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%203%2C%20'position'%3A%203%2C%20'logit_difference'%3A%202.6856689453125%2C%20'p_clean_answer'%3A%200.002437145449221134%2C%20'p_corrupt_answer'%3A%200.00016615379718132317%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%203%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-1.34375%2C%20'p_clean_answer'%3A%200.00020214311371091753%2C%20'p_corrupt_answer'%3A%200.0007748937350697815%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%203%2C%20'position'%3A%203%2C%20'logit_difference'%3A%200.5565185546875%2C%20'p_clean_answer'%3A%209.430326463188976e-05%2C%20'p_corrupt_answer'%3A%205.405474075814709e-05%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%203%2C%20'position'%3A%203%2C%20'logit_difference'%3A%200.895263671875%2C%20'p_clean_answer'%3A%200.0001781375758582726%2C%20'p_corrupt_answer'%3A%207.276918040588498e-05%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%203%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-1.1141357421875%2C%20'p_clean_answer'%3A%200.00035266217309981585%2C%20'p_corrupt_answer'%3A%200.0010745382169261575%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%203%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-1.34375%2C%20'p_clean_answer'%3A%200.00020214311371091753%2C%20'p_corrupt_answer'%3A%200.0007748937350697815%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%203%2C%20'position'%3A%204%2C%20'logit_difference'%3A%200.84033203125%2C%20'p_clean_answer'%3A%209.894199138216209e-06%2C%20'p_corrupt_answer'%3A%204.2700121412053704e-06%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%203%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-0.87255859375%2C%20'p_clean_answer'%3A%203.9202728657983243e-05%2C%20'p_corrupt_answer'%3A%209.381313429912552e-05%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%204%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-1.252685546875%2C%20'p_clean_answer'%3A%200.00022109344718046486%2C%20'p_corrupt_answer'%3A%200.0007737671840004623%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%204%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-1.34375%2C%20'p_clean_answer'%3A%200.00020214311371091753%2C%20'p_corrupt_answer'%3A%200.0007748937350697815%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%204%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-1.31884765625%2C%20'p_clean_answer'%3A%200.00021481336443684995%2C%20'p_corrupt_answer'%3A%200.0008032108889892697%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%204%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-1.75%2C%20'p_clean_answer'%3A%206.665904220426455e-05%2C%20'p_corrupt_answer'%3A%200.0003835963143501431%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%204%2C%20'position'%3A%204%2C%20'logit_difference'%3A%202.8165283203125%2C%20'p_clean_answer'%3A%200.00378618948161602%2C%20'p_corrupt_answer'%3A%200.00022646423894912004%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%204%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-1.34375%2C%20'p_clean_answer'%3A%200.00020214311371091753%2C%20'p_corrupt_answer'%3A%200.0007748937350697815%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%204%2C%20'position'%3A%204%2C%20'logit_difference'%3A%201.42626953125%2C%20'p_clean_answer'%3A%203.3595075365155935e-05%2C%20'p_corrupt_answer'%3A%208.06964817456901e-06%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%204%2C%20'position'%3A%204%2C%20'logit_difference'%3A%202.61376953125%2C%20'p_clean_answer'%3A%202.0397645130287856e-05%2C%20'p_corrupt_answer'%3A%201.494287971581798e-06%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%205%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-1.34375%2C%20'p_clean_answer'%3A%200.00020214311371091753%2C%20'p_corrupt_answer'%3A%200.0007748937350697815%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%205%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-1.34375%2C%20'p_clean_answer'%3A%200.00020214311371091753%2C%20'p_corrupt_answer'%3A%200.0007748937350697815%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%205%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-1.34375%2C%20'p_clean_answer'%3A%200.00020214311371091753%2C%20'p_corrupt_answer'%3A%200.0007748937350697815%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%205%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-1.34375%2C%20'p_clean_answer'%3A%200.00020214311371091753%2C%20'p_corrupt_answer'%3A%200.0007748937350697815%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%205%2C%20'position'%3A%204%2C%20'logit_difference'%3A%202.912353515625%2C%20'p_clean_answer'%3A%200.004140540491789579%2C%20'p_corrupt_answer'%3A%200.00022502873616758734%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%205%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-1.34375%2C%20'p_clean_answer'%3A%200.00020214311371091753%2C%20'p_corrupt_answer'%3A%200.0007748937350697815%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%205%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-0.04510498046875%2C%20'p_clean_answer'%3A%202.5115472453762777e-05%2C%20'p_corrupt_answer'%3A%202.6274243282387033e-05%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%205%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-16.3575439453125%2C%20'p_clean_answer'%3A%201.546693080470438e-17%2C%20'p_corrupt_answer'%3A%201.9651476279580748e-10%7D%5D%2C%20'residual_norms'%3A%20%5B6.926535606384277%2C%209.85919189453125%2C%2010.356697082519531%2C%2011.179608345031738%2C%2016.143373489379883%2C%2092.03011322021484%5D%7D%2C%20%7B'clean'%3A%20'The%20capital%20of%20Italy%20is'%2C%20'corrupt'%3A%20'The%20capital%20of%20Greece%20is'%2C%20'clean_answer'%3A%20'%20Rome'%2C%20'corrupt_answer'%3A%20'%20Athens'%2C%20'clean_tokens'%3A%20%5B'The'%2C%20'%C4%A0capital'%2C%20'%C4%A0of'%2C%20'%C4%A0Italy'%2C%20'%C4%A0is'%5D%2C%20'corrupt_tokens'%3A%20%5B'The'%2C%20'%C4%A0capital'%2C%20'%C4%A0of'%2C%20'%C4%A0Greece'%2C%20'%C4%A0is'%5D%2C%20'positions'%3A%20%5B3%2C%204%5D%2C%20'rows'%3A%20%5B%7B'condition'%3A%20'Clean'%2C%20'layer'%3A%20-1%2C%20'position'%3A%20-1%2C%20'logit_difference'%3A%202.646728515625%2C%20'p_clean_answer'%3A%200.0008934771176427603%2C%20'p_corrupt_answer'%3A%206.333208875730634e-05%7D%2C%20%7B'condition'%3A%20'Corrupted'%2C%20'layer'%3A%20-1%2C%20'position'%3A%20-1%2C%20'logit_difference'%3A%20-0.56689453125%2C%20'p_clean_answer'%3A%200.0002926239976659417%2C%20'p_corrupt_answer'%3A%200.0005158329731784761%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%200%2C%20'position'%3A%203%2C%20'logit_difference'%3A%202.6839599609375%2C%20'p_clean_answer'%3A%200.0007971825543791056%2C%20'p_corrupt_answer'%3A%205.4441337852040306e-05%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%200%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-0.56689453125%2C%20'p_clean_answer'%3A%200.0002926239976659417%2C%20'p_corrupt_answer'%3A%200.0005158329731784761%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%200%2C%20'position'%3A%203%2C%20'logit_difference'%3A%201.5247802734375%2C%20'p_clean_answer'%3A%200.00011289031681371853%2C%20'p_corrupt_answer'%3A%202.4572707843617536e-05%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%200%2C%20'position'%3A%203%2C%20'logit_difference'%3A%201.103515625%2C%20'p_clean_answer'%3A%206.137391756055877e-05%2C%20'p_corrupt_answer'%3A%202.0357907487777993e-05%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%200%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-0.634521484375%2C%20'p_clean_answer'%3A%200.0003284046542830765%2C%20'p_corrupt_answer'%3A%200.0006194104207679629%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%200%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-0.56689453125%2C%20'p_clean_answer'%3A%200.0002926239976659417%2C%20'p_corrupt_answer'%3A%200.0005158329731784761%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%200%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-0.3017578125%2C%20'p_clean_answer'%3A%200.0017463015392422676%2C%20'p_corrupt_answer'%3A%200.0023614075034856796%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%200%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-4.5218505859375%2C%20'p_clean_answer'%3A%203.3553880030012806e-07%2C%20'p_corrupt_answer'%3A%203.087148434133269e-05%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%201%2C%20'position'%3A%203%2C%20'logit_difference'%3A%202.6199951171875%2C%20'p_clean_answer'%3A%200.0007483805529773235%2C%20'p_corrupt_answer'%3A%205.448451338452287e-05%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%201%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-0.56689453125%2C%20'p_clean_answer'%3A%200.0002926239976659417%2C%20'p_corrupt_answer'%3A%200.0005158329731784761%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%201%2C%20'position'%3A%203%2C%20'logit_difference'%3A%201.555419921875%2C%20'p_clean_answer'%3A%208.325278031406924e-05%2C%20'p_corrupt_answer'%3A%201.7574720914126374e-05%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%201%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-0.076171875%2C%20'p_clean_answer'%3A%202.1386196749517694e-05%2C%20'p_corrupt_answer'%3A%202.307887189090252e-05%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%201%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-0.58056640625%2C%20'p_clean_answer'%3A%200.0003615020541474223%2C%20'p_corrupt_answer'%3A%200.0006460223812609911%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%201%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-0.56689453125%2C%20'p_clean_answer'%3A%200.0002926239976659417%2C%20'p_corrupt_answer'%3A%200.0005158329731784761%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%201%2C%20'position'%3A%204%2C%20'logit_difference'%3A%200.3721923828125%2C%20'p_clean_answer'%3A%203.796036617131904e-05%2C%20'p_corrupt_answer'%3A%202.616310666780919e-05%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%201%2C%20'position'%3A%204%2C%20'logit_difference'%3A%200.5045166015625%2C%20'p_clean_answer'%3A%200.00011763974180212244%2C%20'p_corrupt_answer'%3A%207.103056850610301e-05%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%202%2C%20'position'%3A%203%2C%20'logit_difference'%3A%202.546630859375%2C%20'p_clean_answer'%3A%200.0006700000376440585%2C%20'p_corrupt_answer'%3A%205.249126843409613e-05%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%202%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-0.56689453125%2C%20'p_clean_answer'%3A%200.0002926239976659417%2C%20'p_corrupt_answer'%3A%200.0005158329731784761%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%202%2C%20'position'%3A%203%2C%20'logit_difference'%3A%201.061767578125%2C%20'p_clean_answer'%3A%205.919741670368239e-05%2C%20'p_corrupt_answer'%3A%202.0473069525905885e-05%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%202%2C%20'position'%3A%203%2C%20'logit_difference'%3A%200.6181640625%2C%20'p_clean_answer'%3A%201.8874567103921436e-05%2C%20'p_corrupt_answer'%3A%201.0172126167162787e-05%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%202%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-0.5078125%2C%20'p_clean_answer'%3A%200.0004022792272735387%2C%20'p_corrupt_answer'%3A%200.0006684482214041054%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%202%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-0.56689453125%2C%20'p_clean_answer'%3A%200.0002926239976659417%2C%20'p_corrupt_answer'%3A%200.0005158329731784761%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%202%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-0.3221435546875%2C%20'p_clean_answer'%3A%200.001148705487139523%2C%20'p_corrupt_answer'%3A%200.0015853088116273284%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%202%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-0.8779296875%2C%20'p_clean_answer'%3A%200.00021564339112956077%2C%20'p_corrupt_answer'%3A%200.0005188193754293025%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%203%2C%20'position'%3A%203%2C%20'logit_difference'%3A%202.509765625%2C%20'p_clean_answer'%3A%200.0006377698155120015%2C%20'p_corrupt_answer'%3A%205.184258407098241e-05%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%203%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-0.56689453125%2C%20'p_clean_answer'%3A%200.0002926239976659417%2C%20'p_corrupt_answer'%3A%200.0005158329731784761%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%203%2C%20'position'%3A%203%2C%20'logit_difference'%3A%201.18310546875%2C%20'p_clean_answer'%3A%208.506407175445929e-05%2C%20'p_corrupt_answer'%3A%202.6057337890961207e-05%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%203%2C%20'position'%3A%203%2C%20'logit_difference'%3A%200.120849609375%2C%20'p_clean_answer'%3A%201.0392931471869815e-05%2C%20'p_corrupt_answer'%3A%209.209875315718818e-06%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%203%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-0.4931640625%2C%20'p_clean_answer'%3A%200.00042447124724276364%2C%20'p_corrupt_answer'%3A%200.0006950670504011214%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%203%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-0.56689453125%2C%20'p_clean_answer'%3A%200.0002926239976659417%2C%20'p_corrupt_answer'%3A%200.0005158329731784761%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%203%2C%20'position'%3A%204%2C%20'logit_difference'%3A%201.39208984375%2C%20'p_clean_answer'%3A%203.484515764284879e-05%2C%20'p_corrupt_answer'%3A%208.660949788463768e-06%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%203%2C%20'position'%3A%204%2C%20'logit_difference'%3A%201.23583984375%2C%20'p_clean_answer'%3A%204.303562946006423e-07%2C%20'p_corrupt_answer'%3A%201.2505749680258305e-07%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%204%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-0.4681396484375%2C%20'p_clean_answer'%3A%200.000313681666739285%2C%20'p_corrupt_answer'%3A%200.0005009560263715684%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%204%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-0.56689453125%2C%20'p_clean_answer'%3A%200.0002926239976659417%2C%20'p_corrupt_answer'%3A%200.0005158329731784761%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%204%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-0.4801025390625%2C%20'p_clean_answer'%3A%200.00029381661443039775%2C%20'p_corrupt_answer'%3A%200.00047487823758274317%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%204%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-0.8463134765625%2C%20'p_clean_answer'%3A%200.00015750282909721136%2C%20'p_corrupt_answer'%3A%200.00036714502493850887%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%204%2C%20'position'%3A%204%2C%20'logit_difference'%3A%202.5517578125%2C%20'p_clean_answer'%3A%200.0008376878686249256%2C%20'p_corrupt_answer'%3A%206.529319216497242e-05%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%204%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-0.56689453125%2C%20'p_clean_answer'%3A%200.0002926239976659417%2C%20'p_corrupt_answer'%3A%200.0005158329731784761%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%204%2C%20'position'%3A%204%2C%20'logit_difference'%3A%202.6104736328125%2C%20'p_clean_answer'%3A%204.381401959108189e-05%2C%20'p_corrupt_answer'%3A%203.2203183764067944e-06%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%204%2C%20'position'%3A%204%2C%20'logit_difference'%3A%202.60595703125%2C%20'p_clean_answer'%3A%204.3775358790298924e-05%2C%20'p_corrupt_answer'%3A%203.2320417631126475e-06%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%205%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-0.56689453125%2C%20'p_clean_answer'%3A%200.0002926239976659417%2C%20'p_corrupt_answer'%3A%200.0005158329731784761%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%205%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-0.56689453125%2C%20'p_clean_answer'%3A%200.0002926239976659417%2C%20'p_corrupt_answer'%3A%200.0005158329731784761%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%205%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-0.56689453125%2C%20'p_clean_answer'%3A%200.0002926239976659417%2C%20'p_corrupt_answer'%3A%200.0005158329731784761%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%205%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-0.56689453125%2C%20'p_clean_answer'%3A%200.0002926239976659417%2C%20'p_corrupt_answer'%3A%200.0005158329731784761%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%205%2C%20'position'%3A%204%2C%20'logit_difference'%3A%202.646728515625%2C%20'p_clean_answer'%3A%200.0008934771176427603%2C%20'p_corrupt_answer'%3A%206.333208875730634e-05%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%205%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-0.56689453125%2C%20'p_clean_answer'%3A%200.0002926239976659417%2C%20'p_corrupt_answer'%3A%200.0005158329731784761%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%205%2C%20'position'%3A%204%2C%20'logit_difference'%3A%200.4127197265625%2C%20'p_clean_answer'%3A%202.0489895177888684e-05%2C%20'p_corrupt_answer'%3A%201.356119173578918e-05%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%205%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-9.8231201171875%2C%20'p_clean_answer'%3A%209.144730573166536e-23%2C%20'p_corrupt_answer'%3A%201.6877098386817432e-18%7D%5D%2C%20'residual_norms'%3A%20%5B7.046963691711426%2C%209.859155654907227%2C%2010.483169555664062%2C%2011.39472484588623%2C%2015.967036247253418%2C%2091.78528594970703%5D%7D%2C%20%7B'clean'%3A%20'The%20capital%20of%20Canada%20is'%2C%20'corrupt'%3A%20'The%20capital%20of%20Russia%20is'%2C%20'clean_answer'%3A%20'%20Ottawa'%2C%20'corrupt_answer'%3A%20'%20Moscow'%2C%20'clean_tokens'%3A%20%5B'The'%2C%20'%C4%A0capital'%2C%20'%C4%A0of'%2C%20'%C4%A0Canada'%2C%20'%C4%A0is'%5D%2C%20'corrupt_tokens'%3A%20%5B'The'%2C%20'%C4%A0capital'%2C%20'%C4%A0of'%2C%20'%C4%A0Russia'%2C%20'%C4%A0is'%5D%2C%20'positions'%3A%20%5B3%2C%204%5D%2C%20'rows'%3A%20%5B%7B'condition'%3A%20'Clean'%2C%20'layer'%3A%20-1%2C%20'position'%3A%20-1%2C%20'logit_difference'%3A%201.9915771484375%2C%20'p_clean_answer'%3A%200.000249287870246917%2C%20'p_corrupt_answer'%3A%203.4022810723399743e-05%7D%2C%20%7B'condition'%3A%20'Corrupted'%2C%20'layer'%3A%20-1%2C%20'position'%3A%20-1%2C%20'logit_difference'%3A%20-5.2939453125%2C%20'p_clean_answer'%3A%207.749201358819846e-06%2C%20'p_corrupt_answer'%3A%200.001543079037219286%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%200%2C%20'position'%3A%203%2C%20'logit_difference'%3A%201.727783203125%2C%20'p_clean_answer'%3A%200.000198131674551405%2C%20'p_corrupt_answer'%3A%203.5203607694711536e-05%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%200%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-5.2939453125%2C%20'p_clean_answer'%3A%207.749201358819846e-06%2C%20'p_corrupt_answer'%3A%200.001543079037219286%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%200%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-2.55615234375%2C%20'p_clean_answer'%3A%207.68435620557284e-06%2C%20'p_corrupt_answer'%3A%209.902170131681487e-05%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%200%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-1.98291015625%2C%20'p_clean_answer'%3A%201.0834867680387106e-05%2C%20'p_corrupt_answer'%3A%207.870286935940385e-05%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%200%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-5.1505126953125%2C%20'p_clean_answer'%3A%209.333541129308287e-06%2C%20'p_corrupt_answer'%3A%200.0016102218069136143%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%200%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-5.2939453125%2C%20'p_clean_answer'%3A%207.749201358819846e-06%2C%20'p_corrupt_answer'%3A%200.001543079037219286%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%200%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-4.3345947265625%2C%20'p_clean_answer'%3A%204.410222027217969e-05%2C%20'p_corrupt_answer'%3A%200.003364735981449485%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%200%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-4.1114501953125%2C%20'p_clean_answer'%3A%202.190113491451484e-06%2C%20'p_corrupt_answer'%3A%200.00013367393694352359%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%201%2C%20'position'%3A%203%2C%20'logit_difference'%3A%201.60693359375%2C%20'p_clean_answer'%3A%200.00018559007730800658%2C%20'p_corrupt_answer'%3A%203.721108805621043e-05%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%201%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-5.2939453125%2C%20'p_clean_answer'%3A%207.749201358819846e-06%2C%20'p_corrupt_answer'%3A%200.001543079037219286%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%201%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-2.63134765625%2C%20'p_clean_answer'%3A%204.751538654090837e-06%2C%20'p_corrupt_answer'%3A%206.601065251743421e-05%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%201%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-1.353759765625%2C%20'p_clean_answer'%3A%201.3753314306086395e-05%2C%20'p_corrupt_answer'%3A%205.325222809915431e-05%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%201%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-5.082275390625%2C%20'p_clean_answer'%3A%209.889869033941068e-06%2C%20'p_corrupt_answer'%3A%200.0015936564886942506%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%201%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-5.2939453125%2C%20'p_clean_answer'%3A%207.749201358819846e-06%2C%20'p_corrupt_answer'%3A%200.001543079037219286%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%201%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-3.106689453125%2C%20'p_clean_answer'%3A%203.0528742627211614e-06%2C%20'p_corrupt_answer'%3A%206.82224053889513e-05%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%201%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-2.8201904296875%2C%20'p_clean_answer'%3A%201.8729202793110744e-06%2C%20'p_corrupt_answer'%3A%203.142768764519133e-05%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%202%2C%20'position'%3A%203%2C%20'logit_difference'%3A%201.2847900390625%2C%20'p_clean_answer'%3A%200.0001490953000029549%2C%20'p_corrupt_answer'%3A%204.125596387893893e-05%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%202%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-5.2939453125%2C%20'p_clean_answer'%3A%207.749201358819846e-06%2C%20'p_corrupt_answer'%3A%200.001543079037219286%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%202%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-2.1053466796875%2C%20'p_clean_answer'%3A%201.1799415915447753e-05%2C%20'p_corrupt_answer'%3A%209.68725944403559e-05%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%202%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-2.1002197265625%2C%20'p_clean_answer'%3A%201.2542876902443822e-05%2C%20'p_corrupt_answer'%3A%200.00010244976874673739%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%202%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-4.8040771484375%2C%20'p_clean_answer'%3A%201.393308411934413e-05%2C%20'p_corrupt_answer'%3A%200.001699931686744094%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%202%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-5.2939453125%2C%20'p_clean_answer'%3A%207.749201358819846e-06%2C%20'p_corrupt_answer'%3A%200.001543079037219286%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%202%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-3.924072265625%2C%20'p_clean_answer'%3A%204.863757567363791e-05%2C%20'p_corrupt_answer'%3A%200.0024613584391772747%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%202%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-4.6396484375%2C%20'p_clean_answer'%3A%203.1710940220364137e-06%2C%20'p_corrupt_answer'%3A%200.00032823346555233%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%203%2C%20'position'%3A%203%2C%20'logit_difference'%3A%201.1851806640625%2C%20'p_clean_answer'%3A%200.0001394124119542539%2C%20'p_corrupt_answer'%3A%204.261710637365468e-05%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%203%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-5.2939453125%2C%20'p_clean_answer'%3A%207.749201358819846e-06%2C%20'p_corrupt_answer'%3A%200.001543079037219286%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%203%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-2.3052978515625%2C%20'p_clean_answer'%3A%201.0947535884042736e-05%2C%20'p_corrupt_answer'%3A%200.00010977275087498128%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%203%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-2.5252685546875%2C%20'p_clean_answer'%3A%207.5007337727583945e-06%2C%20'p_corrupt_answer'%3A%209.371604392072186e-05%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%203%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-4.6798095703125%2C%20'p_clean_answer'%3A%201.5385352526209317e-05%2C%20'p_corrupt_answer'%3A%200.0016577648930251598%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%203%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-5.2939453125%2C%20'p_clean_answer'%3A%207.749201358819846e-06%2C%20'p_corrupt_answer'%3A%200.001543079037219286%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%203%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-1.06689453125%2C%20'p_clean_answer'%3A%203.9897677197586745e-06%2C%20'p_corrupt_answer'%3A%201.1595619980653282e-05%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%203%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-0.5289306640625%2C%20'p_clean_answer'%3A%201.5792038539075293e-05%2C%20'p_corrupt_answer'%3A%202.6800931664183736e-05%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%204%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-5.163818359375%2C%20'p_clean_answer'%3A%208.411583621636964e-06%2C%20'p_corrupt_answer'%3A%200.0014706035144627094%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%204%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-5.2939453125%2C%20'p_clean_answer'%3A%207.749201358819846e-06%2C%20'p_corrupt_answer'%3A%200.001543079037219286%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%204%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-5.1756591796875%2C%20'p_clean_answer'%3A%208.221449206757825e-06%2C%20'p_corrupt_answer'%3A%200.0014544829027727246%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%204%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-5.2236328125%2C%20'p_clean_answer'%3A%203.665065833047265e-06%2C%20'p_corrupt_answer'%3A%200.0006802627467550337%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%204%2C%20'position'%3A%204%2C%20'logit_difference'%3A%201.839599609375%2C%20'p_clean_answer'%3A%200.00022672058548778296%2C%20'p_corrupt_answer'%3A%203.6021599953528494e-05%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%204%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-5.2939453125%2C%20'p_clean_answer'%3A%207.749201358819846e-06%2C%20'p_corrupt_answer'%3A%200.001543079037219286%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%204%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-1.8505859375%2C%20'p_clean_answer'%3A%202.8020317586197052e-06%2C%20'p_corrupt_answer'%3A%201.7830861906986684e-05%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%204%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-3.47607421875%2C%20'p_clean_answer'%3A%204.356832903340546e-07%2C%20'p_corrupt_answer'%3A%201.4086747796682175e-05%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%205%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-5.2939453125%2C%20'p_clean_answer'%3A%207.749201358819846e-06%2C%20'p_corrupt_answer'%3A%200.001543079037219286%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%205%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-5.2939453125%2C%20'p_clean_answer'%3A%207.749201358819846e-06%2C%20'p_corrupt_answer'%3A%200.001543079037219286%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%205%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-5.2939453125%2C%20'p_clean_answer'%3A%207.749201358819846e-06%2C%20'p_corrupt_answer'%3A%200.001543079037219286%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%205%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-5.2939453125%2C%20'p_clean_answer'%3A%207.749201358819846e-06%2C%20'p_corrupt_answer'%3A%200.001543079037219286%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%205%2C%20'position'%3A%204%2C%20'logit_difference'%3A%201.9915771484375%2C%20'p_clean_answer'%3A%200.000249287870246917%2C%20'p_corrupt_answer'%3A%203.4022810723399743e-05%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%205%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-5.2939453125%2C%20'p_clean_answer'%3A%207.749201358819846e-06%2C%20'p_corrupt_answer'%3A%200.001543079037219286%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%205%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-0.98699951171875%2C%20'p_clean_answer'%3A%201.2674412573687732e-05%2C%20'p_corrupt_answer'%3A%203.400762579985894e-05%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%205%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-9.03704833984375%2C%20'p_clean_answer'%3A%207.568397582009324e-19%2C%20'p_corrupt_answer'%3A%206.3642049190847615e-15%7D%5D%2C%20'residual_norms'%3A%20%5B7.054368019104004%2C%209.872750282287598%2C%2010.385741233825684%2C%2011.390432357788086%2C%2016.195310592651367%2C%2091.97669982910156%5D%7D%2C%20%7B'clean'%3A%20'The%20capital%20of%20Portugal%20is'%2C%20'corrupt'%3A%20'The%20capital%20of%20France%20is'%2C%20'clean_answer'%3A%20'%20Lisbon'%2C%20'corrupt_answer'%3A%20'%20Paris'%2C%20'clean_tokens'%3A%20%5B'The'%2C%20'%C4%A0capital'%2C%20'%C4%A0of'%2C%20'%C4%A0Portugal'%2C%20'%C4%A0is'%5D%2C%20'corrupt_tokens'%3A%20%5B'The'%2C%20'%C4%A0capital'%2C%20'%C4%A0of'%2C%20'%C4%A0France'%2C%20'%C4%A0is'%5D%2C%20'positions'%3A%20%5B3%2C%204%5D%2C%20'rows'%3A%20%5B%7B'condition'%3A%20'Clean'%2C%20'layer'%3A%20-1%2C%20'position'%3A%20-1%2C%20'logit_difference'%3A%20-0.0050048828125%2C%20'p_clean_answer'%3A%200.0002310975978616625%2C%20'p_corrupt_answer'%3A%200.00023225712357088923%7D%2C%20%7B'condition'%3A%20'Corrupted'%2C%20'layer'%3A%20-1%2C%20'position'%3A%20-1%2C%20'logit_difference'%3A%20-4.726318359375%2C%20'p_clean_answer'%3A%201.722963497741148e-05%2C%20'p_corrupt_answer'%3A%200.0019448678940534592%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%200%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-0.1097412109375%2C%20'p_clean_answer'%3A%200.000205121366889216%2C%20'p_corrupt_answer'%3A%200.00022891323897056282%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%200%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-4.726318359375%2C%20'p_clean_answer'%3A%201.722963497741148e-05%2C%20'p_corrupt_answer'%3A%200.0019448678940534592%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%200%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-4.471923828125%2C%20'p_clean_answer'%3A%203.965176802012138e-06%2C%20'p_corrupt_answer'%3A%200.00034705185680650175%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%200%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-4.05078125%2C%20'p_clean_answer'%3A%202.254277205793187e-06%2C%20'p_corrupt_answer'%3A%200.00012949090159963816%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%200%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-4.6053466796875%2C%20'p_clean_answer'%3A%201.9812881873804145e-05%2C%20'p_corrupt_answer'%3A%200.00198163790628314%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%200%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-4.726318359375%2C%20'p_clean_answer'%3A%201.722963497741148e-05%2C%20'p_corrupt_answer'%3A%200.0019448678940534592%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%200%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-6.358154296875%2C%20'p_clean_answer'%3A%201.5533585610683076e-05%2C%20'p_corrupt_answer'%3A%200.00896567665040493%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%200%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-4.5860595703125%2C%20'p_clean_answer'%3A%203.0882301871315576e-06%2C%20'p_corrupt_answer'%3A%200.0003029772487934679%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%201%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-0.2073974609375%2C%20'p_clean_answer'%3A%200.0001777676079655066%2C%20'p_corrupt_answer'%3A%200.0002187379723181948%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%201%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-4.726318359375%2C%20'p_clean_answer'%3A%201.722963497741148e-05%2C%20'p_corrupt_answer'%3A%200.0019448678940534592%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%201%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-4.3712158203125%2C%20'p_clean_answer'%3A%201.618573605810525e-06%2C%20'p_corrupt_answer'%3A%200.0001280935830436647%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%201%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-3.488037109375%2C%20'p_clean_answer'%3A%201.1298652680125087e-06%2C%20'p_corrupt_answer'%3A%203.6971057852497324e-05%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%201%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-4.541748046875%2C%20'p_clean_answer'%3A%202.278800093336031e-05%2C%20'p_corrupt_answer'%3A%200.0021387613378465176%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%201%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-4.726318359375%2C%20'p_clean_answer'%3A%201.722963497741148e-05%2C%20'p_corrupt_answer'%3A%200.0019448678940534592%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%201%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-6.08984375%2C%20'p_clean_answer'%3A%201.8905900560639566e-06%2C%20'p_corrupt_answer'%3A%200.0008344165980815887%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%201%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-4.4884033203125%2C%20'p_clean_answer'%3A%206.413524261006387e-06%2C%20'p_corrupt_answer'%3A%200.0005706706433556974%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%202%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-0.380859375%2C%20'p_clean_answer'%3A%200.00017902175022754818%2C%20'p_corrupt_answer'%3A%200.00026200583670288324%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%202%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-4.726318359375%2C%20'p_clean_answer'%3A%201.722963497741148e-05%2C%20'p_corrupt_answer'%3A%200.0019448678940534592%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%202%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-4.1737060546875%2C%20'p_clean_answer'%3A%203.926917088392656e-06%2C%20'p_corrupt_answer'%3A%200.00025507580721750855%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%202%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-4.936279296875%2C%20'p_clean_answer'%3A%201.8976224964717403e-06%2C%20'p_corrupt_answer'%3A%200.00026424616225995123%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%202%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-4.4349365234375%2C%20'p_clean_answer'%3A%202.077850695059169e-05%2C%20'p_corrupt_answer'%3A%200.0017525998409837484%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%202%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-4.726318359375%2C%20'p_clean_answer'%3A%201.722963497741148e-05%2C%20'p_corrupt_answer'%3A%200.0019448678940534592%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%202%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-4.041015625%2C%20'p_clean_answer'%3A%200.00020550392218865454%2C%20'p_corrupt_answer'%3A%200.01168990321457386%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%202%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-3.605712890625%2C%20'p_clean_answer'%3A%201.7486372883013246e-07%2C%20'p_corrupt_answer'%3A%206.436368948925519e-06%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%203%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-0.481201171875%2C%20'p_clean_answer'%3A%200.00020049703016411513%2C%20'p_corrupt_answer'%3A%200.0003244075633119792%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%203%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-4.726318359375%2C%20'p_clean_answer'%3A%201.722963497741148e-05%2C%20'p_corrupt_answer'%3A%200.0019448678940534592%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%203%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-4.09765625%2C%20'p_clean_answer'%3A%204.207162419334054e-06%2C%20'p_corrupt_answer'%3A%200.0002532671205699444%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%203%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-4.1517333984375%2C%20'p_clean_answer'%3A%205.153463462193031e-06%2C%20'p_corrupt_answer'%3A%200.0003274719638284296%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%203%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-4.2991943359375%2C%20'p_clean_answer'%3A%201.964739931281656e-05%2C%20'p_corrupt_answer'%3A%200.0014468432636931539%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%203%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-4.726318359375%2C%20'p_clean_answer'%3A%201.722963497741148e-05%2C%20'p_corrupt_answer'%3A%200.0019448678940534592%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%203%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-3.3099365234375%2C%20'p_clean_answer'%3A%203.3058024655474583e-06%2C%20'p_corrupt_answer'%3A%209.052407403942198e-05%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%203%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-3.127197265625%2C%20'p_clean_answer'%3A%207.2276470746146515e-06%2C%20'p_corrupt_answer'%3A%200.00016486232925672084%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%204%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-4.6561279296875%2C%20'p_clean_answer'%3A%201.7825590475695208e-05%2C%20'p_corrupt_answer'%3A%200.0018757483921945095%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%204%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-4.726318359375%2C%20'p_clean_answer'%3A%201.722963497741148e-05%2C%20'p_corrupt_answer'%3A%200.0019448678940534592%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%204%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-4.9031982421875%2C%20'p_clean_answer'%3A%201.5437568436027505e-05%2C%20'p_corrupt_answer'%3A%200.002079748548567295%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%204%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-5.0360107421875%2C%20'p_clean_answer'%3A%201.3213696547609288e-05%2C%20'p_corrupt_answer'%3A%200.002032993594184518%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%204%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-0.0762939453125%2C%20'p_clean_answer'%3A%200.00022222462575882673%2C%20'p_corrupt_answer'%3A%200.0002398425422143191%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%204%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-4.726318359375%2C%20'p_clean_answer'%3A%201.722963497741148e-05%2C%20'p_corrupt_answer'%3A%200.0019448678940534592%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%204%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-2.8394775390625%2C%20'p_clean_answer'%3A%202.4132782527885865e-06%2C%20'p_corrupt_answer'%3A%204.128352884436026e-05%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%204%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-4.4090576171875%2C%20'p_clean_answer'%3A%204.190845857010572e-07%2C%20'p_corrupt_answer'%3A%203.4445387427695096e-05%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%205%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-4.726318359375%2C%20'p_clean_answer'%3A%201.722963497741148e-05%2C%20'p_corrupt_answer'%3A%200.0019448678940534592%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%205%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-4.726318359375%2C%20'p_clean_answer'%3A%201.722963497741148e-05%2C%20'p_corrupt_answer'%3A%200.0019448678940534592%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%205%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-4.726318359375%2C%20'p_clean_answer'%3A%201.722963497741148e-05%2C%20'p_corrupt_answer'%3A%200.0019448678940534592%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%205%2C%20'position'%3A%203%2C%20'logit_difference'%3A%20-4.726318359375%2C%20'p_clean_answer'%3A%201.722963497741148e-05%2C%20'p_corrupt_answer'%3A%200.0019448678940534592%7D%2C%20%7B'condition'%3A%20'Patched'%2C%20'layer'%3A%205%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-0.0050048828125%2C%20'p_clean_answer'%3A%200.0002310975978616625%2C%20'p_corrupt_answer'%3A%200.00023225712357088923%7D%2C%20%7B'condition'%3A%20'No-op%20control'%2C%20'layer'%3A%205%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-4.726318359375%2C%20'p_clean_answer'%3A%201.722963497741148e-05%2C%20'p_corrupt_answer'%3A%200.0019448678940534592%7D%2C%20%7B'condition'%3A%20'Zero%20control'%2C%20'layer'%3A%205%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-1.751220703125%2C%20'p_clean_answer'%3A%201.0088283488585148e-05%2C%20'p_corrupt_answer'%3A%205.812497693113983e-05%7D%2C%20%7B'condition'%3A%20'Random%20norm-matched%20control'%2C%20'layer'%3A%205%2C%20'position'%3A%204%2C%20'logit_difference'%3A%20-10.49072265625%2C%20'p_clean_answer'%3A%202.72739842459549e-24%2C%20'p_corrupt_answer'%3A%209.813219715401338e-20%7D%5D%2C%20'residual_norms'%3A%20%5B7.041056156158447%2C%209.729270935058594%2C%2010.59091854095459%2C%2011.194535255432129%2C%2015.990078926086426%2C%2091.74433898925781%5D%7D%5D%7D%0A%20%20%20%20config%3D%7B'id'%3A'u12-llm'%2C'controls'%3A%5B%5D%7D%0A%20%20%20%20return%20config%2C%20recorded%0A%0A%0A%40app.cell%0Adef%20_(mo)%3A%0A%20%20%20%20mo.md(%22%22%22%0A%20%20%20%20%23%20Inside%20a%20pretrained%20language%20model%0A%0A%20%20%20%20**Year%2012%20%C2%B7%2055%20minutes%20%C2%B7%20Recorded%20Pythia%20measurements%2C%20with%20an%20optional%20live%20native%20rerun**%0A%0A%20%20%20%20%23%23%20Can%20restoring%20an%20internal%20state%20recover%20a%20language%20model's%20answer%20preference%3F%0A%0A%20%20%20%20Explore%20real%20activation-patching%20measurements%20collected%20from%20EleutherAI's%20Pythia-70M-deduped.%20Six%20fixed%20capital-city%20prompt%20pairs%20keep%20the%20comparison%20inspectable.%20Candidate%20probabilities%20use%20the%20entire%20vocabulary%2C%20not%20just%20the%20two%20answer%20choices.%0A%0A%20%20%20%20**Browser%20mode%3A**%20Python%20analyses%20recorded%20model%20runs%3B%20moving%20these%20controls%20does%20not%20rerun%20the%20language%20model.%20**Native%20mode%3A**%20download%20this%20notebook%20to%20repeat%20the%20measured%20experiment%20on%20a%20CPU%20or%20your%20own%20H100.%20Neither%20mode%20is%20a%20reproduction%20of%20Goodfire's%20models%20or%20a%20representative%20factual-accuracy%20test.%0A%20%20%20%20%22%22%22)%0A%20%20%20%20return%0A%0A%0A%40app.cell%0Adef%20_(mo)%3A%0A%20%20%20%20project_file%20%3D%20mo.ui.file(filetypes%3D%5B'.json'%5D%2C%20multiple%3DFalse%2C%20max_size%3D2000000%2C%20label%3D'Open%20a%20saved%20Brightlab%20notebook%20project')%0A%20%20%20%20mo.vstack(%5Bmo.md('**Keep%20your%20work%3A**%20download%20a%20project%20before%20leaving.%20Reopen%20it%20here%20to%20restore%20settings%2C%20comparisons%20and%20writing.%20Files%20are%20read%20inside%20this%20notebook%3B%20use%20fictional%20classroom%20data.')%2Cproject_file%5D)%0A%20%20%20%20return%20(project_file%2C)%0A%0A%0A%40app.cell%0Adef%20_(config%2C%20json%2C%20mo%2C%20project_file)%3A%0A%20%20%20%20restored%20%3D%20%7B%7D%0A%20%20%20%20if%20project_file.contents()%3A%0A%20%20%20%20%20%20%20%20try%3A%0A%20%20%20%20%20%20%20%20%20%20%20%20_candidate%20%3D%20json.loads(project_file.contents().decode('utf-8'))%0A%20%20%20%20%20%20%20%20%20%20%20%20def%20_tree(value%2Cdepth%3D0)%3A%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20if%20depth%3E16%3Araise%20ValueError('Project%20nesting%20is%20too%20deep.')%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20if%20isinstance(value%2Cdict)%3A%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20if%20len(value)%3E100%3Araise%20ValueError('Too%20many%20object%20fields.')%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20for%20child%20in%20value.values()%3A_tree(child%2Cdepth%2B1)%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20elif%20isinstance(value%2Clist)%3A%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20if%20len(value)%3E20000%3Araise%20ValueError('Too%20many%20rows.')%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20for%20child%20in%20value%3A_tree(child%2Cdepth%2B1)%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20elif%20isinstance(value%2Cfloat)%20and%20not%20(-1e100%3Cvalue%3C1e100)%3Araise%20ValueError('Project%20contains%20a%20non-finite%20or%20excessive%20number.')%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20elif%20isinstance(value%2Cstr)%20and%20len(value)%3E200000%3Araise%20ValueError('Project%20text%20is%20too%20long.')%0A%20%20%20%20%20%20%20%20%20%20%20%20_tree(_candidate)%0A%20%20%20%20%20%20%20%20%20%20%20%20if%20not%20isinstance(_candidate%2Cdict)%20or%20_candidate.get('version')!%3D1%3Araise%20ValueError('Choose%20a%20supported%20version%201%20notebook%20project.')%0A%20%20%20%20%20%20%20%20%20%20%20%20if%20_candidate.get('format')%20!%3D%20'brightlab-notebook-project'%20or%20_candidate.get('lesson')%20!%3D%20config.get('id'%2Cconfig.get('lesson_id'))%3A%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20raise%20ValueError('Choose%20a%20project%20for%20this%20notebook.')%0A%20%20%20%20%20%20%20%20%20%20%20%20if%20not%20isinstance(_candidate.get('settings')%2Cdict)%20or%20not%20isinstance(_candidate.get('prediction')%2Cstr)%20or%20not%20isinstance(_candidate.get('saved_runs'%2C%5B%5D)%2Clist)%3A%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20raise%20ValueError('The%20project%20is%20missing%20its%20settings%2C%20prediction%20or%20comparison%20list.')%0A%20%20%20%20%20%20%20%20%20%20%20%20for%20_spec%20in%20config%5B'controls'%5D%3A%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20_value%3D_candidate%5B'settings'%5D.get(_spec%5B'key'%5D%2C_spec%5B'default'%5D)%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20if%20_spec%5B'kind'%5D%3D%3D'slider'%20and%20(type(_value)%20not%20in%20%5Bint%2Cfloat%5D%20or%20not%20_spec%5B'min'%5D%3C%3D_value%3C%3D_spec%5B'max'%5D)%3Araise%20ValueError('A%20saved%20slider%20is%20outside%20its%20allowed%20range.')%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20if%20_spec%5B'kind'%5D%3D%3D'choice'%20and%20(type(_value)%20is%20not%20int%20or%20not%200%3C%3D_value%3Clen(_spec%5B'options'%5D))%3Araise%20ValueError('A%20saved%20choice%20is%20invalid.')%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20if%20_spec%5B'kind'%5D%3D%3D'text'%20and%20(not%20isinstance(_value%2Cstr)%20or%20len(_value)%3E1000)%3Araise%20ValueError('A%20saved%20text%20control%20is%20invalid.')%0A%20%20%20%20%20%20%20%20%20%20%20%20if%20not%20isinstance(_candidate.get('conclusion'%2C'')%2Cstr)%20or%20len(_candidate.get('saved_runs'%2C%5B%5D))%3E6%3Araise%20ValueError('Invalid%20writing%20or%20comparison%20count.')%0A%20%20%20%20%20%20%20%20%20%20%20%20if%20len(_candidate.get('prediction'%2C''))%3E10000%20or%20not%20isinstance(_candidate.get('conclusion'%2C'')%2Cstr)%20or%20len(_candidate.get('conclusion'%2C''))%3E20000%3Araise%20ValueError('Invalid%20writing%20in%20the%20saved%20project.')%0A%20%20%20%20%20%20%20%20%20%20%20%20_runs%3D_candidate.get('saved_runs'%2C%5B%5D)%0A%20%20%20%20%20%20%20%20%20%20%20%20if%20len(_runs)%3E6%3Araise%20ValueError('A%20project%20may%20contain%20up%20to%20six%20comparisons.')%0A%20%20%20%20%20%20%20%20%20%20%20%20for%20_run%20in%20_runs%3A%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20if%20not%20isinstance(_run%2Cdict)%20or%20not%20isinstance(_run.get('settings')%2Cdict)%3Araise%20ValueError('Invalid%20saved%20comparison.')%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20_rows%3D_run.get('measurements'%2C_run.get('result'%2C%7B%7D).get('rows')%20if%20isinstance(_run.get('result')%2Cdict)%20else%20None)%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20if%20not%20isinstance(_rows%2Clist)%20or%20not%20all(isinstance(row%2Cdict)%20for%20row%20in%20_rows)%3Araise%20ValueError('A%20saved%20comparison%20needs%20a%20measurement%20table.')%0A%20%20%20%20%20%20%20%20%20%20%20%20restored%20%3D%20_candidate%0A%20%20%20%20%20%20%20%20%20%20%20%20mo.output.replace(mo.md('Project%20read.%20Save%20the%20restored%20prediction%20to%20continue%20your%20investigation.'))%0A%20%20%20%20%20%20%20%20except%20(ValueError%2CTypeError%2CUnicodeError%2CKeyError%2CAttributeError)%20as%20_error%3A%0A%20%20%20%20%20%20%20%20%20%20%20%20mo.output.replace(mo.callout(mo.md('Could%20not%20open%20this%20project%3A%20'%2Bstr(_error))%2Ckind%3D'warn'))%0A%20%20%20%20return%20(restored%2C)%0A%0A%0A%40app.cell%0Adef%20_(mo)%3A%0A%20%20%20%20reset_controls%20%3D%20mo.ui.button(value%3D0%2Con_click%3Dlambda%20count%3Acount%2B1%2Clabel%3D'Reset%20controls%20to%20starting%20settings')%0A%20%20%20%20return%20(reset_controls%2C)%0A%0A%0A%40app.cell%0Adef%20_(mo%2C%20restored)%3A%0A%20%20%20%20prediction%3Dmo.ui.text_area(value%3Drestored.get('prediction'%2C'')%2Clabel%3D'My%20prediction%20%E2%80%94%20which%20patch%20will%20recover%20the%20clean%20preference%3F'%2Cplaceholder%3D'Compare%20the%20final%20token%20after%20the%20last%20block%20with%20an%20earlier%20patch.'%2Cfull_width%3DTrue).form(submit_button_label%3D'Save%20prediction%20%26%20open%20the%20lab'%2Cvalidate%3Dlambda%20v%3A%20None%20if%20v%20and%20len(v.strip())%3E%3D3%20else%20'Write%20a%20testable%20prediction%20first.')%0A%20%20%20%20prediction%0A%20%20%20%20return%20(prediction%2C)%0A%0A%0A%40app.cell%0Adef%20_(mo)%3A%0A%20%20%20%20get_attempt%2Cset_attempt%3Dmo.state(None)%0A%20%20%20%20return%20get_attempt%2C%20set_attempt%0A%0A%0A%40app.cell%0Adef%20_(mo%2C%20prediction%2C%20sys)%3A%0A%20%20%20%20mo.stop(prediction.value%20is%20None)%0A%20%20%20%20native_run%3Dmo.ui.dropdown(options%3D%5B'CPU'%2C'H100%20CUDA'%5D%2Cvalue%3D'CPU'%2Clabel%3D'Device%20for%20a%20fresh%20model%20run').form(submit_button_label%3D'Repeat%20the%20model%20experiment'%2Cclear_on_submit%3DFalse)%0A%20%20%20%20mo.vstack(%5Bmo.md('%23%23%23%20Optional%3A%20repeat%20the%20original%20measurements%5CnThe%20first%20native%20run%20downloads%20the%20pinned%20model%20weights%20(roughly%20300%20MB)%20from%20Hugging%20Face.%20CPU%20is%20sufficient.%20H100%20uses%20your%20own%20runtime%20and%20may%20incur%20your%20provider%E2%80%99s%20charges.%20Nothing%20provisions%20a%20GPU%20automatically.%20The%20browser%20copy%20cannot%20run%20PyTorch.')%2Cmo.md('Download%20the%20Python%20notebook%20and%20run%20%60uv%20run%20marimo%20edit%20--sandbox%20u12-llm.py%60%2C%20then%20submit%20this%20section.%20The%20%5BH100%20setup%20guide%5D(https%3A%2F%2Fbrightlab-ai-creators.ian347727.chatgpt.site%2Fmarimo-labs%2Fh100)%20explains%20runtime%20access%2C%20device%20verification%20and%20shutdown.')%20if%20sys.platform%3D%3D'emscripten'%20else%20native_run%5D)%0A%20%20%20%20return%20(native_run%2C)%0A%0A%0A%40app.cell%0Adef%20_()%3A%0A%20%20%20%20%22%22%22Pinned%20Pythia%20activation%20patching%20on%20fixed%2C%20teacher-reviewable%20prompts.%0A%20%20%20%20CPU%20works%3B%20H100%20is%20optional.%20No%20arbitrary%20text%20generation%20or%20remote%20model%20code.%0A%20%20%20%20%22%22%22%0A%20%20%20%20import%20time%0A%20%20%20%20from%20datetime%20import%20datetime%2C%20timezone%0A%20%20%20%20MODEL_ID%3D'EleutherAI%2Fpythia-70m-deduped'%0A%20%20%20%20PAIRS%3D%5B('The%20capital%20of%20France%20is'%2C'The%20capital%20of%20England%20is'%2C'%20Paris'%2C'%20London')%2C('The%20capital%20of%20Germany%20is'%2C'The%20capital%20of%20Spain%20is'%2C'%20Berlin'%2C'%20Madrid')%2C('The%20capital%20of%20Japan%20is'%2C'The%20capital%20of%20China%20is'%2C'%20Tokyo'%2C'%20Beijing')%2C('The%20capital%20of%20Italy%20is'%2C'The%20capital%20of%20Greece%20is'%2C'%20Rome'%2C'%20Athens')%2C('The%20capital%20of%20Canada%20is'%2C'The%20capital%20of%20Russia%20is'%2C'%20Ottawa'%2C'%20Moscow')%2C('The%20capital%20of%20Portugal%20is'%2C'The%20capital%20of%20France%20is'%2C'%20Lisbon'%2C'%20Paris')%5D%0A%0A%20%20%20%20def%20collect(revision%2Cdevice%3D'cpu'%2Ccache_dir%3DNone)%3A%0A%20%20%20%20%20%20%20%20import%20os%0A%20%20%20%20%20%20%20%20os.environ.setdefault(%22HF_HUB_DISABLE_XET%22%2C%221%22)%0A%20%20%20%20%20%20%20%20import%20torch%2C%20transformers%0A%20%20%20%20%20%20%20%20from%20transformers%20import%20AutoTokenizer%2CAutoModelForCausalLM%0A%20%20%20%20%20%20%20%20if%20device%3D%3D'cuda'%3A%0A%20%20%20%20%20%20%20%20%20%20%20%20if%20not%20torch.cuda.is_available()%3Araise%20RuntimeError('CUDA%20is%20unavailable%3B%20select%20CPU%20or%20attach%20your%20GPU%20runtime.')%0A%20%20%20%20%20%20%20%20%20%20%20%20if%20'H100'%20not%20in%20torch.cuda.get_device_name(0)%3Araise%20RuntimeError('The%20selected%20GPU%20is%20not%20an%20H100.%20Select%20CPU%20or%20attach%20an%20H100.')%0A%20%20%20%20%20%20%20%20torch.manual_seed(51)%3Btorch.set_num_threads(2)%0A%20%20%20%20%20%20%20%20tokenizer%3DAutoTokenizer.from_pretrained(MODEL_ID%2Crevision%3Drevision%2Ccache_dir%3Dcache_dir%2Ctrust_remote_code%3DFalse)%0A%20%20%20%20%20%20%20%20model%3DAutoModelForCausalLM.from_pretrained(MODEL_ID%2Crevision%3Drevision%2Ccache_dir%3Dcache_dir%2Ctrust_remote_code%3DFalse%2Cuse_safetensors%3DTrue%2Cattn_implementation%3D'eager').to(device).eval()%0A%20%20%20%20%20%20%20%20def%20hidden(output)%3Areturn%20output%5B0%5D%20if%20isinstance(output%2Ctuple)%20else%20output%0A%20%20%20%20%20%20%20%20def%20replace(output%2Cnew)%3Areturn%20(new%2C)%2Boutput%5B1%3A%5D%20if%20isinstance(output%2Ctuple)%20else%20new%0A%20%20%20%20%20%20%20%20records%3D%5B%5D%3Bstart%3Dtime.perf_counter()%0A%20%20%20%20%20%20%20%20with%20torch.inference_mode()%3A%0A%20%20%20%20%20%20%20%20%20%20%20%20for%20clean%2Ccorrupt%2Ca%2Cb%20in%20PAIRS%3A%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20x%3Dtokenizer(clean%2Creturn_tensors%3D'pt').to(device)%3By%3Dtokenizer(corrupt%2Creturn_tensors%3D'pt').to(device)%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20assert%20x%5B'input_ids'%5D.shape%3D%3Dy%5B'input_ids'%5D.shape%2C'Unequal%20token%20positions%20require%20an%20explicit%20alignment%20policy.'%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20aid%3Dtokenizer.encode(a%2Cadd_special_tokens%3DFalse)%3Bbid%3Dtokenizer.encode(b%2Cadd_special_tokens%3DFalse)%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20assert%20len(aid)%3D%3Dlen(bid)%3D%3D1%2C'This%20experiment%20requires%20single-token%20answer%20candidates.'%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20aid%2Cbid%3Daid%5B0%5D%2Cbid%5B0%5D%3Bsaved%3D%7B%7D%3Bhandles%3D%5B%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20for%20layer%2Cblock%20in%20enumerate(model.gpt_neox.layers)%3A%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20def%20capture(module%2Cargs%2Cout%2Cindex%3Dlayer)%3Asaved%5Bindex%5D%3Dhidden(out).detach().clone()%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20handles.append(block.register_forward_hook(capture))%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20original%3Dmodel(**x).logits%5B0%2C-1%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20for%20h%20in%20handles%3Ah.remove()%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20bad_saved%3D%7B%7D%3Bbad_handles%3D%5B%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20for%20layer%2Cblock%20in%20enumerate(model.gpt_neox.layers)%3A%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20def%20capture_bad(module%2Cargs%2Cout%2Cindex%3Dlayer)%3Abad_saved%5Bindex%5D%3Dhidden(out).detach().clone()%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20bad_handles.append(block.register_forward_hook(capture_bad))%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20broken%3Dmodel(**y).logits%5B0%2C-1%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20for%20h%20in%20bad_handles%3Ah.remove()%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20metric%3Dlambda%20logits%3Adict(logit_difference%3Dfloat((logits%5Baid%5D-logits%5Bbid%5D).item())%2Cp_clean_answer%3Dfloat(logits.softmax(-1)%5Baid%5D.item())%2Cp_corrupt_answer%3Dfloat(logits.softmax(-1)%5Bbid%5D.item()))%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20rows%3D%5Bdict(condition%3D'Clean'%2Clayer%3D-1%2Cposition%3D-1%2C**metric(original))%2Cdict(condition%3D'Corrupted'%2Clayer%3D-1%2Cposition%3D-1%2C**metric(broken))%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20positions%3Dsorted(set((x%5B'input_ids'%5D%5B0%5D!%3Dy%5B'input_ids'%5D%5B0%5D).nonzero().flatten().tolist()%2B%5Bx%5B'input_ids'%5D.shape%5B1%5D-1%5D))%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20for%20layer%2Cblock%20in%20enumerate(model.gpt_neox.layers)%3A%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20for%20position%20in%20positions%3A%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20random_vector%3Dtorch.randn_like(saved%5Blayer%5D%5B%3A%2Cposition%2C%3A%5D)%3Brandom_vector%3Drandom_vector%2Frandom_vector.norm()*saved%5Blayer%5D%5B%3A%2Cposition%2C%3A%5D.norm()%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20interventions%3D%5B('Patched'%2Csaved%5Blayer%5D%5B%3A%2Cposition%2C%3A%5D)%2C('No-op%20control'%2Cbad_saved%5Blayer%5D%5B%3A%2Cposition%2C%3A%5D)%2C('Zero%20control'%2Ctorch.zeros_like(random_vector))%2C('Random%20norm-matched%20control'%2Crandom_vector)%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20for%20condition%2Cvector%20in%20interventions%3A%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20def%20patch(module%2Cargs%2Cout)%3A%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20new%3Dhidden(out).clone()%3Bnew%5B%3A%2Cposition%2C%3A%5D%3Dvector%3Breturn%20replace(out%2Cnew)%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20handle%3Dblock.register_forward_hook(patch)%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20try%3Apatched%3Dmodel(**y).logits%5B0%2C-1%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20finally%3Ahandle.remove()%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20rows.append(dict(condition%3Dcondition%2Clayer%3Dlayer%2Cposition%3Dposition%2C**metric(patched)))%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20records.append(dict(clean%3Dclean%2Ccorrupt%3Dcorrupt%2Cclean_answer%3Da%2Ccorrupt_answer%3Db%2Cclean_tokens%3Dtokenizer.convert_ids_to_tokens(x%5B'input_ids'%5D%5B0%5D.tolist())%2Ccorrupt_tokens%3Dtokenizer.convert_ids_to_tokens(y%5B'input_ids'%5D%5B0%5D.tolist())%2Cpositions%3Dpositions%2Crows%3Drows%2Cresidual_norms%3D%5Bfloat(saved%5Bi%5D%5B0%2C-1%5D.norm().item())%20for%20i%20in%20range(len(saved))%5D))%0A%20%20%20%20%20%20%20%20if%20device%3D%3D'cuda'%3Atorch.cuda.synchronize()%0A%20%20%20%20%20%20%20%20return%20dict(model%3DMODEL_ID%2Crevision%3Drevision%2Ctorch%3Dtorch.__version__%2Ctransformers%3Dtransformers.__version__%2Cdevice%3Dtorch.cuda.get_device_name(0)%20if%20device%3D%3D'cuda'%20else%20'CPU'%2Cinference_seconds%3Dtime.perf_counter()-start%2Ccollected_at_utc%3Ddatetime.now(timezone.utc).isoformat()%2Cprecision%3D'float32'%2Cmethod%3D'Clean%20post-block%20residual%20vectors%20patched%20into%20corrupted%20prompts%20at%20changed-token%20and%20final-token%20positions.%20No%20weight%20changes.%20Controls%20are%20interventions%2C%20not%20guaranteed%20null%20effects.%20Candidate%20probabilities%20use%20the%20full%20vocabulary%20denominator.'%2Climits%3D'Six%20selected%20capital-city%20prompt%20pairs%2C%20with%20no-op%2C%20zero%20and%20one%20seeded%20norm-matched%20random%20intervention%20control%20in%20a%20small%20pretrained%20research%20model.%20Not%20a%20representative%20accuracy%20evaluation%2C%20a%20Goodfire%20replication%20or%20a%20frontier-model%20result.'%2Crecords%3Drecords)%0A%0A%0A%20%20%20%20return%20(collect%2C)%0A%0A%0A%40app.cell%0Adef%20_(Path%2C%20collect%2C%20mo%2C%20native_run%2C%20recorded%2C%20set_attempt%2C%20sys)%3A%0A%20%20%20%20if%20sys.platform!%3D'emscripten'%20and%20native_run.value%20is%20not%20None%3A%0A%20%20%20%20%20%20%20%20try%3A%0A%20%20%20%20%20%20%20%20%20%20%20%20with%20mo.status.spinner(title%3D'Loading%20pinned%20model%20and%20collecting%20matched%20interventions%E2%80%A6')%3A%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20_data%3Dcollect(recorded%5B'revision'%5D%2C'cuda'%20if%20native_run.value%3D%3D'H100%20CUDA'%20else%20'cpu'%2Cstr(Path.home()%2F'.cache'%2F'brightlab-models'))%0A%20%20%20%20%20%20%20%20%20%20%20%20set_attempt(%7B'data'%3A_data%2C'error'%3ANone%7D)%0A%20%20%20%20%20%20%20%20except%20Exception%20as%20_error%3A%0A%20%20%20%20%20%20%20%20%20%20%20%20set_attempt(%7B'data'%3ANone%2C'error'%3Astr(_error)%7D)%0A%20%20%20%20return%0A%0A%0A%40app.cell%0Adef%20_(get_attempt%2C%20recorded)%3A%0A%20%20%20%20attempt%3Dget_attempt()%0A%20%20%20%20dataset%3Dattempt%5B'data'%5D%20if%20attempt%20and%20attempt%5B'data'%5D%20else%20recorded%0A%20%20%20%20origin%3D'Fresh%20native%20model%20run'%20if%20attempt%20and%20attempt%5B'data'%5D%20else%20'Recorded%20'%2Brecorded%5B'device'%5D%2B'%20measurements%3A%20'%2Brecorded%5B'collected_at_utc'%5D%0A%20%20%20%20return%20attempt%2C%20dataset%2C%20origin%0A%0A%0A%40app.cell%0Adef%20_(attempt%2C%20dataset%2C%20mo%2C%20origin%2C%20prediction%2C%20reset_controls%2C%20restored)%3A%0A%20%20%20%20mo.stop(prediction.value%20is%20None%2Cmo.md('Save%20a%20prediction%20to%20inspect%20the%20measurements.'))%0A%20%20%20%20_saved%3Drestored.get('settings'%2C%7B%7D)%20if%20not%20reset_controls.value%20else%20%7B%7D%0A%20%20%20%20_pair%3D_saved.get('pair'%2C0)%0A%20%20%20%20if%20type(_pair)%20is%20not%20int%20or%20not%200%3C%3D_pair%3Clen(dataset%5B'records'%5D)%3A_pair%3D0%0A%20%20%20%20control%3Dmo.ui.dictionary(%7B'pair'%3Amo.ui.dropdown(options%3D%7Br%5B'clean'%5D%2B'%20%2F%20'%2Br%5B'corrupt'%5D%3Ai%20for%20i%2Cr%20in%20enumerate(dataset%5B'records'%5D)%7D%2Cvalue%3Ddataset%5B'records'%5D%5B_pair%5D%5B'clean'%5D%2B'%20%2F%20'%2Bdataset%5B'records'%5D%5B_pair%5D%5B'corrupt'%5D%2Clabel%3D'Matched%20prompt%20pair'%2Cfull_width%3DTrue)%2C'position'%3Amo.ui.dropdown(options%3D%7B'Changed%20country%20token'%3A0%2C'Final%20token'%3A1%7D%2Cvalue%3D'Changed%20country%20token'%20if%20_saved.get('position')%3D%3D0%20else%20'Final%20token'%2Clabel%3D'Patched%20token%20position')%2C'metric'%3Amo.ui.dropdown(options%3D%5B'logit_difference'%2C'p_clean_answer'%2C'p_corrupt_answer'%5D%2Cvalue%3D_saved.get('metric')%20if%20_saved.get('metric')%20in%20%5B'logit_difference'%2C'p_clean_answer'%2C'p_corrupt_answer'%5D%20else%20'logit_difference'%2Clabel%3D'Measurement')%2C'condition'%3Amo.ui.dropdown(options%3D%5B'Patched'%2C'No-op%20control'%2C'Zero%20control'%2C'Random%20norm-matched%20control'%5D%2Cvalue%3D_saved.get('condition')%20if%20_saved.get('condition')%20in%20%5B'Patched'%2C'No-op%20control'%2C'Zero%20control'%2C'Random%20norm-matched%20control'%5D%20else%20'Patched'%2Clabel%3D'Intervention%20or%20control')%7D)%0A%20%20%20%20mo.vstack(%5Bmo.callout(mo.md('**Current%20evidence%3A**%20'%2Borigin)%2Ckind%3D'info')%2Cmo.callout(mo.md('The%20fresh%20run%20failed%3B%20recorded%20evidence%20remains%20selected.%20'%2Battempt%5B'error'%5D)%2Ckind%3D'warn')%20if%20attempt%20and%20attempt%5B'error'%5D%20else%20mo.md('')%2Creset_controls%2Ccontrol.vstack()%5D)%0A%20%20%20%20return%20(control%2C)%0A%0A%0A%40app.cell%0Adef%20_(control%2C%20dataset)%3A%0A%20%20%20%20record%3Ddataset%5B'records'%5D%5Bcontrol.value%5B'pair'%5D%5D%0A%20%20%20%20position%3Drecord%5B'positions'%5D%5Bcontrol.value%5B'position'%5D%5D%0A%20%20%20%20rows%3D%5Br%20for%20r%20in%20record%5B'rows'%5D%20if%20r%5B'condition'%5D%3D%3Dcontrol.value%5B'condition'%5D%20and%20r%5B'position'%5D%3D%3Dposition%5D%0A%20%20%20%20reference%3Drecord%5B'rows'%5D%5B%3A2%5D%0A%20%20%20%20den%3Dreference%5B0%5D%5B'logit_difference'%5D-reference%5B1%5D%5B'logit_difference'%5D%0A%20%20%20%20analysis_rows%3D%5B%7B**r%2C'normalised_recovery'%3A(r%5B'logit_difference'%5D-reference%5B1%5D%5B'logit_difference'%5D)%2Fden%20if%20abs(den)%3E1e-6%20else%20None%7D%20for%20r%20in%20rows%5D%0A%20%20%20%20chart%3D%7B'kind'%3A'line'%2C'rows'%3Aanalysis_rows%2C'x'%3A'layer'%2C'y'%3Acontrol.value%5B'metric'%5D%7D%0A%20%20%20%20return%20analysis_rows%2C%20chart%2C%20position%2C%20record%2C%20reference%0A%0A%0A%40app.cell%0Adef%20_(html%2C%20math%2C%20np)%3A%0A%20%20%20%20def%20draw(out)%3A%0A%20%20%20%20%20%20%20%20%22%22%22Accessible%20SVG%20views%2C%20with%20exact%20values%20duplicated%20in%20tables.%22%22%22%0A%20%20%20%20%20%20%20%20esc%3Dlambda%20s%3Ahtml.escape(str(s)%2Cquote%3DTrue)%0A%20%20%20%20%20%20%20%20marks%3D%5B%5D%3Bkind%3Dout%5B'kind'%5D%3Brows%3Dout.get('chart_rows'%2Cout%5B'rows'%5D)%3Bx%3Dout%5B'x'%5D%3By%3Dout%5B'y'%5D%0A%20%20%20%20%20%20%20%20if%20kind%3D%3D'circle'%3A%0A%20%20%20%20%20%20%20%20%20%20%20%20marks.append('%3Ccircle%20cx%3D%22290%22%20cy%3D%22180%22%20r%3D%22130%22%20fill%3D%22none%22%20stroke%3D%22%23778d85%22%20stroke-width%3D%222%22%2F%3E')%0A%20%20%20%20%20%20%20%20%20%20%20%20for%20i%20in%20range(12)%3A%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20a%3D2*math.pi*i%2F12%3Blabel%3Dout.get('labels'%2C%5Bstr(i*30)%2B'%C2%B0'%20for%20i%20in%20range(12)%5D)%5Bi%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20marks.append(f'%3Ctext%20x%3D%22%7B290%2B152*math.cos(a)%3A.1f%7D%22%20y%3D%22%7B185-152*math.sin(a)%3A.1f%7D%22%20text-anchor%3D%22middle%22%3E%7Besc(label)%7D%3C%2Ftext%3E')%0A%20%20%20%20%20%20%20%20%20%20%20%20points%3D'%20'.join(f'%7B290%2B130*r%5Bx%5D%3A.2f%7D%2C%7B180-130*r%5By%5D%3A.2f%7D'%20for%20r%20in%20rows)%0A%20%20%20%20%20%20%20%20%20%20%20%20marks.append(f'%3Cpolyline%20points%3D%22%7Bpoints%7D%22%20fill%3D%22none%22%20stroke%3D%22%239f3e20%22%20stroke-width%3D%224%22%2F%3E')%0A%20%20%20%20%20%20%20%20%20%20%20%20for%20i%2Cr%20in%20enumerate(rows)%3Amarks.append(f'%3Ccircle%20cx%3D%22%7B290%2B130*r%5Bx%5D%3A.2f%7D%22%20cy%3D%22%7B180-130*r%5By%5D%3A.2f%7D%22%20r%3D%224%22%20fill%3D%22%239f3e20%22%3E%3Ctitle%3EStep%20%7Bi%7D%3A%20(%7Br%5Bx%5D%3A.3f%7D%2C%20%7Br%5By%5D%3A.3f%7D)%3C%2Ftitle%3E%3C%2Fcircle%3E')%0A%20%20%20%20%20%20%20%20%20%20%20%20marks.append('%3Ctext%20x%3D%22485%22%20y%3D%22160%22%3EOrange%3A%20trajectory%3C%2Ftext%3E%3Ctext%20x%3D%22485%22%20y%3D%22185%22%3EGrey%3A%20unit%20circle%3C%2Ftext%3E')%0A%20%20%20%20%20%20%20%20elif%20kind%3D%3D'line'%3A%0A%20%20%20%20%20%20%20%20%20%20%20%20values%3D%5Bfloat(r%5By%5D)%20for%20r%20in%20rows%5D%3Blo%3Dmin(0%2Cmin(values))%3Bhi%3Dmax(.01%2Cmax(values))%3Bpx%3Dlambda%20v%3A70%2B(v-float(rows%5B0%5D%5Bx%5D))%2Fmax(1e-9%2Cfloat(rows%5B-1%5D%5Bx%5D)-float(rows%5B0%5D%5Bx%5D))*570%3Bpy%3Dlambda%20v%3A300-(v-lo)%2Fmax(1e-9%2Chi-lo)*245%0A%20%20%20%20%20%20%20%20%20%20%20%20points%3D'%20'.join(f'%7Bpx(float(r%5Bx%5D))%3A.2f%7D%2C%7Bpy(float(r%5By%5D))%3A.2f%7D'%20for%20r%20in%20rows)%0A%20%20%20%20%20%20%20%20%20%20%20%20marks.append(f'%3Cpolyline%20points%3D%22%7Bpoints%7D%22%20fill%3D%22none%22%20stroke%3D%22%239f3e20%22%20stroke-width%3D%223%22%2F%3E')%0A%20%20%20%20%20%20%20%20%20%20%20%20for%20r%20in%20rows%3Amarks.append(f'%3Ccircle%20cx%3D%22%7Bpx(float(r%5Bx%5D))%3A.2f%7D%22%20cy%3D%22%7Bpy(float(r%5By%5D))%3A.2f%7D%22%20r%3D%224%22%20fill%3D%22%239f3e20%22%3E%3Ctitle%3E%7Besc(x)%7D%20%7Br%5Bx%5D%7D%3A%20%7Br%5By%5D%3A.4f%7D%3C%2Ftitle%3E%3C%2Fcircle%3E')%0A%20%20%20%20%20%20%20%20%20%20%20%20for%20frac%20in%20%5B0%2C.5%2C1%5D%3A%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20val%3Dlo%2Bfrac*(hi-lo)%3Bmarks.append(f'%3Ctext%20x%3D%2260%22%20y%3D%22%7Bpy(val)%2B5%7D%22%20text-anchor%3D%22end%22%3E%7Bval%3A.3g%7D%3C%2Ftext%3E')%0A%20%20%20%20%20%20%20%20%20%20%20%20marks.append(f'%3Cpath%20d%3D%22M70%2040%20V300%20H645%22%20fill%3D%22none%22%20stroke%3D%22%2359756b%22%2F%3E%3Ctext%20x%3D%2270%22%20y%3D%22325%22%3E%7Brows%5B0%5D%5Bx%5D%7D%3C%2Ftext%3E%3Ctext%20x%3D%22640%22%20y%3D%22325%22%20text-anchor%3D%22end%22%3E%7Brows%5B-1%5D%5Bx%5D%7D%3C%2Ftext%3E%3Ctext%20x%3D%22340%22%20y%3D%22350%22%20text-anchor%3D%22middle%22%3E%7Besc(x)%7D%3C%2Ftext%3E%3Ctext%20x%3D%2270%22%20y%3D%2225%22%3E%7Besc(y)%7D%3C%2Ftext%3E')%0A%20%20%20%20%20%20%20%20else%3A%0A%20%20%20%20%20%20%20%20%20%20%20%20vals%3D%5Bfloat(r%5By%5D)%20for%20r%20in%20rows%5D%3Blo%3Dmin(0%2Cmin(vals))%3Bhi%3Dmax(.001%2Cmax(vals))%3Bscale%3Dlambda%20v%3A200%2B440*(v-lo)%2Fmax(1e-9%2Chi-lo)%3Bzero%3Dscale(0)%3Bheight%3Dmin(46%2C270%2Fmax(1%2Clen(rows)))%0A%20%20%20%20%20%20%20%20%20%20%20%20for%20i%2Cr%20in%20enumerate(rows)%3A%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20yy%3D50%2Bi*height%3Bend%3Dscale(float(r%5By%5D))%3Bmarks.append(f'%3Ctext%20x%3D%22187%22%20y%3D%22%7Byy%2B14%7D%22%20text-anchor%3D%22end%22%3E%7Besc(r%5Bx%5D)%7D%3C%2Ftext%3E%3Crect%20x%3D%22%7Bmin(zero%2Cend)%3A.2f%7D%22%20y%3D%22%7Byy%7D%22%20width%3D%22%7Bmax(.5%2Cabs(end-zero))%3A.2f%7D%22%20height%3D%22%7Bheight*.65%7D%22%20rx%3D%222%22%20fill%3D%22%2326735c%22%2F%3E%3Ctext%20x%3D%22650%22%20y%3D%22%7Byy%2B14%7D%22%3E%7Bfloat(r%5By%5D)%3A.3g%7D%3C%2Ftext%3E')%0A%20%20%20%20%20%20%20%20%20%20%20%20marks.append(f'%3Ctext%20x%3D%22200%22%20y%3D%2225%22%3E%7Besc(y)%7D%3C%2Ftext%3E')%0A%20%20%20%20%20%20%20%20graphic%3Df'%3Csvg%20viewBox%3D%220%200%20720%20365%22%20role%3D%22img%22%20aria-label%3D%22%7Besc(y)%7D%20chart%3B%20exact%20values%20in%20the%20evidence%20table%22%20style%3D%22width%3A100%25%3Bmin-width%3A560px%3Bbackground%3A%23f7f8f2%3Bborder-radius%3A12px%3Bfont%3A14px%20system-ui%3Bfill%3A%23173d36%22%3E%3Ctitle%3E%7Besc(y)%7D%20chart%3C%2Ftitle%3E%7B%22%22.join(marks)%7D%3C%2Fsvg%3E'%0A%20%20%20%20%20%20%20%20if%20out.get('heat')%20is%20not%20None%3A%0A%20%20%20%20%20%20%20%20%20%20%20%20heat%3Dnp.asarray(out%5B'heat'%5D)%3Bmaxv%3Dmax(1e-9%2Cfloat(np.max(np.abs(heat))))%3Bcells%3D%5B%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20for%20i%2Crow%20in%20enumerate(heat)%3A%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20cells.append(f'%3Ctext%20x%3D%2280%22%20y%3D%22%7B44%2Bi*24%7D%22%20text-anchor%3D%22end%22%3E%7Besc(out%5B%22heat_labels%22%5D%5Bi%5D)%7D%3C%2Ftext%3E')%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20for%20j%2Cval%20in%20enumerate(row)%3A%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20opacity%3D.1%2B.9*abs(float(val))%2Fmaxv%3Bcolor%3D'%2326735c'%20if%20val%3E%3D0%20else%20'%23a94322'%3Bcells.append(f'%3Crect%20x%3D%22%7B90%2Bj*32%7D%22%20y%3D%22%7B27%2Bi*24%7D%22%20width%3D%2229%22%20height%3D%2221%22%20fill%3D%22%7Bcolor%7D%22%20opacity%3D%22%7Bopacity%3A.3f%7D%22%3E%3Ctitle%3ERow%20%7Bi%7D%2C%20column%20%7Bj%7D%3A%20%7Bval%3A.5f%7D%3C%2Ftitle%3E%3C%2Frect%3E')%0A%20%20%20%20%20%20%20%20%20%20%20%20for%20j%20in%20range(heat.shape%5B1%5D)%3Acells.append(f'%3Ctext%20x%3D%22%7B96%2Bj*32%7D%22%20y%3D%2218%22%3E%7Bj%7D%3C%2Ftext%3E')%0A%20%20%20%20%20%20%20%20%20%20%20%20graphic%2B%3Df'%3Csvg%20viewBox%3D%220%200%20720%20%7B65%2B24*len(heat)%7D%22%20role%3D%22img%22%20aria-label%3D%22Value%20heat%20map.%20Green%20positive%2C%20rust%20negative.%20Exact%20values%20in%20the%20heat%20map%20table.%22%20style%3D%22width%3A100%25%3Bmin-width%3A560px%3Bfont%3A12px%20system-ui%3Bfill%3A%23173d36%22%3E%3Ctitle%3EHeat%20map%20with%20indexed%20columns%3C%2Ftitle%3E%7B%22%22.join(cells)%7D%3C%2Fsvg%3E'%0A%20%20%20%20%20%20%20%20return%20'%3Cdiv%20tabindex%3D%220%22%20role%3D%22group%22%20aria-label%3D%22Scrollable%20experiment%20charts%22%20style%3D%22max-width%3A100%25%3Boverflow-x%3Aauto%22%3E'%2Bgraphic%2B'%3C%2Fdiv%3E'%0A%0A%20%20%20%20return%20(draw%2C)%0A%0A%0A%40app.cell%0Adef%20_(analysis_rows%2C%20chart%2C%20draw%2C%20mo%2C%20position%2C%20record%2C%20reference)%3A%0A%20%20%20%20mo.vstack(%5Bmo.md('**Clean%20prompt%3A**%20'%2Brecord%5B'clean'%5D%2B'%5Cn%5Cn**Corrupted%20prompt%3A**%20'%2Brecord%5B'corrupt'%5D%2B'%5Cn%5Cn**Candidate%20comparison%3A**%20%60'%2Brecord%5B'clean_answer'%5D%2B'%60%20minus%20%60'%2Brecord%5B'corrupt_answer'%5D%2B'%60.%20Patched%20position%3A%20'%2Bstr(position)%2B'.')%2Cmo.Html(draw(chart))%2Cmo.ui.table(analysis_rows%2Cselection%3DNone)%2Cmo.accordion(%7B'Clean%20and%20corrupted%20baselines'%3Amo.ui.table(reference%2Cselection%3DNone)%2C'Token%20alignment'%3Amo.ui.table(%5B%7B'position'%3Ai%2C'clean'%3Aa%2C'corrupted'%3Ab%7D%20for%20i%2C(a%2Cb)%20in%20enumerate(zip(record%5B'clean_tokens'%5D%2Crecord%5B'corrupt_tokens'%5D))%5D%2Cselection%3DNone)%7D)%5D)%0A%20%20%20%20return%0A%0A%0A%40app.cell%0Adef%20_(mo%2C%20restored)%3A%0A%20%20%20%20get_runs%2Cset_runs%3Dmo.state(restored.get('saved_runs'%2C%5B%5D)%5B%3A6%5D)%0A%20%20%20%20return%20get_runs%2C%20set_runs%0A%0A%0A%40app.cell%0Adef%20_(analysis_rows%2C%20control%2C%20mo%2C%20set_runs)%3A%0A%20%20%20%20_snapshot%3D%7B'settings'%3Adict(control.value)%2C'measurements'%3Aanalysis_rows%7D%0A%20%20%20%20keep%3Dmo.ui.button(label%3D'Save%20this%20run%20for%20comparison'%2Con_click%3Dlambda%20_%3Aset_runs(lambda%20old%3A(old%2B%5B_snapshot%5D)%5B-6%3A%5D))%0A%20%20%20%20clear%3Dmo.ui.button(label%3D'Clear%20saved%20comparisons'%2Con_click%3Dlambda%20_%3Aset_runs(%5B%5D))%0A%20%20%20%20mo.hstack(%5Bkeep%2Cclear%5D)%0A%20%20%20%20return%0A%0A%0A%40app.cell%0Adef%20_(get_runs%2C%20mo)%3A%0A%20%20%20%20mo.vstack(%5Bmo.md(f'Saved%20comparisons%3A%20%7Blen(get_runs())%7D%20%2F%206.%20A%20seventh%20replaces%20the%20oldest.')%2Cmo.accordion(%7Bf'Run%20%7Bi%2B1%7D'%3Amo.ui.table(r%5B'measurements'%5D%2Cselection%3DNone)%20for%20i%2Cr%20in%20enumerate(get_runs())%7D)%5D)%0A%20%20%20%20return%0A%0A%0A%40app.cell%0Adef%20_(mo%2C%20restored)%3A%0A%20%20%20%20reflection%3Dmo.ui.text_area(value%3Drestored.get('conclusion'%2C'')%2Clabel%3D'My%20conclusion%20%E2%80%94%20intervention%2C%20evidence%20and%20scope'%2Cplaceholder%3D'Which%20positive%20control%20worked%3F%20What%20does%20this%20experiment%20leave%20unresolved%3F'%2Cfull_width%3DTrue%2Cdebounce%3DFalse)%0A%20%20%20%20mo.vstack(%5Bmo.md('%23%23%20Challenge%20the%20explanation%5CnCompare%20changed-token%20and%20final-token%20patches%20across%20all%20six%20blocks.%20The%20post-final-block%20final-token%20patch%20should%20recover%20the%20clean%20score%3A%20it%20replaces%20the%20state%20read%20by%20the%20output%20head.%20Treat%20that%20as%20a%20positive%20control%2C%20not%20discovery%20of%20a%20complete%20circuit.%20Repeat%20across%20all%20six%20prompt%20pairs.%5Cn%5Cn**Make%20an%20intervention%20report%3A**%20name%20the%20checkpoint%2C%20prompt%20pair%2C%20token%20position%2C%20layer%2C%20metric%20and%20normalised%20recovery.%20Recovery%20can%20exceed%200%E2%80%931%3B%20a%20near-zero%20baseline%20gap%20makes%20it%20unstable.')%2Creflection%5D)%0A%20%20%20%20return%20(reflection%2C)%0A%0A%0A%40app.cell%0Adef%20_(%0A%20%20%20%20analysis_rows%2C%0A%20%20%20%20control%2C%0A%20%20%20%20dataset%2C%0A%20%20%20%20get_runs%2C%0A%20%20%20%20html%2C%0A%20%20%20%20json%2C%0A%20%20%20%20mo%2C%0A%20%20%20%20origin%2C%0A%20%20%20%20prediction%2C%0A%20%20%20%20reference%2C%0A%20%20%20%20reflection%2C%0A)%3A%0A%20%20%20%20journal%3D%7B'format'%3A'brightlab-notebook-project'%2C'version'%3A1%2C'saved_runs'%3Aget_runs()%2C'lesson'%3A'u12-llm'%2C'evidence_origin'%3Aorigin%2C'prediction'%3Aprediction.value%2C'settings'%3Acontrol.value%2C'measurements'%3Aanalysis_rows%2C'baselines'%3Areference%2C'conclusion'%3Areflection.value%2C'provenance'%3A%7Bk%3Av%20for%20k%2Cv%20in%20dataset.items()%20if%20k!%3D'records'%7D%7D%0A%20%20%20%20_e%3Dhtml.escape%0A%20%20%20%20def%20_readable(value)%3A%0A%20%20%20%20%20%20%20%20if%20isinstance(value%2Cdict)%3Areturn%20'%3Cdl%3E'%2B''.join('%3Cdt%3E%3Cstrong%3E'%2B_e(str(k).replace('_'%2C'%20'))%2B'%3C%2Fstrong%3E%3C%2Fdt%3E%3Cdd%3E'%2B_readable(v)%2B'%3C%2Fdd%3E'%20for%20k%2Cv%20in%20value.items())%2B'%3C%2Fdl%3E'%0A%20%20%20%20%20%20%20%20if%20isinstance(value%2Clist)%3A%0A%20%20%20%20%20%20%20%20%20%20%20%20if%20value%20and%20all(isinstance(row%2Cdict)%20for%20row%20in%20value)%3A%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20_keys%3Dlist(dict.fromkeys(k%20for%20row%20in%20value%20for%20k%20in%20row))%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20return%20'%3Ctable%3E%3Cthead%3E%3Ctr%3E'%2B''.join('%3Cth%3E'%2B_e(str(k).replace('_'%2C'%20'))%2B'%3C%2Fth%3E'%20for%20k%20in%20_keys)%2B'%3C%2Ftr%3E%3C%2Fthead%3E%3Ctbody%3E'%2B''.join('%3Ctr%3E'%2B''.join('%3Ctd%3E'%2B_readable(row.get(k%2C''))%2B'%3C%2Ftd%3E'%20for%20k%20in%20_keys)%2B'%3C%2Ftr%3E'%20for%20row%20in%20value)%2B'%3C%2Ftbody%3E%3C%2Ftable%3E'%0A%20%20%20%20%20%20%20%20%20%20%20%20return%20'%3Col%3E'%2B''.join('%3Cli%3E'%2B_readable(v)%2B'%3C%2Fli%3E'%20for%20v%20in%20value)%2B'%3C%2Fol%3E'%0A%20%20%20%20%20%20%20%20return%20_e(str(value))%20if%20value%20is%20not%20None%20else%20'Not%20recorded'%0A%20%20%20%20_report%3D'%3Chtml%20lang%3D%22en-AU%22%3E%3Cmeta%20charset%3D%22utf-8%22%3E%3Ctitle%3EMy%20Pythia%20investigation%3C%2Ftitle%3E%3Cstyle%3Ebody%7Bfont%3A18px%2F1.6%20system-ui%3Bmax-width%3A950px%3Bmargin%3Aauto%3Bpadding%3A25px%7Dtd%2Cth%7Bborder%3A1px%20solid%20%23999%3Bpadding%3A8px%7Dtable%7Bborder-collapse%3Acollapse%7D%3C%2Fstyle%3E%3Ch1%3EMy%20Pythia%20investigation%3C%2Fh1%3E%3Cp%3E'%2B_e(origin)%2B'%3C%2Fp%3E%3Ch2%3EPrediction%3C%2Fh2%3E%3Cp%3E'%2B_e(prediction.value%20or%20'')%2B'%3C%2Fp%3E%3Ch2%3ESettings%3C%2Fh2%3E%3Cp%3E'%2B_e(str(control.value))%2B'%3C%2Fp%3E%3Ctable%3E'%2B''.join('%3Ctr%3E'%2B''.join('%3Ctd%3E'%2B_e(str(v))%2B'%3C%2Ftd%3E'%20for%20v%20in%20row.values())%2B'%3C%2Ftr%3E'%20for%20row%20in%20analysis_rows)%2B'%3C%2Ftable%3E%3Ch2%3EConclusion%3C%2Fh2%3E%3Cp%3E'%2B_e(reflection.value)%2B'%3C%2Fp%3E%3Ch2%3EScope%3C%2Fh2%3E%3Cp%3E'%2B_e(dataset%5B'limits'%5D)%2B'%3C%2Fp%3E%3C%2Fhtml%3E'%0A%20%20%20%20_report%3D_report.replace('%3C%2Fhtml%3E'%2C'%3Ch2%3EBaselines%3C%2Fh2%3E'%2B_readable(reference)%2B'%3Ch2%3ESaved%20comparisons%3C%2Fh2%3E'%2B_readable(get_runs())%2B'%3C%2Fhtml%3E')%0A%20%20%20%20mo.hstack(%5Bmo.download(data%3Djson.dumps(journal%2Cindent%3D2%2Callow_nan%3DFalse).encode()%2Cfilename%3D'u12-llm-project.json'%2Clabel%3D'Download%20resumable%20notebook%20project')%2Cmo.download(data%3D_report.encode()%2Cfilename%3D'u12-llm-report.html'%2Clabel%3D'Download%20readable%20report%20%2F%20print')%5D)%0A%20%20%20%20return%0A%0A%0A%40app.cell%0Adef%20_(dataset%2C%20mo)%3A%0A%20%20%20%20mo.accordion(%7B'Teacher%20lesson%20plan'%3Amo.md('**Preparation%3A**%20browser%20runtime%20access%3B%20no%20personal%20text%2C%20accounts%20or%20GPU%20needed%20for%20the%20recorded-data%20lesson.%20Review%20the%20six%20fixed%20prompt%20pairs.%5Cn%5Cn**0%E2%80%938%20min%3A**%20distinguish%20a%20recorded%20observation%20from%20a%20new%20intervention%20run%3B%20predict%20the%20final-block%20positive%20control.%20**8%E2%80%9325%3A**%20compare%20positions%20and%20layers.%20**25%E2%80%9340%3A**%20repeat%20across%20prompt%20pairs%20and%20inspect%20token%20alignment.%20**40%E2%80%9355%3A**%20write%20a%20claim%20with%20checkpoint%20and%20scope.%5Cn%5Cn**Assessment%20(0%E2%80%932%20each)%3A**%20matched%20comparison%3B%20correctly%20defined%20logit%20difference%3B%20positive-control%20interpretation%3B%20limited%20claim.%20Support%20with%20the%20clean%2Fcorrupted%20baseline%20table.%20Extend%20by%20repeating%20the%20native%20experiment%2C%20then%20adding%20one%20teacher-reviewed%20pair%20with%20equal%20token%20lengths%20and%20single-token%20answer%20candidates.%20Preserve%20the%20original%20data%20and%20reserve%20new%20pairs%20for%20evaluation.%5Cn%5Cn**Misconceptions%3A**%20high%20probability%20is%20not%20truth%3B%20patch%20recovery%20does%20not%20identify%20a%20unique%20circuit%3B%20a%20small%20research%20model%20does%20not%20represent%20all%20LLMs.%20H100%20acceleration%20is%20unnecessary%20for%20this%20small%20workload.')%2C'Glossary%20and%20method'%3Amo.md('**Activation%3A**%20a%20measured%20intermediate%20tensor.%20**Residual%20patch%3A**%20replacing%20an%20internal%20vector.%20**Logit%3A**%20an%20unnormalised%20token%20score.%20**Recovery%3A**%20(patched%20%E2%88%92%20corrupted)%2F(clean%20%E2%88%92%20corrupted)%2C%20using%20the%20chosen%20candidate%20logit%20difference.%20**Recorded%20trace%3A**%20values%20from%20a%20previous%20identified%20model%20run%2C%20not%20a%20live%20model%20response.')%2C'Provenance'%3Amo.json(%7Bk%3Av%20for%20k%2Cv%20in%20dataset.items()%20if%20k!%3D'records'%7D)%2C'Read%20the%20research'%3Amo.md('%5BPythia%20model%20card%5D(https%3A%2F%2Fhuggingface.co%2FEleutherAI%2Fpythia-70m-deduped)%20%C2%B7%20%5BGoodfire%3A%20Mixing%20Mechanisms%5D(https%3A%2F%2Fwww.goodfire.com%2Fresearch%2Fmixing-mechanisms)%20%C2%B7%20%5BGoodfire%3A%20Replicating%20Circuit%20Tracing%5D(https%3A%2F%2Fwww.goodfire.com%2Fresearch%2Freplicating-circuit-tracing-for-a-simple-mechanism)%20%C2%B7%20%5BCommunity%20circuit%20research%20guide%5D(https%3A%2F%2Fwww.neuronpedia.org%2Fgraph%2Finfo)%5Cn%5CnThese%20papers%20motivate%20controlled%20interventions%3B%20this%20notebook%20uses%20a%20different%20model%20and%20task.%20It%20is%20independently%20authored%20and%20is%20not%20endorsed%20by%20Goodfire.')%7D)%0A%20%20%20%20return%0A%0A%0Aif%20__name__%20%3D%3D%20%22__main__%22%3A%0A%20%20%20%20app.run()%0A