Writing
- I Thought Deleting Your Data Had Exactly One Right Answer
For several months I tested AI models on one small legal decision: whether a company should erase someone's data under India's new data protection law. This is what I saw, what others working on law and AI are finding, and how my idea of a right answer changed along the way.
- My LLM Eval Was Reading the Answer From the IDs
In July I measured what an LLM does when it replaces a rule-based legal system. The ids in my prompts were carrying part of the answer, and my isolation tests passed anyway, because they checked the data, not the prompt. This is the correction, with a re-measurement on 350 cases.
- Should an AI Decide What Gets Erased? I Measured It
I built a system that adjudicates data-erasure requests with no model in the decision path, then measured what a capable model would have done in its place. The model almost never broke the law. What it did instead was more interesting. (Corrected: see note.)