LLMs and Contextual Integrity
I have been thinking a lot about AI and integrity. Part of that is contextual integrity. I recently found two papers on the topic.
“CIMemories: A Compositional Benchmark for Contextual Integrity of Persistent Memory in LLMs“:
Abstract: Large Language Models (LLMs) increasingly use persistent memory from past interactions to enhance personalization and task performance. However, this memory introduces critical risks when sensitive information is revealed in inappropriate contexts. We present CIMemories, a benchmark for evaluating whether LLMs appropriately control information flow from memory based on task context. CIMemories uses synthetic user profiles with over 100 attributes per user, paired with diverse task contexts in which each attribute may be essential for some tasks but inappropriate for others. Our evaluation reveals that frontier models exhibit up to 69% attribute-level violations (leaking information inappropriately), with lower violation rates often coming at the cost of task utility. Violations accumulate across both tasks and runs: as usage increases from 1 to 40 tasks, GPT-5’s violations rise from 0.1% to 9.6%, reaching 25.1% when the same prompt is executed 5 times, revealing arbitrary and unstable behavior in which models leak different attributes for identical prompts. Privacy-conscious prompting does not solve this—models overgeneralize, sharing everything or nothing rather than making nuanced, context-dependent decisions. These findings reveal fundamental limitations that require contextually aware reasoning capabilities, not just better prompting or scaling.
“Contextual Integrity in LLMs via Reasoning and Reinforcement Learning“:
Abstract: As the era of autonomous agents making decisions on behalf of users unfolds, ensuring contextual integrity (CI)—what is the appropriate information to share while carrying out a certain task—becomes a central question to the field. We posit that CI demands a form of reasoning where the agent needs to reason about the context in which it is operating. To test this, we first prompt LLMs to reason explicitly about CI when deciding what information to disclose. We then extend this approach by developing a reinforcement learning (RL) framework that further instills in models the reasoning necessary to achieve CI. Using a synthetic, automatically created, dataset of only 700 examples but with diverse contexts and information disclosure norms, we show that our method substantially reduces inappropriate information disclosure while maintaining task performance across multiple model sizes and families. Importantly, improvements transfer from this synthetic dataset to established CI benchmarks such as PrivacyLens that has human annotations and evaluates privacy leakage of AI assistants in actions and tool calls.
Subscribe to comments on this entry
GregW • August 18, 2026 10:51 AM
Regarding “thinking about AI and integrity”, while it depends on what you mean by integrity, in the domain of information processing (not storage itself), the “Integrity” assurances in a deterministic computational system are pretty different from “Integrity” assurances that can be made in a nondeterministic system where many or all steps of data processing occur via nondeterministic LLM agents.
There are ways to request certain models to be deterministic but this issue of nondeterminism gets pretty fundamental with certain types of use cases pretty fast. You start to have to deal with m-of-n voting, distilling and checksumming key outputs, etc to patch over the integrity-of-processing issues (just like with unreliable hardware) or carefully anticipate which steps of a process can or cannot be allowed to be nondeterministic.