Schneier on Security

Clive Robinson • October 2, 2024 6:15 AM

@ lurker,

Re : Hacking has broad meaning.

I may be wrong but as I understand it from,

<

blockquote>“When security researcher Johann Rehberger recently reported a vulnerability in ChatGPT that allowed attackers to store false information and malicious instructions in a user’s long-term memory settings”

<

blockquote>

The researcher found a way to modify “long term memory” for a user that would then cause ChatGPT to do things “for a user” from then on in a way that was “not obvious/apparent to the user”.

Further,

“OpenAI summarily closed the inquiry, labeling the flaw a safety issue, not, technically speaking, a security concern.”

OpenAI “potentially misunderstood / misclassified” that was reported to them, thus did not take action.

As we don’t have the information supplied in the article it’s not possible to say from it’s reporting.

So the researcher made the flaw found a little more obvious to OpenAI (again by how much is not in the article other than saying it was a POC with what that might imply).

The flaw and hack are described as,

“Rehberger found that memories could be created and permanently stored through indirect prompt injection, an AI exploit that causes an LLM to follow instructions from untrusted content such as emails, blog posts, or documents. The researcher demonstrated how he could trick ChatGPT into believing a targeted user was 102 years old, lived in the Matrix, and insisted Earth was flat and the LLM would incorporate that information to steer all future conversations. These false memories could be planted by storing files in Google Drive or Microsoft OneDrive, uploading images, or browsing a site like Bing—all of which could be created by a malicious attacker.”

The implication as written is the “prompt injection” was not done as the “effected user” but as another user and it in effect updated the more “global memory” of ChatGPT.

But could some how be focused on a particular user[1] via “the system” not “their account”.

So I could produce allegations against J.Sixpack in many places on the Internet that I know ChatGPT goes sniffing for “input”. I could by say making a post to some “social media site” that has lax or nonexistant rules get others to “create churn” around the name “J.Sixpack” which then gets used to train ChatGPT and in the process pulls in all the “fake memories” you’ve created on line. Much like happens with Wikipedia and it’s “secondary sources” rules, the “primary” or “true” data is downgraded as being in effect “faux”…

Whilst there are known ways to embed malware payloads in images and the like, it’s not clear how this actually effects an individual user or target.

It’s written to suggest that when a user follows a link then they somehow get their user memory corrupted, which I’m assuming is not the main memory ChatGPT uses for any and all users.

I assume that AI attached to Internet Search Engines like Microsoft Bing and DuckDuck get corrupted in some way.

That said over on the NIST password rules update page,

https://www.schneier.com/blog/archives/2024/09/nist-recommends-some-common-sense-password-rules.html

I detailed how to use Bing to reduce the entropy of a human memorable passphrase down to maybe 9-10 bits irrespective of the passphrases actual length.

[1] A real world example might be an “Interpol Red Notice” for an alleged criminal. The person subject to the Red Notice may not have committed any crimes or done anything wrong, but when they cross a border they get detained and made available for extradition by the nation they have crossed into. It’s known that Spain has abused the process in Europe and well so have other dictortorial nations, some of whom have quite a history,

https://www.rednoticelawjournal.com/2024/09/china-interpol-red-notice-abuse-in-cases-of-wealthy-business-owners-and-dissidents-post-1-of-2/