Simon Willison has been playing with the video processing capabilities of the new Gemini Pro 1.5 model from Google, and it’s really impressive.
Which means a lot of scary new video prompt injection attacks. And remember, given the current state of technology, prompt injection attacks are impossible to prevent in general.
Apokrif • February 22, 2024 4:27 PM
I don’t understand why the IA treats text in images as instructions.
Bob • February 22, 2024 5:09 PM
@Apokrif
Then you’ve basically missed out on the entire history of software development.
vas pup • February 22, 2024 6:05 PM
Israeli deepfake detection startup raises first $16 million in seed funding
https://www.timesofisrael.com/israeli-deepfake-detection-startup-raises-first-16-million-in-seed-funding/
“The sophistication of deepfake technology backed by advanced AI models makes it difficult to distinguish between real versus manipulated or fabricated online content. The creation of synthetic media is being abused for the spread of misinformation and propaganda, including during the ongoing war with the Hamas terror group.
Clarity, an Israeli AI cybersecurity startup that has developed software to detect and protect against deepfakes, said on Thursday that it has raised its first $16 million in seed money. The financing round will enable the startup to double the number of its workers and expand its research and development operations.
Clarity was planning to focus much of its attention on developing software for the detection of deepfakes often used to spread misinformation to sway public opinion in election campaigns, and thereby help mitigate the threat these pose to democracies. The startup has 15 employees.
The startup’s patent-pending technology detects and analyzes AI manipulations in videos, images and audio, and authenticates media using various techniques such as image forensics, metadata analysis and AI-based algorithms.
Clarity has been working with the Israeli government and has recently partnered with an Israeli-founded video software firm Kaltura to verify and authenticate sensitive video footage and testimonials of hostages from the October 7 terror assault.
“The ongoing war has made it clear that warfare transcends into the digital space and requires governments to leverage technology and work with startups, where it is for using drones, face recognition, or detecting deepfakes.”
Deepfake technology uses deep learning, a form of artificial intelligence involving machine learning to create seemingly realistic renderings of real people and their voices. The recent explosion of generative AI models and chatbots such as ChatGPT are assisting bad actors to generate deepfakes quickly, using sophisticated tools that are widely accessible, and at no cost.
“Our digital lives are under attack by shockingly accurate representations of people saying things they never said, and doing things they never did; we are just at the beginning of the invasion of deceit,” said Matias, who is the son of Yossi Matias, Managing Director of Google’s R&D center in Israel. “We want to create a layer of trust for digital media and content.”
JonKnowsNothing • February 22, 2024 7:09 PM
All
It does not take technology per se to create an image that passes as real but is a fake.
Forgers do it all the time. Great Masterpieces, honored by prime viewing spots in National Museums, get stashed in the base-ment once their “attributed to” ends up being a contemporary artist.
The US Military, CIA are masters at doing this long long before Deep Fake became the applicable term. The USA entered a good number of wars based on forged, fraudulent imagery, text and altered analysis.
The danger isn’t so much that the technique can be done by everyone-plus-dog, it’s that the CIA is losing a prime source of NUDGE in their never ending pursuit of … well, I don’t really know what the CIA is pursuing but whatever it is, it is not what is printed in their manual or oaths.
It’s going to be much harder to get anyone to believe the crying woman, sobbing her heart out about the incubator babies being left to die on the floor(2) when the image becomes the latest emoji.
===
1)
ht tps://en.wi kipedia.org/wiki/To_Tell_the_Truth
The panelists are each given a period of time to question the challengers.
Once the votes are in, the host asks, “Will the real [person’s name] please stand up?” The central character then stands, often after some brief playful feinting and false starts among all three challengers.
2) It was a critical theatrical scene played in from of the US Congress at the time.
However, in one of the current war zone versions (2024), they at least, had a line up of “real” infants in advance of the movie trailer.
lurker • February 22, 2024 7:31 PM
@Bob, Apokrif
Stuff we wish we had missed out on the entire history of software development: like when people wanted a fancy header, or math formula on a web page, and they used .gif, yuck. I’m OK with the bot being able to read almost white text on a white ground which is invisible to the 90th percentile human audience; and I’m aware of the old saw about data being code and v.v.
But my human semantics tell me if the machine is given explicit instructions, then goes and executes instructions it found some random place else， then that is a design fault that needs fixing before it is exploited for bad purpose.
Clive Robinson • February 22, 2024 7:39 PM
@ Bruce, ALL,
Re : Godël waved a pen.
“prompt injection attacks are impossible to prevent in general.”
It’s a funny old world… in that this was not just known but proved prior to what we loosely call computers and very nearly a century ago.
In essence,
“A computer can not describe it’s self.”
That is it can not tell what it’s internal state is because to do so requires the internal state to be known.
Yup and,
“Whee turtles all the way down.”
For those that don’t quite get this you need to think like an attacker. When you ask the computer to do something it goes and uses a set of instructions to perform the task. These are held in memory and the computer can not see if these instructions are real and valid or not, nor can it see if it’s starting them at the correct point or not.
To try to stop these issues all manner of tricks have been tried including Call Randomisation. None of them are 100% and they can all be broken.
One example of this issue is the RowHammer attack, which changes bits in memory hardware by shaking things up at or below the logic level of the computing stack. The furthest a CPU can reach down the computing stack is it’s address and data bus registers which is in modern computers way further up the stack, above the “Memory Managment Unit”(MMU) and the “Direct Memory Access”(DMA) controllers. Oh it’s been shown that modern MMU’s and DMA’s can with some tricks become “Ghost CPU’s” thus an attacker can in theory program…
Thus the attacker can use impossible to remove vectors to change the contents of memory that the CPU the user communicates with can not tell has been changed.
A group of people thought up the idea of “tagging memory” and went a long way with developing it. The problem the tags are effectively,
1, Checksums
2, Held in memory
3, Only checked on direct access
The RowHammer attack can change bits of memory without direct access. As is known checksums are not secure and very vulnerable to “bit flipping” attacks.
So,
“Whee turtles all the way down.”
The security model of tagging memory using current technology is broken.
Similar logic applies to all the standard computing hardware models available.
Is there a way around it? Yes but it is not something you can buy off the shelf, or in most cases build yourself. Quite some time ago I started looking into this problem and came up with a model that worked by a very ancient riddle about two doors, each guarded where one guard always told the truth and the other always lied.
Clive Robinson • February 22, 2024 9:55 PM
@ Bob,
Re : A problem as old as the hills.
“Everybody’s busy dealing with mass 2FA confusion due to cell outage lmao.”
It was a known problem to me back last century when I proposed using mobile phones as a security side channel. So if you want to call me out it’s best to know if you can do so or not.
Authentication works via a “shared secret” that obviously has to be “initially communicated” in some way. This is a “Turtles all the way down problem” and even Public Key does not solve it (it just makes the work factor large by using “One Way Functions”(OWF) as much crypto does).
As Shannon and others pointed out more than a lifetime ago information can only be communicated via a “channel”.
Further that channels come in two forms “Overt” and “Covert” and this comes about due to the very necessary need for redundancy and this allows channels to be nested within channels. As Simmons pointed out nested channels can be invisible to an observer of a carrying overt channel hence “covert channels”.
Thus as someone wishing to set up an authentication system, you have to use a channel to communicate the shared secret. So you likewise have the choice use an overt channel or a covert channel.
The problem is that transfering secrets via covert channels can fail due to correlation or other attacks.
Thus a more security robust system is to use an overt channel that is not the same as the channel that needs to be authenticated. This second channel is usually called a “side channel”.
When the two parties can meet at some point the side channel can be physically transferred hand to hand etc on a piece of paper. Most banks send out PIN numbers using the postal service that had known security issues back well into the last century hence the development of supposedly secure printing[1] but PINs being easy for criminals to get were just way to insecure for online banking as became increasingly and thus embarrassingly evident in the 1990’s.
Back in the 1990’s mobile phones were not the smart devices they are today and they did not have data connections that most users could use and nor could most phones download user selectable software. However they did have SMS.
Thus rather than post out TAN lists or One Time Passwords they could be sent by SMS to the users phone just before they needed a TAN or OTPassword. Only there was a very well known problem with SMS it was unreliable and untimely. It was a “secondary service” thus might reach the phone within eight hours or not at all. Having worked on developing data services for the old analogue mobile phones I was well aware of why SMS was failing on digital mobile phones. The issue was that the network mostly did not actually know where a mobile phone was nor did it want to due to the data overload that would cause. Whilst the network would search for the phone with a simple algorithm for a primary service of connecting a call it would only try the last known location for a secondary service. Thus the trick to making SMS viable for TAN and OTPasswords was to make a primary service initiation so the network would know the correct location of the handset for when you then sent the SMS.
As long as mobile phones were not smart and had no user downloadable apps or internet connections the security the use of SMS provided was much improved (yes phones could be cloned but banks had transaction limits and a cloned phone showed up very rapidly due to the fact mobile phone companies had been loosing millions due to “phone abroad cheap” shops using cloned phones). So banks went with the “SMS Side Channel”. But it was obvious to them that “loss of service” could happen and it was a point I made, but in comparison to the postal and parcel services it was many many times more reliable (and still is).
So yes it was known loss of service was a potential issue, but due to the way mobile providers earned revenue, unlike the post/parcel services they were very incentivised to keep networks up and running (it’s the same reason Television Networks stay up, they only earn money if the system is running near 100% availability).
However there is an issue which people need to remember, thirty years later the way mobile phone operators earn money has changed. Thus the reliability of the networks they use has changed. Most Mobile Phone companies now nolonger own the actual infrastructure, they lease it and the uptime is part of an SLA. If they are prepared to accept lower availability then the price they pay for the lease can be dramatically less. So to “remain competitive” mobile industry wide availability figures are dropping significantly.
So expect to see the number of outages increase, especially in the US where Energy Infrastructure companies like those in California and Texas are either burning the place down or having deliberate black outs or breakdowns due to near zero preventative maintainance.
It’s just one of the reasons I’ve banged on about –what we now call post C19 “supply chain”– issues for oh about as long as this blog has been around have a search for my name and “Offshoring” and “outsourcing” as well as “evolution” and the 2/3rds efficiency rule of thumb.
But by coincidence and another example of synchronicity this video got posted today,
https://youtube.com/watch?v=xFQu3Pscq7M
It explains why what are being sold as “two way radios” are in fact overly priced near useless for many applications mobile phones in a less than usefully case/form factor. Now ask me who’s being LMAO when you hear that the likes of Police and other first responder, security services, infrastructure support personnel are being moving over on to such systems…
Oh and look up the EU emergency train breaking system specification, it’s based on the use of ETSI and GSM 2G phone standards that date back a long long way and has been phased out in some regions of the EU.
Likewise have a look at many traffic light control systems they to are synchronised and have first responder switching by 2G mobile phone.
But whilst I could give you many other examples how about the control systems for sewerage, water, gas and electricity infrastructure?
Now consider if the electricity control structure goes down it will fail safe. This will cause cascade failures to spread out. Because the control network is dependent on the mobile phone network which in turn is dependent on the electricity supply, it won’t take long for the cascade fail to become “latched up” and thus electricity that all other infrastructure relies on will be down for a very long time as due to neo-con mantra the staff that maintain the network who are the only ones who can safely bring the power grid back up manually have been laid off some years ago because of “shareholder value” and senior management bonuses and benefits etc. Ask those consumers in Texas who were near or actually bankrupted by the cost of energy during the incident Texas had and tell me what they think?
[1] The German Banking system was one of the first to respond to the security failings of PIN’s and the fact that PIN’s only authenticated the channel not the transaction, thus were easily attacked by a Man In The Middle attack. So they developed TANs that were lists of “one time tokens” set out by letter via the postal service. The history of TANs and some of the security vulnerabilities can be read in this more than a decade old article,
https://www.allthingsgerman.net/blog/finance/what-are-the-different-types-of-tan/
lurker • February 22, 2024 10:45 PM
@Bob, Clive Robinson
Reasonable explanation from MSM
“The dirty secret of telecom networks these days is they are just a bunch of wires and towers connected to the cloud,” McKnight said. “Someone making a mistake, and others on their team — and their automated tools — not catching it, is quite common in cloud computing.”
‘https://www.cnn.com/2024/02/22/tech/att-cell-service-outage/index.html
Clive Robinson • February 22, 2024 11:42 PM
@ Bruce, ALL,
Re : Google Gemini “woke” claims
It is not a good day for Gemini it appears,
https://www.bbc.co.uk/news/business-68364690
“Users said the firm’s Gemini bot supplied images depicting a variety of genders and ethnicities even when doing so was historically inaccurate.”
How Google will correct this is going to be a curiosity in it’s self.
Because those that study history know that things are not the way the “historic record” of the day shows.
In the UK for instance things like parish birth, marriage and death records and census records really do not align with other contemporary records.
So how do you make a picture of a historic event?
Do you use the likes of parish records that accurately present the population, or do you make it in line with the “mores of a strata of society”, ie basically as seen by those that were self entitled and had certain views about how people were classified but controlled what we might call the media of the day…
Clive Robinson • February 23, 2024 1:30 AM
@ Bruce, ALL,
ChatGPT channels Manuel…
It appears that the last 24hours or so has not been good for big AI…
“ChatGPT has meltdown and starts sending alarming messages to users”
“AI system has started speaking nonsense, talking Spanglish without prompting, and worrying users by suggesting it is in the room with them”
Why are we all missing the fun?
The article sounds “OMG the AIs are on the march” but the comments are way lighter.
Winter • February 23, 2024 1:32 AM
@Clive
In the UK for instance things like parish birth, marriage and death records and census records really do not align with other contemporary records.
That can be extended to every aspect of society. To go back to North American settlers, Graeber and Wengrow [1] discuss at length how native Americans had a strong influence on the ideas on Humanism and “Freedom” in Europe. It looks like Thomas Moore’s *Utopia” is less of a fairytale than a summary of influential Native North American political philosophy.
Other sources showed that quite a number of women preferred living with native tribes than with the European settlers. In the early days they were better off in Native American society in almost every respect. Not quite the story depicted in the average Western.
The history of Chevalier de Saint-Georges [2] is another example of how history was bleached, this time by no lesser tyrant than Napoleon Bonaparte.
[1] The Dawn of Everything
‘https://en.m.wikipedia.org/wiki/The_Dawn_of_Everything
[2] It is not possible to describe his career in a few words. Enough to say Mozart visited him to learn from his music.
‘https://en.m.wikipedia.org/wiki/Chevalier_de_Saint-Georges
Winter • February 23, 2024 1:50 AM
@Clive
Re : Google Gemini “woke” claims
What’s in a word.
Woke can be “translated” as “anti-discrimination activism”
This aligns very well with those who are anti-woke. These anti-woke people are invariably advancing discriminatory policies. [1]
[1] ‘https://www.reuters.com/legal/government/white-fragility-lawsuit-exposes-emptiness-anti-woke-movement-2023-02-13/
Vles • February 23, 2024 4:29 AM
I don’t find this funny. I draw a lot of comfort on social media from being honest in my responses to people there when they share something. I do my best to write something from me as a person, looking for things that resonate within me.
You could call my favorite past time on social media prompt injection attacks(but with kindness+honesty+vulnerability)
Are we going to kill off all human interaction over the wire? Just tune in to the internet and let AI write dialogue between us that we are going to watch unfold like TV? Sad! I’m really sad about this thought.
What do you fellas think?
echo • February 23, 2024 9:02 AM
The claims picked up speed in right-wing circles in the US, where many big tech platforms are already facing backlash for alleged liberal bias.
This is straight out of the far right playbook. They have a loose grasp of facts and turbo boost and Ctrl-C Ctrl-V anything which fits their agenda. They also rarely to never correct anything leaving a long tail of disinformation to bounce around and be recycled forever by a pool of “low information” marks in the media and on the internet.
Under parachuted in Tory management the BBC is known to have developed a weakness for having a far right tilt so it’s no surprise the article was a bit off. The far right at all levels like to hide behind “the bants”. It’s a “we’re only joking”. No it’s not. It’s another way of pushing an agenda and/or pretending they don’t mean it really. Everyone with a brain knows it’s an attack. It’s hard to prove whether the BBC fell for it or went hunting for it so they could recycle this content unrestricted. I just know what I think.
How do you know the far right grifter Matt Walsh is a man?
Answer: Women put their hands over their glasses when he walks past.
Bob • February 23, 2024 9:39 AM
@lurker
That’s what it’s always been. Last I could tell, the outage was related to BGP. The problems with BGP are well-established and are most certainly not due to it being the newest, latest and greatest cloud thing.
Clive Robinson • February 23, 2024 10:16 AM
@ Bob, lurker, ALL,
Re : It went down town…
“Last I could tell, the outage was related to BGP.”
All we really know is when it started many started in on Cyber-attack with the subtext of,
“It woz Russia wot dun it in”
Others then blamed an X-Class Solar Storm.
But AT&T kept shtum even with the FBI,
“Locking and loading for bear”.
Then another story started leaking out that AT&T had kind of shot themselves in the foot.
So then we see some MSM dog piling in the way the CNN quote does above.
I suspect none are true and AT&T legals are in “cost minimisation mode” right now.
What I’d actually like to know is if AT&T actually run the network or if they’ve subed it out on lease back etc. And if the latter who they’ve subed it out to…
I guess we are going to have to wait on the legal people and accountants.
Bob • February 23, 2024 10:55 AM
@Clive
That’s ridiculous. It was obviously a tower upgrade so they can use the 5G to activate our COVID vax microchips.
Clive Robinson • February 23, 2024 2:12 PM
@ Bob,
Re : 5G has passed you by.
“It was obviously a tower upgrade so they can use the 5G to activate our COVID vax microchips.”
Some forms of humour don’t work well on the Internet…
“And Trumper One has promised me that Billy Bob over at Micro$haft will be given the ‘Dec37 insurrection treatment’ for his VaxenChips if I just give $100 to the Old Trumper’s fighting fund.”
Would be another example.
echo • February 23, 2024 3:42 PM
With regard to Google’s disaster and the reactive BBC roundup of far right aligned opinions:
http://lcfi.ac.uk/projects/ai-narratives-and-justice/ai-and-gender/
The ‘Gender and AI’ research stream develops feminist and queer approaches to AI that are informed by critical race theory, postcolonial/decolonial theory, Asian American/Asian diaspora studies, crip theory, and areas of justice-oriented knowledge and work.
While this project stream includes critical approaches to the state of women’s representation in the AI industry, it goes beyond ‘doing a head count’ to examine how gender as a set of relations shapes AI.
In this project stream, we ask: how does AI reanimate or exacerbate sexism, racism, transphobia, homophobia, colonialism, and other forms of discrimination and harm? How does AI codify or reinvent gendered and racialised taxonomies and categories? How does gender and sexism shape and define what ‘counts’ as AI, the parameters of AI’s use, and the kinds of technological futures we imagine? Can AI be directed towards more feminist futures, or is feminist AI an impossible project? What kinds of feminist knowledge and practices have been erased from the history of AI and intelligent machines? How are feminists hacking, adapting, or remaking technologies like AI to suit their own ends? How can technologies like AI galvanise or transform our understanding of feminism, or even gender itself?
[…]
Eleanor Drage’s monograph The Planetary Humanism of European Women’s Science Fiction: An Experience of the Impossible (Routledge 2023), argues that utopian science fiction written by European women has, since the seventeenth century, played an important role in exploring the racial and gender possibilities of the outer limits of the humanist imagination.
Drage and McInerney’s podcast,The Good Robot, asks: what is good technology? Is it even possible? And how can feminism help us work towards it? As the creators and producers of the podcast, Drage and McInerney have interviewed a wide range of leading feminist thinkers, activists and technologies about the possibilities for feminist technology.
Yet all the BBC did was barf up the loudest voices in the room…
JonKnowsNothing • February 23, 2024 4:22 PM
@All
re: Another round of theatrics coming to Congress
Aside from the normal theatrics in the US (and others), we will soon be seeing a new rendition of
this time with increased flavoring by having
The atrocities of war and deprivation of human rights and plights of the displaced, disadvantaged persons everywhere is disserviced by such theatrical arrangements.
The propaganda NUDGE is to authorize increased military spending (everywhere; on everything MILSPEC). The actual expenditures on providing food, shelter, safe havens, repatriation will be close to NIL.
The UK cannot even repatriate their own citizens, rendered stateless by government decree.
The long term storage of refugees is in doubt. Stockpiles and stockades of people waiting for Godot.
The theatrics are sure to raise tears in all viewers and great pathos will be on display.
Clive Robinson • February 24, 2024 3:48 AM
@ Jonknowsnothing, ALL,
Re : Ancient Punishments.
Not all punishments of old were physically nasty and entertaining for the masses.
The church had “Excommunication”
The crown had “Banishment”
And to lesser degrees various organisations had what we now call “Being sent to Coventry”.
In short,
“To be cast out.”
In effect it was a form of mental torture but also an economic deprivation.
That is the person was pushed out of the society they were a part of and earned their status and living in. Reduced if you like to “vagrant beggar” amongst the lowest position in any place or time. Barred even from the use of “common land”.
It was very effective because without land to work or have others work for you, you had no position in any society. “Land really was life” back then.
Now consider your comment of,
“The UK cannot even repatriate their own citizens, rendered stateless by government decree.
Firstly your comment is not accurate,
“It’s ‘will not’ not ‘can not’.”
It’s a very deliberate policy of “banishment” as a “life sentence for treason”. It’s cheap it’s simple, there is no appeal for the condemned.
Thus it appeals to those currently sitting on the majority side of the UK Parliament. Along with clearing out other stateless people to Rwanda and internment camps on military islands.
The fact the UK MSM has not put it together and raised a public outcry about this fairly obvious policy measure might also tell you something about what is going to happen in the future if it’s allowed to succeed at it’s early stages.
After all politically “doubling down on a bad idea” is often seen as a vote winner hence the nonsense that is “hard on crime” etc including their ideas they have to push via “Think of the Children” rhetoric lest people call them out for what they really are.
ResearcherZero • February 26, 2024 4:19 AM
@Clive Robinson
No one ever built a family business without a little slave labour. Small town politics is the best way to get those children working for well below the minimum wage. None of them are going to university, and they won’t be dreaming about it without “those” kinds of books. If you are going to snub the sunlight out in someone, it’s best done young.
ResearcherZero • February 26, 2024 4:44 AM
@Clive Robinson
Remote identification via fingerprint, psychological assessment, data pooling of medical details, beliefs, attachments and views. These tools will greatly aid parenting.
Now the fantastic thing about LLMs and AI tools is that they can confirm exactly what you want to hear. Hence enhancing self delusions and assumptions inline with our perceptions.
“”””
Most of our assumptions about the world are (were) completely upside down. “Until now.”
SCL provided the proof of how powerful these tools are long before the new batch was rolled out. The insight and power gained from demographics and financial data alone is astounding.
ResearcherZero • February 26, 2024 4:55 AM
With parents transfixed by the glow of burning books, they will never even notice how enmeshed Big Tech is in the education of their children, or the collection of that data.
Dick and Jane just signed on with their fingerprints…[demented laughter]
Clive Robinson • February 26, 2024 8:07 AM
@ ResearcherZero, ALL,
Re : AI and LLMs as surveillance tools on steroids.
“Now the fantastic thing about LLMs and AI tools is that they can confirm exactly what you want to hear. Hence enhancing self delusions and assumptions inline with our perceptions.”
Hence why I say,
“Bedazzle, Beguile, Bewitch, Befriend and Betray”
The Microsoft etc “Five point B plan” for growing their business which is why it’s like the old Monty Python sketch about Spam with everything.
Only it’s not funny in the slightest as thousands are at B1 and many are at B3 going into B4 but at every step Microsoft in particular have rigged a hidden path to B5…
The thing is B5 “Betray” being implicit for anyone using the Code generator as it can give away business secrets and be akin to breaching of NDAs, Client Privilege, oh and then there is “insider trading” and “industrial espionage” which are specifically legislated against, but Microsoft and Co have shown a proficiency of “buying their way out” in one way or another…
Subscribe to comments on this entry
Sidebar photo of Bruce Schneier by Joe MacInnis.
Bob • February 22, 2024 1:46 PM
Everybody’s busy dealing with mass 2FA confusion due to cell outage lmao.