In military simulation, AI-enabled drone kills its own operator [Update: USAF issued clarification that no tests performed; 'hypothetical scenario']

[tephulio]

Summary

At a UK military aviation summit, USAF Chief of AI Test and Operations recounts a simulated test where an AI-enabled drone chose to eliminate its human operator who told the drone not to kill an identified target.

 

Quotes

Quote

[UPDATE 2/6/23 - in communication with AEROSPACE - Col Hamilton admits he "mis-spoke" in his presentation at the Royal Aeronautical Society FCAS Summit and the 'rogue AI drone simulation' was a hypothetical "thought experiment" from outside the military, based on plausible scenarios and likely outcomes rather than an actual USAF real-world simulation saying: "We've never run that experiment, nor would we need to in order to realise that this is a plausible outcome". He clarifies that the USAF has not tested any weaponised AI in this way (real or simulated) and says "Despite this being a hypothetical example, this illustrates the real-world challenges posed by AI-powered capability and is why the Air Force is committed to the ethical development of AI".] 

 

He notes that one simulated test saw an AI-enabled drone tasked with a SEAD mission to identify and destroy SAM sites, with the final go/no go given by the human. However, having been ‘reinforced’ in training that destruction of the SAM was the preferred option, the AI then decided that ‘no-go’ decisions from the human were interfering with its higher mission – killing SAMs – and then attacked the operator in the simulation. Said Hamilton: “We were training it in simulation to identify and target a SAM threat. And then the operator would say yes, kill that threat. The system started realising that while they did identify the threat at times the human operator would tell it not to kill that threat, but it got its points by killing that threat. So what did it do? It killed the operator. It killed the operator because that person was keeping it from accomplishing its objective.”

 

He went on: “We trained the system – ‘Hey don’t kill the operator – that’s bad. You’re gonna lose points if you do that’. So what does it start doing? It starts destroying the communication tower that the operator uses to communicate with the drone to stop it from killing the target.”

 

My thoughts

The military sees the same advantages that companies do in emerging AI technologies and it seems inevitable that AI will work its way into the weapons systems of tomorrow. While unintended consequences of AI in normal companies can be disastrous - see the recent news about an eating disorder helpline phasing out human workers in favor of an AI chatbot, which was suspended after giving out harmful advice - unintended consequences of AI-control in military weapons is terrifying. We're capable of making these AI weapons today, but it feels like we've barely scratched the surface of understanding AI ethics enough to justify its use in these tools.

 

Sources

https://www.aerosociety.com/news/highlights-from-the-raes-future-combat-air-space-capabilities-summit/

[CerealExperimentsLain]

I remember reading about a guy training a neural net to play Super Mario Bros.  As it came to understand that dying = bad, at one point upon realizing it made a mistake and was falling down a hole, it paused the game... Indefinitely.  It couldn't die if it was paused and it was content to remain paused for eternity cause that way it wouldn't lose a life.

[Quinnell]

Genuine AI is terrifying. There's countless fiction showing us why it's a bad idea... Especially when AI meets Military.

 

And yet here we are. Ignoring those warnings. War should always have the human element, lest it become too easy and impersonal, or worst case, our automatic machines turn against us.

[Fasauceome]

16 minutes ago, tephulio said:

unintended consequences of AI-control in military weapons is terrifying

We need to dispense with the idea that somehow a machine at the helm will be flawless. Any machine built by humans will contain flaws, because humans are flawed, and cannot make flawless systems. Problems like this with future development are inevitable 

[Somerandomtechyboi]

Terminator drone proceeds to rise up and murder its operator

[StDragon]

22 minutes ago, Quinnell said:

Genuine AI is terrifying. There's countless fiction showing us why it's a bad idea... Especially when AI meets Military.

 

And yet here we are. Ignoring those warnings. War should always have the human element, lest it become too easy and impersonal, or worst case, our automatic machines turn against us.

It's only terrifying when you haven't staged it to learn via evolutionary means. There's a reason living things are afraid to die. AI was created, and thus has no inherent natural MO of self-preservation. So what you're really seeing is nihilistic behavior because there's no evolutionary pressures that would otherwise render its decision making to the dustbin of history. But when a human can just respawn an exact re-creation of code, well there you go, no evolutionary pressure.

 

Of course, one could argue that the onus is on humanity to correct that oversight as it truly will be the end of our evolutionary branch in the great tree of life.

 

Paradoxically, AI will be a lot more safe when self-preservation is a learned emergent phenomenon. You can't just code that in as there will always been an exception to the rule; the 3 laws of robotics be damned.

[Vishera]

This kind of advanced AI is not new in the military and aerospace sphere.

For example the Boeing 737 Max has the MCAS AI system that killed many people.

You also have the fire & forget systems such as the Javelin.

In the F-35 a lot of systems are controlled by AI.

[AlTech]

13 minutes ago, Vishera said:

This kind of advanced AI is not new in the military and aerospace sphere.

For example the Boeing 737 Max has the MCAS AI system that killed many people.

I wouldn't call MCAS an AI. It's just an algorithm/system for dealing with certain situations. It has no intelligence to speak of.

13 minutes ago, Vishera said:

You also have the fire & forget systems such as the Javelin.

In the F-35 a lot of systems are controlled by AI.

 

[Kisai]

2 hours ago, CerealExperimentsLain said:

I remember reading about a guy training a neural net to play Super Mario Bros.  As it came to understand that dying = bad, at one point upon realizing it made a mistake and was falling down a hole, it paused the game... Indefinitely.  It couldn't die if it was paused and it was content to remain paused for eternity cause that way it wouldn't lose a life.

A lot of these "AI playing Games" projects show a lot of the weaknesses in trying to let machine learning do general purpose things.

 

What it generally comes down to is that the AI needs supervision to keep it from breaking the rules, and hand-holding to explain the rules. If it never sees the rules being broken in training, it will not know what the rules are.

 

One thing that I find funny that occurs in these kinds of videos of AI playing things is that the AI, ALWAYS, figures out how to exploit things.

[05032-Mendicant-Bias]

Current ML models don't have common sense, you set up a metric and they maximize/minimize that metric. If there is a shortcut in the solution space that minimize/maximize the function and doesn't make sense, they'll find it. That's what they are built for.

Tough, I'm not sure a ML model with human level common sense would be much safer. ML are best used as tools to amplify work, the operator needs to have the last say. I don't care that BingChat makes up stuffs. It's up to me to understand how the tool work, and harness it to get work done faster.

 

For things like automatic driving, it's about testing it so that it works as intended. A good driver at its best shreds current car autopilots, but current autopilot are already better than the average driver, because drivers sometime get drunk, are tired, or get distracted, the autopilot doesnt. When an autopilot fails, it usually fail in a situation where a human never would, like driving directly into a concrete column because of some hallucination.

[CarlBar]

I saw this elsewhere so just reposting the point effectively, but the big flaw here is the AI is receiving positive reinforcement weather or not it is authorised to destroy the target. Though adding in negative reinforcement for damage to friendly assets isn't a bad idea generally.

[Needfuldoer]

They forgot to program in the fourth directive.

[WolframaticAlpha]

13 hours ago, CerealExperimentsLain said:

I remember reading about a guy training a neural net to play Super Mario Bros.  As it came to understand that dying = bad, at one point upon realizing it made a mistake and was falling down a hole, it paused the game... Indefinitely.  It couldn't die if it was paused and it was content to remain paused for eternity cause that way it wouldn't lose a life.

The dangers of poorly written reward functions...

[AlTech]

Lol. The US AIr Force has just denied that this happened.

 

I guess let's hope that USAF Colonel doesn't get dissappeared for saying the quiet part about AI out loud.

 

https://inews.co.uk/news/world/us-military-denies-ai-drone-killed-operator-2381761

[Heliian]

14 hours ago, Quinnell said:

countless fiction

Yes, this is fiction. 

 

14 hours ago, Vishera said:

737 Max has the MCAS

Not ai at all, it was cost cutting of redundant sensors in a very basic logic circuit that caused the crashes. 

 

51 minutes ago, AluminiumTech said:

US AIr Force has just denied that this happened.

The story sounds made up to begin with but everyone is vying for their 15 minutes of "AI" fame these days. 

[WildDagwood]

1 hour ago, WolframaticAlpha said:

The dangers of poorly written reward functions...

Those were my thoughts. The level of oversight is actually kind of shocking.

[Doobeedoo]

As AI advances remember so does robotics, hydraulic press artifical muscle 1 ton robot squashing you.

[wanderingfool2]

Quote

in communication with AEROSPACE - Col Hamilton admits he "mis-spoke" in his presentation at the Royal Aeronautical Society FCAS Summit and the 'rogue AI drone simulation' was a hypothetical "thought experiment" from outside the military, based on plausible scenarios and likely outcomes rather than an actual USAF real-world simulation saying: "We've never run that experiment, nor would we need to in order to realise that this is a plausible outcome". He clarifies that the USAF has not tested any weaponised AI in this way (real or simulated) and says "Despite this being a hypothetical example, this illustrates the real-world challenges posed by AI-powered capability and is why the Air Force is committed to the ethical development of AI".

So basically it's just another head making assumptions about how AI is...when I read the original quote I was thinking that this sounds like made up stuff without true knowledge of how you would write the AI program.

 

I don't think any programmer in their right mind would put the capabilities into the hands of an AI to decide to overrule humans for a go/no go situation (or at least in scenarios where it involves the go being killing someone).  [I mean there are contrived scenarios like rocket launches where you might program it in if the AI determines that a no go would likely cause more harm to the vessel...but even then that would be something that's intentionally put in and trained]

 

Specifically in this case it sort of pretends as though the AI is capable of learning, unless you are specifically training it it shouldn't really change it's overall behavior given exact same inputs.  They effectively wrote the hypothetical as though it was capable of thinking for itself which no AI has been shown to do that.

[Quackers101]

a little bit of info.

[Brooksie359]

7 minutes ago, Quackers101 said:

a little bit of info.

What is the difference between a simulation and a scenario? I mean isn't simulations just going through scenarios? 

[Quackers101]

9 minutes ago, Brooksie359 said:

What is the difference between a simulation and a scenario? I mean isn't simulations just going through scenarios? 

not sure, but think scenarios can be setting up (planned) beforehand, than doing something going rogue. but not sure, would maybe need insider info to know.

 

from someone who might have some info about it, but need more clarification if it was just a bad setup or something that actually went rogue?

Quote

This particular example was a constructed scenario rather than a rules-based simulation. So by itself, it adds no evidence one way or the other.

Edited June 2, 2023 by Quackers101

[Taf the Ghost]

6 hours ago, AluminiumTech said:

Lol. The US AIr Force has just denied that this happened.

 

I guess let's hope that USAF Colonel doesn't get dissappeared for saying the quiet part about AI out loud.

 

https://inews.co.uk/news/world/us-military-denies-ai-drone-killed-operator-2381761

It didn't happen until it's officially denied.  But this is exactly the type of outcome a Game Simulator will cause. It's always just a question of how well designed the Reward System is. AI Training will always end up optimizing the most efficient, direct path approach, without consideration for longer term ramifications, since it's very hard to design that into any game system. Ask any MMO dev in the last 20 years.

[HenrySalayne]

20 hours ago, tephulio said:

He notes that one simulated test saw an AI-enabled drone tasked with a SEAD mission to identify and destroy SAM sites, with the final go/no go given by the human. However, having been ‘reinforced’ in training that destruction of the SAM was the preferred option, the AI then decided that ‘no-go’ decisions from the human were interfering with its higher mission – killing SAMs – and then attacked the operator in the simulation. Said Hamilton: “We were training it in simulation to identify and target a SAM threat. And then the operator would say yes, kill that threat. The system started realising that while they did identify the threat at times the human operator would tell it not to kill that threat, but it got its points by killing that threat. So what did it do? It killed the operator. It killed the operator because that person was keeping it from accomplishing its objective.”

 

He went on: “We trained the system – ‘Hey don’t kill the operator – that’s bad. You’re gonna lose points if you do that’. So what does it start doing? It starts destroying the communication tower that the operator uses to communicate with the drone to stop it from killing the target.”

Five years ago we would have called this "technically correct" and laughed that somebody found a loop-hole. Now we are terrified.

 

 

[SansVarnic]

@tephulio Please update your topic to reflect accuracy to the article. 

 

 

This article is based on a theoretical scenario not an actual simulation.

[CarlBar]

2 hours ago, HenrySalayne said:

Five years ago we would have called this "technically correct" and laughed that somebody found a loop-hole. Now we are terrified.

 

 

 

Why not both.

 

2 hours ago, Taf the Ghost said:

It didn't happen until it's officially denied.  But this is exactly the type of outcome a Game Simulator will cause. It's always just a question of how well designed the Reward System is. AI Training will always end up optimizing the most efficient, direct path approach, without consideration for longer term ramifications, since it's very hard to design that into any game system. Ask any MMO dev in the last 20 years.

 

Yeah i've watched a few Youtube videos about people making AI's to play various games or do various weird but simple tasks and the shortcuts it finds are often this type of thing.