In military simulation, AI-enabled drone kills its own operator [Update: USAF issued clarification that no tests performed; 'hypothetical scenario']

[Quackers101]

business insider might have recently gotten an update

https://www.businessinsider.com/ai-powered-drone-tried-killing-its-operator-in-military-simulation-2023-6?r=US&IR=T

[wanderingfool2]

12 hours ago, Taf the Ghost said:

It's always just a question of how well designed the Reward System is. AI Training will always end up optimizing the most efficient, direct path approach, without consideration for longer term ramifications, since it's very hard to design that into any game system.

But it's also important to note that all the output from the NN will ultimately be run against a piece of code that takes the results and either feeds it into another NN or decides against something.  As an example, a Go/No go, the NN wouldn't "have" access to the API to fire the missile...which would mean it can never override something it never really had true access to.

 

nn_fire_on_target = collectresults.run(parameters);
if(nn_fire_on_target && human_authorization)
	fire();

No matter what happens, the NN will never be able to fire without a human giving final authorization; which is why I really hate when the military or news acts as though a programmer would give it that ability to override (even some missile systems require 2 humans to authorize)

 

13 hours ago, Brooksie359 said:

What is the difference between a simulation and a scenario? I mean isn't simulations just going through scenarios? 

In general a scenario is effectively a story where you setup guidelines of what happened (and then maybe setting objectives or planning how one would deal with it)

 

While a simulation is testing to see whether or not a scenario would play out or how it actually plays out.  You effectively create scenarios so you know what to do in certain situations or devise mitigations

 

So as an example, you can have a scenario "Canada invades the US using a joke so deadly it kills anyone who hears it".  To have a simulation you would have mock Canadians shouting something on a microphone and if the US people can hear it they report they died.  Then the simulation tells you things like the distance at which people would be affected and that $2 headphones blocks out the jokes.

 

In this case the "simulation" that actually wasn't a simulation was that they were 'training with the AI drones to fight targets', where the scenario in that case would be enemy targets xyz are in this area find them and eliminate them without hurting civilians.  The issue with it being claimed as a simulation is that a drone would have acted outside of the scope in which it was meant to act, which would have been very concerning if it did happen during a simulation (even if it was filled with dummy rounds)...but in this case it was just the scenario.  So it's essentially say assume this happened.

[Isuck Assimov]

And this is why autonomous transport vehicles are gonna kill a lot of people in the future as long as killing a pedestrian results in lower losses than avoiding the pedestrian and risking loss of the transported goods