The AI powered deep fake audio scams have arrived

[TheFutureIsStupid]

Summary

AI powered audio deep fakes are now being used to target people pretending to be family in destress, in need of money. And unfortunately its working

 

Quotes

Quote

"Although impostor scams come in many forms, they essentially work the same way: a scammer impersonates someone trustworthy — a child, lover or friend — and convinces the victim to send them money because they’re in distress. But artificially generated voice technology is making the ruse more convincing. Victims report reacting with visceral horror when hearing loved ones in danger."... 

[Benjamin Perkin]'s voice-cloning nightmare started when his parents received a phone call from an alleged lawyer, saying their son had killed a U.S. diplomat in a car accident. Perkin was in jail and needed money for legal fees. 

The lawyer put Perkin, 39, on the phone, who said he loved them, appreciated them and needed the money. A few hours later, the lawyer called Perkin’s parents again, saying their son needed $21,000 ($15,449) before a court date later that day.

Perkin’s parents later told him the call seemed unusual, but they couldn’t shake the feeling they’d really talked to their son.

 

The voice sounded “close enough for my parents to truly believe they did speak with me,” he said. In their state of panic, they rushed to several banks to get cash and sent the lawyer the money through a bitcoin terminal."

 

 

My thoughts

Anyone following developments in the film industry along with developments in deep fake technology probably knew this was coming, but I didnt think it'd be here this soon. From the Washington Post article, it seems all that is needed is about 30 seconds worth of voice sample that can be taken from just about anywhere (tiktok, youtube, recorded professional conference, hacked/leaked video call recordings etc), and to top if off if the attacker knows the voice's phone number they can spoof it and make it look legitimate. Best case scenario seems to be it doesnt get any worse (unlikely), worst case scenario its going to ruin phone calls. One has to assume deepfake video / facetime calls are also on the horizon. 

 

On the up side, this probably means they can now recreate Majel Barrett's Star Trek "Computer" voice without much trouble now... 

 

Sources

https://www.washingtonpost.com/technology/2023/03/05/ai-voice-scam/

[Somerandomtechyboi]

33 minutes ago, TheFutureIsStupid said:

Although impostor scams

[wanderingfool2]

This is the unfortunate future we face, and not really much that we can do to prevent it.  The one presented is a pretty tame one at the moment, but I could see this getting a lot worse very quickly (one where the "red flags" of a scam are a lot more subtle).

 

There was the "hostage" scam that was pulled off, not too long ago iirc.  They would target people living here with rich family overseas, they would "distract" or steal the person's phone and tell the family overseas something along the lines of their family members had been kidnapped and if they didn't pay the ransom they'd be killed.  (At which point they would try contacting the family member, and get no response...and told not to contact the policy).  Imagine that scam, except faking the persons plea for help.

 

I really do hope new laws come into effect that setup punishments for impersonations, otherwise things could get out of control really quickly.  Like imagine knowing an owner of a business who is wealthy, and emulating their voice to send a wire transfer or bitcoin transfer.  While there should be internal processes in place, if the owner of a company phones and is mad/demanding of something to be done quickly it can be quite convincing (especially if it's paired with spoofing their number).

[StDragon]

Remember, the future is "Password-less" (combined with biometrics no doubt too).

AI blurred that line between fake and what is real. Never assume anyone remote is whom they claim to be without proof. Voiceprints and written signatures don't count.

[TheFutureIsStupid]

22 minutes ago, StDragon said:

Remember, the future is "Password-less" (combined with biometrics no doubt too).

AI blurred that line between fake and what is real. Never assume anyone remote is whom they claim to be without proof. Voiceprints and written signatures don't count.

unfortunately without massive government efforts and international cooperation, there is a not too distant future where we're all going to have to have word or colors of the day/week that we physically mail to relatives that contain key words for an emergency. And those words will never be able to be digitized lest you have to assume they're compromised. A secure future will be an OPSEC future.

 

I have sever doubts the majority of people will have the patience for such procedures. 

 

I also worry that biometrics will be steal-able in digital form (though I suppose properly designed hardware would help). 

 

I feel like this would all be a little more tolerable if we at least had flying cars by now (helicopters dont count)

[Kisai]

4 hours ago, TheFutureIsStupid said:

unfortunately without massive government efforts and international cooperation, there is a not too distant future where we're all going to have to have word or colors of the day/week that we physically mail to relatives that contain key words for an emergency. And those words will never be able to be digitized lest you have to assume they're compromised. A secure future will be an OPSEC future.

 

Nope. Passkeys and physical biometrics (particularly those that can't be duplicated without the person voluntarily giving it, and can't be created from a photo eg a tongue or teeth print.)

 

Deep fake audio scams however are really not clever. They'll fool people who don't understand the tech exists, and don't really know their loved ones that well.

 

https://m.imdb.com/video/vi118145817/

Like the Simpsons hit this on the nose.

 

All the AI is doing is making it more convincing to someone who might actually believe it. The scam itself has been around for decades, and was more convincing during a time when telephones didn't sound as good.

 

Never volunteer information over the phone. If you volunteer someone's name, they'll use that to make it more convincing.

 

[Brooksie359]

10 hours ago, wanderingfool2 said:

This is the unfortunate future we face, and not really much that we can do to prevent it.  The one presented is a pretty tame one at the moment, but I could see this getting a lot worse very quickly (one where the "red flags" of a scam are a lot more subtle).

 

There was the "hostage" scam that was pulled off, not too long ago iirc.  They would target people living here with rich family overseas, they would "distract" or steal the person's phone and tell the family overseas something along the lines of their family members had been kidnapped and if they didn't pay the ransom they'd be killed.  (At which point they would try contacting the family member, and get no response...and told not to contact the policy).  Imagine that scam, except faking the persons plea for help.

 

I really do hope new laws come into effect that setup punishments for impersonations, otherwise things could get out of control really quickly.  Like imagine knowing an owner of a business who is wealthy, and emulating their voice to send a wire transfer or bitcoin transfer.  While there should be internal processes in place, if the owner of a company phones and is mad/demanding of something to be done quickly it can be quite convincing (especially if it's paired with spoofing their number).

Yeah as soon as I heard the lawyer asking for money in bitcoin I would 100% assume it was a scam. No actual lawyer would do business this way. Also I feel like in the future if you aren't sure just ask them a question that an impersonator wouldn't know. I feel like that would easily double check that who you are speaking to is really the person you know. 

[HenrySalayne]

12 hours ago, wanderingfool2 said:

This is the unfortunate future we face, and not really much that we can do to prevent it. The one presented is a pretty tame one at the moment, but I could see this getting a lot worse very quickly (one where the "red flags" of a scam are a lot more subtle).

[...]

I really do hope new laws come into effect that setup punishments for impersonations, otherwise things could get out of control really quickly.  Like imagine knowing an owner of a business who is wealthy, and emulating their voice to send a wire transfer or bitcoin transfer.  While there should be internal processes in place, if the owner of a company phones and is mad/demanding of something to be done quickly it can be quite convincing (especially if it's paired with spoofing their number).

A future that would have been preventable. Why is it legal that pictures of your face or samples of your voice are collected and processed without your consent? "You have a profile picture on Facebook? You are now in our face recognition data base, get over it."

And everybody laughed about GDPR....

[Doobeedoo]

No surprise. First imagery then audio then complete made up video.

[Zodiark1593]

3 hours ago, HenrySalayne said:

A future that would have been preventable. Why is it legal that pictures of your face or samples of your voice are collected and processed without your consent? "You have a profile picture on Facebook? You are now in our face recognition data base, get over it."

And everybody laughed about GDPR....

Given the tools available to obfuscate what data is actually used (encryption is one means, but you may also put images through an algorithm, similar to a hash, that generates an output that describes the photo, without having to reference the original, and do so without leaving evidence), consent was never a factor.

Laws only provide the consequence, but if they are impractical, or impossible to enforce (and given the complexity of the tech at hand, investigations are sure to be an expensive endeavor), they’re not worth the paper they’re printed on. 
 

Always assume that anything uploaded to a server you don’t directly control, can and will be used by third parties, regardless of laws or jurisdiction at play. If you don’t want a photo to be used by companies or other outsiders, the only sure protection is to not upload it. 

[HenrySalayne]

13 minutes ago, Zodiark1593 said:

Laws only provide the consequence, but if they are impractical, or impossible to enforce (and given the complexity of the tech at hand, investigations are sure to be an expensive endeavor), they’re not worth the paper they’re printed on. 

Are you saying the GDPR is not worth the paper it is printed on?

15 minutes ago, Zodiark1593 said:

Always assume that anything uploaded to a server you don’t directly control, can and will be used by third parties, regardless of laws or jurisdiction at play. If you don’t want a photo to be used by companies or other outsiders, the only sure protection is to not upload it. 

That sounds an awful lot like "if you don't want to get robbed, don't walk around in public". It's obviously your own fault, if entities abuse your data...

[BuckGup]

17 hours ago, StDragon said:

Remember, the future is "Password-less" (combined with biometrics no doubt too).

AI blurred that line between fake and what is real. Never assume anyone remote is whom they claim to be without proof. Voiceprints and written signatures don't count.

This goes for law enforcement too. They assume all data they collect has intent or was even made by a human

[CarlBar]

6 hours ago, HenrySalayne said:

Are you saying the GDPR is not worth the paper it is printed on?

That sounds an awful lot like "if you don't want to get robbed, don't walk around in public". It's obviously your own fault, if entities abuse your data...

 

The issue is the way the internet fundamentally works and what that means for enforcement. The internet is mostly "public by default" in nature. Whatever you put up on it is going to be available to billions if they want to look at it. Things like GDPR can affect companies operating in a certain area, but they do nothing to stop private entities from gathering the data, especially if they don't care about the law in the first place. For that you either need a system that makes it impossible for private entities to gather the data, or you need to make gathering it such a nuisance criminals aren't willing to go through the effort. But neither is really something that applies to the internets public side.

 

And this goes double in a situation like is being described here, even if there is some law in place that makes it illegal for them to access the data needed to produce a solid voice match, (they're going to need somthing to make the algorithm spit out somthing useful), they're unlikely to obey it unless breaking ti is a sufficiently large nuisance for them.

 

Now you might say stronger laws on companies might have prevented the underlying algorithms being created, but that relies on the idea that very few people would ever willing provide data for such purposes. The best a law like that might have done is slow them down, they'd have appeared eventually short of a complete ban on such things, but that would need to be global and thats just not practical at all.

[wanderingfool2]

7 hours ago, HenrySalayne said:

Are you saying the GDPR is not worth the paper it is printed on?

I don't think it's really that great honestly.  It has also made sites a lot more annoying with those pop-ups, to the point that most people just click through them...or in cases like myself where I have no-script sometimes I have to enable it to access a site.

 

One example of where some of the privacy laws I think go too far is looking at WhiteCastle.  They could face a $17 billion fine for using fingerprint readers to access their pay stubs (instead of passwords)...but lets say it was for something like accessing a POS machine.  Under the law if they didn't get correct written consent it would literally could be a penalty so severe it would put any company out of business.

 

Then you have the concept of less targeted ads which also means they have to make their money some other way (so either more ads or getting into subscription models).

 

 

[HenrySalayne]

7 hours ago, wanderingfool2 said:

I don't think it's really that great honestly.  It has also made sites a lot more annoying with those pop-ups, to the point that most people just click through them...or in cases like myself where I have no-script sometimes I have to enable it to access a site.

Ehh, you know that's not the regulation, right?

Website operators are butt-hurt they cannot use targeted ads any more (which pays less) and they make the most atrocious interface so people like you are annoyed about GDPR. These pop-ups are literally designed to agitate you and make you hate GDPR, they don't have to be like this.

7 hours ago, wanderingfool2 said:

One example of where some of the privacy laws I think go too far is looking at WhiteCastle.  They could face a $17 billion fine for using fingerprint readers to access their pay stubs (instead of passwords)...but lets say it was for something like accessing a POS machine.  Under the law if they didn't get correct written consent it would literally could be a penalty so severe it would put any company out of business.

Clearview AI and al these other companies abusing public data would be gone. What a terrible world to live in...

Remember the lawyer who was thrown out of Madison Square garden because a picture of her was on a website the owner didn't like - illegal with GDPR.

There is no honest business which would struggle with GDPR. So many companies where whining about it and what happened? They all found a way to operate under it.

7 hours ago, wanderingfool2 said:

Then you have the concept of less targeted ads which also means they have to make their money some other way (so either more ads or getting into subscription models).

Utter hypocrisy. Apple blocks targeted ads (to favour their own targeted ads) and the world is cheering, the EU regulates targeted ads and everybody just loses their mind.

[LAwLz]

On 3/7/2023 at 12:50 AM, wanderingfool2 said:

I really do hope new laws come into effect that setup punishments for impersonations, otherwise things could get out of control really quickly.  Like imagine knowing an owner of a business who is wealthy, and emulating their voice to send a wire transfer or bitcoin transfer.  While there should be internal processes in place, if the owner of a company phones and is mad/demanding of something to be done quickly it can be quite convincing (especially if it's paired with spoofing their number).

We don't need new laws because our current laws already makes this illegal.

Sadly, the thing about criminals is that they don't follow the law.

[HenrySalayne]

12 hours ago, CarlBar said:

And this goes double in a situation like is being described here, even if there is some law in place that makes it illegal for them to access the data needed to produce a solid voice match, (they're going to need somthing to make the algorithm spit out somthing useful), they're unlikely to obey it unless breaking ti is a sufficiently large nuisance for them.

Now you might say stronger laws on companies might have prevented the underlying algorithms being created, but that relies on the idea that very few people would ever willing provide data for such purposes. The best a law like that might have done is slow them down, they'd have appeared eventually short of a complete ban on such things, but that would need to be global and thats just not practical at all.

No, that's not what I'm saying.

Sometimes the question is when it becomes illegal. Scamming is illegal. Building tools and databases which are used for scamming is not illegal. Using bombs is illegal but building bombs is also illegal. We all know laws cannot stop things from happening, but outlawing precursive steps can reduce the likeliness. Under GDPR building the database would have been illegal and that's basically a side-effect of privacy regulation. Criminals probably don't care much about this, but the judiciary has another lever to pull.

[Spotty]

Seems like a more advanced and targeted version of the "Hi mum" scam where scammers send sms messages pretending to be family members saying their phone was lost or stolen and this is their new phone number before asking for money. Same principal of impersonating a family member and then asking for money. Only difference here is that they're use voice instead of text messages and it must be a much more targeted attack if they're getting audio of the family member they're impersonating.

 

It'll be interesting to know how they're getting not only audio of the person they are impersonating to train the model on but also the contact details of their family member they are targeting in the scam. I wonder if the impersonated person is getting caught up in a different scam themselves and unknowingly handing over information like their own and family members details to the scammers which the scammers are then using to scam their associates. Maybe it's something like a fake job offer scam where they are offered a job and the scammers request family contact details for emergency contacts and the scammers are grabbing an audio sample through a fake job interview video call?

[HenrySalayne]

1 hour ago, Spotty said:

It'll be interesting to know how they're getting not only audio of the person they are impersonating to train the model on but also the contact details of their family member they are targeting in the scam. I wonder if the impersonated person is getting caught up in a different scam themselves and unknowingly handing over information like their own and family members details to the scammers which the scammers are then using to scam their associates. Maybe it's something like a fake job offer scam where they are offered a job and the scammers request family contact details for emergency contacts and the scammers are grabbing an audio sample through a fake job interview video call?

That's an interesting idea but probably too sophisticated. In this case it was most likely all publicly available information. The message they received was probably pre-recorded ( I can't find any clue that they actually interacted with the voice pretending to be their son) and they never called their son to check.

 

You just need to find a Facebook post with the parents interacting with their son ("Best son in the world trimming the hedges"), you may find the phone number on the parents profile, you may find the voice of the son in some video he posted and you are good to go. You can stalk some people so easily on the internet, it's astonishing.

[Kisai]

2 hours ago, Spotty said:

 

 

It'll be interesting to know how they're getting not only audio of the person they are impersonating to train the model on

The voice mail greeting.

2 hours ago, Spotty said:

but also the contact details of their family member they are targeting in the scam.

The address book on the phone.

2 hours ago, Spotty said:

I wonder if the impersonated person is getting caught up in a different scam themselves and unknowingly handing over information like their own and family members details to the scammers which the scammers are then using to scam their associates. Maybe it's something like a fake job offer scam where they are offered a job and the scammers request family contact details for emergency contacts and the scammers are grabbing an audio sample through a fake job interview video call?

Like there's more than a few ways. People who post tiktoks and youtubes are prime targets since both services tie your phone number to an account.

 

Now if they don't have the person's phone, of course that's a bit harder to pull off. But this is another reason to ensure old phones are wiped/remote-wiped.

 

[ouroesa]

"they rushed to several banks to get cash and sent the lawyer the money through a bitcoin terminal" People need to know (be educated) that anyone asking for money via crypto or gift cards is 99.9% likely to be a scam. This really should be general knowledge unfortunatelly the majority of people out there is not aware of this simple fact. 

Fun fact, in Australia, they've scammed businesses to pay money into a different bank account and yet the police are unable to find these people or freeze the account. They 'are aware' of bad actors using this account but simply let it go. No wonder Asutralia is one of the top countries for scammers.

[ouroesa]

On 3/7/2023 at 10:20 AM, wanderingfool2 said:

I really do hope new laws come into effect that setup punishments for impersonations, otherwise things could get out of control really quickly.  Like imagine knowing an owner of a business who is wealthy, and emulating their voice to send a wire transfer or bitcoin transfer.  While there should be internal processes in place, if the owner of a company phones and is mad/demanding of something to be done quickly it can be quite convincing (especially if it's paired with spoofing their number).

There are already multiple laws in place in most countries that prohibit impersonation of another person. For instance, in the US, Section 53a-130 criminalizes impersonation, while in Australia, the CRIMINAL CODE 1899 - SECT 514 deals with impersonation, and in the UK, the Fraud Act 2006 (FA 2006) addresses impersonation, among others. The severity of the penalties imposed for impersonation varies based on who is being impersonated. In the US, for example, impersonating a US citizen carries even more severe consequences. However, before prosecuting those who impersonate others, law enforcement agencies need to catch them first. Unfortunately, many law enforcement agencies still rely on outdated methods, which can hinder their ability to apprehend impersonators.

[Kisai]

2 hours ago, ouroesa said:

The severity of the penalties imposed for impersonation varies based on who is being impersonated. In the US, for example, impersonating a US citizen carries even more severe consequences.

This is where you have to draw the correct line, because "impersonation in order to defame/libel/satire" and "impersonation in order gain access to information/money/etc that only that person would be entitled to receive" are extremely different.

 

If someone makes a deepfake of someone they don't like, but all they do is meme on it on twitter, that's really of no consequence to anyone who actually knows that person, but could cause some reputational damage if it's put in front of the wrong audience, just like tweets pretending to be a company. Ultimately satire is immune from libel because it's not meant to be believed, where as libel is trying to pass a misinformation as having come from the source itself.

 

Unfortunately a lot of laws were written during a time when credit cards weren't even a thing, so the idea that someone could create a convincing impression of another, and transfer their assets electronically out of their hands into a scammer was not something that was in mind when the laws were written. They had in mind "Catch Me If You Can", where someone is passing fake checks off.

 

[StDragon]

15 hours ago, LAwLz said:

We don't need new laws because our current laws already makes this illegal.

Sadly, the thing about criminals is that they don't follow the law.

Emphasis mine.

Legislature passes more laws (often compounding or overlapping existing ones) for that feel-good measure to placate the electorate. But laws are absolutely worthless unless there's consequences for breaking them.

I'm a simple guy. Breaking a few kneecaps or a solid Singapore style caning often suffices.

[HenrySalayne]

5 hours ago, Kisai said:

If someone makes a deepfake of someone they don't like, but all they do is meme on it on twitter, that's really of no consequence to anyone who actually knows that person, but could cause some reputational damage if it's put in front of the wrong audience, just like tweets pretending to be a company. Ultimately satire is immune from libel because it's not meant to be believed, where as libel is trying to pass a misinformation as having come from the source itself.

With deepfakes you will have a hard time passing it as satire and not defamation (especially if it's demeaning the person).