Block's human-sounding Dessa needs a fraud plan before a launch date

Is the most impressive thing about a voice AI that you can't tell it's a machine? For a payments company, that should be the most uncomfortable question in the room.

Block, Jack Dorsey's company behind Square and Cash App, has unveiled Dessa, a speech model that, as The Verge reports, it says was trained from scratch on nothing but human conversation. According to CoinDesk, it listens while it speaks, so it handles interruptions, laughter and shifts in tone. It is compact enough to run locally on a MacBook Pro with an M4 Max or better. For now it is a research preview, with no public release date, and the audio samples are company demos rather than independent tests.

The reaction split exactly where you'd expect. Some listeners called it sci-fi. Filmmaker Justine Bateman asked why AI voices need to sound human at all, writing, "So you can trick people into lowering their guard." Journalist Shiv Aroor wrote: "Scam calls about to attain pandemic level." We think the skeptics are asking the better question.

Naturalness is the feature and the attack surface

Block's pitch is that today's voice AI converts speech to text and throws away what matters: sarcasm, a sigh, a cracking voice. That's a fair critique of stilted assistants. But the properties that make a conversation feel real are the same ones a phone scammer needs: a pause that sounds like hesitation, a laugh that sounds spontaneous, a voice that reacts to what you say in real time.

The local-execution detail deserves more attention than it's getting. Block presents it as a feature. We'd note that a model running on a laptop is a model with no server on the other end to rate-limit, log or shut down. That's an opinion about incentives, not a claim about what Block plans, and the company hasn't said how it would handle misuse. But "runs on a high-end laptop" and "trust the voice on the phone" are not comfortable neighbors.

The safety argument is real, but it points the wrong way

Block gives three reasons for building Dessa: curiosity, better human-AI understanding, and safety. The safety case is that AI is nearly deaf to emotion, and a system that missed someone in serious pain would be a problem. We find that sincere and reasonable. It is also a safety story about the user in front of the model, not about the person receiving a call from it.

The company is offering early access to a small group of AI safety researchers to find risks it hasn't anticipated. That's a good step. But the fraud risk is not unanticipated. Two reactions in the first day named it. A review process that waits for outsiders to discover the obvious is not the same as a plan.

What Block should do, since it's a payments company

Few companies are better placed to treat voice authenticity as a product problem. Block runs consumer payments and money-transfer services, and its customers are exactly who gets hit when a voice persuades someone to send money. Block hasn't said anything about provenance in the material we reviewed. Our recommendation, before any public release:

• Build disclosure in: a model that identifies itself as synthetic when it speaks to a third party, and says so by default rather than as an opt-in.
• Ship provenance tooling alongside the model, so that audio it generates can be detected, at least by Block's own systems.
• Publish the red-team findings from the safety researchers, not just the praise.

Reasonable people can object that watermarks can be stripped and that bad actors won't comply. True, and nobody should claim this solves fraud. But defaults matter at scale, and the honest versions of this technology shouldn't be the ones that make deception easy.

Verdict

Dessa may well be a genuine technical advance; Block says on its Dessa page that the model learned language from hundreds of times fewer words than LLMs need, though we can't verify that independently. Admiration is fair. But a research preview is the cheapest moment to decide how a technology can be abused, and the most expensive moment to ignore it is after release. Block wants the world to hear a voice that feels like a person. Our view: until it shows how anyone can tell when it isn't, the company has a fraud problem before it has a product.