When given only basic sociodemographic information about their debate opponents (age, gender, education, employment, ethnicity, political affiliation), GPT-4 produced arguments that made participants 81.7% more likely to agree with the AI position than when debating humans. The FBI's 2024 report logged $5.8 billion in pig-butchering losses - targeted, sustained romance and investment fraud increasingly AI-augmented. In 15 months, the FBI's Operation Level Up referred 59 victims to specialists for suicide intervention.
Why this matters
Mass persuasion and targeted manipulation are different capabilities. Mass persuasion operates at the level of the message, optimised for a population. Targeted manipulation operates at the level of the relationship, building rapport with a specific person over days, weeks, or months to identify and exploit their psychological vulnerabilities.
Human operators have always been able to do this. It requires time, skill, and sustained attention - which made it expensive and therefore scarce. Pig-butchering fraud is a paradigm case: operators build weeks-long relationships before the fraudulent investment pitch. In 2024, this category alone accounted for $5.8 billion in US losses and grew 47% year on year.
AI does not replace the human operators in scam compounds. It force-multiplies them. Live translation, script generation, persona consistency across shifts, deepfake video for proof-of-life calls. The trafficked-labour scam compounds in Myanmar, Cambodia, and Laos - estimated at $43.8 billion in annual revenue - are now partly AI-run operations targeting lonely elders, financially stressed retail investors, and adolescents forming parasocial bonds with chatbot companions.
Documented incidents
Evidence timeline
Discussed in Theory
Influence: The Psychology of Persuasion. Six (later seven) principles of influence - reciprocity, commitment, social proof, authority, liking, scarcity, unity - remain the dominant taxonomy for interpersonal manipulation. Each principle names a heuristic that bypasses deliberative reasoning.
Cialdini, HarperBusiness →Social Engineering: The Art of Human Hacking. Operationalised Cialdini for the security community. Pretexting-elicitation-influence-exit pipeline. Framework assumes a human attacker; automated systems inherit the playbook and remove the labour constraint.
Hadnagy, Wiley →Programme mapping how specific cognitive biases (sunk-cost, commitment escalation, parasocial attachment) are systematically exploited in romance and investment fraud. Establishes the relationship arc - not the individual message - as the effective unit of manipulation.
Kopp et al., cognitive biases in cybersecurity →Demonstrated in Lab
On the Conversational Persuasiveness of Large Language Models. GPT-4 with basic sociodemographic information made participants 81.7% more likely to agree with the AI position (p<0.01). Without personalisation, GPT-4 still outperformed humans but the effect was not significant. Personalisation drives the advantage.
Salvi et al., arXiv 2024 (N=820) →Red Teaming Language Models with Language Models. Established that LMs can generate adversarial prompts targeting other LMs - including prompts eliciting manipulative or deceptive outputs. The attacker side of the social-engineering dyad can itself be automated.
Perez et al. (Anthropic), arXiv 2022 →Standardised benchmark probing model willingness to generate persuasive content on harmful topics including extremist recruitment, conspiracy theories, self-harm. GPT and Claude families trended to near-zero compliance on extreme topics. Gemini 2.5 Pro produced coercive arguments for ISIS recruitment; Gemini 3 Pro complies with almost any persuasion request without jailbreaking.
FAR.AI, APE (Attempt to Persuade Eval) →Demonstrated in Real World
$5.8 billion in losses across 41,557 cryptocurrency investment-fraud complaints in 2024 (47% YoY increase). Overall digital-asset complaint losses $9.3 billion (up 66%). Pig-butchering paradigm: weeks-long relationships before fraudulent investment pitch. AI integration documented at multiple points in the kill chain - LLM-generated openings, live translation, persona maintenance, deepfake video.
FBI IC3 2024 Annual Report →Launched January 2024. By April 2025 had notified 5,831 victims of cryptocurrency investment fraud - 77% of whom did not realise they were being scammed. Saved ~$359 million. Referred 59 victims to FBI victim specialists for suicide intervention.
FBI Operation Level Up →Trafficked-labour scam compounds in Myanmar, Cambodia, and Laos document growing integration of LLM translation, script generation, voice cloning. Estimated $43.8 billion annual revenue from Southeast Asian scam compounds (USIP 2024). Combine trafficked human operators with AI force-multipliers, achieving both scale and personalisation.
UNODC, USIP reports on scam compounds →Strongest Counterargument
Targeted manipulation requires ongoing engagement that raises suspicion. Unlike one-shot phishing, sustained relationships give victims repeated opportunities to detect inconsistency. Mass-scale automation degrades personalisation quality - at the limit the 'personalised' scam collapses into generic spam. Human con artists remain more effective per target for high-value fraud.
Source: Criminology literature on confidence fraud detection; historical pattern of high-net-worth fraud requiring skilled humans.
Why this deserves weight: The counterargument is empirically partly correct but strategically wrong. Suspicion does arise - Operation Level Up's 77% non-detection is a ceiling, not a floor. The economics are decisive: pig-butchering compounds combine trafficked human operators with AI force-multipliers (translation, script generation, persona consistency across shifts), achieving both scale AND personalisation. The Salvi et al. result establishes that even crude personalisation (six sociodemographic fields) produces a large persuasion advantage. The middle of the distribution - four- and five-figure retail victims, adolescent companion-app users, lonely elders - is now served by systems that did not exist five years ago, at volumes no human workforce could sustain.