An AI provocation! How biased are WE on AI? Fascinating paper…

I work exclusively in this area but as soon as I mention my work, the mere mention of the two letters ‘AI’ result in an emotional reaction, often expressed as “but surely it’s all biased", "we’ll lose the ability to think" whatever… alarmist opinions are thrown about with little or no evidence or analysis. I wrote about our human biases when first encountering AI in my book 'Artificial Intelligence in Learning', as I'd experienced it so often.

STUDY

So it was interesting to come across this strange but fascinating paper that investigated how bias affects the perception of AI-generated versus human-generated content. (Thanks Rod @rodjnaquin)

They conducted three Experiments:

1. Participants evaluated reworded passages
2. Summaries of news articles were assessed
3. Evaluations of persuasive essays were gathered.

Some texts were simply labeled as either ‘AI Generated’ or ‘Human Generate’, other texts were presented without any labels.

RESULTS

First, in blind tests (unlabeled content), raters could not reliably differentiate between AI and human-generated texts.

With labeled Content, things got far more interesting. Participants showed a strong preference for content simply labeled as ‘Human Generated’ over ‘AI Generated’. This preference was over 30% higher for texts labeled as human-created. The same bias persisted even when the labels were intentionally swapped, indicating a preconceived bias rather than an assessment based on content quality.

Oddly, for those who bang on about bias in AI, the study reveals a significant human bias against AI-generated content, not based on content quality but on the label assigned.

CONSEQUENCES

I believe that much of the debate around some topics on ethics and AI follows this pattern. As soon as people hear those two letters their own bias kicks in. People come with confirmation bias around human exceptionalism, the belief that AI can't match up to human writing skills. This research uncovers these biases dives into whether people's biases are messing with their judgments in the realm of writing.

As human biases affect perceptions of AI-generated text. This leads people to assume that humans outperform AI in creative writing. Their blind tests, with deliberately swapped labels, assessed the depth of that bias.

This really matters, and this is an area that is really worthy of m orer research, rather than the outpouring of alarmist rhetoric. By shedding light on these biases, we can pave the way for better collaboration between humans and AI, especially in creative fields.

Paper: https://arxiv.org/pdf/2410.03723