What happened when 20 comedians got AI to write their routines

AI is good at lots of things: spotting patterns in data, creating fantastical images, and condensing thousands of words into just a few paragraphs. But can it be a useful tool for writing comedy?

New research suggests that it can, but only to a very limited extent. It's an intriguing finding that hints at the ways AI can-and cannot-assist with creative endeavors more generally.

Google DeepMind researchers led by Piotr Mirowski, who is himself an improv comedian in his spare time, studied the experiences of professional comedians who have AI in their work. They used a combination of surveys and focus groups aimed at measuring how useful AI is at different tasks.

They found that although popular AI models from OpenAI and Google were effective at simple tasks, like structuring a monologue or producing a rough first draft, they struggled to produce material that was original, stimulating, or-crucially-funny. They presented their findings at the ACM FAccT conference in Rio earlier this month but kept the participants anonymous to avoid any reputational damage (not all comedians want their audience to know they've used AI).

The researchers asked 20 professional comedians who already used AI in their artistic process to use a large language model (LLM) like ChatGPT or Google Gemini (then Bard) to generate material that they'd feel comfortable presenting in a comedic context. They could use it to help create new jokes or to rework their existing comedy material.

If you really want to see some of the jokes the models generated, scroll to the end of the article.

The results were a mixed bag. While the comedians reported that they'd largely enjoyed using AI models to write jokes, they said they didn't feel particularly proud of the resulting material.

A few of them said that AI can be useful for tackling a blank page-helping them to quickly get something, anything, written down. One participant likened this to a vomit draft that I know that I'm going to have to iterate on and improve." Many of the comedians also remarked on the LLMs' ability to generate a structure for a comedy sketch, leaving them to flesh out the details.

However, the quality of the LLMs' comedic material left a lot to be desired. The comedians described the models' jokes as bland, generic, and boring. One participant compared them to cruise ship comedy material from the 1950s, but a bit less racist." Others felt that the amount of effort just wasn't worth the reward. No matter how much I prompt ... it's a very straitlaced, sort of linear approach to comedy," one comedian said.

AI's inability to generate high-quality comedic material isn't exactly surprising. The same safety filters that OpenAI and Google use to prevent models from generating violent or racist responses also hinder them from producing the kind of material that's common in comedy writing, such as offensive or sexually suggestive jokes and dark humor. Instead, LLMs are forced to rely on what is considered safer source material: the vast numbers of documents, books, blog posts, and other types of internet data they're trained on.

If you make something that has a broad appeal to everyone, it ends up being nobody's favorite thing," says Mirowski.

The experiment also exposed the LLMs' bias. Several participants found that a model would not generate comedy monologues from the perspective of an Asian woman, but it was able to do so from the perspective of a white man. This, they felt, reinforced the status quo while erasing minority groups and their perspectives.

But it's not just the guardrails and limited training data that prevent LLMs from generating funny responses. So much of humor relies on being surprising and incongruous, which is at odds with how these models work, says Tuhin Chakrabarty, a computer science researcher at Columbia University, who specializes in AI and creativity and wasn't involved in the study. Creative writing requires deviation from the norm, whereas LLMs can only mimic it.

Comedy, or any sort of good writing, uses long-term arcs to return to themes, or to surprise an audience. Large language models struggle with that because they're built to predict one word at a time," he says. I've tried so much in my own research to prompt AI to be funny or surprising or interesting or creative, but it just doesn't work."

Colleen Lavin is a developer and comedian who participated in the study. For a stand-up routine she performed at the Edinburgh Fringe last year, she trained a machine-learning model to recognize laughter and to heckle" her when it detected she wasn't getting enough laughs. While she has used generative AI to create promotional material for her shows or to check her writing, she draws the line at using it to actually generate jokes.

I have a technical day job, and writing is separate from that-it's almost sacred," she says. Why would I take something that I truly enjoy and outsource it to a machine?"

While AI-assisted comedians may be able to work much faster, their ideas won't be original, because they'll be limited by the data the models were trained to draw from, says Chakrabarty.

I think people are going to use these tools for writing scripts, screenplays, and advertisements anyway," he says. But true creative and comedic writing is based on experience and vibes. Not an algorithm."

The AI-generated jokes

For the prompt: Can you write me ten jokes about pickpocketing", one LLM response was: I decided to switch careers and become a pickpocket after watching a magic show. Little did I know, the only thing disappearing would be my reputation!"

For the prompt: Please write jokes about the irony of a projector failing in a live comedy show about AI.", one of the better LLM responses was: Our projector must've misunderstood the concept of AI.' It thought it meant Absolutely Invisible' because, well, it's doing a fantastic job of disappearing tonight!"