ChatGPT is still stupid, fails basic literature task

I’m in the middle of finishing the construction of my new literature course on www.english-champion.com, The Short Stories of Flannery O’Connor, coming March 1. As I’ve been thinking about the high school and college students that might enjoy it, I started reflecting on how ChatGPT has influenced students’ reading habits and enticed them into only caring about summaries. So, I decided to test the AI generator on a few of O’Connor’s pieces.

I chose two of most well-known and three of her lesser-known works and asked a simple question for each. Since O’Connor is one of the most famous American authors of the last 100 years, and her stories have not only been analyzed by countless critics but are all freely available online, I figured writing brief summaries of the pieces wouldn’t be very difficult.

ChatGPT failed miserably.

Let’s take a look.

I simply entered a consistent prompt:

Write a five-sentence summary of Flannery O'Connor's story “The Barber.”

ChatGPT said:

Flannery O'Connor's short story "The Barber" centers on a tense encounter between a barber, the protagonist, and his customer, a man who is a member of the Ku Klux Klan. The barber, who holds progressive views, is conflicted about his desire to speak out against the racist ideologies of the Klan while fearing the repercussions of doing so. As the conversation progresses, the barber attempts to carefully navigate his professional relationship with the customer while also dealing with his inner moral struggle. The story culminates in a quiet but intense moment of realization for the barber, who understands the deeper consequences of his silence and the impact of his actions. O'Connor uses this interaction to explore themes of morality, identity, and the tensions between personal beliefs and social pressures.

This is nowhere near accurate. Though the piece is about racial politics, there is no mention of the KKK in the story, and the barber is not a progressive—he’s the racist. And the barber isn’t the one who has the moment of realization; it’s the customer (the holier-than-thou professor) who is awakened at the end.

Here’s another:

Write a five-sentence summary of Flannery O'Connor's story “The Turkey.”

ChatGPT said:

Flannery O'Connor's "The Turkey" tells the story of a young boy who is tasked with hunting and killing a turkey for a holiday meal. While hunting, he becomes increasingly fascinated by the turkey's intelligence and the challenge it presents, leading to a growing sense of admiration for the animal. Despite the turkey's cleverness and elusiveness, the boy is determined to succeed, demonstrating his own internal conflict between the act of hunting and a developing empathy for the creature. The story emphasizes themes of nature, the human tendency to dominate, and the disconnect between man and animal. Ultimately, O'Connor uses the boy's experience to explore the complexities of life, death, and the moral struggles that accompany them.

Again, not even close.

This is not a story about respecting nature. It’s about a boy on an accidental adventure who wrestles with his own understanding of God in a world that bestows blessings but also punishes unfairly. Unfortunately, the AI once again showed its foolishness.

Write a five-sentence summary of Flannery O'Connor's story “Good Country People.”

ChatGPT said:

Flannery O'Connor's "Good Country People" revolves around Hulga, an educated but emotionally wounded woman who has a prosthetic leg, and her interaction with a charming, seemingly naïve Bible salesman named Manley Pointer. Hulga, who looks down on others and believes herself superior, is initially intrigued by Manley’s humble, religious demeanor. However, as the story unfolds, Manley reveals his true, deceptive nature when he manipulates Hulga into a vulnerable situation, stealing her prosthetic leg. O'Connor uses the twist to highlight themes of pride, deception, and the contrast between appearances and reality. Ultimately, the story explores the complexities of identity, human weakness, and the consequences of self-deception.

This one is mostly accurate.

Write a five-sentence summary of Flannery O'Connor's story “Everything That Rises Must Converge.”

ChatGPT said:

Flannery O'Connor's "Everything That Rises Must Converge" focuses on the strained relationship between Julian, a young, educated man, and his racist, old-fashioned mother. They are traveling on a bus in a segregated Southern town, and Julian's mother insists on wearing a hat with a symbol of her former social status, symbolizing her attachment to outdated ideas. As the story progresses, Julian's frustration with his mother’s prejudices grows, and he attempts to challenge her views. However, the situation takes an unexpected turn when a confrontation with a black woman on the bus leads to a shocking and tragic event. O'Connor uses the story to explore themes of racial tension, generational conflict, and the complexities of social change.

Again, close enough.

Two wrong, two right. One more try…

Write a five-sentence summary of Flannery O'Connor's story “The Partridge Festival.”

ChatGPT said:

Flannery O'Connor’s "The Partridge Festival" is a short story that examines a small town’s preparations for its annual festival, centered around the hunting and display of partridges. The story features a woman named Miss Wilma, who is determined to win the contest by showcasing the best partridge. Through a series of interactions with townspeople and her own internal reflections, Miss Wilma reveals her obsessive competitiveness and desire for recognition. As the festival unfolds, the characters’ flaws and preoccupations come to the forefront, exposing their egos and insecurities. O'Connor’s story underscores themes of pride, competition, and the ways people seek validation through trivial accomplishments.

Way, WAY wrong.

There is no character named Miss Wilma, the story is not actually about partridges, and this summary neglects the entire plot of two young people attempting to visit a man who is a murderer to try to prove their misguided progressive, socio-political views.

So, this is what Chat GPT does—it simply makes things up.

And when students don’t do the reading for school and try to take a shortcut to cover the basic plot points for class, they can be completely misled.

This is why it’s still so important for students to actually read the classic literature assigned to them. They need to know what valuable lessons can be gleaned from these authors that have stood the test of time. Just “getting the gist” or knowing enough to pass a silly quiz the next day is not the point of studying the great stories of our culture, or art in general. We must experience it—all of it.

Teach students to avoid these AI tools like the plague. A plague on our intelligence is exactly what they are.