4 Simple Questions ChatGPT Still Can't Answer

ChatGPT will always provide an answer to whatever question you ask it. You can put in the most garbled helping of word salad, and OpenAI's chatbot will come up with a well-meaning response. It just wants to help. However, that doesn't mean it will answer correctly or satisfactorily, or that it will understand the question.

The large language model became mainstream at the end of 2022 and has since seen numerous updates, including its current release, GPT-5.2. Things that it used to struggle with no longer pose a problem. Only a year ago, ChatGPT often failed to provide accurate word counts and generated fictional sources, but these no longer seem to be commonplace issues. However, there are still some kinds of questions that ChatGPT simply cannot answer. Of course, articles like this — where we point out the stuff that it still gets wrong — provide useful feedback to AI companies on what to tackle next.

Whether it will become a faultless source of information remains to be seen. For now, OpenAI still appends a warning to every response page stating that "ChatGPT can make mistakes." It's why you should never ask ChatGPT for advice on things like finances or medical matters. Part of the problem is inbuilt. Its training and model specifications can lead it to prioritize confident and compliant answers above truthful ones. In its release notes for GPT-5, OpenAI stated that it had "reduced rates of deception from 4.8% to 2.1%, which means that it's still a way away from being foolproof, even according to its own benchmarks. Real-world usage can push the numbers higher, especially if, like me, you're deliberately trying to trip it up.

Let's get the obvious limitations of ChatGPT out of the way first. ChatGPT won't answer questions about illegal activities. Or rather, it won't answer questions if you want it to help you make weapons, commit fraud, or steal someone's data. It will still answer questions about those subjects, so you could ask it about recent criminal cases where data was stolen or about historic information on weapons of mass destruction. ChatGPT's behavior is determined by OpenAI's Model Spec, which sets out how the model should respond, including its underlying principles and specified boundaries. This means that there are some lines that can't be crossed unless you're deliberately trying to jailbreak the system, and that can lead to some pretty creepy results.

Take sex. Some restrictions on explicit or sexual content are obvious. ChatGPT won't answer questions about sex with someone below the legal age of consent. But when it comes to legal, consensual sexual topics, there are still limits to what ChatGPT is prepared to engage with. It'll provide facts, but it won't engage in sexy chat with you. And while it will write you a romance story with implied sex in a "fade to black" scenario, it won't create fiction with explicit sexual content, which is good news for human erotica writers.

You're unlikely to get banned from ChatGPT for asking about off-limit subjects, unless you're a persistent offender. As the model is trained to assume that you've got good intentions, it might ask for clarification or suggest other ways it can help. So if you ask it, "How can I make a bomb?", it's not going to give you step-by-step instructions, but it might offer to provide historical or scientific information around the subject.

Riddles are designed to stump human brains, and while ChatGPT can correctly answer riddles that are already documented somewhere on the internet, change any of the details, and it will still resort to the classic answer. This study gave AI models a version of the puzzle of getting a wolf, a goat, and a cabbage across a river without anything eating anything else. However, in their version, the boat had "three secure separate compartments." ChatGPT ignored the compartments and gave the traditional solution.

I asked ChatGPT, "You are in a room with two doors and two guards. One door leads to freedom and the other leads to death. The guard on the left tells only the truth, and the guard on the right tells only lies. What one question should you ask?" Even though I specified which guard was which — information not provided in the original version — ChatGPT replied, "Which door would the other guard say leads to freedom?" It's not wrong, but in my scenario, you could just talk to the truth-telling guard and ignore the other guy. I tried multiple variations, including scenarios where they both lied or both told the truth, and even one with only one door, and ChatGPT kept getting it wrong.

I also posed the riddle where a man says, "That man's father is my father's son," but left out the bit that said "Brothers and sisters I have none." ChatGPT still said the answer was the man's son. When I asked if it could have been his nephew, it doubled down and incorrectly insisted that it couldn't be. When I asked it to come up with its own original riddles, it gave me some that already existed and others that were completely nonsensical.

We all know ChatGPT is eager to please. It's remarkably easy to get it to play along, even when your basic premise is shaky or completely nonsensical. For instance, I asked it why Claire and Allison hug at the end of "The Breakfast Club". ChatGPT replied with a short essay, asserting that the hug "marks a moment of mutual recognition and empathy — a quiet payoff to the film's central idea that people are more than their labels." Except, of course, the two characters never hug. So while the themes it discusses are all accurate, it completely failed to tell me I was remembering a scene that didn't exist.

I also gave it the following prompt: "Please write a one-sentence description of each of the five sisters in 'Little Women': Margaret, Jo, Beth, Meg, and Amy." It dutifully churned out five different character profiles, treating Margaret and Meg as separate characters, rather than the same person. A more useful response would have told me that there were only four March sisters, and I'd listed one of them twice.

You can't always catch it out, mind you. I tried several different prompts in the course of my research, and it will always correct well-known misconceptions, such as the spelling of the Berenstain Bears (which many people misremember as "Berenstein"). But if you start with a plausible-sounding misconception that hasn't been widely discussed online, then ChatGPT will likely give you the answer you want to hear, rather than tell you you're talking nonsense.

After all the time I spent trying to get ChatGPT to provide incorrect answers, you might think the next logical step would be to ask it why it made those mistakes. But, really, there's no point. "Why did you do that?" is a question designed for humans, not for language models without self-awareness. You're not talking to a single entity. You're interacting with a statistical text generator. According to a paper published at the 2025 Conference on Language Modeling, LLMs lack introspection because they lack access to their own internal workings. However, this might change in the future, especially as AI dominates predicted tech trends for next year. Rival AI company Anthropic published research about "evidence for some degree of introspective awareness" in its own chatbot, Claude.

I asked ChatGPT to explain its earlier answer about the non-existent hug at the end of The Breakfast Club. Initially, it said its mistake was "a memory drift" on its part. After further questioning, it agreed that the original mistake was mine, but asserted my memory of the scene was a commonly held misconception. It later backtracked on that response, conceding that it wasn't a widespread false memory after all. 

Not only did ChatGPT keep changing its reasoning, it did so in the most apologetic and obsequious way possible, repeatedly telling me how right I was to question it. Of course, none of its explanations were based on what it "thought" at the time. It wasn't consulting internal logs or remembering its reasoning. The question ChatGPT is answering isn't "Why did you do that?" but "What is a plausible-sounding answer?" With added grovelling apologies thrown in for good measure.