Your student asks ChatGPT to write a history essay about the First Continental Congress. The AI generates five paragraphs that sound authoritative. The kid turns it in. It's coherent. It sounds right.
It's also missing a key date, misnames a delegate, and confidently asserts something that's not actually true.
This is the gap: your students don't know that AI can sound confident while being wrong. They think if it says it clearly, it's true.
What happens inside an AI model
This is worth teaching because it explains *why* AI fails:
Models are trained on patterns in text. They learn "what words usually follow what other words." They're very good at it. So good they can sound authoritative about things they've literally never seen.
This is called a "hallucination." The model isn't lying. It doesn't know it's wrong. It's just predicting the next word, and sometimes that prediction is plausible but false.
Example: "The capital of Iceland is Copenhagen." No. But the model "knows" that Scandinavian countries have capitals, and Copenhagen is a Scandinavian capital, so... maybe?
Your student needs to know this isn't a search engine. It's a text predictor that sounds like it's answering questions.
What AI is prone to
Hallucinations: Making up facts, citations, dates that sound real but aren't.
"Write a historical analysis of the 1847 Great Famine Relief Act." The model generates several paragraphs about a real famine and a plausible-sounding act that never existed.
Bias: Training data reflects the world as it was (often biased). So the model learns those biases.
"Describe a teacher." The AI might default to "she" for elementary, "he" for high school. Reflects data, not reality.
Outdated info: Models have a knowledge cutoff. They don't know about recent events.
"What's the current president?" If the model's training ended in 2023, it doesn't know.
Overconfidence: The model has no uncertainty. It says wrong things with the same tone as right things.
"Is glucose a protein or a carbohydrate?" The model doesn't hedge. It states. Even when wrong.
How to teach students to check
1. Fact-check claims against a primary source or reputable database. The AI says the Treaty of Versailles was signed in 1918? Find the original treaty. Check.
2. Check citations. "The AI cited this book. Does that quote actually appear in that book?" Have students spot-check one or two citations from every AI-generated essay.
3. Ask the AI to explain itself. "Why did you choose that date?" Sometimes the model will reveal it's guessing. Sometimes it'll double down confidently even when wrong. Either way, the student learns something.
4. Run the same prompt twice. Ask the AI the same question. If it gives different answers, something's off. AI shouldn't contradict itself on factual questions.
5. Assume recent information needs verification. If it's about the last 6 months, the AI probably doesn't know. Check news sources.
The assignment that teaches this
Don't just tell them. Show them:
"Find the hallucination" exercise:
Give students 5 claims from an AI response. 3 are true. 2 are hallucinations (made up).
Their job: determine which is which using a search engine or database.
They'll learn fast: AI sounds confident about things that are just... false.
Then, have them rewrite the AI response with corrections and citations.
This teaches: 1) AI can be wrong. 2) Your job is to verify. 3) Here's how to verify.
The rule to teach
"AI is a draft assistant, not an answer machine. It sounds smart. But smart-sounding and accurate aren't the same. When you use AI, your job becomes: is this actually true? That's thinking. That's what you need to do."
The [academic paper outline tool](/academic-paper-outline) helps students structure their research *before* they write, so they have a backbone of facts before the AI fills in. And the [discussion questions tool](/discussion-questions) models the kind of thinking that doesn't rely on AI—it relies on your own questions.
