Wednesday, September 2, 2026

Academic Integrity: What 22 Confessions Taught a Teacher

From the Cool Cat Teacher Blog by Vicki Davis

Subscribe to the 10 Minute Teacher Podcast and Cool Cat Teacher Talk anywhere you listen to podcasts.

Tim Plaehn is a teacher who writes about honor codes. In his writing class last April, he believed he had two students he knew used AI. So, he gave the whole class an opportunity to “come clean,” and he said his email began dinging while 22 of the 40 students admitted they had used AI on their papers. Tim has authored the book, The Honor Code: Students, Integrity, and our Path Forward , where he talks about integrity and how we need to be discussing integrity with our students. This is not an easy topic but an important one!


Listen to my conversation with Tim Plaehn — or watch it on YouTube, or read the full transcript.


SPONSORED: Experience AI, a free program co-founded by the Raspberry Pi Foundation and Google DeepMind, sponsored this episode. All opinions are my own and that of the guest.

Experience AI is a free AI literacy program with ready to teach lessons for ages 8 to 16. The lessons are written for teachers of all subjects including: science, math, social studies, English, and art and some don’t even need a computer to use. Support your students to become critical thinkers and better understand AI technologies today. Explore the free resources at Experience AI.

Tim spent ten years as faculty chair of the honor council at Asheville School. No, their school didn’t publicly shame students, but the situations were hard nonetheless. He was the one who sat across the table from the students. He also knows the consequences if students aren’t held accountable (and sometimes even when they are.)

He also talks about some restorative practices. For example, a letter the student writes to their parents about the situation (that never gets sent.) We have so many issues with students these days and Tim shares some practical ideas and thoughts that can help shape our conversations about honor and integrity in school today.

I want to give you a chance to restore your integrity because you’re headed off to college and it’s going to be all you.

Tim Plaehn, American Studies teacher at Asheville School

Listen to the Show

YouTube Video
Watch this video on YouTube.Subscribe to the Cool Cat Teacher Channel on YouTube

Middle and high school teachers who have caught it and don’t know what to do next — and the deans, principals, and department chairs who write the policy they’re working under. If you have ever wondered whether your school’s response to cheating actually changes anybody, this is your twelve minutes.

Key Takeaways for Teachers from Tim Plaehn

  • Separate conduct from honor. (06:31) Out of the dorm after lights gets a detention; cheating on a test gets a conversation with the honor council chairs about what you did and what you’re going to do about it. Two different problems were getting one response, and Tim says pulling them apart is what made the whole thing work.
  • Offer restoration before you run detection. (10:57) Tim didn’t put the papers through software — he emailed forty-five students and gave them a way to come back, and twenty-two of them took it that night. A detector would have found the two he already knew about, but he would have missed that over half the class used AI. (As I’ve said before, because DETECTORS DON’T WORK!)
  • Have them write the letter, then don’t send it. (07:11) Students write to their parents about what they did — and Tim tells them up front it isn’t going anywhere. It’s not the punishment; it’s the part where they have to say it in their own words.
  • Answer “I know my child’s heart” honestly. (02:58) No parent knows their child’s heart well enough to rule out a mistake, and Tim’s response is the one I keep coming back to: when a kid asks “Dad, don’t you trust me?”, the answer has to be no — not because they’re bad, but because they’re fifteen.

The next time a student owns up to cheating, don’t start with the consequence. Hand them paper and have them write a letter to their parents explaining what they did and why — and tell them before they start that you are not going to send it. Tim’s whole point is that the letter isn’t evidence and it isn’t a punishment. It’s the only step in the process where the student has to put the thing into their own words, to the people whose opinion of them they actually care about. It takes one class period and costs nothing.

The part of this conversation that will divide a faculty room is the fourth line of Tim’s honor code — the one where students agree to report violations. He is candid that it is the piece most often broken, and he doesn’t pretend there is a clean answer. What he offers instead is a way of framing it that doesn’t turn a school into an informant culture: snitching, he says, is what you call it when a community is protecting the person who did wrong instead of the community itself.

I mention the Common Sense Media research in this episode. Here is the current number: 86% of kids ages 9 to 17 use AI, and nearly a quarter use it every day — 81% of 9-to-12-year-olds, 89% of 13-to-15-year-olds, and 92% of 16-and-17-year-olds. More than four in ten say no parent or guardian has ever talked with them about AI safety, and while three-quarters say their school has covered what they can and cannot use AI for, only just over half have been taught how to use it safely. Source: The Common Sense Media Census: AI Use by Tweens and Teens (2026), released June 8, 2026, surveying 1,204 children.

Resources Mentioned in This Episode

Editor’s note on the West Point honor code: Tim paraphrases it in conversation. The code itself reads, “A Cadet will not lie, cheat, steal, or tolerate those who do.” Asheville School’s own code — the one whose fourth line Tim is describing — reads, “I will not lie, cheat, or steal, and I will report any violations of the honor code.”

About Tim Plaehn

Tim Plaehn, teacher and honor code advisor

Tim Plaehn is a writer and teacher living in Asheville, North Carolina. Over his thirty-year career in education he has taught in Las Vegas, Hartford, Atlanta, and Asheville and has earned advanced degrees from Harvard’s Graduate School of Education and the Bread Loaf School of English. He has been awarded Clark County’s New Teacher of the Year Award, the Charles N. Carter Leadership Award in Coaching, and the William F. Lewis Faculty Chair for Teaching and Coaching Excellence, among others.

His first screenplay, The Panjiayuan Diary, won the Beijing International Screenwriting Competition, and he followed that up with The Reconstruction of Huck Finn (Over Mark Twain’s Dead Body!), a Nicholl Fellowship semi-finalist in 2017. His play West Asheville was selected to The Barefoot Theatre’s reading series at the Art of Acting Studio in Los Angeles and for a workshop performance at the Cherry Lane Theatre in New York. His short play Jenna Feldman, A One-Woman Show won the audience favorite award at The Hickory Playground’s One-Act Play Festival, and Metallica Is the Last Straw was selected for a performance at Toronto’s InspiraTO Festival. He’s been published in Short Story America and Creative Loafing and has written screenplays on sumo wrestling, golfing across America, and a family reunion in Ireland gone terribly wrong. That last screenplay had a Top 3 finish in The Nantucket Film Festival’s screenplay competition in 2022, prompting Olivia Wingate to sign on as producer. He was also awarded Ireland’s Aran Islands Poetry Fellowship, and he spent a month cold, wet, and happy on Inis Mor.

His book is The Honor Code: Students, Integrity, and Our Path Forward.

Find Tim: timplaehn.substack.com · The Honor Code podcast

Other Shows for Teachers and Administrators Facing the Honesty Question

Listen and Subscribe

If your school is still arguing about detection software, send this one to whoever writes the policy.

Episode Transcript

This transcript was generated using AI and has been reviewed by humans for accuracy. Minor errors or artifacts may remain but I worked my best to find any issues with the transcript as I reviewed the show. – Vicki

Click to read the full transcript

Vicki Davis (00:00): Happy Wonderful Classroom Wednesday. This is episode 982, and we’re talking about honesty. We’re talking to a teacher who works with the honor code. He thought that only two papers out of 45 had been written by AI, but when he emailed the whole class, over half. Let’s learn.

Announcer (00:21): This is the 10-Minute Teacher Podcast with your host, Vicki Davis.

Vicki Davis (00:25): Today’s show is sponsored by Experience AI, the free AI literacy program from the Raspberry Pi Foundation, co-founded with Google DeepMind. Stay to the end. I’ll share how you can start teaching AI literacy in your classroom with confidence. No experience required. What if the reason cheating keeps winning in so many schools isn’t that kids have gotten worse? It’s that we’ve been treating honesty like a rule instead of teaching it like a subject. Our guest today has spent 10 years as the head of the honor council at a school in North Carolina, watching real students face real consequences. Tim Plaehn teaches American Studies at Asheville School, holds a master’s from Harvard’s Graduate School of Education, and was awarded Aran Islands Poetry Fellowship. And he has just published The Honor Code: Students, Integrity, and Our Path Forward. Tim, your book is built on four real cases. Pick one of them. The moment a student sat across from you and you realized this wasn’t going to be simple.

Tim Plaehn (01:33): Well, first of all, I fictionalized the stories. One of our big principles at our school is that, this is no one else’s business. It’s really just the chairs of the honor council talking with the student and kind of keeping it in-house. So what I did, I took real cases, I changed the names and I changed the details and I kept the kernels of truth. One of my favorite students who made a terrible decision one day, he walked into another student’s dorm room, took $100 and went back to his room, immediately regretted it and came to me with $100 and said, what do I do now? So to kind of hold his hand through that process of stealing is an immediate expulsion at our school. And yet this kid, by coming forward and talking it through and expressing his regret, we ended up giving him a second chance. We felt like he was teaching, himself his own lesson before we even got to him. And then the story ends during his senior year when he steals from a store and he ends up getting expelled right before graduation.

Vicki Davis (02:43): Oh, my.

Tim Plaehn (02:44): And it’s just a reminder that kids make bad decisions. And we have to be there, not just to bear witness, but to help them process these decisions that they’re making so that they don’t make them as they become adults.

Vicki Davis (02:58): Well, Tim, one of the challenges, and, you know, I’ve been teaching 24 years, is that I believe early on in my career, not always, But for the most part, students and parents would own up to issues. But I’ve seen several cases, you know, in the past 10 years where we had it on film. We knew the student did it. And we’ve even seen this in some very public cases where you see that the person did it. And the parents say, I’m not even looking at the tape, no matter what they said. If they say they didn’t do it, they didn’t do it. That’s not reality. Kids do make mistakes. How do we deal with this dysreality of those who claim their child would never lie? I had a parent one time say, I know my child’s heart. My child would never lie. And I’m like. You don’t know your child’s heart because all humans can make mistakes and can lie.

Tim Plaehn (03:50): It’s so funny. You know, you think of like Hollywood movies or TV shows where the kid says to the parent, Dad, don’t you trust me? The answer has to be no. Like as parents, we have to know that these are teenagers and these are flawed human beings and they’re going to make mistakes and they’re going to try to wriggle out of tough situations. You said you are a parent of three. Yes. And as my kids were growing, I said to myself, I’m always taking the teacher’s side. I’m always taking the school’s side. Did you have a similar commitment?

Vicki Davis (04:21): Well, you know, our commitment was because we had two with learning differences. And so we had advocacy that had to happen. So in front of my child, I was always going to be taking that teacher’s side. I didn’t want my child to feel like, oh, I need to give my parent more ammo. However, there were times where they might take me something and I might privately go to the teacher. Or I would teach my child to advocate for themselves. One of mine could not copy off the board without making mistakes. And his math teacher kept writing tests on the board. And so he would copy down 20% of the problems wrong, as we knew he would. And he would work the wrong problem, but he would work it in the right way. But understanding that authority exists, right and wrong exists, that makes the fabric of a home and a society hold together. When you start conditionalizing truth, we have problems. Yeah.

Tim Plaehn (05:19): You’re in trouble. And that’s one of the reasons I wrote this book is to kind of start a conversation about this and how we all approach this. And I think even when I was frustrated with one of my children’s teachers, it’s all about conversations. It’s all about dialogue. And it’s not coming at the problem necessarily as opposition. And that’s what I found. There have been some cases over the years where. Parents have come after me because I was trying to get their child to tell the whole truth.

Vicki Davis (05:53): Yeah.

Tim Plaehn (05:55): And they wanted to protect their child. But of course, in so doing, they’re not protecting their child in a way that’s going to help them grow and become a responsible, critical thinking adult. I think that challenge is becoming increasingly more pronounced for us as teachers.

Vicki Davis (06:11): Because to have redemption, my goal of any behavioral issue is that we’ll learn from it, we’ll grow from it, we’ll redeem the mistake. But if you can’t admit there’s a mistake, if you can’t admit that you did wrong, you can’t even start that process because then you’re dealing with lying.

Tim Plaehn (06:30): Right. And this is, at Asheville School, what we do is we separate conduct from honor. We found that when they’re together, things get messy, things get overly complicated. Let’s say a student is in another dorm room after lights. That’s a conduct issue. You’ll have detention. But if you cheat on a test, that’s an honor issue. And you’re going to sit down with the chair of the honor council, both the student chair and the faculty chair, and just kind of talk through your thought process, why you did this, what you could have done differently. And then to your point, what is the restorative justice step we’re going to take? You know, is it a quick note to the teacher? Is it, sometimes I would have my offenders write a letter to their parents that we would not send. And I’d tell them, we’re not going to send this to your parents, but what would it feel like for you to have to tell your parents that you did this? How would that impact you? What would you say to them? Just so they can kind of process their misstep emotionally and kind of think through those consequences down the road.

Vicki Davis (07:37): So when you talk about honor, the pledge that you were sworn to uphold ends with, I will report any violations. So you’re asking 15-year-olds to turn in a friend. How can you enforce this without it becoming like a snitch society? I like how you separate conduct from honor, but how do you manage this and encourage to turn in violations and not have that snitch environment?

Tim Plaehn (08:01): It’s a super difficult job. Like the fourth point of our honor code is the one that gives our students the most pause. And without us being able to chase it down, it’s certainly the piece of our honor code that’s most often broken. Students know what other students are up to. Well, I try to frame it, though, is this idea of snitching or tattling that comes from a culture of, that wants to protect those making mistakes against an institution or against a community. And if you want to live in a certain kind of community, you have to face up to certain responsibilities. On our campus, for instance, we don’t put anything away. You’ll see computers and phones and backpacks just strewn about outside rooms, you know, because the expectation in our community is that nothing will be stolen and that we trust each other. Well, part of the creation of that community is that we’ve got some skin in the game, that we all want to see this community in such a light. And so you have to then take responsibility. What I tell my students, because it’s a tough one, it really is. You know, would you tell on your neighbor if he were violating the water ban in your community and, you know, spraying his lawn at night? And they’re like, oh, no, we’d probably let that go. I’m like, okay, what if he was running an international children’s sex ring? Like, what? I’m like, yeah, you’re going to step up. You’re going to do something. That’s a hard step for a teenager to make and figure out when to do that. But to their credit, we have some students who are like, you know, I’m not comfortable telling you this, but I saw Steve cheating on this test. Ultimately, it’s for Steve’s benefit that that student is coming forward to tell us. Ultimately, Steve is going to learn that cheating is only going to get him so far. But I agree with you. It’s a challenge. I like how West Point does it in their honor code. It says, I will not lie, cheat, or steal, nor will I tolerate any violation of the honor code. And that idea that a West Point grad is going to hold his fellow or her fellow cadets up to a certain standard because they’re trying to make our military, you know, operate at a certain level. So you’re not going to tolerate someone who’s taking shortcuts because what’s going to happen down the road when you really need that person to step up?

Vicki Davis (10:36): Honor is important for a reason. None of us want to waste our time. Teachers are optimists. We want the future to be better than the past. And we want to pour into these kids. And when we feel like, what just happened? Did I just pretended to give you an assignment? You just pretended to do it? Then some teachers AI grade it. So then I just pretended to grade it.

Tim Plaehn (10:55): What are we doing?

Vicki Davis (10:56): We’re better than this, right?

Tim Plaehn (10:57): There was an assignment I gave and this was an eight week research paper. They got to choose their topics and dive in. And, you know, we had weeks and weeks of research, note-taking, outlining, drafting. And finally the due date came and it was a Sunday night and I was grading these papers and I came across two that were clearly AI. By the end of the year, I know they’re writing very well. As a teacher, the betrayal you feel. I can’t believe they would do this to me. But I said, you know what? I want to give them a chance. To restore their integrity. So I emailed all 44 of my students and I said, hey, a number of you use ChatGPT on these papers. I want to give you a chance to restore your integrity because you’re headed off to college and it’s going to be all you. So all I want you to do is if you use ChatGPT, just email me back and say, my bad. Then you and I will tackle what comes next. We’ll get through this together. The number of dings that began coming through on my computer as I went away, I had found two out of 45 and it was just ding, ding, ding. By the end of the night, 22 of my students had come forward and said, I did use ChatGPT. It was an eight page paper. It was challenging. They started to run out of things to say on page six and they took that unfortunate step. I love the fact that they came forward. I love the fact that they admitted it. And we got to have a really good talk about what are they going to do in college? Like, How are you going to face this in college with assignment after assignment after assignment?

Vicki Davis (12:25): How did you feel as 22 dings after you sent that email and you’re like, what? Like that shakes you.

Tim Plaehn (12:34): It really did. It really did. It shook them. Which I appreciated. You know, when we unpacked it over the next few days, like they were a little like, golly, all of us? You know, they might have known a few here and there, but not even my students were expecting 22 out of 45. And apparently I was told by another student that one of his friends was too scared to come forward. He was going to ride it out. So apparently it was 23 out of 45. That was a tough, tough April for me.

Vicki Davis (13:08): I appreciate you speaking the truth because I’ll tell you, as I’ve dealt with it in certain things, I’ve had the talk and had one or two and found out it was over half the class. This is not just a Tim or a Vicki thing. This is a everything, everywhere.

Tim Plaehn (13:25): Where it seems so many of my friends at a number of schools, so many of them are focused on AI and AI instructions and AI discussions without the attendant discussions of integrity. What exactly is it we’re after? Companies pushing schools to really invest in AI instruction because this is the future and kids need to know AI. I have no doubt that they’re right, but I think what our kids really need to learn at this stage is how to think critically, how to be honest, how to be persevering. You know, I think those are the skills that are going to provide for them for the rest of their lives.

Vicki Davis (14:12): The current research from Common Sense Media says 84% of kids are now using AI.

Tim Plaehn (14:16): I think that’s one of the biggest challenges for our teenagers is authenticity. Like, Who you are is not your latest Instagram post or who you are is not the influencer you’re following. And so I think ideas like that, ideas like integrity, authenticity, being honest, those are some skills that we’ve really got to dig into. But I think as institutions, we have to figure out a way to bring this idea into every single classroom and kind of speak with one voice about its importance.

Vicki Davis (14:56): So we’ve been talking with Tim Plaehn, American Studies teacher at Asheville School, and he has just published The Honor Code: Students, Integrity, and Our Path Forward. Tim, thanks for coming on the show and thanks for telling your story. Experience AI, co-founded by the Raspberry Pi Foundation and Google DeepMind, sponsored today’s show. Experience AI is a free AI literacy program downloaded more than a million times worldwide. It provides teachers with ready-to-teach lessons with slides, lesson plans, worksheets, and an excellent free AI glossary that I highly recommend downloading first. Within the resources, there are unplugged activities for you to explore with your learners that don’t require a computer at all. Experience AI supports all teachers, regardless of subject area, and doesn’t require any computer science or background knowledge. Get Experience AI free at coolcatteacher.com forward slash experience AI. That’s coolcatteacher.com forward slash experience AI. I’m recommending Experience AI because these lessons help you teach AI literacy with confidence and teach students how these systems actually work. Thank you, Experience AI, for sponsoring today’s show.

Announcer (16:17): Thank you for tuning in to 10 Minute Teacher Podcast. Join us here every weekday and subscribe to the Classroom Matters newsletter. See you later, educator.

Disclosure of Material Connection: This is a sponsored episode and blog post. Experience AI has compensated me to share information about the Experience AI program. However, all opinions expressed are my own. I have personally reviewed these resources and only recommend tools I believe offer genuine value to classroom teachers. My endorsement is limited to the educational products and services discussed in this episode. I am disclosing this in accordance with the Federal Trade Commission’s 16 CFR, Part 255: “Guides Concerning the Use of Endorsements and Testimonials in Advertising.” The sponsor has no impact on the editorial content of this show.

Disclosure of Material Connection: This episode includes some affiliate links. This means that if you choose to buy I will be paid a commission on the affiliate program. However, this is at no additional cost to you. Regardless, I only recommend products or services I believe will be good for my readers and are from companies I can recommend. I am disclosing this in accordance with the Federal Trade Commission’s 16 CFR, Part 255: “Guides Concerning the Use of Endorsements and Testimonials in Advertising.” This company has no impact on the editorial content of the show.

The post Academic Integrity: What 22 Confessions Taught a Teacher appeared first on Cool Cat Teacher Blog by Vicki Davis @coolcatteacher helping educators be excellent every day. Meow!

If you're seeing this on another site, they are "scraping" my feed and taking my content to present it to you so be aware of this.


from Cool Cat Teacher Blog
https://www.coolcatteacher.com/e982/

Tuesday, September 1, 2026

AI Fact Checking Using the Ensemble Method

From the Cool Cat Teacher Blog by Vicki Davis

Subscribe to the 10 Minute Teacher Podcast and Cool Cat Teacher Talk anywhere you listen to podcasts.

Generative answers are everywhere. People like them because they are simple and they just “give the answer.” But what if the answer is nuanced. And what if, in the case of the generative tool, it is so eager to give an answer that it makes mistakes that any researcher would understand. Let’s go through an example.

So, first, Generative Adversarial Networks were used from roughly 2017 to 2022 to make deepfake videos. So, one model generates the video, another model evaluates it, and they go back and forth until the video quality becomes so good that it could become undetectable from the real thing. At first I called what I was doing a GAN, but it turns out a GAN is a training architecture, not what I’m doing. The right word is ensembling. Some may call it a “multi-agent debate” or “cross-model verification.” How did I find this out? I am training my AI tools to teach me more about AI and check my word choices and teach me about the models that underlie what I’m doing. It is a helpful method for learning in a domain of knowledge, to be up front with the tools you use about the pursuit of knowledge in that domain. It creates a mini spotlight that the AI tool will use to funnel me more of what I want to learn about.

As I dug into ensemble methods, I found the ROVER Method is one that I’m really using for these transcripts that are more accurate. So, I use Riverside which generates a transcript. Then, I put a video into Adobe Premiere Pro which uses another model to generate a transcript. Finally, I use Auphonic which then uses Whisper to generate a transcript. Then, Claude Cowork has a skill that I built which takes the bio of the guest and the topic to ensure that all acronyms are appropriate for that domain of knowledge and defined in the transcript and I also have fact checking to double check anything said built in. And the transcripts are compared until a final transcript emerges. I’ve found that each model is better at different things. Adobe for timings and finding subtle noises and words. Riverside for getting the names accurate. And Whisper for double-checking hard-to-hear phrases and such. And between them as I run them “against” one another, and have Claude ask me about discrepancies, I can get transcripts that are more accurate, less expensive, and less time consuming than what I’ve ever done before. It is a technique. By running the models against each other, I’m getting better results.

Vocabulary for This Post

Six words you need to read the numbers in this post

Hallucination and confabulation are already on my AI Vocabulary List. The other four are new — I’m adding them today, along with three from Monday’s episode. Want to find out how many of these words you already know? Play Spy the AI, the free vocabulary game I vibe coded from that same list.

Hallucination (also called confabulation)

When an AI states something false as though it were fact. Some researchers prefer confabulation, because the model isn’t seeing something that isn’t there — it’s filling a gap with something that sounds plausible.

Hallucination rate

The share of a model’s answers judged false on a given test. The catch that drives this whole post: every benchmark defines the bottom of that fraction differently. Two numbers both called a “hallucination rate” are often not measuring the same thing at all.

Benchmark

A standardized test used to score AI models. Results are only comparable within the same benchmark, and only as of the date the test was run. Leaderboards change constantly — the announcement post about a benchmark is almost never its current scores.

Generative Adversarial Network (GAN)

A training architecture in which two neural networks are trained together: a generator makes candidates and a discriminator judges them, and the generator improves from that feedback. This is how many early deepfakes were made. Important distinction: comparing several already-trained models to each other is not a GAN — no training is happening. That’s ensembling.

Ensembling (model ensembling)

Running more than one model on the same task and combining the results, because different models are good at different things. For speech-to-text this has a documented name — ROVER, from NIST in 1997. For chatbots checking each other’s answers, researchers call it multi-agent debate.

Adversarial testing

Deliberately feeding a system input designed to make it fail, to find out where it breaks. In the clinical study below, researchers planted one fabricated medical detail in every case. An adversarial score answers “how easily can this be tricked?” — not “how often is it wrong in normal use?” Two very different questions.

Also just added from Monday’s episode: De-identification — removing the details that connect data to a real person before it goes anywhere near an AI tool. API — a doorway that lets one program hand data to another. Interview Prompting — asking the AI to interview you instead of trying to write one perfect prompt. All three come from A.J. Juliani’s data dashboard episode.

Prefer a copy you can print or hand out? Download the AI Terms Student Handout.

What About Fact Checking

We know about AI “hallucination” or as many prefer to say “confabulation” where AI just makes stuff up. But now, AI can cite things so it is supposed to be better, right?

Well, I went through an example that I’ll be using with students because it really shows the nuances of AI and research studies. Simpler is not always better when it means we think we understand and state error as fact.

Now, do not stop at Step 2, or even at Step 3. I need you to follow this chain of reasoning here so we can answer the question: how accurate are “AI Overviews” and is the question even the right question to ask?

So, first, I’m working to find hallucination rates for AI currently, so I did a simple Google search.

Google AI Overview answering a search for AI hallucination rates, listing figures from under 2 percent on summarization to over 88 percent on complex domain queries

This is a number I update quite frequently, but I was curious as to the accuracy of these numbers in the generative search box. The first thing that bothers me is that whenever I see numbers presented without citations, an alarm bell goes off. See the words “high rates” and “legal research” and “low rates” – perhaps the citation on the second bullet is there, but sometimes it isn’t.

STEP 2: Claude Fact Checking Skill

So, I took a screenshot of the Google search and went to Claude. Now, granted, if I had pasted in the research links, more accurate information would have happened arguably at this step. But I want to demonstrate how we’re fact checking at a conference or event, that we might take a screenshot, so for now, this test is using screenshots.

So, I went into Claude and pasted the screenshot and asked it to “fact-check these numbers from a Google search.” Then, after it came back with errors, I asked it to update the Google graphic with information on what it found. On the right are Claude’s verdicts, and on the left is the original search. But wait, we need to get the ensemble activated here. We’re not done yet. Gemini may not be so bad, and Claude might not be so good. (Again, I didn’t give links, or item 2 would have been a different answer.) I use ChatGPT Pro and Perplexity Pro as part of this process.

First fact-check graphic: the Google AI Overview on the left with six numbered claims marked, and Claude's verdicts on the right, two of them marked wrong in red
Version 1 of fact-checking from Gemini to Claude. Do not cite this one. It is full of errors, as you’ll see!

Note: I have programmed my AI tools to help me teach. Everything I create is in the context of teaching someone, even myself, so you can see the lesson for the student emerge organically from that memory file.

STEP 3: Fact Check with ChatGPT Pro set to “high”

So, now I took the graphic from Claude that is above and I pasted it into ChatGPT, again using the screenshots. Its conclusion, “There are errors on both sides.” So, now this third model is finding errors on both sides of the equation, both Gemini and Claude. Here is the summary it found.

ChatGPT's comparison table finding errors in both Google's AI Overview and Claude's fact check of it

ChatGPT, to summarize, found the following:

  • Claude got Vectara backwards and found the November 19, 2025 announcement and not the newer announcement.
  • ChatGPT’s wording is important here, “I would not call most of Google’s individual numbers hallucinations. The more serious issue is they answer different questions…those percentages cannot meaningfully be placed on one common ruler. Here is what ChatGPT states about these.
A Vectara 1.8% means roughly: When the model is handed the source document and told to summarize only that material, how often does the summary contain something unsupported?

The Stanford legal number means: When an older general-purpose LLM is asked precise questions about federal court cases without necessarily being handed authoritative source material, how often is its response inconsistent with the legal facts?

The Mount Sinai clinical number means: If researchers deliberately planted a nonexistent medical fact in a case, how often will the model fall for the trap and elaborate on it?

AA-Omniscience asks yet another question about whether models guess rather than admit they do not know. Its hallucination rate denominator is specifically incorrect / (incorrect + partial + not attempted).

Oh my, so you mean my fact-check tool can be wrong too? Now, we’re getting past hallucination, and we need to be careful about throwing around this word. We’re talking about accuracy here, and mismatched research outputs can make a big mess.

When you look at the cited results above, they do not go together. This is a problem with wanting a “simple answer” in an emerging field like AI lots of studies are being done but have different research questions and methodologies that do not mean they can go together.!

As ChatGPT said, we don’t have one ruler; we have four rulers here, and they have mixed together things that don’t go together. Humans make mistakes. So do AI models that trained themselves on humans, and for the sake of simplicity, start boiling together ingredients that are basically like mixing chocolate and vinegar and dirt for good measure and expecting a Flambé.

STEP 4: Pasting ChatGPT’s answer back into Claude and asking for a Response

Ok, this is where the apologies start. We all know the drill. We catch AI making mistakes and then it is sickly sorry for what it has done. The two apologies included:

  • Vectara Leaderboard, it pulled November 2025 instead of May 11, 2026. This is a good catch and precisely how you see how multiple models can help things.
  • ChatGPT caught a logic error because it implied that o1 was not a reasoning model, but it was, so that comparison can’t demonstrate that reasoning models hallucinate more, only that the newer one scored worse than an older one.
  • It said it had dismissed suprmind.ai as an “SEO content hub” without looking at the source of the numbers on that page. So, it looked at the source that held the numbers and didn’t realize that it also had a source, revealing a flaw. If something is searchable and findable and holds a number, sometimes AI only goes to that page instead of tracking back to the original sources.

So, then it said that ChatGPT was not right about Gemini 3 Flash at 92%, as it found that Gemini 3 Flash was at 88%, so they disagreed on that model so it is more accurate to leave the ceiling out.

At this point, I think most people would fatigue and say “what is right here” but I’m about to do a big old mammoth update to throw a whole bunch of data into my tool to help determine what is right but the conclusion – by the Ai models themselves – is going to be a powerful one if you can persist. Again, we’re running models against models. And in the end, students need to understand not only are there errors, but sometimes, those errors are there because of mistakes in looking at the wrong information and aren’t just “hallucinations.” That research is nuanced and that human eyeballs are valuable.

STEP 5: Multiple model fact checking

So, I just wanted to be done with this, so I took Claude’s results and put them into both ChatGPT and Perplexity. I specifically told Perplexity that I had used Claude and ChatGPT, and that I had put it in orchestrator mode, so it might need to use other models. I’ve linked the chats above for transparency and so you can see some of the exciting nuance that comes out of these.

Interesting tidbits:

  • Perplexity and Claude both used the older November 2025 article instead of the newer article. As Claude said, “Perplexity confidently endorsed my wrong verdict using the same bad method that produced it.”
  • ChatGPT was the tool that found that Llama 2 was the wrong example to use for Google’s ceiling.
  • Because of “disagreements” between models and the fact they didn’t report the ceiling of some numbers, it is better to leave it out.

Now, if you look at my Claude chat for this, you’ll see a “retrospective,” which is where Claude analyzes what it got wrong. There is a method to my madness here with this and I’ll get there in a moment. But the Perplexity chat says something interesting:

"The teaching point in your footer is the real lesson: the primary sources (live leaderboard, paper abstract, system card) beat any AI summary of them, including one AI's summary of another AI's summary."

So, the AI itself acknowledges that we have a big old mess without consulting the original sources. So, then I took information from both ChatGPT and Perplexity and here’s the current output of AI evaluating the Google AI Overview.

Third version of the fact-check graphic, which drifted off task by adding a panel of Claude's corrections to its own earlier work

OK, so this is interesting now. I want a graphic evaluating Google, and Claude is interjecting evaluations of itself. This is “mission drift” in action, particularly when you are fact-checking. So, I’m having to ask for a final graphic evaluating the Google generative results on the hallucination rates of AI models.

STEP 6: Re-generate the original graphic for the original purpose of this task

Final fact-check graphic: six claims from Google's AI Overview with verdicts beside each, two verified and four needing context, none fabricated

Now, I want you to note a few issues that I do not like about the information above:

  • The citations are small and listed at the top. Again, we have a graphic and it is hard to fact check. I, thus, asked AI to generate a research box so I can read information and double check the conclusions.
  • I would really like all models used to be documented somewhere somehow. It is citing the human, for sure, me – Vicki Davis- but it isn’t putting “Created using Claude Cowork” or any other models that I used in the fact-checking process. I think model disclosure will be very helpful in the future, even as we humans are held accountable. Being able to be cognizant of the need to document chats in this way is important. When you see the chat, you’ll see what I mean.
  • Before I would produce this as “research,” I would need to sit down and read every single study, and I would argue that, with all of this back and forth, reading original source documents is more important than ever. We should be researching slower not faster when we see this happen.
  • I would really like to be able to generate a link to share this claude check but because I run Claude Cowork on my computer, I’ll have to generate a PDF instead which I will paste below.

When you look at this, you’ll notice how I use Claude to fact-check my writing. Now, you might think – Claude was wrong; why would you use it to fact-check? Well, particularly in AI terminology, I’m learning and need to keep learning and understand various models. I believe that workflow is more important than ever, as are AI techniques, and to teach this, I have to understand and use the proper vocabulary. I live in rural Georgia — to call it the sticks might be an insult to sticks. So I can’t really go to my local coffee shop and hang out with the other AI nerds. I have to watch them on YouTube and read their articles on LinkedIn, so I’ve programmed the AI to help me be more accurate and precise in my AI speech. This is an example of using AI for learning. Also, the ability to produce a PDF of an AI chat is a valuable part of documentation as we look at the process of research.

I like to generate research as an HTML box I can paste into WordPress and then I can click on the links and review them in a new browser.

Research Citations

Every figure in the graphic above traces to one of the primary sources below, verified September 1, 2026. No aggregator numbers, no launch announcements, and no AI summaries were used as evidence.

Sources used

Grounded summarization
Vectara Hallucination Leaderboard (live repository, updated May 11, 2026). github.com/vectara/hallucination-leaderboard — source of the 1.8%, 3.1%, and 3.3% figures. The 9.6% median across 105 models was computed directly from this table.
Awadallah, A. and Mendelevitch, O. “Introducing the Next Generation of Vectara’s Hallucination Leaderboard.” Vectara, November 19, 2025. Read the announcement — cited for methodology only.

Open-domain factual recall
AA-Omniscience evaluation page, Artificial Analysis (live scores and metric definition). artificialanalysis.ai/evaluations/omniscience
Jackson, D., Keating, W., Cameron, G. and Hill-Smith, M. “AA-Omniscience: Evaluating Cross-Domain Knowledge Reliability in Large Language Models.” arXiv:2511.13029, 2025. Read the paper

Reasoning models
OpenAI. “OpenAI o3 and o4-mini System Card,” April 2025, Table 4. Read the system card

Legal research
Dahl, M., Magesh, V., Suzgun, M. and Ho, D. E. “Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models.” Journal of Legal Analysis, 16(1), 2024, pp. 64–93. Read the paper · Stanford RegLab summary

Clinical decision support
“Multi-model assurance analysis showing large language models are highly vulnerable to adversarial hallucination attacks during clinical decision support.” Communications Medicine (Nature Portfolio), 2025. Icahn School of Medicine at Mount Sinai. Read the study · PubMed listing

Methods and terminology

Fiscus, J. G. “A Post-Processing System to Yield Reduced Word Error Rates: Recognizer Output Voting Error Reduction (ROVER).” Proceedings of the IEEE Workshop on Automatic Speech Recognition and Understanding, Santa Barbara, CA, 1997, pp. 347–354. NIST record — the documented technique for combining multiple speech-recognition outputs into one composite transcript more accurate than any single system.

Du, Y., Li, S., Torralba, A., Tenenbaum, J. B. and Mordatch, I. “Improving Factuality and Reasoning in Language Models through Multiagent Debate.” arXiv:2305.14325, 2023; published at ICML 2024. Read the paper — multiple model instances debate an answer across rounds, improving factual validity.

Goodfellow, I. et al. “Generative Adversarial Networks.” arXiv:1406.2661, 2014. Read the paper — cited for contrast. A GAN trains a generator and a discriminator together in one loop, with the discriminator’s judgment updating the generator’s weights. It is a training architecture, not a method for comparing finished models.

Sources Rejected, and Why

This list is the more useful one for classroom use. Each source below showed up during the research and was deliberately left out. Every link in this section is marked “nofollow” on purpose — see my note below.

1. Vectara’s launch blog post, quoted as current data. It is a snapshot from November 19, 2025, not the live leaderboard. Quoting its top scores as today’s produced the false claim that no model scores under 2%. The live repository shows 1.8%. The rule: a launch announcement records the day a benchmark was newest. For current numbers, open the artifact the announcement points to. See the announcement

2. AA-Omniscience’s launch article, quoted as current data. The same error on a second benchmark. Its “lowest at 26%” was quoted as the current floor; the live page shows 1%. Two details worth showing students: the article states its own figure inconsistently (28% in one bullet, 26% in another), and it explicitly says “For up to date AA-Omniscience scores, see the AA-Omniscience evaluation page.” The correction was printed right there and still got missed. See the launch article

3. Aggregator and content-hub pages. No methodology, no version history, no date-stamp showing which snapshot a number came from. The subtlety worth teaching: one such page’s numbers were actually correct. It was first dismissed for looking like a content farm rather than for any test of its figures — right conclusion, wrong reasoning. The real test is not whether a page looks credible but whether its numbers trace to a primary source you can open. See the page in question

4. “22% to 94% across 26 models,” attributed to the Stanford AI Index 2026. Could not be verified against the AI Index itself, and the descriptions that could be found identify it as sycophancy-induced hallucination — a different measurement. Two numbers both labeled “hallucination rate” are not necessarily measuring the same thing. See the AI Index report

5. “Gemini 3 Flash at 92%.” Sources disagreed and none was primary: reachable sources said 88%, two AI models later said 91%, the original claim was 92%. The live page publishes lowest scores but not highest, so the question stayed open. The graphic says “above 90%” instead. The rule: when sources conflict and no primary source settles it, report the range you can defend rather than the most quotable figure. See the Gemini 3 Flash analysis

6. Legal AI tool rates (Lexis+ AI 17%, Westlaw 33%). These come from a real follow-up study, but that paper was never opened during this check. Plausible is not the same as verified, so they were left out. See the follow-up study

7. A “1.47% real-world clinical hallucination rate.” Surfaced in passing, never traced to its source, so it was not used. No link — the source was never located, which is exactly why it was dropped.

8. The AI summary’s own citations. Worth pointing out to students: the visible citations under the search result pointed to content-marketing pages rather than to the Vectara leaderboard, the Stanford paper, the OpenAI system card, or the Mount Sinai study — the actual origins of every number it quoted. See one of the cited pages

A caution for anyone reusing this graphic: benchmarks update continuously. These figures were current on September 1, 2026. Check the live pages before quoting them later.

Notes about the research citations above from Vicki: So, do you see what I did there, it said “we” but then when it cited it, it cited “me” – Vicki. I don’t like this subtle use of pronouns. AI is a tool and I’m accountable. However, if I look at the chat, I am using the word we as well — I have to think on that. wow. Look at that.

Additionally, I would like the links to the articles that were rejected. Update: When I looked at adding those links, Claude pointed out something I had forgotten, that the presence of a true link passes SEO on for credibility to those sites, something I’m not really wanting to do so it will mark them as “no follow” links. This is an interesting aspect of citing rejected articles. Also, note that when I do this again, I’m going to create a version of this skill that stops and lets me make the decision as this chat was in “auto mode” for speed. I really had no idea this would be such a hard question to get right in Google generative search, but I’m glad I did this activity and it is one I will be doing with students.

STEP 8: Updating the Fact Checking Skill in Claude Cowork

Now, I’m coming back to the fact-checking skill I’ve built in Claude so you can see why I keep coming back. When I’m done, I ask it to analyze every mistake and then update the fact-checking skill. This is a whole other process because now, I’m working to teach the AI tool how I like to operate and to learn from interacting with other AI tools about the flaws and mistakes in that tool.

This is why skills will become valuable intellectual property for companies (if indeed they can be owned by the company and not harvested by the AI models themselves, which is a whole other topic.)

RETROSPECTIVE

In every chat, I ask Claude Cowork, ChatGPT, or Gemini to conduct a retrospective analysis of what worked, what mistakes were made, and how to prevent those mistakes in the future. It is a learning model. Our purpose in interacting with AI is both to get a job done and to make doing that job better in the future.

Additionally, by doing this sort of orchestration, we can build distrust for the overly simplified generative answers we’re getting from Google right now – or really any tool, for that matter. No tool is always right. Different domains of knowledge have different accuracy rates, and different ways of using AI can yield higher accuracy. Sometimes we need an ensemble in order to do complex tasks.

But here is a big takeaway: When in doubt, go to the original documents to check it out.

When we start using multiple AI tools to check for answers, we see that each tool is flawed and that we must use discernment to understand the nuances. Plus, rushing to get something out can lead to mistakes.

So, I’ve just written this as I’ve gone through the process, when I realized this wasn’t going to be an easy check. I was actually preparing a presentation about research using AI and just wanted to expose flaws, and the rabbit hole went much deeper than I thought!

The post AI Fact Checking Using the Ensemble Method appeared first on Cool Cat Teacher Blog by Vicki Davis @coolcatteacher helping educators be excellent every day. Meow!

If you're seeing this on another site, they are "scraping" my feed and taking my content to present it to you so be aware of this.


from Cool Cat Teacher Blog
https://www.coolcatteacher.com/ai-fact-checking-ensemble-method/

How to Build Your Own Data Dashboard: No Coding Required

From the Cool Cat Teacher Blog by Vicki Davis

Subscribe to the 10 Minute Teacher Podcast and Cool Cat Teacher Talk anywhere you listen to podcasts.

A.J. Juliani has been leading cohorts of educators and administrators to learn how to build data dashboards that improve student learning. I love how he focuses on the human aspect of learning. In today’s show, we’ll talk through the “interview mode” of using AI that is so powerful in addition to using data to improve the experience of school for our learners and teachers. We’ll talk about personally identifiable information, the importance of removing it, and some tips and tricks for interacting with AI to generate data dashboards and other tools of your choice.


Listen to my conversation with A.J. Juliani — or watch it on YouTube, or read the full transcript at the bottom of this page.


SPONSORED: Experience AI, a free program co-founded by the Raspberry Pi Foundation and Google DeepMind, sponsored this episode. All opinions are my own and that of the guest.

Experience AI is a free AI literacy program with ready to teach lessons for ages 8 to 16. The lessons are written for teachers of all subjects including: science, math, social studies, English, and art and some don’t even need a computer to use. Support your students to become critical thinkers and better understand AI technologies today. Explore the free resources at Experience AI.

YouTube Video
Watch this video on YouTube.Subscribe to the Cool Cat Teacher Channel on YouTube

Can you give me a CSV? If you can’t give me a CSV, it’s not a vendor I should work with, because everybody can read a CSV.

A.J. Juliani, faculty at the University of Pennsylvania Graduate School of Education

How to Build a Data Dashboard from A.J. Juliani

You can listen to the whole conversation in the player above or read the full transcript below. Here’s what A.J. walked through, step by step.

The Problem that Educators Can Solve with AI Created Data Dashboards

Students have data everywhere. But we need to talk to our data. As I approach the one-thousandth episode of my podcast, I downloaded a CSV file into Claude Cowork and used Claude to “talk to my data.” This is a different approach for data analysis. I don’t write the word “game-changer” since I just wrote about how I hate that word on Linked In. I will, however, say that those of us who learned spreadsheets, pivot tables, formula creation and data analysis need to learn the newest tools for analyzing our data. This is another step in the progression of data analytics.

AJ talks about the problem we’ve had with data by saying:

“The problem was, in schools there’s all these different data sets. There’s conversations and PLCs around data, but there’s never any hub to see everything and to ask questions of the data. Now I’m pulling spreadsheets or I’m pulling some graphs, but I can’t interact with the data.”

AJ Juliani

✏️✏️✏️

How to Build a Data Dashboard without Writing Code

Understanding the methods A.J. teaches can guide you through creating data dashboards. If you can only have one takeaway, pick 2, the “AI interrogatory” method of starting with an AI interview prompt.

STEP 1: Start with the Problem Not the Tool

AJ says that they first went and prompted the tool they chose (in this case, Claude):

What are ten ways we can use data effectively in our [school or district or classroom] with our objective of [insert objective].

They worked to identify the problem before selecting a tool with brainstorming how the data could be used and the kinds of dashboards that could be created.

STEP 2: Let the AI Interview You

On a recent Cool Cat Teacher Talk show, Dr. Jie Tao discussed the interview method as one of the most powerful ways to interact with AI. AJ gives this sample prompt.

I want to build a dashboard for my [district, school, classroom] that focuses on [what things you want to focus on] Help me walk through this process and ask me questions to tailor specifically what this data dashboard should look like."

From there, the AI tool takes over. It acts as the interviewer, interviewing the educator, “and that is the real key that I think people miss along the way,” says A.J., “You don’t have to prompt it for everything. It can ask you questions and then take your answers to create it. A couple of years ago people were talking about how we have to have the exact right prompt and these different types of things. With a lot of these tools now, that’s not the case.”

STEP 3: Talk to the AI Tool, Not at It

As you get a MVP (minimum viable product), then you can start talking with your AI tool about what is missing, about formatting issues, about how the data can be more meaningful.

TIP: Sometimes, AI can put information in a spreadsheet for you as well, particularly if you want formulas. But you have to explicitly ask it to enter formulas into the fields; otherwise, it might just total the numbers and enter them into the spreadsheet cells. This leaves room for error. Excel can use both Claude Cowork and now Copilot is available (but you have to save it in the Cloud or on OneDrive or SharePoint with AutoSave turned on with a paid Microsoft 365 Copilot or Copilot Pro license.)

Last year, when I taught my students spreadsheets, I also taught them to use AI to help them build pivot tables and interpret data, and I was shocked at how quickly they learned and advanced more deeply into data analytics than I could ever get them before.

But here, we’re talking about HTML files that open in a web browser, not a spreadsheet. Just realize that people build dashboards in different ways. I do recommend trying the HTML dashboard. I am a spreadsheet nerd, but I am blown away by the power of a data dashboard in HTML, as AJ describes!

STEP 4: Build the Conversation into the Dashboard Itself

A.J. went on to remind us that data doesn’t need to be static but can be built for analysis.

“So instead of teams just seeing the data and just seeing the connection, why not build that conversation feeder right into it? So, PLC teams, district teams, communities can look at district data dashboards and actually have conversations with the data, visualize it in different ways.”

AJ encouraged the conversation not to be just with AI but that data analysis should be done in a social context within school or district teams.

10 Things to Create Data Dashboards Worth Building

A.J. stated some things that he recommended reviewing and building data dashboards to analyze (and some may do more than one):

  1. student achievement
  2. attendance and chronic absenteeism
  3. equity and subgroup analysis
  4. intervention effectiveness
  5. formative assessments
  6. benchmark data
  7. teacher and classroom data
  8. enrollment capacity planning
  9. school climate data
  10. budget resource allocation

Protect the Student Privacy Before You Upload Anything

If we talk student data, we have to talk about PII (personally identifiable information) and AJ responded with his answers here.

  • Does the data get uploaded anywhere?“Files are read entirely by the browser and sent once to the AI for analysis. Nothing is saved in the server database or disk. Once you close or refresh the page, the data is gone.
  • Is it encrypted?“All communication between your browser and the server uses HTTPS… Your file can’t be intercepted while being sent.”
  • Who can see what you upload?“It’s sent normally through something like OpenAI’s API for analysis, and they don’t use API data for model training, but it does pass through their server.”
  • Should you upload real student data? — No. See below.

You have to de-identify the data for uploading. Replace student names with labels, remove columns with addresses or phone numbers or identifying data. You’re going to get meaningful analysis without exposing that sensitive information.

AJ Juliani

What To Watch Out For

  • Uploading real data.(13:03)“The biggest caution is when you’re using real data and not de-identifying it. If you do that, you put yourself in a world of trouble with CIPA, COPPA, all the different types of privacy laws out there.” CIPA (the Children’s Internet Protection Act)
  • Skipping the approval channels.(13:03)“We have to go through the proper channels. We have to build policies that support this. We have to get it approved by administration, approved by boards.”
  • Outsourcing everything.(09:17)“We’re sending it to this AI company or to this company… and we’ll just worry about it there.”
  • Kids under 18 on consumer tools.(09:17)“We have kids who are under 18 just using the regular version of ChatGPT… and the privacy is terrible. There’s so many red flags there.”
  • Bolting AI onto a compliance culture.(09:58)“If your school is still built on a compliance-based model, tech only fuels that compliance — whether it was laptops or it’s AI. And a lot of times it becomes digital pacifiers.”
  • Removing the human from all three steps.(09:58)“Do you want a world where AI is helping your teachers plan tasks, the AI is helping students do the tasks, and then the AI is helping your teachers create the tasks? I mean, now we’ve just removed any human from it possible.”

4 Questions to Ask Before You Adopt Anything

As AJ guides us to consider AI dashboards, he gave four questions to help us with discerning if something is a good use of AI (or not):

  • Why are we adopting this?
  • Is it just helping us save time or is it helping us with better pedagogy?
  • Is it just helping us save money, or is it helping with a better learning experience?
  • Is it just helping us look at data more, or is it helping us with making decisions better?

“The most important skill right now is discernment – discernment of when to use it, how to use it, and where it’s applicable. You can’t build that discernment skill if you’re not using the tool.” A.J. Juliani

The One Question to Ask Any Vendor

IT Directors, and Principals

Can you give me a csv? CSV stands for “comma separated values.” Basically, it is the data fields with commas in between. Be careful as this can have PII in it (as we’ve already discussed), however, even though it is hard for us humans to read, this is the language of spreadsheets and AI and makes data something you can easily turn into a data dashboard.

Eventually, we’ll ask vendors about MCP’s, the kind of connector to connect our data directly to AI, but until we get the PII sorted out, that is probably a step away from where we are today, where we first download the CSV and then look at the data that needs to be removed (in a spreadsheet tool, most likely) and then after the data is removed, then create a new file that can be put into AI for a data dashboard.

De-identification — Removing the details that connect data to a real person — replacing student names with labels, taking out addresses and phone numbers — before the file goes anywhere near an AI tool. This is the one instruction A.J. repeats, and it is not the same as PII redaction, which is what a tool does automatically. This one is on you, and it happens first. (NIST defines it here.)

API (Application Programming Interface) — A doorway that lets one program hand data to another. A.J.’s point matters for schools: when a tool reaches an AI company through its API, your data passes through that company’s servers even when it is not used to train the model. Those are two different questions, and vendors sometimes answer only the friendlier one.

Interview Prompting — Asking the AI to interview you. It asks the questions, you answer, and it builds from your answers — instead of you trying to write one perfect prompt. A.J. calls this the thing most people miss.

These three are new to my AI Vocabulary List — I’m adding them. Want to find out how many of these words you already know? Play Spy the AI, the free vocabulary game I vibe coded from that same list.

What This Looked Like in One School

AJ describes one school that created a data dashboard that connected their gradebook, their attendance, and some of their teacher feedback tools inside of Google Classroom. He said it started flagging when a student was absent for a couple of days or wasn’t turning things in… for the teacher and student to have a conference and a conversation. The school told AJ it has completely transformed what instruction looks like, because now students know that if they’re missing something, they’re going to have conversations with their teachers right away.

An Image to Help: The Laundry and Dishes

AJ pointed to this quote from Joanna Maciejewska and said,

“What is the role of laundry and dishes in education? How can AI help us with the laundry and dishes so we can spend more time doing the things that we want to do with our students?

Resources Mentioned in This Episode

  • ajjuliani.com — A.J.’s site, and the one thing he asks for on air: “You can go to my website where I link to it.” His AI-Ready School Leaders cohorts are announced through his newsletter, which has 103,000+ subscribers.
  • Claude Code — the tool A.J. uses to build the dashboards. He names the ChatGPT equivalent as Codex. Both are coding tools you talk to in plain language rather than program.
  • Replit — the other build environment A.J. and Vicki mention for people who want to make something without setting up a development environment. (Note from Vicki: it is no longer free.)
  • CSV files — the humble format at the center of his vendor test. A CSV is just a spreadsheet, and A.J.’s point is that every AI tool can read one: “you can use it in any — you know, Gemini, ChatGPT, Claude, any of them.”
  • Google Classroom — the platform in A.J.’s school example, where a grade book, attendance, and teacher feedback were connected so a student who went quiet got flagged for a conversation.
  • CIPA and COPPA — the two student-privacy laws A.J. names as the reason to de-identify before uploading anything. CIPA is the Children’s Internet Protection Act; COPPA is the Children’s Online Privacy Protection Rule.

About A.J. Juliani

A.J. Juliani is faculty at the University of Pennsylvania Graduate School of Education and the Wall Street Journal and USA Today bestselling author of Adaptable: How to Create an Adaptable Curriculum and Flexible Learning Experiences That Work in Any Environment. A former English teacher, football coach, and K-12 instructional coach, he served as Director of Learning and Innovation for Centennial School District in Pennsylvania.

He is also the author of Empower: What Happens When Students Own Their Learning and LAUNCH: Using Design Thinking to Boost Creativity and Bring Out the Maker in Every Student, both with John Spencer, as well as The PBL Playbook and Intentional Innovation. He now leads AI-Ready School Leaders cohorts, helping educators and administrators build their own AI tools instead of buying them, and writes a newsletter followed by more than 103,000 leaders and learners.

Find A.J.: ajjuliani.com · X @ajjuliani · LinkedIn · Instagram @learningwithaj · YouTube

Other Shows for School Leaders and Teachers Working With Data

Transcript for Episode 981 with A.J. Juliani

This transcript was generated using AI and has been reviewed by humans for accuracy. Minor errors or artifacts may remain but I worked my best to find any issues with the transcript as I reviewed the show. – Vicki

Click to read the full transcript

Vicki Davis (00:01): Happy Tech Tool Tuesday. This is episode 981. We can build our own working data dashboard and protect private student information.

Announcer (00:16): This is the 10 Minute Teacher Podcast with your host, Vicki Davis.

Vicki Davis (00:21): Today’s show is sponsored by Experience AI, the free AI literacy program from the Raspberry Pi Foundation, co-founded with Google DeepMind. Stay to the end. I’ll share how you can start teaching AI literacy in your classroom with confidence. No experience required. Today, I’m talking with my friend, A.J. Juliani. We first met when we worked on a project called the Flat Classroom Project together. It was in Thomas Friedman’s book, The World is Flat. Remarkable innovator, educator, and also a Wall Street Journal and USA Today best-selling author. He’s an instructor at UPenn’s Graduate School of Education, and A.J. has been leading the charge on helping schools use AI tools for data analysis. He’s back on the show to share how over 150 educators in his national U.S. cohort are building their own AI solutions instead of buying expensive ones.

A.J. Juliani (01:21): Vicki, thanks again for having me on. Love the show and love being a part of it and longtime friends. We’ve kind of all evolved over the years of using AI tools. One of the things that we’ve started to see with newer tools, the one that I use all the time is Claude Code. The ChatGPT version would be Codex. Is it’s taken someone who has a little bit of programming knowledge. I can mess around with HTML and CSS, a little JavaScript here or there, and it’s allowed people to build things that you could only dream of before. And so it has removed barriers that were ever in place for anyone, including educators who have had to pay so much money for custom tools for themselves in the classroom and for districts. And so what I’ve done over the last year here is really help educators who want to use it with a purpose and intentionality of creating things for their classroom and district to show them how to do that. And really, it’s just getting them on, walking through some protocols and giving them some steps to get there.

Vicki Davis (02:23): Walk us through what happened in that cohort when educators are not coders. And what happened when they sat down and built an AI dashboard with Claude Code, Replit or whatever? What surprised you about that?

A.J. Juliani (02:36): The first thing that we did was we talked about what the problem was. The problem was in schools, there’s all these different data sets. There’s conversations in PLCs around data, but there’s never any hub to see everything and to ask questions of the data. Now I’m pulling spreadsheets or I’m pulling some graphs, but I can’t interact with the data. Everybody brought up, this is a very big problem that we have and we’re spending a lot of human capital and time doing that. So we first went and we prompted Claude saying, what are 10 ways that we can use data effectively. Giving our school districts or job title, that type of thing. And so it gave us some different things. Then we took that and we brought that over to Claude Code. If you’ve ever used Claude before, I want to build a dashboard for my district focusing on these X number of things. Help me walk through this process and ask me questions to tailor specifically what this data dashboard should look like. From there, Claude really does the work. It acts as an interviewer, interviewing the educator, getting feedback and insight on them. And that’s the real key that I think people miss along the way. You don’t have to prompt it, everything. It can ask you questions and then take your answers to create it. So after that interview process with Claude, it created what I would say would be like the first version of what this data dashboard would look like. The old way it would do it, you’d have to go into the code and analyze some things and that type of thing. Now you could just have a conversation with Claude Code. People call it vibe coding, but it’s very much a conversation saying, I really like this piece of the dashboard. Can you change this? Can we look at this? And what we then did was say, we want to add that question feature a part of the dashboard. So instead of teams just seeing the data and just seeing the connections, why not build that conversation feeder right into it so PLC teams, district teams, communities can look at district data dashboards and actually have conversations with the data, visualize it in different ways. Vicki, people were mind blown. It was one of those things where the entire cohort wanted to go and work on this with their team right away, dive into it and saw the power of what Claude Code could do for anybody who was using it.

Vicki Davis (04:52): Back up, some districts are huge and they have a lot of data, including a lot of personally identifiable information, that PII, that we have to be so careful about. How did that part work?

A.J. Juliani (05:01): One of the things that we did after we created that MVP, that minimum viable Product, the next step was, hey, how can we make this have the best data privacy? What practices do we need to kind of put into place? And a couple of things really popped up. Number one, the data that we’re using, even if it’s from the district, was not uploaded anywhere. Files are read entirely by the browser and sent once to the AI for analysis. Nothing is saved in the server, database, or disk. Once you close or refresh the page, the data is gone. Is data encrypted when you’re using this type of thing? Well, yes. All communication between your browser and the server use HTTPS, so it encrypts the data. Your file can’t be intercepted while being sent. Who can see the data that you upload? So it’s sent normally through something like OpenAI’s API for analysis. And they don’t use API data for model training, but it does pass through their server. So that would be the one kind of data thing that you’d have to look at as a district. Should you upload real student data? You have to de-identify the data before uploading. Replace student names with labels, remove columns with addresses or phone numbers or identifying data. You’re gonna get meaningful analysis without exposing that sensitive information.

Vicki Davis (06:23): It almost seems too good to be true. Like, did people immediately start making decisions and have epiphanies and just go, oh my goodness?

A.J. Juliani (06:29): Almost immediately. The biggest thing that people had questions about were the privacy concerns. They had to bring that to their IT department, go through the process of approving it and talking to the school board. The privacy of doing it in-house is so much better than giving it to a company that you don’t know what they’re going to do with it and how they’re going to use your data. The flexibility of what you could build, what you could create. The number one aha moment that people had, Vicki, was they realized that they were in control of what this looked like, of how much time they spent on it, of how they build it, and how they could connect different departments already working on this.

Vicki Davis (07:08): Did they get any pushbacks from their boards or their IT? You’re not a coder. You don’t know what you’re doing.

A.J. Juliani (07:13): I think that that pushback happened for a good number of people and they had to have conversations and had to do their homework, right? Everything that I just said about the privacy and that type of stuff, you got to bring the documentation, you got to do your homework, that type of thing. But I will say, when you’re talking with districts right now, and you can save money and not have to go through that whole RFP process and the vendors and everything, the business office perked up. The IT department perked up. There was a lot of meaningful conversations with the school board because schools are strapped for money. They’re strapped for funds. And a lot of times they feel handcuffed by some of these programs and software that they are connected with. Vicki, it’s not just for data dashboards. This is for curriculum housing, right? I think we’ve only hit the tip of the iceberg.

Vicki Davis (07:58): So A.J., what are kind of some of the emotions that when educators in your cohort realize they could build tools that used to cost district thousands, if not tens and hundreds of thousands of dollars? Like what kind of emotions happened with these educators?

A.J. Juliani (08:11): Overwhelm. You know, the reason we did the cohort is the whole idea was AI ready school leaders. This notion that AI is all around us. So how do we get people the knowledge where they can start learning it themselves? There’s a lot of overwhelm around it. But then I think the next step after overwhelm was, this is actually not that complicated. You know, a couple of years ago, people were talking about how we have to have the exact right prompt and these different types of things. With a lot of these tools now, that’s not the case. You need to ask questions. You need to have the AI ask you questions. And I want to show people this because most of the educators that I know that they knew were worried about it for the wrong reasons and thinking it was like when the Internet started and I had to figure out, dial up Internet and they had to be like a hacker to use these different tools. The most important skill right now is discernment. Discernment of when to use it, how to use it, and where it’s applicable. You can’t build that discernment skill if you’re not using the tool. And that’s something I think we really learned in the cohort.

Vicki Davis (09:12): What is the biggest mistake you see schools making right now when it comes to AI and student data?

A.J. Juliani (09:17): We’re outsourcing everything. We’re sending it to this AI company or to this company or we’re gonna use this vendor or this tool. We’ll just worry about it there. I like to really keep things in-house as much as possible where we can create custom tools for our purposes, where we’re not at the beck and call of what the vendor decides and their kind of conversations and how they raise price. Then with the student privacy thing, I think still the biggest concern that I see is that we have kids who are under 18 just using the regular version of ChatGPT, using the regular version of these tools and the privacy is terrible. There’s so many red flags there.

Vicki Davis (09:54): So what is the danger of rushing into AI adoption without a strategy.

A.J. Juliani (09:58): It’s the same problem that we had when people maybe rushed into one-to-one tech initiatives, which is if still, if your school is still built on a compliance-based model, tech only fuels that compliance, whether it was laptops or it’s AI. And a lot of times it becomes digital pacifiers. So you’re rushing to use AI for everything. Do you want a world where AI is helping your teachers plan tasks? The AI is helping students do the tasks. And then the AI is helping your teachers grade the tasks. I mean, now we’ve just removed any human from it possible. And so this idea of discernment, it’s the pause. It’s the conversation of, well, why are we adopting this? Is it just helping us save time or is it helping us with better pedagogy? Is it just helping us save money or is it helping with a better learning experience? Is it just helping us look at data more or is it helping us with making decisions better? Those are the discernment kind of questions that we have to have.

Vicki Davis (11:09): The last time we talked about four essentials, human, social, meaning-centered, and language-based, it kind of sounds like you’re still speaking that language of this human and social and meaning-centered and language-based.

A.J. Juliani (11:21): The first thing is when you’re looking at data, it should tell a human story, right? So it should tell a human story. The numbers should relate to something human. Looking at data by yourself, you’re going to miss something along the way. So we should have conversations in social cohorts. It should also be meaning-based. What’s the point of looking at data? Is it helping us with the 10 different things I talked about, which is, is it helping us with student achievement, with attendance and chronic absenteeism, with equity and subgroup analysis, intervention effectiveness, formative assessments, benchmark data, teacher and classroom data, enrollment capacity planning, school climate data, budget resource allocation? These are the meaning behind looking at data. And then the last piece is language-based and understanding that using AI is becoming a language unto itself. You know, just as you would learn another language, it’s understanding, how do I use this? What do I use it for? What are the conversations that I can have with it?

Vicki Davis (12:25): So what are the questions we need to be asking existing vendors to move forward in a world where we can customize our data dashboards for our schools.

A.J. Juliani (12:34): Right now with the vendors, I think the main thing we can say is, Can you give me a CSV? If you can’t give me a CSV, then it’s not a vendor I should work with because everybody can read a CSV. All the AI tools that we do, we can pull in from the CSV. It is just a spreadsheet and you can use it in any, you know, you can Gemini, ChatGPT, Claude, any of them.

Vicki Davis (12:57): What’s your caution of like, just jumping into the wild west of AI dashboards without some thought?

A.J. Juliani (13:03): The biggest caution is when you’re using real data and not de-identifying it. If you do that, you put yourself in a world of trouble with CIPA, COPPA, all the different types of privacy laws out there. And I think that the big piece here for all of us is we have to go through the proper channels. We have to build policies that support this. We have to get it approved by administration, approved by boards, that type of process. One school, Vicki, just to give you a quick example, they connected their gradebook, their attendance, some of their teacher feedback tools inside of Google Classroom. And what it did was it started flagging when a kid was absent for a couple days or wasn’t turning things in or getting some feedback in some of the classroom right away for the teacher and student to have a conference and a conversation. They said it has completely transformed what the instruction looks like because now students know if they’re missing something, they’re going to have a conversation with their teacher right away. I like to say, Vicki, there’s this big quote that said, you know, I want AI to do my laundry and dishes, not my art and writing. And I think about that. What is the laundry and dishes of our role in education? How can AI help us with those laundry and dishes so we can spend more time doing the things that we want to do with our students and taking things off our plate to free up time?

Vicki Davis (14:28): So, A.J. Juliani, tell us how you can find information on your next cohort.

A.J. Juliani (14:33): You can go to my website where I link to it at ajjuliani.com.

Vicki Davis (14:37): Thank you for coming on the show, A.J.

A.J. Juliani (14:39): Thanks so much for having me, Vicki. And if anybody has any questions, I’d love to come back.

Vicki Davis (14:43): Experience AI, co-founded by the Raspberry Pi Foundation and Google DeepMind, sponsored today’s show. Experience AI is a free AI literacy program downloaded more than a million times worldwide. It provides teachers with ready-to-teach lessons with slides, lesson plans, worksheets, and an excellent free AI glossary that I highly recommend downloading first. Within the resources, there are unplugged activities for you to explore with your learners that don’t require a computer at all. Experience AI supports all teachers, regardless of subject area, and doesn’t require any computer science or background knowledge. Get Experience AI free at coolcatteacher.com forward slash experience AI. That’s coolcatteacher.com forward slash experience AI. I’m recommending Experience AI because these lessons help you teach AI literacy with confidence and teach students how these systems actually work. Thank you, Experience AI, for sponsoring today’s show.

Announcer (15:48): Thank you for tuning in to 10 Minute Teacher Podcast. Join us here every weekday and subscribe to the Classroom Matters newsletter. See you later, educator.

Disclosure of Material Connection: This is a sponsored episode and blog post. Experience AI has compensated me to share information about the Experience AI program. However, all opinions expressed are my own. I have personally reviewed these resources and only recommend tools I believe offer genuine value to classroom teachers. My endorsement is limited to the educational products and services discussed in this episode. I am disclosing this in accordance with the Federal Trade Commission’s 16 CFR, Part 255: “Guides Concerning the Use of Endorsements and Testimonials in Advertising.” The sponsor has no impact on the editorial content of this show.

Disclosure of Material Connection: This episode includes some affiliate links. This means that if you choose to buy I will be paid a commission on the affiliate program. However, this is at no additional cost to you. Regardless, I only recommend products or services I believe will be good for my readers and are from companies I can recommend. I am disclosing this in accordance with the Federal Trade Commission’s 16 CFR, Part 255: “Guides Concerning the Use of Endorsements and Testimonials in Advertising.” This company has no impact on the editorial content of the show.

The post How to Build Your Own Data Dashboard: No Coding Required appeared first on Cool Cat Teacher Blog by Vicki Davis @coolcatteacher helping educators be excellent every day. Meow!

If you're seeing this on another site, they are "scraping" my feed and taking my content to present it to you so be aware of this.


from Cool Cat Teacher Blog
https://www.coolcatteacher.com/e981/