From the Cool Cat Teacher Blog by Vicki Davis
Follow @coolcatteacher on Twitter
Perhaps the first thing teachers, students, and just about anyone who has used Chat GPT thinks is,
"Oh no, we're in trouble. What will happen when kids figure out they don't have to write their paper -- that Chat GPT can do it for them."
After I share the information with you in this post, you'll be even more concerned, but perhaps will find some ways that you can work with students until we have AI to help us with this problem (which will be used by education when it is invented.)
The Problem of Plagiarism and Chat GPT is Acknowledge by Open AI
If you listened to Steve Dembo in yesterday's post, you'd learn that Chat GPT is still not as good as a human in many cases. Perhaps this is why Open AI produced the Educator Considerations for Chat GPT. They state in this paper
“We recognize that many school districts and higher education institutions do not currently account for generative AI in their policies on academic dishonesty. We also understand that many students have used these tools for assignments without disclosing their use of AI. Each institution will address these gaps in a way and on a timeline that makes sense for their educators and students. We do however caution taking punitive measures against students for using these technologies if proper expectations are not set ahead of time for what users are or are not allowed.”
Open AI “Educator Considerations for Chat GPT”
Using Text Classifier from Open AI
Then, they recommend this tool AI Text Classifier. So, I decided to test it out on something I knew was initially drafted by Chat GPT. What answer did it give me?
Using Chat GPT 3.5 from Open AI
OK, so it says it wasn't AI-generated. But when I paste the text into the GPT 3.5 chat, I get the answer on the exact text I pasted into AI Text Classifier as being written by Chat GPT. Strike one against the AI Text Classifier, or is it a false positive for Chat GPT?
Using Chat GPT 4.0 from Open AI
So, then I pressed it forward. I went into Chat GPT 4 and pasted in the same paper, and asked if it wrote the paper. This is the answer I received from GPT 4
Observations and Applications for Detecting Student Chat GPT Use
So, either Chat GPT has a multi-personality disorder or is only trained to recognize its own algorithm. So, for example, Chat GPT 3.5 (which was used to write this paper as the open free version) can recognize Chat GPT 3.5, and Chat GPT 4.0 is trained to recognize itself.
I also had a student who took a paper he had written two years ago. He pasted it into Chat GPT 3.5, and it said it wrote the paper. When he pushed back, the algorithm quickly backed down and said it could be wrong.
Recommendation: If you're looking at student work, I recommend pasting it into Chat GPT 3.5 and Chat GPT 4.0. However, you cannot rely on this alone, as there are confirmed false positives and false negatives. All it seems to detect is possible algorithm use.
AI Detection Services
So, next, let's look at some of the popular AI detection services. We'll look at the results of my sample, which was generated by Chat GPT and partially edited, and discuss some recommendations and questions we need to ask as we move forward.
GPT Zero
So, first, I went to GPT Zero and pasted it into the text. As a result, I received a significant portion of the highlighted paper and received this response. Again, even here, they state that you need a “holistic assessment of student work.” More on that later.
CopyLeaks AI Content Detector
Copyleaks AI Content Detector is pretty awesome. After you paste the words in, you move your cursor over the text and can see what likelihood percentage it is that it was written by a human versus AI Bot. So, when I moved my mouse over part of the text, I received this comment.
Then, when I moved my cursor over most of the paper, I received this response.
So, this is a much better tool than the next option below because it treats sections of the text independently.
Writer AI Content Detector
So, some apps like Writer's AI Content Detector will tell a person what percentage of the paper says it was human created according to the algorithm. This particular site encourages humans who use AI to generate their content to make it seem more human so that search engines won't punish it for AI-generated content. (Sorry, bloggers. You can't get easy content by letting Chat GPT do the work for you.) So, let's look at how this responds.
AI Generated Content That Has Been Edited
I pasted part of the paper from above which had been generated by Chat GPT and then was edited somewhat, and this was the response I received.
So, this number is even higher than the previous stating that 83% is human-generated content. Note that it was drafted by Chat GPT and altered. So, it is reasonable to expect students also to use these tools.
Slightly Edited Chat GPT Generated Content at the End of the Paper
Then, I pasted in the end of the essay which had been slightly edited but was pasted directly from Chat GPT, and this was the response I received.
Recommendation: Different portions of the paper may be edited differently. Instead of pasting the entire paper into a detection tool, you should use a tool that highlights suspicious portions like AI Copyleaks detector. Tools that create an overall percentage may not tell you the true story about content. Considerations: This also might influence our policies that we have as sometimes the revision of papers using AI bots might cause a detector to say that a paper was written by AI, when in reality it was written by a human and revised by AI. Questions to Discuss: And, isn't Grammarly AI? Is revision by Grammarly considered different than revision by Chat GPT? Also remember that Chat GPT in combination with Quillbot might be a challenging combination to break. However, that said, in the end, if we want good writing, do we consider if a student does all of this work to do good writing that they can produce high quality "original writing" by AI to get them to where we want them to be? Is the purpose good writing or understanding of content or both? Do we consider students with learning difference in written expression in a different category? Do we want them to use tools to help them express themselves better?
AI Content Detector
So, now we're about to see concerns with aggregating data. I pasted the paper into Crossplag AI Content Detector which says “Originality has a new threat and this is the solution.” I got these results with the same paper I had used all along.
So, now I took in the part that was a direct cut and paste and received an 18% human above. In this case Crossplag doesn't do much better and says this portion of the text is mostly human written.
Sapling AI Content Detector
So, then I headed over to Sapling's AI Content Detector. I paste the WHOLE paper in and get a different result.
So, here's a different tool with a different algorithm. However, if you scroll down the page, they also have a disclaimer that should alarm us.
Content Detector.ai
I looked at ContentDetector.ai and pasted in the content. The result — 78.6 AI!
GPT Radar
So, GPT Radar touts itself to be “the best GPT detector.” I pasted in the same content. And, you guessed it… a totally different result.
Winston.AI
So, an academic solution, Winston.AI says it is the best for academics and education. Not so, according to my results.
Recommendation: Use many tools. Realize that the AI bots are rapidly evolving. You may have false positives and false negatives. We have to go further. You cannot discipline a student on the basis of an AI detector alone.
What's Next?
So, we know now that we're not going to have an accurate tool. The tools will conflict with each other. If you base your decision on just AI checkers, you're setting yourself up for a fall. There will be a wealth of tools that will disagree with you. So, you need to go to the next level. Here's where I've started.
Check the Grade Level
Use the Hemingway app or another similar app to look at the grade level of the paper. Even better, look at the grade level of other papers turned in. You should see a pattern. It does make sense that at some point, a company will invent AI that gets to know a student's pattern of writing and will be able to give an indicator of the likelihood that a student wrote something.
Recommendation: Remember that students can ask Chat GPT to write at a certain grade level. This is not always going to be an indicator, but if you see a fourth grader writing at a doctoral level, which may be a strong indicator for you.
Know Your Students
Remember how the apps are talking about holistically looking at student work? You have to know them. Their style. Their work. How do they write?
Recommendation: You have to relate to educate. Now more than ever. Sometimes they should write on paper in class. Have many writing samples, drafts, and outlines. Writing as a process is teachable but if you are hands off and just look at the final paper, there may be issues.
Talk to Your Students
I've been teaching for twenty-one years. I've seen parents, siblings, friends, girlfriends, and boyfriends write papers for students and even papers purchased on the Internet. The time-tested method is to sit down and talk to a student about their paper. (Of course, this hinges upon moving rapidly on grading. If you wait, they could argue (possibly rightly so) that they have forgotten what they wrote about.)
Talk to them about word choice and vocabulary. Talk to them about content. You'll have to rest on more than just one word they don't know.
Recommendation: Consider writing into policies that all written work can and will be "spot checked" with conversations about the written work. Then do it. It is good to have conversations anyway about work as part of teaching. This isn't about gotcha moments and catching students but more about gotcha moments where students show they have understood the concepts you're teaching and they're learning.
Be Proactive
As I said on Day 2, Chat GPT is the fastest-growing software in history. Listen to what Steve Dembo did with his class, the policies they wrote, and their conversations. Discuss policies with parents and students.
Now is the time to have a meeting with your Language Arts department. You will have an issue before the end of the school year — guaranteed. You already have issues and don't know it. And how you react is essential. I know often schools may want to wait and deal with this over the summer. You don't have time.
You could have some handwritten papers. You can flip your classroom. You can *gasp* discuss Chat GPT with your students. I have.
Recommendation: Start the conversations with your team now. Have teachers start using Chat GPT to learn to recognize the output. There is a pattern in what it writes.
Update Your Policies
Update policies relating to the use of AI tools. Disclose the use of AI tools when you use them. For example, I check everything with Grammarly, but I have not used Chat GPT in this article except to check the paper of concern.
There's also a privacy concern. Open AI reserves the right to read chats. Nothing HIPPA-related or IEP-related should be pasted into chat GPT if there is a risk of disclosing private information. Strategic information, personal information, and student information should be kept out of Chat GPT. I know some educators who have used it to help write hard emails to parents. If you're using names of people or specific instances, this is a very bad idea. Can Chat GPT and other AI tools help us revise: Yes. Are there times we should keep the information out of an AI tool: Yes. Other tools like Notion may have more privacy supports than Chat GPT.
Recommendation: Update policies to include acceptable AI usage for both teachers and students. Include a requirement to disclose. Include what information may not be pasted into a Chat GPT post. Understand that AI is being built into many tools, so other AI tools (like Notion AI) may be acceptable to use while Chat GPT may not. You're going to have to do some checking on the policy of the site you use. Be clear about which tools you allow and what type of information can and cannot be used in them.
Why Don't You Just Block It?
Well, wouldn't that be nice? Except you can't. AI writing tools are being built into everything. Coming to Google Docs and Microsoft Word soon. Students will need to use these tools and indeed there are many uses, which I'll be writing about as we go through these 80 days of AI (this is only Day 3!)
In my opinion, it is a mistake to block Chat GPT. There are many positive uses for this tool. But at this point, the disruption to our writing programs will be significant among students with integrity issues. (I've talked about the integrity gap before.)
Writing Helps Us Learn
I have so much to learn. When I need to learn, I write.
And therein is the importance of writing. I have learned far more about this topic by writing about it than I would have if I had just read. Which is why good writing can and always should be part of a good education.
How we move forward will be a challenge.
And now to my final recommendation.
Recommendation: True experts don't exist on this topic. What I'm posting today about these tools could change tomorrow. Education in a world of AI will take work, conversations, comparing notes, and some policies at the highest levels to help us move forward.
We have many opportunities and many conundrums. We don't have them in the fall, we have them now. I hope that as I share what I'm learning that it will help some of you who follow these posts as you're working through these issues.
But please, contact me via Twitter or email me at vicki at coolcatteacher dot com and share with me what you're learning, what you disagree with, what you know and what will help me. Is there any other tools or topics that a classroom teacher like me needs to learn? Please share!
Happy day 3 of 80 Days of AI and HI. Can you see how much human intelligence is needed?
The post What Do I Do If I Think Chat GPT Wrote my Student’s Paper? appeared first on Cool Cat Teacher Blog by Vicki Davis @coolcatteacher helping educators be excellent every day. Meow!
from Cool Cat Teacher Blog
https://www.coolcatteacher.com/what-do-i-do-if-i-think-chat-gpt-wrote-my-students-paper/
No comments:
Post a Comment