Educators on the cutting edge of generative AI and its application in the classroom are discovering fascinating new ways to assess learning, uprooting centuries-old reading- and writing-based assessments as more and more students employ ChatGPT and related tools.
“Any language-based assessment or any language-based discipline is going to be affected, [including] lab reports, research papers and speech writing,” says Mike Kentz, founder of AI For Schools (soon to be Zainetek Educational Advisors), an AI literacy training consultancy. “Literally anything that is going to involve the production of language is going to need to be rethought.”
Despite the evidence that AI will deeply impact the sector, a new report from Ithaka S+R declares that “high levels of uncertainty and deep pockets of pessimism” persist among faculty teaching at four-year institutions. Of the 5,259 faculty surveyed, 42% of instructors completely prohibit their students from using generative AI.
Kentz, on the other hand, made generative AI tools a cornerstone of his assessment grading at Benedictine Military School in Georgia, where he taught this past academic year. The English professor had students interact with a chatbot and assessed how well they could guide the conversation and refine their interaction based on the bot’s output. Kentz also assigned students to perform close reading on their classmates’ chatbot transcripts to assess the quality of their queries.
The consultant, professor and adjunct faculty member at Point University isn’t alone in recognizing the increasing futility of traditional open-ended writing assignments and essays. Researchers at the University of Reading in the U.K. created fake student profiles and armed them with ChatGPT to answer psychology exam questions. The study discovered that 94% of AI-generated submissions went undetected and routinely outperformed students.
“As many of you have realized or experienced this year in the classroom, our traditional assessments are less relevant than they used to be if students can use AI,” Kentz said in a recent Course Hero webinar featuring K12 and college educators. “We must shift our assessment focus on the process of learning rather than our traditional outputs.”
Kentz’s framework not only assesses student learning but also tackles AI literacy. Students unfamiliar with a topic and who blindly trust a chatbot’s output are likely to fall victim to AI’s proclivity to “hallucinate” wrong information to satisfy the user.
“We could have done a better job at teaching students how to engage with information produced on the internet,” he says. “I hope that we’ve learned from that.”
President moves: 3 hires as another leader changes her mind
Here’s his outline of how educators can build AI literacy in their classrooms and grade chat transcripts effectively:
- Teach principles for effective AI use. Ask AI for suggestions and options—not answers—and reflect critically on what AI is producing. Additionally, show students what a strong AI query looks like.
- Teachers generate purpose statements. What is the purpose of the assessment that teachers are permitting students to interact with AI on? What will students gain?
- Students interact with AI for an assessment grade. Provide students with a rubric involving engagement quality, critical analysis, ethical considerations and query strategies.
- Students complete traditional output. Decrease the value of their essay. Instead, assess students’ understanding of the project’s objective, task and content through a closed-notes verbal presentation.
- The AI reflection: Task students with reflecting on their use of AI using the “What-Why-How” approach—What did AI suggest? Why did it make this suggestion? How did AI use impact the overall product?
SchoolAI, Magicschool.AI, and Khanmigo provide educators direct access to their student’s transcripts with a chatbot, allowing them to intervene and steer the conversation if students fall off course.
Upending the business case study
Noah Askin, faculty member at the University of California Irvine’s Paul Merage School of Business, has a hard time trusting any written assignment turned in by his students, questioning whether what he’s read is spat out from a generative AI tool. However, the assistant professor has found a way to integrate AI into his business case assignments without the technology becoming a crutch.
A case is a foundational assessment for business students that requires them to identify, address, and solve an organizational problem. While the traditional assignment requires students to a read briefing on the issue and respond via a written assignment, Askin piloted Breakout Learning’s AI software to create a new kind of business case. The edtech service provides a multimedia assignment and facilitates student-led discussions on business operation and strategy.
“It moves away from the classic written case, which generations now in school are less interested in,” he says.
Breakout Learning’s AI monitors the student discussion and evaluates each of their contributions, grading them based on the professor’s rubric. With access to discussion overviews and the AI’s evaluation, professors can tailor class discussions around students’ grasp of the material.
“I still assigned a final project that was a written assignment,” he says, “but my interest is less about what they wrote and more about whether they understood the concepts and utilized them.”
Askin has found that the AI platform generated class discussion and student comprehension that was superior to the outcomes from traditional assignments.
Avoiding student learning blindspots
As empowering as AI can be, teacher input is essential to the process, the educators say. AI tasked with grading student discussion could potentially miss student insights that aren’t explicitly covered in the rubric, Askin contends. “That’s obviously a concern where a human grader has the judgment and wherewithal to recognize a really interesting point.”
And while generative AI bots have computational power that extends far beyond the human imagination, they are also as deeply fallible as we are, Kentz says. “We need to treat its responses the same way we treat a family member at Thanksgiving dinner who’s talking confidently about Russia,” he explains. “It doesn’t matter how smart your uncle is or how many PhDs he has, you still have to be critical. That’s what you should be doing when AI produces responses.”
