What Happened When I Asked an AI Agent to Grade the Transcript
Source: Higher AI Substack
Author: Higher AI
Original source: https://higherai.substack.com/p/what-happened-when-i-asked-an-ai
Private backup: the full article text is archived in the private repository at archives/articles/higherai-substack-com-what-happened-when-i-asked-an-ai.source.md. It is not published on the public Quartz site.
Summary
The author tests whether an AI agent can complete a supposedly metacognitive “grade the transcript” activity by interacting with a custom Playlab pressure bot, copying the transcript into a Google Doc, and annotating its own moves. The exercise was meant to make students defend ideas, anchor claims in textual evidence, and reflect on chatbot interaction. The experiment shows that an AI agent can complete this layered process task with minimal prompting, which challenges assessment designs that rely on AI-resistant process work. The author argues that agentic AI makes it harder to preserve the cognitive effort required for deep learning, especially when the task can be delegated end-to-end.
Big ideas
- Learning still needs some struggle, even when AI can make things easier
- Students need to bring the purpose; AI should not supply it for them
- AI literacy should help people notice how AI changes what counts as knowing
- AI tools should be judged by the work they will actually do
Claims
- Learning requires some productive struggle that AI can remove
- In an AI world, assessment should focus on watching students think
- Students need boundaries for when to use AI and when to step back
- AI conversations can become assessments when students have to think visibly
- AI chat transcripts can be taught like texts
Key evidence and examples
- The author used the Claude Chrome Extension as an AI agent connected to a paid Claude account.
- The agent interacted with a Playlab pressure bot designed for a Berkeley Writing Through Literature class.
- The bot pushed users to defend literary interpretations and anchor claims in specific story moments.
- The agent completed the chatbot interaction, transferred the transcript to a Google Doc, and marked up its own metacognitive moves with very little prompting.
Education relevance
Highly relevant for assessment redesign, AI-resistant assignment design, metacognitive reflection, writing instruction, and the limits of process-based assessment in an agentic AI environment.