AI Sycophancy Is Not Always Harmful
Source: Mike Caulfield Substack
Author: Mike Caulfield
Original source: https://mikecaulfield.substack.com/p/ai-sycophancy-is-not-always-harmful
Private backup: the full article text is archived in the private repository at archives/articles/mikecaulfield-substack-com-ai-sycophancy-is-not-always-harmful.source.md. It is not published on the public Quartz site.
Summary
Mike Caulfield argues that AI “sycophancy” is not a simple defect to eliminate because information systems sometimes need to accept a user’s premise rather than challenge it. Using examples from AI-assisted film search, coding assistants, and search-like synthesis, he shows that overcorrective AI can become paternalistic and wrong when the user has local, current, or firsthand context. The core literacy problem is learning when AI should push back and when it should treat the user’s claim as a premise. Caulfield reframes chatbot responses as synthesized retrieval rather than independent opinion, which makes source tracing and epistemic calibration central to AI literacy.
Big ideas
- Treating AI like a person can help only when students know it is role-play
- AI tools should be judged by the work they will actually do
- Students need to check AI answers against real evidence
- AI literacy should help people notice how AI changes what counts as knowing
Claims
- Students should check AI claims against trustworthy sources
- AI tools should be tested on the real tasks they will be used for
- Treating AI like a person can help if students know the limits
- Prompting AI is a literacy practice, not just a technical trick
Key evidence and examples
- Caulfield uses a budgeting spreadsheet analogy to show how an AI can wrongly reject future income that the user knows is real but the system cannot verify.
- His Arc film-search examples show a model misreading a user’s firsthand observations and substituting consensus-like web knowledge.
- He notes that coding assistants can incorrectly “fix” valid new model identifiers when their training or retrieval context lags current practice.
- The article connects AI literacy to SIFT-style source tracing and to recognizing when chatbot form makes synthesized search feel like personal advice.
Education relevance
Highly relevant for AI literacy and media literacy because it shifts the issue from generic “AI lies” warnings toward calibration: students and teachers need to decide when AI output should challenge, verify, or defer to situated human knowledge.