When AI Says This Quote Is Accurate
Source: Nick Potkalitsky Substack
Author: Nick Potkalitsky
Original source: https://nickpotkalitsky.substack.com/p/when-ai-says-this-quote-is-accurate
Private backup: the full article text is archived in the private repository at archives/articles/nickpotkalitsky-substack-com-when-ai-says-this-quote-is-accurate.source.md. It is not published on the public Quartz site.
Summary
Potkalitsky argues that generative AI systems are unreliable for exact quotation and citation verification because they operate through probabilistic reconstruction rather than deterministic text matching. Even when a source document is uploaded, a model may paraphrase, compress, or subtly alter language while presenting the result as a verbatim quote. Asking the same model to verify its own quotation often compounds the problem because the verification uses the same plausibility-based process. RAG systems can improve traceability but do not eliminate retrieval, chunking, and faithfulness failures, so AI literacy must distinguish semantic approximation from literal fidelity.
Big ideas
- Students need to check AI answers against real evidence
- AI literacy should help people notice how AI changes what counts as knowing
- AI tools should be judged by the work they will actually do
Claims
- AI cannot be trusted to verify exact quotes by itself
- Students should check AI claims against trustworthy sources
- AI tools should be tested on the real tasks they will be used for
Key evidence and examples
- AI may generate plausible paraphrases while presenting them as direct quotes.
- The article contrasts semantically similar phrasing with exact source language to show why “close enough” is not quotation fidelity.
- Self-verification can fail because the same model is asked to judge the plausibility of its own output.
- Potkalitsky discusses RAG failure modes including retrieval misses, chunking problems, probabilistic paraphrase, and unfaithful citations.
Education relevance
Very high for research instruction, source-based writing, citation practice, teacher workflows, student academic integrity, and professional judgment around exact textual evidence.