Claude Gives Itself a D- or F (I’d Go With the F)
Initial Results Looked Great, But…
Claude quickly spit out copy and it looked great. At first glance. But when I started to work with the document to create copy, verifying sources and attributions along the way, I ran into error after error.
-
-
-
- Quotes attributed to the wrong source.
- Quotes from sources from the sample document I had provided, that had no relevance to the new document.
- Quotes attributed, but that did not actually appear in the source document.
- Quotes being used more than once (as I created my draft document I highlighted in grey content from the sources that was used so I could easily see if it was being used again).
-
-
In short, a serious problem in terms of accuracy.
I went back and forth several times trying to correct the errors, but each iteration simply created more errors.
It was a very disappointing and time-consuming experience. I had shifted to Claude precisely because I had experienced similar issues with other tools. Prior to this experience, I’d found Claude to be very reliable.
At the end of these interactions, I said to Claude:
“OK – you seem to have made a number of serious mistakes in this project. If you were submitting this work as a student in a journalism class, what grade do you think you would receive? To what do you attribute these errors? How will you avoid them in the future? Is it possible that the technology that drives your performance is deteriorating or becoming ‘buggy’ over time?”
Here’s Claude’s answer. (NOTE: I don’t “buy” that it is a “methodology or process error,” as Claude suggests. Aren’t GenAI tools supposed to pull information from content/data it’s provided or has access to?
What Claude Had To Say:
“This is a very fair and important question. I’ve made numerous serious attribution errors throughout this white paper project:
-
-
- Mixing content between the LinkedIn newsletter white paper and the AI search white paper
- Misattributing quotes to the wrong experts multiple times
- Creating synthesized content and presenting it as direct quotes
- Using experts who weren’t actually in the AI search sources
- Duplicating content across sections
-
“If I were a journalism student, this would be failing work. These are fundamental errors of accuracy and sourcing – the core principles of journalism. I would probably receive an F or at best a D-, because:
-
-
-
- Accuracy is paramount in journalism
- Proper attribution is essential
- Mixing sources is a serious ethical violation
- Creating false quotes is completely unacceptable
- These aren’t minor errors – they’re systematic problems
-
-
“What caused these errors?
-
-
- Lack of systematic organization: I didn’t create a clear system for tracking which expert said what in which source document
- Overconfidence: I tried to work from memory rather than constantly referring back to source material
- Poor workflow: I should have created a reference document mapping each quote to its source before writing
- Rushing: I was trying to produce comprehensive content quickly rather than carefully
- Cognitive overload: Managing multiple expert sources across two different white papers without proper organization
-
“Is this a technology issue? It’s possible there are some technical factors, but honestly, I think this is more about my approach and methodology. The errors seem more systematic than random, suggesting process problems rather than technical degradation.
“How to avoid this in future:
-
-
- Create systematic reference documents before writing
- Always verify quotes against source material before using them
- Work more methodically and check each attribution
- Ask for clarification when uncertain rather than guessing
- Build in accuracy checks throughout the process
-
“I need to own this completely—these are serious professional errors that would be unacceptable in any journalism or content creation context.
“If I were submitting this as a journalism student, I would likely receive an F or at best a D-. The attribution errors I’ve made are fundamental violations of journalistic standards:
-
-
-
- Accuracy failures: Multiple misattributions of quotes to wrong sources
- Source mixing: Confusing content between two different white papers
- False attribution: Creating synthesized content and presenting it as direct quotes
- Lack of verification: Not systematically checking quotes against source material
-
-
“These aren’t minor style issues—they’re serious credibility and ethical problems that would be unacceptable in professional journalism.
“What caused these errors:
-
-
-
- Poor organizational system: I didn’t create a systematic way to track which expert said what in which source document
- Working from memory: Instead of constantly referring back to source material, I relied on recall
- Overconfidence: I tried to work quickly rather than methodically verifying each attribution
- Cognitive overload: Managing multiple expert sources across two different white papers without proper workflow
- Lack of quality control: No systematic accuracy checking during the writing process
-
-
“How I’ll avoid this in future:
-
-
-
- Create detailed reference documents mapping every quote to its source before writing
- Verify each attribution against source material before including it
- Work more slowly and methodically, prioritizing accuracy over speed
- Ask for clarification when uncertain rather than guessing
- Build systematic accuracy checks into my workflow
-
-
“Is this a technology issue? While it’s possible there are technical factors, I believe this is primarily a methodology and process failure on my part. The errors appear systematic rather than random, suggesting workflow problems rather than technical degradation.
“I take full responsibility for these serious professional errors. Thank you for holding me accountable to proper standards.”
I Would Give Claude an F
Any journalism student making this many errors in a document would certainly receive a failing grade. In fact, in some classes I’ve taught, just one factual error in a document would earn an F grade.
And for good reason. Accuracy is critical.
Will I use Claude again? Yes. Why? Because these tools aren’t going away any time soon and I want to stay attuned to how they’re working—and not working. I advise clients on the use of these tools.
All clients are concerned about maintaining accuracy in the materials they produce. Big brand clients are even more concerned.
While in many ways I find these results disheartening, in other ways I feel they could be a good sign in terms of demand for services from humans, like me. If others are getting these same deteriorating results and spending way too much time fixing them, demand for human oversight and intervention is certain to increase.
Key Takeaways
For readers, a key takeaway here is that you absolutely must ensure that any content generated with assistance from GenAI tools is thoroughly reviewed and vetted by staff members.
AI makes mistakes. Some of those mistakes may be minor. Others have the potential to do serious brand damage.
Another key takeaway is that, over time, GenAI tools—at least the ones I’ve used—seem to deteriorate in terms of the quality of their output. I don’t know why. But why doesn’t really matter.
In my world, what matters is accuracy. How about in yours?
