An eval tells you which outputs are weak. The step Jeremy MacDonald called particularly clever is using that information to improve the work that produced them. In his BrightonSEO talk it was the third and last takeaway: create a feedback loop. (Part four of my series. Last time: build your first agentic eval with the RISE framework.)
How the loop works
- Generate with a RISE prompt, for example keyword research from a brand PDF and product details.
- Evaluate each output with a second prompt that scores it against named criteria.
- Analyze the failures. A third prompt, with the role of an AI solution architect, reads the graded outputs and looks at the low scorers to understand why each one failed.
- Write the learnings. It produces a short set of lessons, such as which kinds of keywords keep scoring poorly on brand fit.
- Feed them back. Add the lessons to the original brief as constraints, and add the weak outputs as examples of what to avoid.
- Run again. The next generation starts from a better brief.
If you build this as a reusable skill, Jeremy said, it will learn and learn. Essentially, the skill teaches itself. His blunt view was that teams who aren’t working this way are falling behind.
What to put in the feedback
- Patterns, not single items. “Keywords with competitor names score low on brand fit” is useful; one bad keyword isn’t.
- Both sides of the scale. Include what scored highest so the brief reinforces good output.
- Missing examples. If the analysis shows the original prompt had no examples of a case, add some. Jeremy noted that this is the stage where gaps in the examples show up.
Guardrails
These guardrails are my additions, not points Jeremy made. A self-improving loop can also drift. Don’t let it rewrite itself unattended:
- Version your brief so you can roll back a change that made scores worse.
- Keep a frozen test set of inputs you re-run after every change, to confirm that the score gains are real.
- Review proposed changes before they go live, particularly anything that touches compliance or brand rules.
- Watch for the eval being gamed: if scores climb while a human reviewer sees no improvement, the grader needs work. That’s the subject of the next post in this series.
This is also why I think senior people matter more, not less, as AI takes on more of the work: someone has to know when the loop is drifting.
Where to apply it
Jeremy’s examples went beyond keywords. The same pattern works for content briefs, support-ticket analysis and any multi-step process where each step can be checked. His former team used to finish work and hand it to him. A looped system checks every step on the way instead of only at the end.
Want help turning a one-off prompt into a workflow that improves itself? Get in touch.
Leave a Reply