Fifty-one testers killed a feature. Hodman | How To Build With AI almost shipped it again anyway.
That is the story that anchors this whole piece, and it is the reason I wanted to sit down with Hodman, who writes two Substack newsletters, "Between Thinking and Doing" and "The Data Letter."
Hodman is a solo founder running her own AI company, Asaura AI, which means she is also her own product team, her own QA department, and her own worst source of confirmation bias.
Most solo founders don't lack ideas. They lack anyone in the room who'll say no.
👋 Julley, I'm Dheeraj, an AI systems builder.
I build production-grade AI systems at work by day and ship my own products by night, 9 and counting, including SubflowAI and the Content OS Agents Toolkit. This newsletter is the bridge between those two worlds. Every system, every build, documented step by step.
Join 2,000+ builders getting the exact AI setups, prompts, and workflows that actually work in your business.
She built a Claude Projects system to fix that last part, and on a recent Substack Live, Wyndo and I walked through exactly how it works.
By the end of this, you'll have the core system for free:
The actual project instructions
The four-file structure that gives Claude something real to argue with
The exact three-part prompt sequence
The one field in her intake template that does most of the work
I wanted this breakdown because most of us building alone don't have a co-founder to say "wait, didn't we try this already?" We have our own excitement, a study we read once, and a launch button.
Hodman's system replaces the missing co-founder with a Claude Project that has been explicitly told not to be nice to you.
Why Does AI Default to Agreeing With You?
AI models default to agreeing with you, every single time you ask. That's not a flaw you can prompt around once and forget. It's a standing bias that has to be countered every time you ask an AI to evaluate your own idea, not just someone else's.
"AI in nature is sycophantic, meaning its default is to agree with you and make you feel good about your decisions." - Hodman Murad
Frontier labs have acknowledged this openly, and it isn't a secret failure mode. It's a known default that has to be countered explicitly and repeated, not assumed.
This matters more for solo founders than for teams. A team has a skeptical cofounder, a blunt engineer, or an investor asking uncomfortable questions before a feature ships. A solo founder has whoever they're talking to.
If that's an AI defaulting to encouragement, every decision gets a rubber stamp instead of a stress test.
The Feature That Proved the Problem
Hodman's case study is the priming overlay: a breathing exercise that appeared before users started a task in her app. Her reasoning felt solid at the time, backed by a study she remembered and her own lived experience with the practice.
She had read a study out of India linking yoga and meditation to better ADHD outcomes, and breathwork had helped her own ADHD and anxiety for years.
She shipped it to 51 beta testers. Within a week, it was dead. Users weren't confused about the science, they were annoyed by the friction.
"I already tried meditation apps. That's not why I'm here." - a beta tester, months before the feature even existed
That line was sitting in her feedback log the whole time. Nobody had connected it to the roadmap.
A study plus personal conviction felt like evidence. It wasn't user evidence, and that gap is exactly what her system now exists to catch before code gets written, not after 51 people confirm it for you.
If you’re the only person checking your own product calls, the real cost isn’t the bad decision, it’s the weeks of build time spent before you find out.
That’s the gap this system is built to close, and it’s the same gap I try to close for paying readers inside PluggedIn, where I share the actual working prompts and project setups I run myself.
How Do You Build a Claude Project That Argues Back?
A Claude Project that argues back needs two explicit instruction lines written into its system prompt: don't cheerlead, and actively surface the failure modes and dismissed positions being avoided. Naming the sycophancy problem directly in the instructions is what overrides Claude's default agreeableness.
Hodman calls this project "Decision Studio." Here is close to what she uses, adapted for anyone starting from zero:
You are my decision partner for product and business calls at [your company].
Don't cheerlead. Your job is to surface failure modes I haven't seen
and make the case for positions I have dismissed.
Before responding to any decision, check the uploaded knowledge files
for related past decisions, relevant user feedback, and known
anti-patterns. If this decision resembles one already made, say so
before doing anything else.
Never tell me a decision is good just because I want it to be. Tell
me what would have to be true for it to work, and what evidence I
already have that it isn't.Setting this up in Claude Projects takes about ten minutes: create a new project, paste instructions like the ones above, and you're already ahead of a plain chat window that forgets everything the moment you close the tab.
I run a version of the same "don't cheerlead" instruction across a few of my own recurring Claude Projects, though I only got there after I stopped juggling several one-off Projects and consolidated into one unified Claude Project instead of juggling several for my own content work.
The difference in how often I get pushed back on is not subtle, and it's part of why I run every PluggedIn build decision through a version of this before I commit to it.
Hodman’s system works because it never argues from opinion. It argues from your own documented evidence: your past decisions, your users’ actual words, and the mistakes you’ve already named. That is the only thing that lets an AI push back on a founder hard enough to matter.
What Four Files Turn a Nice Chatbot Into a Sharp One?
Four uploaded knowledge files give Claude Decision Studio the ammunition to argue with you using your own evidence instead of generic advice: a jobs-to-be-done file, a user feedback log, a decisions log, and an anti-patterns list. The "don't cheerlead" instruction has nothing to push back with otherwise.
Here's what each file does:
Jobs-to-be-done file
Best for: naming what users actually hire your product to do
Setup: one sentence per core job, written in plain language
Compounds: no, it's mostly static once written
User feedback log
Best for: dated, tagged quotes from real users
Setup: a running markdown file you add to after every support conversation or beta round
Compounds: yes, every entry makes future decisions sharper
Decisions log
Best for: recording what you decided, when, and why
Setup: one dated entry per meaningful product call
Compounds: yes, this is what caught the repeat mistake
Anti-patterns list
Best for: naming your own recurring failure modes
Setup: short, named patterns like "the works-for-me fallacy"
Compounds: yes, grows sharper with every near-miss
Hodman was direct about why this matters: "Claude can only push me this hard because I loaded evidence into the project. My anti-pattern doc, my feedback log, my decision history gave it the ammunition." The instruction sets the tone. The files supply the facts.
If you're starting from nothing, I asked Hodman the question a lot of readers will have first: how do you build framework files like jobs-to-be-done if you've never used the framework at all.
"Write down whatever you find yourself constantly repeating to AI chats about your product, your writing tone, or your own working style, and turn those repeated reminders into markdown files you upload to the project." - Hodman Murad
She was equally direct about where to start. "The two files that matter most when starting out are the jobs-to-be-done file and the user feedback log," she said, "decisions log and anti-patterns can come later." Start with two files, not four.
The other two grow on their own once you're using the system.
Learn with me: Claude Code Builder cohort
I run the Claude Code Builder cohort on Maven: four weeks, six live sessions, building a real AI system with me. If you would rather build it alongside me than read about it, this is the room.
Each cohort is a small, hands-on group, so seats are limited.
Build it with me, live. See the syllabus →
What Goes Into the Prompt Before Claude Answers?
A five-field Decision Input Template forces you to write your reasoning down before Claude ever responds. That is what makes its pushback land instead of bouncing off. The fields are: the decision, why now, the evidence for it, the evidence against it, and what happens if you're wrong.
Decision Input Template
The decision - one clear sentence, no hedging
Why now - the trigger that's making this feel urgent
Evidence for - what's actually supporting this call
Evidence against - what you'd rather not look at
What happens if I'm wrong - the real cost of being wrong
That last field is the crux of the whole method. Hodman was blunt about why: "writing 'what happens if I'm wrong' is what lets the studio push back honestly." And skipping the evidence-against field is itself a signal, not an oversight.
If you can't fill it in, that's usually because excitement is blinding you to the counter-evidence sitting in your own feedback log.
With the template filled in, Hodman runs one structured prompt that does three things in sequence:
Using my Decision Input Template below, and cross-referencing my
uploaded jobs-to-be-done, feedback log, decisions log, and
anti-patterns files:
1. PREMORTEM: List 8 ranked failure modes for this decision, most
likely first. Be specific to my actual product and users. For
each one, name the early signal I'd see if it started going
wrong and the cheapest test I could run before committing
further.
2. RED TEAM MEMO: Argue against this decision using my own user
feedback quotes as evidence, with dates.
3. DECISION MEMO: Synthesize a verdict (go / no-go / test first)
plus specific kill criteria I can check against in 30 days.
[paste your filled Decision Input Template here]Two of those eight failure modes matter more than the other six: whichever ones you paired with an early signal you can actually watch for. A failure mode with no signal attached is just a worry sitting in a list. A failure mode with a signal is a checkpoint you can carry into your next decision, whatever that decision turns out to be.
The most striking part of Hodman's live demo wasn't the premortem or the red team memo, it was what Claude did before either one. It cross-referenced her decisions log and flagged the repeat mistake unprompted:
"You are about to reship a feature you already killed four months ago for reasons your own documents predicted, using the same justification you used the first time."
Her anti-pattern doc caught the shape of the mistake by name too, calling out what she'd already labeled "the works-for-me fallacy." No human reviewer said that to her. A file she'd written herself, months earlier, did.
One of the other eight failure modes went straight at Hodman's own evidence: a study-population mismatch. Claude pointed out that the India-based study she'd cited on meditation and ADHD outcomes was run on kids in a supervised setting, not adults using a self-serve app alone.
Her own anti-pattern doc had already named that exact shape of mistake. The fix Claude suggested:
Run a 30-minute literature scan on adult, self-serve mindfulness research
Do that before committing another month of build time, not another month of instinct
The red team memo pulled dated, verbatim complaints straight from her feedback log to argue against the decision in real time. As of July 2026, this system has only been tested on Hodman's own young company, so how well it compounds over a multi-year decisions log is still unproven.
"If I had this memo six months ago, I wouldn't have shipped the overlay at all."
How Do You Close the Loop After the Memo Is Written?
Closing the loop means saving the decision memo into your decisions file with today's date, then scheduling a 30-day check-in to compare the kill criteria against what actually happened. A memo that gets written and forgotten adds nothing, the system only works if you come back to it.
"A memo you write and forget is kind of like a journal entry." - Hodman Murad
That stuck with me. The system only compounds if the outcome gets logged too, not just the decision.
An example kill criterion from her own memo:
Stop testing if task-start rate drops more than 5% over 14 days
Stop testing if negative feedback repeats across multiple testers in that window
That's a concrete, checkable line. Compare that to how the overlay actually died: 51 people had to use it and complain before anyone called it. Written kill criteria would have ended that test far sooner, with far less wasted build time.
Should a Premortem Be a Gate, Not Just a Warning?
A premortem should be a hard go/no-go gate when the cost of being wrong is high and slow to reverse, and a risk-mitigation tool when real-world testing is cheap and fast. I pushed on this directly with Hodman, because I've felt the other side of this system firsthand.
When to treat a premortem as:
A hard gate - the decision is expensive and slow to reverse
A risk list - real-world testing is cheap and fast
When I first started running premortems on my own launches, they scared me out of shipping almost anything. Every idea has eight plausible ways to fail if you ask hard enough, and treating all eight as blockers is its own kind of paralysis, not rigor.
My own use case has shifted toward using the same premortem output to identify risks I can proactively mitigate, rather than treating it as a strict stop sign.
Part of that shift is practical: AI-accelerated A/B testing has made a lot of real-world tests cheap enough that you can just run them instead of arguing about them in a document.
A premortem's job is to make risk visible, not to make you afraid to ship.
A premortem that always says no isn't a sharper tool, it's a misused one. The goal is better decisions, not zero decisions.
Hodman's rigid five-field template is also not a permanent requirement. Once your project files are loaded with your product's context, decisions, and feedback, you can chat with the project unstructured and it still applies the same scrutiny.
The structured template is a teaching tool for building the habit, not a lifetime requirement once the habit exists.
Templates teach the habit. Once the habit sticks, the habit is the system.
We also touched on where this system goes next: instead of copy-pasting the three-part prompt every time, the same method can live as a standing Claude Skill you invoke by name.
I run something similar myself: I went through the process to package it as a reusable Claude Skill installed at the account level so I don't have to remember the exact wording every time a decision comes up.
Why Use a Claude Project Instead of Just Chatting?
A Claude Project persists your context so you never rebuild the same background information every time you want to run a new decision through the system. A plain chat forgets everything the moment you close it, which means every decision starts from zero.
An audience member asked this directly during the live: why use a Project instead of just chatting?
Persistence is the entire value. That was the answer, from both of us.
One project can host unlimited decision-specific chats. Each one shares the same base:
Jobs-to-be-done
Feedback log
Decisions log
Anti-patterns list
You start a new chat per decision rather than letting one long thread collapse under its own weight, and every new chat still has full context without you typing a word of setup, which is also why a plain chat session burns through context and forgets everything once it runs long enough.
Hodman also drew a sharp comparison: baking these instructions permanently into a project's settings is functionally similar to how an AGENTS.md file keeps a coding agent from repeating context, a standing set of rules a coding agent reads before doing anything else.
The parallel holds because the underlying need is the same: don't make me repeat myself, and don't let me forget what I already decided.
Get PluggedIn
Catch the mistake in your own evidence before 51 users have to find it for you.
Every week you skip a system like this is a week you risk repeating the 51-tester kill cycle Hodman lived through firsthand, weeks of build time spent on a decision your own documents already argued against.
Get PluggedIn to go from “shipping on conviction” to catching the repeat mistake before you write a line of code.
Frequently Asked Questions
What are the two files I should build first for this system?
Start with a jobs-to-be-done file, one sentence per job users actually hire your product to do, and a running user feedback log with dated, tagged quotes. These two give Claude enough context to catch bad-fit failure modes.
The decisions log and anti-patterns list can be added later and grow in value with every entry.
Why do you have to tell Claude not to agree with you?
AI defaults to being sycophantic, meaning it tends to agree with you and validate your decisions unless told otherwise. This is an acknowledged default behavior, not a rare glitch. Explicitly instructing it to surface failure modes and argue dismissed positions overrides that default.
What goes in the Decision Input Template?
Five fields: the decision in one sentence, why you're making it now, the evidence for it, the evidence against it, and what happens if you're wrong. The last field matters most because it forces honesty about the real cost of being wrong before Claude ever responds.
Should a premortem always end in a hard no?
Not necessarily. One approach uses it as a strict go/no-go gate before shipping. Another uses the same premortem output purely to surface risks worth mitigating, while still moving fast, since AI-accelerated testing has made some real-world tests cheap enough to just run instead of arguing about them on paper.
Why use a Claude Project instead of a regular chat?
Projects persist context across every conversation, so you don't rebuild the same background each time you want to run a new decision through the system. Each decision can live as its own chat inside one project, all sharing the same base files.
Key Takeaways
Name the sycophancy problem out loud. Two lines telling Claude not to cheerlead and to surface failure modes is what actually changes its behavior.
Start with two files, not four. A jobs-to-be-done file and a feedback log are enough to begin; the decisions log and anti-patterns list grow on their own.
Write down what happens if you're wrong. That single field in the intake template is what lets the system push back with honesty instead of encouragement.
Pair every failure mode with a signal, not just a label. A ranked list of eight ways to fail only becomes useful once each one has an early sign you can watch for and a cheap test you can run before you build anything.
Let Claude check your decisions log before it answers. The repeat-mistake catch only works if past decisions are logged somewhere Claude can read them.
Save every memo with a date. A decision memo you never revisit is a journal entry, not a system.
Don't let a premortem become a permanent no machine. Use it as a gate for expensive, slow-to-reverse decisions and as a risk list for cheap, fast-to-test ones.
Package the method as a skill once the habit sticks. The rigid five-field template is a training wheel, not a lifetime requirement.
Your 15-Minute Challenge
Open Claude and try it yourself:
Start a new project and paste in the two instruction lines from this article: don't cheerlead, surface failure modes and dismissed positions.
Write one paragraph for your jobs-to-be-done file.
Add three real quotes to a new feedback log, pulled straight from your last round of user messages or reviews.
Success looks like this: run one real decision you're currently sitting on through the project, using just those two files, and see whether Claude flags anything you'd been quietly avoiding. If it does, you've already gotten more value than a week of beta testing would have given you for free.
Next up: I’m running my own decisions through a version of this Decision Studio for the next few weeks before writing about what breaks when the files pile up. If you build yours first, tell me what it catches.
Connect with Hodman
Hodman | How To Build With AI writes "Between Thinking and Doing" and "The Data Letter" on Substack if you want to go deeper on structured AI systems for founders and operators.
