I have been tracking competitors manually for years.
Every few weeks I’d open three or four sites, scan recent posts, take notes in a doc somewhere, and close the tab thinking I had a sense of what they were doing. I didn’t. Not really.
The problem isn’t discipline. It’s that manual monitoring can’t run in parallel. You check one site, then another, and 45 minutes later you have a loose list of topics with no priority signal and no authority check attached to them.
You spot the gap after they’ve already published. Or you commit to a topic and find out it’s already in your archive.
It’s not a research problem. It’s a system problem.
In Episode 08 of One Shot Show, Wyndo and I built a competitor intelligence agent using Claude Cowork and a single web research tool called Tavily. No terminal, no code, no API configuration beyond adding a connector in the Claude Desktop app for Claude Cowork.
By the end of this article, you’ll know
how to create an AI Agent in Claude Cowork,
set up the same three-file system for your own site,
run a weekly competitor scan that produces a priority-ranked gap report in minutes, and query the agent on-demand with questions like “should I write about X?” and,
get a full authority check alongside the gap analysis.
👋 Julley, I'm Dheeraj, an AI systems builder.
I build production-grade AI systems at work by day and ship my own products by night, 9 and counting, including SubflowAI and the Content OS Agents Toolkit. This newsletter is the bridge between those two worlds. Every system, every build, documented step by step.
Join 1,800+ builders getting the exact AI setups, prompts, and workflows that actually work in your business.
What Does a Competitor Intelligence Agent Actually Do?
A competitor intelligence agent monitors a defined competitor list on a schedule. It finds content gaps you’re not covering, then produces a ranked report with authority checks attached. This agent runs in three modes:
a weekly scan that fires four parallel searches across your competitor list
an on-demand whitespace check you trigger when a new topic idea comes up
a monthly scan mode that does a pulse-check on all competitors, prune index, surface meta-patterns, recommend 30-day calendar
The Architecture (3.5 Layers)
The agent has three layers. Understanding these before setup matters because how you fill each layer determines how good the output is.
Layer 1: MCP / Tool / Integration layer
Tavily handles all web research. It’s added through the Claude Cowork UI as a single click. No JSON configuration, no separate API key management beyond the Tavily connector auth.
Layer 1.5: Memory layer
memory.md tracks preferences and past scan results across sessions. It gets more precise over multiple scans without requiring manual instruction updates.
Layer 2: Context profile layer
Three markdown files define who you are, who you watch, and what gaps matter. These are what make the agent specific to your site, not generic to any blog. A competitor agent with weak context files produces generic output that could apply to anyone in your niche.
Layer 3: Agent Layer - Operating Instructions (Brain)
CLAUDE.md file that defines three operating modes (weekly scan, on-demand whitespace check, monthly review), the Tavily usage guide (when to use search vs extract vs crawl vs map), the memory protocol (what to write after each run and in what format), output standards (how to format gaps by priority), and cost discipline rules (max Tavily calls per scan).
AI Research Agent v/s Competitor Intelligence Agent in Cowork
That first episode produced a research agent that sharpens your angle once you’ve already chosen a topic.
This one adds the upstream layer: an agent that tells you which topics have genuine whitespace before you commit to writing anything.
Wyndo validated the tool consolidation case at 7:38:
“Tavily has multiple features for deep research, for crawling, for mapping, extracting. It simplifies your tools so you don’t have to have multiple API tools like Perplexity, Firecrawl, or Gemini.”
The previous three-tool stack (Perplexity for search, Firecrawl for crawling, Gemini as backup) is down to one. One tool, one API key, same coverage. Fewer maintenance points, fewer failure modes.
The same agent blueprint works for different purposes. The research agent helps you write a topic you already chose. The competitor agent tells you which topics to choose in the first place. Same architecture, different brain and watchlist.
What the Cowork Agent Found - Scan #1
I built this for my travel blog discoverwithdheeraj.com: an Indian Himalayas travel blog, 562 posts, 17 years of field work. The context profiles took about 30 minutes to write. Then I ran the first scan.
These are real outputs from that first run, not mockups.
Weekly Scan Output (2026-04-29, Scan #1)
### #1 · HIGH · Spiti Valley Budget Road Trip 2026 - Full Route Guide with Atal Tunnel Update
Vargis Khan's Spiti hub dominates this query but is effectively a 2020-era guide
with "2026" in the title. No Atal Tunnel timing data, no current fuel stop
locations, no 2026 prices. Pre-season planning traffic peaks in May. The window
to rank above them is now.
Triggered by: Vargis Khan - Spiti Valley Travel Guide 2026 - broad hub article,
evergreen structure, no 2026-specific operational data
- They missed: Atal Tunnel's effect on Manali-Kaza journey time, current fuel stops
at km markers (Gramphu, Losar, Kaza), 2026 dhaba prices, motorcycle-specific road
conditions for Kunzum La, honest comparison of Shimla route vs Manali route for
first-timers with a car
Our move (high priority - publish by end of May):
Title: "Spiti Valley Road Trip 2026 - Budget Guide for Car and Motorcycle
(Shimla & Manali Routes)"
Angle: 14 years on these roads, actual 2026 prices, fuel stop coordinates, Atal
Tunnel timing reality check - everything Vargis Khan's hub doesn't have
### #2 · MEDIUM · Pangi Valley Road Trip 2026 - Sach Pass Reality for Self-Drivers
[from devilonwheels.com - uncovered in our archive, fits Q2 Himachal expansion goal]
### #3 · MEDIUM · Uttarakhand Road Trip Base-Camp Guide 2026 - Drive to the Trailheads
Triggered by IndiaHikes "11 Beginner Treks in Uttarakhand" (March 28, 2026) —
trekking-only, no road trip angleThat first gap is specific. It names the competitor, explains what they missed, gives a suggested title and angle, and includes a publish timeline. That’s not a spreadsheet of keywords. That’s an editorial brief.
On-Demand Whitespace Check (solo-woman-riders-himalaya)
At 44:02, I ran a live query: “Should I write about solo women riders on Himalayan roads?”
Query: "solo woman motorcycle rider Himalaya road trip safety guide"
Coverage verdict: WHITESPACE - 0 competitors have a motorcycle-specific guide
for solo women riders
Verdict: WHITESPACE - but with a brand constraint to resolve first.
The gap is real. But your brand is built on first-hand E-E-A-T. Writing a
first-person solo women rider guide without that first-hand experience contradicts
the core brand promise.
Three options: Collaborate (co-author with a solo woman rider), Frame as
third-person practical, or Skip.
Wyndo made the contrast with paid tools sharp:
“Those competitor intelligence tools would tell you that hey, this kind of white space exists. But it is not going to tell you. hey, there is a white space but there is a constraint to resolve before you schedule it.” - Wyndo
Paid competitor intelligence SaaS shows you the gap. This agent tells you whether you’re positioned to fill it.
That second answer prevents an entire class of content mistakes: publishing mediocre articles that look like opportunities but will neither rank nor resonate because the author isn’t the right person to write them.
What the Agent Said On Its Own
The agent also flagged two monitoring improvements without me asking:
Devil On Wheels extraction failed cleanly this scan. If it fails again next scan,
consider using tavily_map on their domain to confirm the right crawl entry URL
before spending credits on extracts.
Thrillophilia returned zero editorial content two searches in a row. Consider
demoting to adjacent-only at the next monthly review if this pattern holds.It’s adjusting its own monitoring strategy based on what worked and what didn’t. That kind of signal shows up in the second and third scan, not in any template you write upfront.
Commercial break: Claude Code Builder cohort
The founding batch of my Claude Code cohort starts in July on Maven. Four weeks, six sessions. Learn to build the AI system with me.
Only 12 Seats. When they’re gone, the founding price ($797) closes and Cohort 2 opens at $1,597.
Use code GENAI20 for 20% off. Expires July 9. Check the Syllabus →
Knowledge Layer - How This Agent Gets Smarter Over Time
The memory layer is what separates Scan 1 from Scan 4.
After Scan 1, memory.md is blank except for what the agent logged: which competitors returned useful results, which searches fired duplicates, and any explicit corrections you gave it.
In the demo, the agent re-analyzed my old blog name “Devil on Wheels” as if it were a current competitor. I added a correction note. That error never appeared again.
By Scan 3 or 4, the agent starts surfacing ranked follow-up topics with notes like “don’t publish a 2026 route guide without real 2026 field data” and “run topographic map coverage check before scheduling this.” These aren’t instructions you wrote. They’re patterns the agent detected across sessions.
There’s also a calendar trick built into content-strategy.md. If you list drafts in your pipeline there, the agent skips those gaps automatically. In the weekly scan output above, the Manali-to-Leh October gap got flagged MEDIUM but skipped because it was already scheduled. That filter catches false positives without you reviewing every entry manually.
Session logs are scoped to the past 30 days by default, keeping context size manageable. Keep memory.md under 500 lines. Archive entries older than 30 days to a separate file if it starts growing.
Token Cost Trade-offs
Running this in Claude Cowork costs more tokens than CLI-based approaches. Every MCP connector trigger loads the tool schema plus search results into the context window.
Wyndo flagged this directly at 48:14:
“For MCP that you install inside of Cowork, it basically eats your token by default. And it’s going to eat your token a lot.” - Wyndo
The three-way comparison:
Claude Cowork with MCP connectors - Token cost: highest. Every trigger loads connector schema plus returned data into context. Setup effort: lowest (click to add, no JSON). Best for: getting started, weekly cadence scans.
Claude Code with MCP server - Token cost: moderate. More control over what loads into context. Setup effort: medium (requires terminal, JSON configuration). Best for: power users who want full configurability.
Direct API or CLI-based tool calls - Token cost: lowest. No schema overhead, raw tool output only. Setup effort: highest. Best for: high-frequency automated agents where cost matters at scale.
For heavy agent use, choose the subscription plan over direct API billing. Direct API billing can run approximately 20 times higher than subscription pricing for the same volume of agent work.
This is the honest conversation most Cowork tutorials skip. It came up at 48:32 when a viewer asked about differences between local file search and external integrations. The answer is: connectors are a convenience trade-off, not a cost-optimization tool.
How to Keep the Agent from Making Mistakes
Three practices matter most.
1. Plan mode as the natural review gate.
At 31:24, the agent created a visible to-do list before running any searches. That’s the plan mode review gate. Read it. If a step looks suspicious, stop execution there, annotate the issue, and restart. This intercepts problems before they compound.
2. Git on every project folder.
Version control every file in your project folder, including data files and memory.md. If the agent corrupts a file or removes a context entry you needed, git gives you a clean rollback path. This is not optional.
3. Accurate context files beat all guardrails.
Wyndo closed with this:
“As long as you have a solid data reference for the AI to make an argument, it will be less likely to happen for the hallucinations.” - Wyndo
The more specific and grounded your business-context.md, competitor-watchlist.md, and content-strategy.md, the less the agent needs to guess.
Everything Inside Competitive Research Agent Folder
So I showed you everything about what the system does. Now here’s what you’ll walk away with if you keep reading.
Here’s what the full folder looks like once it’s built:
competitor-intelligence/
├── CLAUDE.md ← the operating brain
├── memory.md ← blank to start, updates itself
├── context-profiles/
│ ├── business-context.md ← who you are
│ ├── competitor-watchlist.md ← who you watch
│ └── content-strategy.md ← what gaps matter
├── logs/
│ └── competitor-index.md ← blank, populated after first scan
├── templates/
│ ├── weekly-scan-template.md
│ ├── topic-whitespace-template.md
│ └── monthly-review-template.md
└── output/
├── weekly-scans/
│ └── _example-weekly-scan.md ← real format example
└── topic-checks/
└── _example-topic-check.md ← real format exampleHere’s what each file does and why it exists. This is the full picture before setup.
CLAUDE.md - the full operating brain. This is where the agent gets its instructions. In Cowork, you define tools with plain English here. Unlike Claude Code’s programmatic tool schemas, CLAUDE.md is where your English instructions replace JSON configuration.
context-profiles/business-context.md - identity, audience, strategic position, quarterly goals. This is the lens for every signal the agent evaluates. The more specific this file is, the more specific the gaps that only apply to you.
context-profiles/competitor-watchlist.md - direct competitors with their crawl URLs, adjacent players (forums, YouTubers, aggregators), the HIGH-gap rule (3+ competitors published on a topic in 60 days, zero equivalent in your archive), and monitoring criteria.
context-profiles/content-strategy.md - content pillars, voice rules, and the calendar trick. The agent reads this to know what to surface and what to skip. List your current draft pipeline here.
logs/competitor-index.md - starts blank. After the first scan, the agent populates this with competitor entries: what they published recently, which queries triggered coverage, and what patterns emerged. By Scan 3 it becomes a useful reference for spotting multi-scan trends without digging through individual scan outputs.
memory.md - starts blank. Updates itself when you give feedback or when the agent detects patterns across sessions. You review it periodically and remove outdated entries. Keep it under 500 lines. This is the file that makes the agent feel progressively more tuned to your site over time.
templates/weekly-scan-template.md, topic-whitespace-template.md, monthly-review-template.md - three output format templates. The agent uses these to structure its responses so every scan looks the same. Consistent format means you can scan the output quickly without re-orienting to a different layout each week.
output/weekly-scans/_example-weekly-scan.md - a real format example from an actual scan. It’s not a mockup. It’s what correct output looks like. The agent references it to calibrate format and detail level on its first run in a new session.
output/topic-checks/_example-topic-check.md - same idea for on-demand whitespace checks. Shows the agent what a properly formatted whitespace check looks like, including the E-E-A-T constraint section and the three-option framework for constrained gaps.
Fine tune context profiles, Open this folder in Claude Cowork. Type
weekly scan. That’s it.
Get PluggedIn
You have the architecture, the data flow, the cost, and the proof. The drop-in scaffold that makes it run is one click away.
Every gap your competitor agent surfaces is an article you still have to research, optimize, and fact-check. Without a system for those steps, you will spend 3-4 hours on each one. PluggedIn members run /research, /seo-research, and /verify and ship in under an hour.
How to Set Up the Claude Cowork Agent (8 Steps)
Setup takes about 30 minutes the first time. No terminal, no coding knowledge required.
Step 1: Download the full system
The full claude-cowork-competitor-agent/ folder is below. Drop it into any directory. Get Tavily API key. Edit three context files. Everything else drops in.
