Robbie Allen @ Tweener Claw: "3 to 5 of My Agents Fail Every Day: Here's What I've Learned"
Robbie shares three hard-won lessons from 60 days building and running AI agents in production: skill stacking, agent monitoring, and using LLMs to check each other's work
Instead of a traditional interview, we're sharing a live talk delivered by Robbie Allen, who is a General Partner at NC Tweener Fund and Managing Director of Automated Consulting Group, at the June 10th Tweener Club meetup in Research Triangle Park. Robbie has spent the last year and a half deploying AI agents inside real mid-market companies, and he came armed with three lessons he's learned in just the last 60 days. Less theory, more field report, and a few things that might surprise you.
We couldn’t bring you 100% original founder-to-founder content like this without the support of our awesome-sauce sponsors:
Loving our content? It wouldn’t be possible without our amazing sponsors and our paid subscribers, so THANK YOU!
Featured Platinum Sponsor
Featured Gold Sponsor
Featured Silver Sponsor
And special thanks to our friends at Walk West for partnering with us to produce this pod!
Summary
Robbie Allen, the Founder and Managing Director of Automated Consulting Group and General Partner at NC Tweener Fund, gave a live talk at the June 10th Tweener Club meetup in Research Triangle Park. In it, he shares three lessons from 60 days of building and running AI agents in production with real clients across the Triangle. The talk covers skill building (mapping your own workflows into reusable AI packages), agent monitoring (why 3–5 of his 25 agents fail on any given day and how he built a monitor agent to catch it), and using LLMs to check each other's work (his "Codex Opinion" and "Gemini Opinion" skills, and an LLM Council that synthesizes outputs from multiple models). Robbie is about as far out on the cutting edge of real-world agent deployment as anyone in the Triangle, and he's sharing what's actually working, what keeps breaking, and what he'd wire in from the start.
Stick around for the highlights below. 👇
Highlights
“The sawdust of business”: Robbie’s term for meeting transcripts: raw, unstructured data that reveals how a company actually thinks and operates. He argues that as AI approaches superhuman capability, organizations will need this context to train and ground models, yet most companies aren’t systematically collecting it.
Skills as reusable modules: He treats AI skills like software packages: reusable blocks of functionality that can contain prompts, tools, or both. His approach is to map a human workflow and then build skills to automate each step.
A practical skills pipeline: Robbie created a three-skill workflow for meeting follow-up:
Detect new Otter transcripts
Extract action items
Run a weekly review of unfinished tasks
The entire system uses composable skills rather than custom-built software.
Agent reliability remains a challenge: In his setup of 25 agents, 3–5 fail every day, resulting in a 12–20% daily failure rate. Common causes include API outages, LinkedIn scraping restrictions, and dependency failures. He argues that claims of running hundreds of agents without a support team are unrealistic.
“Build costs are down, support costs aren’t”: AI has dramatically reduced development costs, but maintenance remains expensive. Non-deterministic systems introduce new forms of brittleness, making operational support just as important as ever.
Monitor agents need monitors: To manage failures, Robbie built a 26th agent dedicated to monitoring the other 25. His takeaway: observability should be designed in from day one, with comprehensive logging across the system.
“Einstein and Feynman” multi-LLM strategy: Robbie compares using multiple AI models to consulting multiple brilliant scientists. He:
Built a “Codex Opinion” skill to critique Claude Code sessions
Added a “Gemini Opinion” skill for additional perspectives
Created an “LLM Council” that runs Sonnet, Gemini, and Codex in parallel, with a leader model synthesizing the results
He reports that the additional models consistently uncover blind spots.
Skills marketplace security is unresolved: Robbie highlights a largely overlooked risk: downloaded skills are effectively third-party code running in your environment. He expects enterprise security teams to scrutinize this area heavily in the coming years.
Microsoft Copilot skepticism: Based on experience with Microsoft-centric clients, Robbie believes Copilot’s agent environment still lags behind expectations despite Microsoft’s early access to OpenAI models. He remains cautious about announcements until features reach general availability (GA).
Every practitioner has lessons they've learned the hard way. Robbie's just willing to share his before the bruises fade. Enjoy the conversation.
Visit Tweener Talks to listen for all audio podcast players
We have a video version on YouTube (make sure to subscribe!)
📌 What’s Next
Subscribe to NC Tweener Talks for more conversations with North Carolina founders, operators, and builders shaping the future of the ecosystem. Paid subscribers help us keep creating more episodes, events, and founder-focused content across the state.