20 real engineering tasks over a month, from a Notion-to-Sheets sync to debugging, handed to Devin, the autonomous AI engineer. It closed with 3 successes, 14 failures and 3 inconclusive results.
Done withDevinSpotted· by Answer.AI team· Oct 20, 2026
Result It once spent over a day chasing an impossible task
“Out of 20 tasks, we had 14 failures, 3 successes (including our 2 initial ones), and 3 inconclusive results.”
Spotted: found in the wild. Details are as reported by the source above and have not been independently verified.
What they did
A team at Answer.AI tested Devin, an autonomous AI software engineer that works through Slack with its own sandboxed computer, shell, browser and code editor. Over a month they gave it 20 real tasks across four categories: creating new projects, research, analyzing existing code, and modifying existing projects.
How it went
Devin succeeded on 3 tasks, including pulling data from Notion into Google Sheets and building a planet tracker, but failed on 14 and left 3 inconclusive, often producing overly complex or unusable code, or pressing ahead on tasks that were not actually possible, like deploying multiple apps to one Railway deployment. One team member spent several hours on a feature Devin could not finish before writing it themselves in about 90 minutes.
Try it yourself with Devin
Before adopting an autonomous coding agent like Devin for your team, run a structured month-long trial: give it a real mix of at least [number] tasks across creating new projects, research, analyzing existing code and modifying existing code, document each attempt's outcome as success, failure or inconclusive, and have multiple team members try it to see whether failures follow a predictable pattern.
A starting prompt written by Agents in the Wild. Fill in the [brackets], and check what the agent does before it sends, pays for or deletes anything.
A client's CRM needed a new deals feature. Devin opened a first pull request in under 10 minutes, but its PRs carried unneeded changes, it gave a false answer about a library, and it hit its usage limit.
Engineering managers needed more QA, reliability and DevOps capacity than their teams had. Litera added Devin agents in those roles, and test coverage rose 40%, according to Cognition.
After $45 in GitHub Actions overage in one month, a solo founder put an idle homelab to work as CI runners, with Claude making it easy to set up. Daily spend dropped from $2-4 a day to under $0.25.
A first homemade synth has 36 hall-effect keys and buttons and 7 rotary encoders. The firmware started by hand, then gentle prodding at Claude got everything needed much faster. Rev 2 of the PCB is nearly done.
Markdown files should just open, so they asked Claude Code for a viewer. It opens instantly: faster, they say, than any other macOS app, and faster than Bitwarden's Chrome extension shows its fill menu.
Migrating a password reset flow, Claude Code and Opus produced code on par with a mid level developer. It needed close supervision throughout, and cost roughly 2 months of Pro subscription for about half a day's work.