Today I’m launching a new blog series focused on a frustrating and underexamined part of the AI landscape: in-website and in-app chatbots.
We now have general-purpose agents like Claude Cowork and Claude Code that are genuinely impressive. They reason well, adapt to context, and often feel like real collaborators. And yet, the AI assistants embedded in the websites and apps we actually spend time and money on are, more often than not, remarkably bad. These should be the easiest wins: narrow scope, rich first-party data, clear user intent. Instead, they routinely underperform.
In this series, I’ll review these embedded agents with a critical but fair lens. When they fail, I’ll try to unpack why: model choice, agent design, retrieval quality, guardrails, or product decisions. When they succeed, I’ll explain what they’re doing right and where they still fall short of what’s technically possible today, using only publicly available information.
The first two reviews will cover the E-ZPass NY customer service chatbot, which earns a straightforward one-star, and the Monarch app assistant, which is usable and occasionally helpful but leaves significant value on the table. More reviews will follow. If there are products you think are especially bad, surprisingly good, or just interesting case studies, I’m open to suggestions.
No comments:
Post a Comment