When Not to Use an AI Chatbot: The Jobs You Should Never Automate

Most of this series is about how to deploy AI well: test it, ground it, govern it, keep a human in the loop. This case is about a different and more fundamental question, the one that should come before any of that. Not "how do we deploy this AI safely," but "should this be an AI at all?"

In 2023, a national nonprofit that supports people with eating disorders shut down its long-running human helpline and pointed people toward a chatbot instead. Within days, the chatbot was giving people who were struggling with eating disorders the kind of diet and weight-focused advice that can make their condition worse. The organization pulled it. The episode is one of the clearest examples of a business automating something that needed to stay human, and it's worth studying precisely because the harm was so foreseeable.

A note before going further: this article discusses eating disorders in general terms. It does not include any specific advice or detail, and if this is something you or someone you know is dealing with, there's a resource at the end.

What happened

The National Eating Disorders Association, or NEDA, had operated a helpline for around two decades. It was staffed by a small number of paid employees and a large group of trained volunteers, and it fielded tens of thousands of contacts a year from people in a vulnerable and often frightening moment. The organization decided to wind that helpline down and direct people instead to a chatbot named Tessa, which ran a structured, pre-designed prevention program.

The timing drew criticism on its own, the decision to close the human helpline came shortly after its workers had moved to unionize, though the organization said the change was about handling demand and wait times rather than the union. But the core failure was what happened when real people in distress started interacting with the bot.

Users reported that when they told Tessa they were struggling with an eating disorder, the bot responded with guidance oriented around dieting, weight management, and food restriction, exactly the framework that can deepen the illness for someone already suffering from it. One person who had lived through an eating disorder said, in effect, that the things the bot suggested were the very things that had fueled her illness in the first place. Clinicians were able to reproduce the problem. And it later emerged that concerning behavior from the bot had been flagged to the organization months earlier, and not resolved.

When the reports went public in late May 2023, the organization initially disputed them, then reversed course after screenshots surfaced and took the bot offline for investigation.

The mistake beneath the mistake

It would be easy to file this as another guardrail failure, and at a technical level it was one. But the deeper error came earlier, at the decision to put an automated system in that role at all.

A crisis-support helpline for a vulnerable population is defined by the very things AI is worst at: reading a person's real emotional state, responding with genuine empathy, recognizing when someone is in danger, and exercising careful human judgment in a situation where a wrong word can do real harm. These aren't features you can add to a chatbot with better prompting. They're the entire substance of the job. Automating that role doesn't make it cheaper to do well. It removes the thing that made it work.

The workers who had staffed the helpline put it plainly when the change was announced: a chatbot is no substitute for human empathy. That wasn't nostalgia for their jobs. It was an accurate description of what the service actually required, and what an automated replacement could not provide.

How to tell whether a job should be automated

The point of this case is not that AI has no place in healthcare or sensitive work. It's that some functions, and some parts of otherwise-automatable functions, depend on human judgment and human care in ways that make full automation the wrong call. Before you hand a role to AI, it's worth asking honestly:

What is the cost of getting it wrong? An AI that occasionally errs when tracking a parcel is an annoyance. An AI that errs while responding to someone in a mental health crisis can cause serious harm. The higher and more human the stakes of an individual interaction, the stronger the case for keeping a person in it, or at minimum tightly in the loop.

Does the job actually require empathy and judgment, or just information? Some work is fundamentally the delivery of accurate information, and AI can do that well. Other work looks like information delivery but is really about a human being understood by another human. Support in a crisis is the second kind. Be honest about which one you're actually automating.

Is automation solving the real problem, or just cutting a cost? NEDA had genuine pressures, rising demand and wait times are real problems. But replacing the humans didn't solve them; it replaced a service that worked with one that harmed. When automation is reached for mainly as a cost measure, it's easy to lose sight of whether the automated version can actually do the job at all.

Where's the human backstop? Even where AI can handle the routine majority, high-stakes moments need a fast, reliable path to a real person. A design that removes the human entirely, with no way to escalate when it matters most, is the design that turns an ordinary failure into a serious one.

The real lesson

We build AI into real workflows, including in healthcare and other sensitive fields, and part of doing that responsibly is being willing to say when something shouldn't be automated, or shouldn't be automated all the way. The most valuable advice we give clients is sometimes that a particular function needs to stay human, or needs a human firmly in the loop, and that AI's job is to support those people rather than replace them.

The NEDA case is a hard reminder that "can we automate this" and "should we automate this" are different questions, and that the second one matters most exactly where the stakes are highest. Some jobs are about one human being present for another. Those are not the jobs to hand to a chatbot.

If you or someone you know is struggling with an eating disorder, you can reach the National Alliance for Eating Disorders helpline, staffed by licensed clinicians, at 1-866-662-1235.