Former pastor claims he almost died from ChatGPT’s inaccurate medical advice

This may be evolution's survival of the fittest taking place. The fact that this guy felt that this was the right path to receiving medical advice to a serious condition is beyond me.

While in general I would agree that using AI to advise treatment over getting that advice from a doctor is pretty stupid, I’ll point out that many more people have died following a doctor’s advice than advice from AI. Doctors are hardly all-knowing, and they make mistakes. Of course, they probably make less mistakes than AI at this point.

Personally, I think it’s only a matter of time before AI’s ability to diagnose from symptoms will be in most cases superior to that of any human. That speaks to AI’s strengths (i.e. collating and evaluating data, while considering thousands more sources than any doctor will have read). However, as AI is still very poor at judgment or discernment, and I don’t foresee that really changing, I would still want human doctors to go over any AI diagnosis results when discussing treatment options, even if AI is ever considered the primary tool used by doctors for diagnosis.

....several months of "conversations" with AI, and he never once thought "hey, maybe I ought to give the nurse line a call and see what they think?" Yes, I'm with Mr. Szweda on the Darwinian implications here to a degree. I am reminded of the time I had a slight strain in my left pectoral after a kid in my church's youth group wrestled me (he lost FWIW), and the nurse put me through full heart attack triage. Well done, nurse! I'm pretty sure that if he'd called his doctors' office, he'd have undergone the same, and they'd have recommended he get a ride to the hospital, if not even a ride with blinking lights. My RN daughters note that they put people through heart attack triage routinely--it's like a reflex action.

A couple of other questions come to mind here. First, if he's trusting AI for his medical care, are the churches who employed him totally skimping on his health insurance? Second, if he's trusting AI for his medical care, I have to wonder how much of his sermons were "less than original" as well.

It’s a trust issue for sure.

I know a lot of people struggle with ‘trusting doctors.’ Maybe that was a factor. But trust is always a bounded and nuanced thing. People don’t ‘trust doctors’ to fix their cars or pick the right lawn fertilizer. And even in their sphere of expertise, we should only ‘trust doctors’ with one additional categorical constraint: ‘trust doctors to be humans who have some special expertise.’

I don’t know the details of this case but it might be possible for a team of medical professionals to make the same recommendations, depending on how good the team was.

As for AI’s, we’re all old enough to know that these things just came to be a little while ago. So, obviously…

But if the case is more about how OpenAI monetizes, he might have a point. I have no idea. I use some of their models pretty regularly at work, but only to do pretty basic repetitive tasks that are a bit quicker that way. Not for advice.

Which is a whole different thing.

On the whole, I’ve found that getting practical advice from AI’s has been a very spotty thing. Why is the grass dying on my lawn over where that stump was removed and what can I do about it? Well, they’ll tell you one thing one day and maybe the opposite the next. AI’s tend to not “feel obligated” to provide continuity! There’s a whole lot of random in the calculations, seems like. They forget context quite a lot. And they aren’t all the same. Some hold on to context a lot better and are way better at self-checking for appropriate levels of certainty and self-correction. Pushing them a lot in that direction in whatever your model uses for standing orders is a good idea.

AI is only as good as the prompt and the context. The better both of these are, the better AI will produce a result. It is highly doubtful that this individual was trained enough to feed it the right level of information. Just asking AI a question is not good enough. This would be no different than going to a doctor and giving them one sentence and asking them to give you a diagnosis and a course of action. They wouldn't do it. In addition, you have to understand enough about AI, to understand drift and other elements to ensure that the output was proper. Again, doubtful that this individual understood those concepts or how to structure the information appropriately to generate the right kind of output. Then layer on the fact that the individual is taking a single piece of information only, on top of the fact that the information was flawed just by looking at the surface. This points to a guy who is not that bright, in my opinion to either understand what he is doing or understand the limitations of what he was doing.