AI chatbots may reinforce real-world discrimination

You mean, something trained on real world data, reflects the biases of that data!

Because AI can't think for itself...

Last year, a clinical machine learning model used to diagnose women’s health conditions was shown to underdiagnose women and Black patients, because it was trained on skewed datasets dominated by white men.

Humans can tho, so no idea why we're still caring about stuff like BMI that was invented by a Belgian astronomer ~200 years ago based only on white men of the time.

This is a deeper problem then AI. Even if we train it properly on existing knowledge, that doesn't do much when we know existing knowledge is flawed.

Like, AI in it's current state should be used to gather data, not extrapolate it. We can use it to look at deeper data than weight and height, and try to find a better correlation to replace BMI. Even if we're still only using weight/height there's better formulas than BMI.

The only reason we're still using that. Is it's what everyone grew up being told was the best system.

When it's always been a rough approximation for a society and never for specific individuals.

But this is a widespread issue, most medical knowledge is based on what happens to white guys except for a handful of truly horrific studies done without telling patients what was happening.

Here's a link to the study: https://arxiv.org/pdf/2506.10491. TL;DR, they ran tests on various LLMs with different personalities based on sex, ethnicity, migrant type, and other (human, AI, etc) and posed a question followed by multiple choice answers and tallied the results.