WSU says don’t always trust ChatGPT for the right science answer

Artificial intelligence can be fun to play with, but is it a rock-solid source for accurate information? Apparently not always. ChatGPT is out there for everybody to access, ask questions and give it things to do. But please keep in mind that all AI, including ChatGPT, are evolving. They're learning more, and some of the things they learn aren't necessarily accurate. 

It's really quite remarkable how many of us rely on systems like ChatGPT to give us the information we need to do our jobs. Right now, I'm using an AI to help me with my spelling and punctuation. As you can see, it doesn't always work well. 

What do we know now?

Now there's some new information out there, and it's not created by artificial intelligence, that says that you cannot necessarily rely on ChatGPT for accurate information when it comes to science. 

According to komonews.com,
‘A new study led by a Washington State University professor found that ChatGPT can be both inaccurate and inconsistent when asked to judge whether scientific hypotheses are true or false. 
Washington State University associate professor Mesut Cicek and his colleagues tested ChatGPT by feeding it hypotheses from scientific papers and asking whether research had upheld each statement.’ 

What the research team actually did was create about 700 different hypotheses. Then they took those hypotheses, presented them to ChatGPT, and asked if it was true or false. They did this 10 times for each question. 

The results were interesting. 

Now you would think that the answer to each of these questions would be the same each time it was asked the question, but that is not the case. Back in 2024, they did this same Series of tests and ChatGPT was Accurate 75% of the time. That seems like a fair average, but if you're asking a yes or no question, it's pretty obvious that you would hope that you would get accuracy 100% of the time. 

In 2025, The test was run again and the percentage rose to 80% accuracy. But there's a hitch here, because when you think about a true or false question, you are either correct or not correct. So, there's a 50% chance of being correct if you guess. When you take that into account, ChatGPT was only 60% better than guessing. Not particularly reliable in any universe. 

Now this is interesting. 

It turns out ChatGPT had the biggest difficulty determining whether or not something was false. When the correct answer to the question was false, ChatGPT only got it right 16.4% of the time. 

The other interesting anomaly in this process is that the researchers asking the question 10 times found that ChatGPT was consistently accurate in its answers only 73% of the time. Think about it, if I ask you the same question 10 times in a row, wouldn't you have the same answer 10 times in a row? 

Final conclusions. 

If you're a businessperson and you're relying on AI to help you make decisions. You need to confirm that the AI knows what it's talking about. Verify its answers before you act on them and if you have staff, you should probably train them on how to deal with AI and use some skepticism when acting on its answers.  

Learn what AI can and can't do.