ChatGPT

ChatGPT isn't human-intelligent. There's no awareness behind what it spits out -- nothing is being experienced, or was experienced, however convincing the sentences sound.

Confident writing can trick people into blindly trusting whatever was used to generate the content. A model can write genuinely useful and/or completely made up things. The realistic approach is to treat these tools as generators that need checking.