This morning, I had an interaction that so clearly illustrated the risks of relying on AI that I had to share it.
As part of our content migration, we want to add a feature that requires a third-party product our company already owns. I know we’re migrating to a different version of the third-party product soon, so before submitting the request, I checked whether the version we’re moving to supports the feature.
The vendor documentation said it does. So, I submitted the request.
However, the application owner told me that the new version doesn’t support the feature.
I respectfully asked them to verify that information and shared the vendor documentation I used to research my request.
They checked again and told me that the AI assistant provided a different answer this time.
Then, they gave me information about the considerations for enabling the feature. We have more exploration and work to do, but the interaction reinforced a concern I have about AI.
This was a small but important example. AI told someone that a feature didn’t exist, and that information was initially accepted as fact.
Whenever I talk to someone about AI, I tell them I use it a lot, with the encouragement of our organization. But I also emphasize the importance of verifying the information it provides.
This is why our documentation team advocates for our documentation to be the authoritative source of information. It’s why I checked the vendor documentation before submitting my request. It’s why I asked the application owner to verify the answer when I was told the feature wasn’t available.
AI can be incredibly helpful. It can make research faster, help us explore possibilities, and point us in directions we might not have considered. But AI can also be wrong, confidently wrong. Although AI can help us find answers, human judgment and verification are essential to determining whether those answers are accurate.
Leave a Reply