Critique of Mention Tracking Tools in AI Responses

Pedro Dias is right:

Tracking prompts. Visibility dashboards in LLM. Mention share metrics in responses. In less than 18 months, a whole category of providers has emerged offering you a metric that shows how often your brand appears in AI-generated responses. This is the Search Console for the chatbot era, and it carries the same comforting message: if the metric is growing, you are winning. If it is falling, buy more of what drives its growth.

I have written about this before, and I will say it again: these tools sell you nonsense with a pencil-drawn confidence interval. When the dashboard reports that your brand “appeared in 73% of relevant AI responses,” it is actually measuring the following: “we sent a few requests to the API, got a few results, and counted the mentions.” This is not ranking. This is a lottery ticket.

The engineers who created these models cannot fully explain why a particular result appeared. But certainly, the SaaS tool sitting atop the Dunning-Kruger mountain with a trend line understands all of this perfectly.

The industry continues to buy because the alternative is to admit that we are operating in the dark. Questioning the data means telling everyone that the “curated” graphs in client presentations are noise disguised as valuable information. No one wants to be that person. So providers keep selling, dashboards keep flashing, and the numbers do not necessarily have to correlate with revenue. They just need to fluctuate enough to maintain the subscription.