When product engagement metrics lie
How to think about engagement in this new AI era
There’s a metric that product teams love to celebrate: engagement.
More messages, longer sessions, higher usage. If the numbers go up, the product is working.
In traditional SaaS, that’s mostly true. A user who spends more time in your app is getting more value from it. More clicks, more actions, more workflows completed. Engagement tracks with success because the product is designed around discrete tasks with clear outcomes.
AI products flip that logic.
Picture two customers using the same AI feature. Both started with the same goal.
Customer A sends 8 messages, gets what they need, and leaves.
Customer B sends 45 messages, keeps rephrasing the same request, gets increasingly frustrated, and eventually gives up.
Your analytics dashboard sees Customer B as the more engaged user. Higher message count. Longer session. If you’re running any kind of usage-based health score, Customer B looks like a power user.
Customer B is about to churn.
This isn’t an edge case. It’s a structural problem with how engagement gets measured in conversational products. The signal is inverted.
In a conversation, high effort often means the product is failing, not succeeding. The customer isn’t choosing to engage more. They’re being forced to work harder because the experience isn’t delivering.
I’ve talked to teams who flag their highest-volume users as their healthiest accounts. They send them case study requests. Meanwhile, those users are evaluating competitors because every interaction with the product feels like a fight.
The fix isn’t to stop measuring engagement. It’s to stop treating engagement as a proxy for success. You need to know whether the customer accomplished what they came to do, not just how much effort they put in. Those are two different questions, and right now most teams only have data to answer the wrong one.
It costs 5 to 25 times more to acquire a new customer than to keep an existing one. If your measurement tools are telling you that your most frustrated customers are your best customers, the math gets ugly fast.