The Problem with Event Analytics for AI Products

You can’t build a funnel when every user takes a different path.

When products have buttons and forms, product teams use Mixpanel or Amplitude to understand user behavior. Click a button → trigger an event → map the funnel. This is Event Analytics, and it works beautifully for traditional SaaS.

But AI products are conversational. Customers don’t click through predetermined paths, they describe what they want in natural language. The variability in inputs creates variable experiences and variable outcomes.

You can’t build a funnel when every user takes a different path.

Same Goal, Completely Different Experiences

Let’s look at two users building the same sales call analysis agent:

User A:

Hey, build me a sales agent that analyzes my notes.

User B:

I need an AI agent that automatically analyzes sales call transcripts to extract actionable intelligence.

SETUP:
- Zoom calls are recorded and transcribed automatically
- Deal data lives in Salesforce (stage, size, industry)
- Follow-up emails go through Gmail

WHAT TO ANALYZE (per call):
1. Strengths & Weaknesses - what went well, what I missed
2. Competitive Intel - competitor names, features discussed, pricing mentioned
3. Value Prop Effectiveness - which pain points/benefits resonated
4. Growth Opportunities - feature requests, new use cases, unmet needs

WORKFLOW:
- Pull transcripts from Zoom after each call
- Cross-reference Salesforce for deal context
- Store analysis back in Salesforce on the Opportunity record
- Send summary email via Gmail to sales manager

OUTPUT FORMAT:
- Structured summary with timestamps for key moments
- Flagged competitive mentions (high priority)
- Recommended next steps
- Track trends over time (week-over-week performance)

TECHNICAL:
- Needs to run automatically within 24 hours of call completion
- Handle multi-speaker transcripts
- Flag if competitor or pricing discussed
- Send Slack notification with highlights to #sales-insights

Both users will probably get their agent built. But their experiences are night and day different:

• User A (vague prompt) burns 5-10 rounds of clarification, risks getting frustrated, and might give up

• User B (thorough prompt) gets 80% of the way there in one shot

In traditional SaaS, you’d track button clicks and optimize the funnel. But here? There are no buttons. Just infinite conversational possibilities.

So how do you design a great AI experience when you can’t predict what users will say?

Why Evals Aren’t Enough

The current answer from engineering teams is: “Build more evals.”

Evals are great for pre-production QA. But they break down once you have real customers using your product in ways you never anticipated. You can’t write test cases for every possible prompt variation.

Even companies with robust engineering processes ship broken experiences. It happened to me last week with Zoom and Replit. Their agents that went completely off the rails despite presumably having extensive eval coverage.

Evals tell you if your agent can work. They don’t tell you if your customers are actually succeeding.

You Need Experience Analytics, Not Event Analytics

AI product teams need to understand:

1. Why users engage (goals and intent)

2. How users experience it (health, effort, and friction)

3. What outcomes it drives (resolution and results)

This requires Experience Analytics.

To build this on your own, you’d have to:

1. Analyze all your agent traces in a database

2. Label the types of requests (vague vs. thorough, simple vs. complex)

3. Identify which interaction patterns lead to success vs. frustration

4. Design interventions (prompts, guardrails, examples) that guide User A toward User B’s experience

That’s what Brixo does for you.

How Brixo Turns Conversations into Insights

Brixo is experience analytics for AI products. We transform unstructured conversational data into actionable product intelligence:

• Automatic labeling of interaction types (so you know your “User A” from “User B”)

• Goal mapping to understand which experiences lead to success

• Pattern detection to identify where users struggle or drop off

• Actionable data for designing better prompts, guardrails, and user flows

We handle the tedious work of analyzing thousands of agent conversations so you can focus on building better AI experiences.Thanks for reading The Masonry! Subscribe for free to receive new posts and support my work.