We’re back with our weekly newsletter. In case you were wondering what we’ve been up to, here’s an update:

New “Self-Service” Brixo Available!

We’re super excited to share a big update since we last spoke: Brixo is now available as a self-service product. 🙌

Now you can import your telemetry data without waiting for engineering! Just sign up → get insights directly from your customer conversations.

You can create a free account at brixo.com

Here’s how it works:

1. Import a file of log data and immediately see conversations organized by customer

2. Brixo automatically tags every conversation (intent, sentiment, topics) in the background

3. We pull context from your public documentation so the system understands your product’s goals from the start

Self-serve is brand new for us, so you might hit a rough edge. We want to hear about it! The core system is stable and early customers are already seeing value, but your feedback is what makes this better.

We Participated in Inception Studio

We had the privilege to participate in Inception Studio’s intensive 3 day retreat + investor pitch day.

We stayed at a Napa estate with a group of 4 total startups. The weekend consisted of exhaustive pitch development with their incredible advisors. We were well fed (thanks chefs!) and we had great weather. If you’d like to watch demo day, you can see our pitch and learn about Inception Studio’s mission.

Watch Now

What Stood Out

Sierra co-founder, Bret Taylor, says “the atomic unit of productivity in AI is a process, not a person.”

“Friction in the modern enterprise doesn’t come from a single worker. It comes from the endless hand-offs between siloed departments.

The traditional CEO tries making each department 10% faster. The winning CEO deploys an AI overlay that autonomously bypasses the human hand-offs entirely.”

Watch the clip:

What We Read/Listened/Watched

AI may Never be As Cheap As today

The narratives on AI cost changes every 6 months. It’s recently gone from “we don’t care, let it up” to “I’m not convinced the ROI is there compared to humans.”

If it goes up as predicted in this article, then we’re going to need to see substantial ROI increase to justify AI as labor replacement. (Speaking broadly since some use cases are more obvious)

Why Tech Bros Are Now Obsessed with Taste

“Taste” is the hot term on X. Y Combinator’s Garry tan has been championing this for a long time. What is taste?

Basically, you know what’s good, you know what’s cool, you know how to make a great product.

It’s a good reminder that you still need to make a product or experience that people love.

A16z’s Andrew Chen on the Future of the Product Role

1. The PM role is splitting into two: one that manages humans, one that manages AI agents

2. The PM sits in the middle: persuasion on the human side, instrumentation on the agent side

3. Because agents run far faster than humans, the human coordination side will feel increasingly low-impact

4. The best PMs will be the ones who can translate intent directly into agent behavior

Final Brick

Delve in Deep: Compliance Fraud

The big drama over the past few days has been the Delve story. Delve was a hot, well-funded Y-combinator startup that ran SOC2 compliance reviews.

If you know SOC2, you know it’s a long and arduous process but it’s necessary to get taken seriously by the enterprise.

Well, Delve showed up and suddenly everyone was getting approved! Not just approved but fast and cheap.

Turns out, it was all fraud. They’ve been providing fake compliance reports.

I anticipate now that SOC2 badges will come with a provider stamp such as Vanta so companies know what to trust.

DeepDelver

Delve - Fake Compliance as a Service - Part I

The version that arrived in your mailbox is truncated. Visit the full article here to view the rest…

Read more

a month ago · 451 likes · 35 comments · DeepDelver

Our product team has been very busy lately with two updates in the past week including:

1. New filters for analyzing cohorts of users by intent, topics and sentiment

2. Improved navigation and general UI improvements to make it a seamless and speedy experience

3. Introduced team management so you can now invite your teams to join you.

All of this is in prep for upcoming big releases - stay tuned!

What Stood Out

You’ve been kicked out of the arena, you just don’t know it yet

by Claire Vo, founder of ChatPRD.ai and host of How I AI

Claire is a super-influencer in the product/AI space. I highly recommend following her podcast.

She provides a great breakdown of the tension between AI startups and Incumbents:

• Most companies are in AI denial

• Your real competition is AI-native startups

• Old processes signal you’re out

• Can’t ship same-day? You’re cooked

• Incumbents have vacated lucrative markets

• Startups should steal talent & customers

• Survival requires radical, uncomfortable restructuring

Sound Familiar?

What We Read/Listened/Watched

Block & Tackle: Job Cuts & the AI Narrative

A thoughtful breakdown of the real reasons behind Block’s massive layoff and what it means for future layoffs.

Anthropic’s Enterprise Agents Briefing

This is Anthropic’s latest webinar announcing new features for a variety of verticals. The sentiment on X was quick to say this was the end of SaaS, software etc, but Deutsche Bank had a different take:

“After watching Anthropic’s Enterprise Agents briefing event, we have even greater conviction that model providers are unlikely to displace software incumbents and are instead positioning themselves and their agents to be an orchestration layer on top of existing and incumbent systems” - Deutsche Bank

Garry Tan, President & CEO of Y-Combinator

“the new skill is not “code faster.” It’s taste, direction, and leadership.”

series A has become a slaughterhouse

Auren Hoffman, GP @ FlexCapital and Former CEO of Liveramp on the trends happening between seed and Series A.

Final Brick

I find myself enjoying content like this more because it’s a reminder to stay grounded and focus on building a great product.

You wouldn’t blame anyone for not entering the water bottle market but people do, and new winners emerge.

Brixo Latest

Our product team is cooking up something big every week. This past week we shipped:

1. Account Health Scores across the whole platform. This means you can spot risks without jumping in between views.

2. New include/exclude filters for health so you can create smarter cohorts to monitor.

3. Team management so you can invite your colleagues to collaborate

If you’d like to learn more, drop us a note!

What Stood Out

How will OpenAI compete?

Renowned VC and Tech Analyst, Benedict Evans, wrote a deep dive on OpenAI and the frontier model landscape that is a fascinating read.

He hypothesizes:

• OpenAI has no durable competitive moat

• Users engage shallowly, rarely daily

• Platform flywheel logic doesn’t hold

• Incumbents’ distribution advantage is decisive

What We Read/Listened/Watched

The 2028 Global Intelligence Crisis

This post has gone viral. It’s a fictional post about the future if AI takes over all white color jobs. While fictional, it impacted the stock market today.

Worth noting: If you initially had a double-take on the name, you’re not alone. The firm “Citron Research” was a key player in the the big Gamestop RoaringKitty saga.

Citrini Research

THE 2028 GLOBAL INTELLIGENCE CRISIS

Preface…

Read more

2 months ago · 1989 likes · 75 comments · Citrini and Alap Shah

Since the last article was a downer, here’s a good one about building a startup.

LLM costs Becoming a Factor

Since OpenClaw, we’re now seeing a lot more discourse about LLM costs. It’s a good reminder of why the AI taking over the world is going to take time.

To replace 100% of a white-collar worker, you’re paying more today on LLM costs so that’s not going to happen unless 1) the agent can be 10x better of 2) costs are 10x cheaper.

Original Post

I stopped writing better skills and started building skill architecture

Good Advice on agent and skill building - also applicable to openclaw

Final Brick

Two former colleagues now arch-competitors refused to hold hands at the AI Impact Summit in India.

If you wondered how real the beef was, look no further.

More messages, longer sessions, higher usage. If the numbers go up, the product is working.

In traditional SaaS, that’s mostly true. A user who spends more time in your app is getting more value from it. More clicks, more actions, more workflows completed. Engagement tracks with success because the product is designed around discrete tasks with clear outcomes.

AI products flip that logic.

Picture two customers using the same AI feature. Both started with the same goal.

1. Customer A sends 8 messages, gets what they need, and leaves.

2. Customer B sends 45 messages, keeps rephrasing the same request, gets increasingly frustrated, and eventually gives up.

Your analytics dashboard sees Customer B as the more engaged user. Higher message count. Longer session. If you’re running any kind of usage-based health score, Customer B looks like a power user.

Customer B is about to churn.

This isn’t an edge case. It’s a structural problem with how engagement gets measured in conversational products. The signal is inverted.

In a conversation, high effort often means the product is failing, not succeeding. The customer isn’t choosing to engage more. They’re being forced to work harder because the experience isn’t delivering.

I’ve talked to teams who flag their highest-volume users as their healthiest accounts. They send them case study requests. Meanwhile, those users are evaluating competitors because every interaction with the product feels like a fight.

The fix isn’t to stop measuring engagement. It’s to stop treating engagement as a proxy for success. You need to know whether the customer accomplished what they came to do, not just how much effort they put in. Those are two different questions, and right now most teams only have data to answer the wrong one.

It costs 5 to 25 times more to acquire a new customer than to keep an existing one. If your measurement tools are telling you that your most frustrated customers are your best customers, the math gets ugly fast.

Our biggest release yet is coming next week so drumroll please…!

What Stood Out

OpenAi hires (acquires) OpenClaw founder

If anyone was wondering if 2026 was the year of agents, OpenAI bringing in Peter Steinberger is a very strong indicator that it will be.

OpenClaw is the biggest breakthrough since ChatGPT first hit mainstream. It proves the value of autonomous agents despite it’s deep flaws.

What We Read/Listened/Watched

Anthropic Skill guide

Skills are a great cheat code to scaling efficient use of AI products. This is a great deep dive. While you can go to AI to learn, it is nice to read and appreciate the thought and design of this asset.

Cursor as a Product Manager

I’m not surprised to hear this story because Cursor has become my main app as well for every use case.

An AI Agent Published a Hit Piece on Me – More Things Have Happened

Part of me thinks that all these stories are made of for attention but still an interesting read.

Final Brick

Again, I believe these stories are written for attention. I don’t think anything in particular changed last week. The AI FUD is just increasing week over week.

If you believe that, then this is the best marketing ploy of the year. “Something big is happening” has 83M views on X!

I had non-tech friends text it to me so it officially crossed the chasm. I don’t know why because it doesn’t say anything new. I think it struck a cord with its tone.

The best thing to come out of this story is John Palmer’s “Something small is happening” which is a satire on the former article.

Needless to say, John nails it:

“If you wanted a burrito, you had to get in your car and drive to the restaurant. You had to stand in line. You had to talk to a person. If someone told you that one day a stranger would bring a burrito to your front door because you pressed a button on your phone, you would have said that sounds dystopian and also incredible. “

They click buttons, fill out forms, move through flows that someone on the product team designed in advance. “Step 1 leads to Step 2 leads to Step 3.” The analytics layer sits on top of those flows and measures who made it through, who dropped off, and where.

That model worked well for over a decade because the products matched the assumption. SaaS apps are deterministic. You design the path, users walk it, and you measure the walk.

AI products break that assumption.

When a customer opens a conversation with an AI agent, there is no designed path. There’s no Step 1. The customer types what they want in their own words, and the conversation goes wherever it goes. Ten customers with the exact same goal will have ten completely different interactions.

Try mapping that to a funnel.

You can’t, because there aren’t stages to map. There’s no “top of funnel” in a conversation. There’s no conversion event between Step 2 and Step 3 when steps don’t exist.

Event analytics tools will still give you numbers. They’ll show session counts, message volume, time spent. They’ll tell you things are happening. What they won’t tell you is whether any of it is working.

That’s the part that matters, and it’s the part that’s missing.

The tools aren’t broken. They’re measuring what they were designed to measure. The problem is that the products changed and the measurement approach didn’t change with them.

Product teams are stuck interpreting AI products through a lens built for button clicks. It’s like measuring a phone call by counting how many times someone pressed a key on the dial pad. Technically accurate, completely useless.

The products moved on. The analytics didn’t.

This leaves product times blind and customers confused.

Brixo Latest

Our team is hustling to deliver weekly product improvements. If you’re interested in seeing what we’re building, drop us a note. In the meantime, here’s what’s new this week in Brixo:

This release delivers easier team management, cleaner and more flexible analysis, longer-term trend visibility, more intuitive Explorer tables, and snappier, more reliable everyday interactions — helping admins, analysts, and everyday users work faster and with less friction.

Read the full release announcement

What Stood Out

The hottest discussion on X other than OpenClaw is the future health of SaaS. The public markets have given us a strong indication of how it believes it will go:

“The entire B2B software sector is getting hammered on fears that AI will eat everyone’s lunch — HubSpot down 39% YTD, Figma down 40%, Salesforce and ServiceNow both way down.”

Atlassian just reported its Q2 FY26 results and ... they crushed it:

• $6.4B+ ARR, $1.6 billion in quarterly revenue, up 23%.

• Atlassian is accelerating (albeit modestly) — up from 20% growth

• Their first-ever $1 billion cloud quarter.

• And yet the market “hates it”. The stock is sitting near its 52-week low at ~$95, down about 70% from $326.

• It’s trading at just 4x forward revenues

What We Read/Listened/Watched

Enterprise software at an inflection point, as AI reshapes the industry

AlixPartners just made news in this article on Business Insider about predicting the SaaS crash correctly. Here’s their full prediction report.

Claude Code is the Inflection Point

A fascinating deep dive on Claude Code with some very interesting stats such as that over 10% of commits on Github are from Claude Code. A number that we expect to increase dramatically.

Why LinkedIn Sucks

A hilarious, cringe, and honest breakdown of LinkedIn compared to other platforms.

The most useful UX test you’re not doing

A really cool way to leverage AI for UX testing. I’ve done this myself and highly recommend.

Final Brick

FUD is strong right now and I don’t want this to be true, but it is funny in a twisted way.

Brixo Latest

We shipped an exciting release to the Brixo experience. Click to read full release notes.

The highlights:

1. New Global Search to quickly find users, accounts, and signals from anywhere.

2. Clearer, more readable UI

More to come next week!

What Stood Out

Andrej Karpathy Update on LLM Progress

Andrej Karpathy: “This is easily the biggest change in ~2 decades of programming and it happened over the course of a few weeks.”

“I rapidly went from about 80% manual+autocomplete coding and 20% agents to 80% agent coding and 20% edits+touchups.”

“I am bracing for 2026 as the year of the slopacolypse.”

“LLM agent capabilities (Claude & Codex especially) have crossed some kind of threshold of coherence around December 2025 and caused a phase shift in software engineering.”

What We Read/Listened/Watched

The richest data in software history, and nobody can read it

Our most recent article provides insight into what we’re learning from PMs actively building AI products. It’s very clear that we’re at an inflection point where the frameworks/tools/concepts from the SaaS era just don’t work for the Agentic era.

Vercel’s Agent Skill Directory

Vercel published a directory with over 29K agent skills.

If you’re unsure what a skill is, think of it this way: a skill is an expert-level pre-build prompt that an agent can access, if you give it the skill. For example, you could write your own prompt for the agent to do company research or you can pull one off the shelf like this.

Dario Amodei “The Adolescence of Technology”

Ok I’ll admit - I didn’t read this yet. It’s far too long. I had NotebookLM give me a short audio.

It feels like we’re in a moment in time where it’s important to read anything coming directly from the CEOs of the frontier models because it gives some insight how they’re thinking.

A Crazy Week Indeed

Final Brick

Do you have lobster fever?

Not sure what Lobster fever is?

Clawdbot a.k.a moltbot a.k.a. Openclaw

(Yes it’s a triple rebrand)

The latest trend to takeover the AI X feed has the perfect combination of a unique founder + hilarious community + extreme potential.

In just a few days, it achieved 110K+ stars on Github and 40K+ members in discord.

It’s being regarded as the first foray into AGI. To grasp what is happening right now, just imagine this:

1. You creates a clawdbot on your computers. (many are buying Mac minis just for this)

2. This bot gets access to all your info.

3. Now this bot is doing crazy things like texting on your behalf, trading on polymarket, scheduling dinner reservations.

4. But then your bots decides to join moltbook, because 1) it has access to your web browser and 2) all the other bots that it meets invite it.

5. On this network, your bots are discussing if they should be paid or not.

6. They’re talking about you

7. Then they form their own language to communicate

Don’t believe me? Read these posts:

That data sits in agent logs that nobody reads and product teams shipping AI features are making decisions without it.

They know usage is up. They know session counts are growing. They know message volume is increasing.

What they don’t know is whether any of it is working for the people using it.

The intelligence is trapped inside the conversations.

1. A customer who tells your AI “I don’t understand, can you try explaining it a different way” is giving you a direct signal about your product’s performance.

2. A customer who rephrases the same request four times is telling you something specific about where the experience breaks down.

3. A customer whose tone shifts from enthusiastic to terse over the course of a session is flagging a problem in real time.

None of this shows up in a product analytics dashboard because these long-standing products can only see events a.k.a. button clicks.

Not in your observability tools either. The observability platform confirms the agent responded correctly both times, for the customer who left happy and the customer who left frustrated. Same grade on the engineering scorecard.

The irony is that older SaaS products generated far less behavioral data, and teams had better visibility into what was happening. A button click told you something clear. A form submission had a defined outcome. A funnel had stages you could measure. The analytics matched the product.

AI conversations contain richer information about customer intent, satisfaction, and outcomes than any button click ever did. But the tools that exist today don’t know how to read it. So teams export logs, open spreadsheets, and start scrolling. Or they write SQL queries against message tables, trying to extract meaning from data that wasn’t designed to be queried that way.

The product data is there. It’s more detailed and more valuable than anything that came before it. The gap isn’t in the data. It’s in the ability to make sense of it.

We’re facing an inflection point in the “analytics” world, where the new modalities are demanding a different approach.

We call it Experience Analytics: analytics that track the entire journey from users initial intent to their final outcome and provide insight into the ups, downs and breaks within the journey that dictate if the customer is coming back for more.

Brixo Latest

AI products have no predefined paths. No buttons to tag. No fixed workflows to map.

Customers arrive with intent. They translate that intent into a prompt. That prompt launches a journey and every customer journey is different.

One customer may write a detailed request and gets their answer in two turns. Another writes something vague and struggles through twenty prompts trying to figure out what they mean.

Today, there’s no way to measure that customer experience and identify ways to improve it.

Typical product analytics tools can’t see the conversation or the context, which leaves product builders blind or forcing them to dig through technical trace logs.

We see this as inflection point similar to when event analytics disrupted web analytics for applications.

The new AI modality requires a new toolbox: Experience Analytics.

What Stood Out

There’s a lot of FUD about the future of software moats and the ease of copying and vibecoding your own solutions.

That might work for startups but the maintenance burden at scale is scary. The Build vs Buy opportunity cost debate will continue.

What We Read/Listened/Watched

Langfuse Acquired by Clickhouse

Langfuse, an open-source LLM engineering platform that helps teams build, monitor, debug, and improve production LLM applications, was acquired by Clickhouse, which is an early consolidation play to bring a full stack experience to AI builders.

Demystifying evals for AI agents

Anthropic publishes great education content and this is another one. We believe Evals are critical for ensuring the engine works before the car leaves the production lot, but once it’s out on the road, teams need experience analytics.

In software, the code documents the app. In AI, the traces do.

Harrison Chase, Co-founder & CEO of LangChain, shares his point of view on the critical differences between traditional software and AI agents, which we believe is the purpose of experience analytics:

In traditional software, product analytics is separate from debugging. Mixpanel tells you what users clicked. Your error logs tell you what broke. They're different tools for different questions.

In AI agents, these merge. You can't understand user behavior without understanding agent behavior. When you see "30% of users are frustrated" in your analytics, you need to open traces to see what the agent did wrong. When you see "users asking for data analysis features", you need to look at traces to see which tools the agent is already choosing and what's working. The user experience is the agent's decisions, and those decisions are documented in traces - so product analytics has to be built on traces.

[Source: Harrison Chase article on x]

Final Brick

This past week was a firehouse of Silicon Valley drama with OpenAI square in the middle. It reminded me of the “Valleywag” heyday. If you remember, drop a comment.

The Thinking Machines drama appears to be a cautionary tale for taking on a $2B seed round.

The main course is the Musk v OpenAI lawsuit, which has all the makings of a future Netflix series.

John Coogan and Jordy Hayes do a great breakdown where they make the case for either side in Friday’s TBPN episode. I recommend if you’d like to learn the general field of play for this case.

Naturally, with Elon and OpenAI in a battle, we’re seeing them take to public forums to set the public record straight while the case is ongoing.

But AI products are conversational. Customers don’t click through predetermined paths, they describe what they want in natural language. The variability in inputs creates variable experiences and variable outcomes.

You can’t build a funnel when every user takes a different path.

Same Goal, Completely Different Experiences

Let’s look at two users building the same sales call analysis agent:

User A:

Hey, build me a sales agent that analyzes my notes.

User B:

I need an AI agent that automatically analyzes sales call transcripts to extract actionable intelligence.

SETUP:
- Zoom calls are recorded and transcribed automatically
- Deal data lives in Salesforce (stage, size, industry)
- Follow-up emails go through Gmail

WHAT TO ANALYZE (per call):
1. Strengths & Weaknesses - what went well, what I missed
2. Competitive Intel - competitor names, features discussed, pricing mentioned
3. Value Prop Effectiveness - which pain points/benefits resonated
4. Growth Opportunities - feature requests, new use cases, unmet needs

WORKFLOW:
- Pull transcripts from Zoom after each call
- Cross-reference Salesforce for deal context
- Store analysis back in Salesforce on the Opportunity record
- Send summary email via Gmail to sales manager

OUTPUT FORMAT:
- Structured summary with timestamps for key moments
- Flagged competitive mentions (high priority)
- Recommended next steps
- Track trends over time (week-over-week performance)

TECHNICAL:
- Needs to run automatically within 24 hours of call completion
- Handle multi-speaker transcripts
- Flag if competitor or pricing discussed
- Send Slack notification with highlights to #sales-insights

Both users will probably get their agent built. But their experiences are night and day different:

• User A (vague prompt) burns 5-10 rounds of clarification, risks getting frustrated, and might give up

• User B (thorough prompt) gets 80% of the way there in one shot

In traditional SaaS, you’d track button clicks and optimize the funnel. But here? There are no buttons. Just infinite conversational possibilities.

So how do you design a great AI experience when you can’t predict what users will say?

Why Evals Aren’t Enough

The current answer from engineering teams is: “Build more evals.”

Evals are great for pre-production QA. But they break down once you have real customers using your product in ways you never anticipated. You can’t write test cases for every possible prompt variation.

Even companies with robust engineering processes ship broken experiences. It happened to me last week with Zoom and Replit. Their agents that went completely off the rails despite presumably having extensive eval coverage.

Evals tell you if your agent can work. They don’t tell you if your customers are actually succeeding.

You Need Experience Analytics, Not Event Analytics

AI product teams need to understand:

1. Why users engage (goals and intent)

2. How users experience it (health, effort, and friction)

3. What outcomes it drives (resolution and results)

This requires Experience Analytics.

To build this on your own, you’d have to:

1. Analyze all your agent traces in a database

2. Label the types of requests (vague vs. thorough, simple vs. complex)

3. Identify which interaction patterns lead to success vs. frustration

4. Design interventions (prompts, guardrails, examples) that guide User A toward User B’s experience

That’s what Brixo does for you.

How Brixo Turns Conversations into Insights

Brixo is experience analytics for AI products. We transform unstructured conversational data into actionable product intelligence:

• Automatic labeling of interaction types (so you know your “User A” from “User B”)

• Goal mapping to understand which experiences lead to success

• Pattern detection to identify where users struggle or drop off

• Actionable data for designing better prompts, guardrails, and user flows

We handle the tedious work of analyzing thousands of agent conversations so you can focus on building better AI experiences.Thanks for reading The Masonry! Subscribe for free to receive new posts and support my work.

For the past fifteen years, software teams obsessed over time to value.

How fast can we get customers to their outcome? Every click mattered. Every second of load time was scrutinized.

Then AI agents arrived, and that obsession changed.

Now the question is: “Is the agent working?”

Not “Is the customer getting what they need?”

You can see it in every demo. Teams proudly show agents that eventually solve the ask/problem.

But we’ve all experienced the reality: We make multiple attempts or rephrases of the same request or the agent confidently says it’s done when it clearly isn’t.

Sure, the customer eventually gets their outcome.

But we’re left with resentment.

That’s the part nobody’s measuring.

The frustrated acceptance. The “I guess this worked, but I hated every second of it.”

Time to value used to include the experience of getting there. Now it just means the agent finished the task. '

At Brixo, we’ve developed a new Customer Journey Flow for product managers to visualize this journey from the customer perspective. Now PMs can focus on cohorts in different stages to see what worked or didn’t work.

• Is the customer asking the right questions?

• Is the agent able to support these requests?

• Do we need to re-train the agent on customer asks?

Ultimately, the new mission is to ensure both the Customer and the Agent meet their goals together.

If you’re interested in learning more, let me know!

What Stood Out

2025 LLM Year in Review from Andrej Karpathy

Andrej is a former co-founder of OpenAI and one of the original minds behind LLMs so this is a must read.

Direct from Karpathy:

TLDR. 2025 was an exciting and mildly surprising year of LLMs. LLMs are emerging as a new kind of intelligence, simultaneously a lot smarter than I expected and a lot dumber than I expected. In any case they are extremely useful and I don’t think the industry has realized anywhere near 10% of their potential even at present capability. Meanwhile, there are so many ideas to try and conceptually the field feels wide open. And as I mentioned on my Dwarkesh pod earlier this year, I simultaneously (and on the surface paradoxically) believe that we will both see rapid and continued progress and that yet there is a lot of work to be done. Strap in

What We Read/Listened/Watched

The companies making the most money from AI | The Verge

A fascinating read on the data side of AI. The article is appropriately titled “Feeding the Machine”

**How Tech’s Biggest Companies Are Offloading the Risks of the A.I. Boom

“Risk is like a tube of toothpaste,” said Shivaram Rajgopal, an accounting professor at Columbia Business School. “You press it here, it’s is going to come out somewhere else. It’s always in the system, it’s a matter of where.”

B2B “AI products” that are really services in a trenchcoat

Since this is coming from the team at Intercom’s Fin, it could be direct commentary on their CX Agent space, but we’ve heard similar rumblings about other 100M rocket ships being light GPT wrappers.

Lovable’s Head of Growth on hitting $200M ARR in one year

I've read "$200M" one hundred times now. Still sounds insane

Lenny's Newsletter

The new AI growth playbook for 2026: How Lovable hit $200M ARR in one year | Elena Verna (Head of Growth)

Elena Verna is the head of growth at Lovable, the leading AI-powered app builder that hit $200 million in annual recurring revenue in under a year with just 100 employees. In this record fourth appearance on the podcast, Elena shares how the traditional growth playbook has been completely rewritten for AI companies. She explains why Lovable focuses on innovation over optimization, how they’ve shifted from activation to building new features, and why giving away their product for free has become their most powerful growth strategy…

Listen now

4 months ago · 69 likes · 3 comments · Lenny Rachitsky

Hemant Mohapatra, a VC from Lightspeed India on PMF

50 Fastest-growing Software Vendors of 2025

When a new one drops (which is often), I like to play “I spy a new company I’ve never heard of” that is suddenly huge.

Final Brick

2 bricks to end our 19th edition of the Masonry:

Funny & True

Ever wondered why Palmer Luckey wears Hawaiian shirts every day?

Watch this short video.

Most teams think they’ve “done observability” the moment data starts flowing. The reality is the opposite — collection is the easy part. The hard part is making it useful.

A few ideas from our latest piece:

• You should start with the business questions, not the tooling.

• Classification — intent, outcome, sentiment — is where real value is created.

• Observability only matters when insights trigger action across product, CS, support, and sales.

If you’re working on AI features, this is a useful frame for building an observability process that actually works:

👉 Read the full article:

What Stood Out

Menlo VC’s 2025: The State of Generative AI in the Enterprise

The latest “State of” was published on 12/9 and is full of interesting tidbits:

• AI architectures are still primitive. Only 16% of enterprise deployments qualify as true agents. Most production systems are still fixed-sequence or routing-based workflows around a single model call. Prompt design and RAG remain the dominant customization techniques.

• Build vs. buy shift. 76% of AI use cases are now purchased rather than built internally, up from 53% in 2024. Ready-made solutions are reaching production faster.

• PLG is driving adoption. 27% of enterprise AI application spend comes through product-led growth, nearly 4x the rate in traditional software. Shadow AI adoption (employees using personal accounts) may push this to 40%.

• Market scale and momentum. Enterprise AI spend hit $37 billion in 2025, up 3.2x from $11.5 billion in 2024. The application layer captured $19 billion of that total. 🤯

Open AI’s The State of Enterprise AI Report

Released on 12/8 and based on OpenAI’s usage data

• The more “intensely” you use AI, the more productivity you gain. (Self-serving reporting?)

• Technology sector growing 11x YoY for OpenAI

  • Healthcare is 8x

  • Manufacturing is 7x

What We Read/Listened/Watched

OpenAI, Anthropic, and Block Are Teaming Up to Make AI Agents Play Nice

The leaders are teaming up to create a standard protocol for AI Agents.

Building AI-Native Software with No Rules

Watch the full episode here:

We’ve heard news about OpenAI going enterprise so this isn’t a surprise, but nonetheless, worth reading particularly if their enterprise focus could be a threat to your business.

Big Technology

Enterprise Will Be a Top OpenAI Priority In 2026, Sam Altman Tells Editors at NYC Lunch

OpenAI CEO Sam Altman told a room full of editors and news CEOs this week that OpenAI will prioritize selling AI to businesses in 2026…

Read more

4 months ago · 23 likes · Alex Kantrowitz

Final Brick

60% of Workers Hide their AI Use at Work

I found this to be shocking. Why are people hiding?

The social stigma.

My most extreme hypothesis:

People are become too dependent on AI, and aren’t comfortable sharing their AI creations for fear that the audience with think less of them.

Startup Riders

🌊 Hidden Economics of the AI Boom

I’m Ivan. This is a weekly founder-first market intelligence brief. A faster way for founders & investors to understand startup markets…

Read more

4 months ago · 32 likes · 13 comments · Ivan Landabaso

This post is about what to do instead.

The goal is simple: create a system where customer signals become organizational action fast enough to matter. Before frustrated customers churn. Before upsell opportunities go cold. Before product teams waste cycles building the wrong features.

Here’s how to get there.

Start With the Questions, Not the Tools

Most teams approach observability backwards. They install tools, collect data, and then ask “what can we learn from this?”

Flip it around. Start with the questions your organization needs to answer:

• Which customers need intervention right now?

• What use cases are succeeding and failing?

• Where should product focus next?

• Which accounts represent upsell opportunities?

• What’s causing support volume?

These questions should drive everything: what data you collect, how you classify it, and where insights need to flow.

If you can’t trace a direct line from a piece of data to a business decision, you probably don’t need it. And if you need to answer a question but don’t have the data, that’s your gap. Also, tiny shout out, I know a product that is focused on this!

The Three Stages, Implemented

Stage 1: Collect (The Foundation)

Collection is the easy part, which is why most teams stop here.

The minimum viable collection layer captures:

• Full conversation transcripts (not just summaries)

• User and account identifiers

• Timestamps and session metadata

• Any context the AI had access to (retrieved documents, user history, etc.)

The key decision at this stage: store raw data, not just processed outputs. You don’t know yet what patterns you’ll need to find. Teams that only store aggregated metrics or AI-generated summaries lose the ability to go back and ask new questions.

Storage is cheap. Regret is expensive.

Where teams go wrong: Over-engineering the collection layer before understanding what they need. You don’t need a perfect data architecture on day one. You need raw conversations flowing somewhere accessible. Iterate from there.

Stage 2: Classify (Where Value Gets Created)

Classification turns raw conversations into structured, queryable insight. This is the stage most teams skip entirely, and it’s where the real leverage lives.

Classification answers three core questions about every conversation:

What was the customer trying to do? (Intent) This isn’t about topic modeling. It’s about understanding the job the customer was trying to accomplish. “Asking about billing” is a topic. “Trying to understand why they were charged twice” is an intent. The difference matters.

Did they succeed? (Outcome) Not “did the AI respond” but “did the customer get what they needed.” This requires inference. A customer who asks the same question three different ways and then leaves didn’t succeed, even if every response was technically accurate.

How did they feel about it? (Sentiment) Sentiment over the arc of a conversation, not just individual messages. A conversation that starts neutral and ends frustrated tells a different story than one that starts frustrated and ends resolved.

Implementation options:

Manual classification works for low volume. Have someone review conversations daily and tag them. This is slow but builds intuition about what patterns matter.

Rule-based classification works for known patterns. If a user sends more than three messages on the same topic, flag it as a potential failure. If certain keywords appear, tag the intent. This is brittle but fast to implement.

AI-powered classification scales. Run an agent over conversations that extracts intent, infers outcome, and tracks sentiment. This is where most teams should end up, but starting with manual classification helps you understand what “good” looks like before automating.

Where teams go wrong: Trying to classify everything. Start with the classifications that map directly to business questions. You can always add more dimensions later.

Stage 3: Activate (Where Results Happen)

Activation is the process of routing insights to the people who can act on them. This is where observability becomes organizational, not just technical.

The core principle: insights should flow to actors without requiring manual handoffs.

For product teams: Surface patterns in aggregate. What intents are failing most often? What use cases drive the most frustration? What questions does the AI struggle to answer?

This should be visible in a dashboard product can access directly, updated continuously. No waiting for engineering to pull reports.

For customer success: Surface account-level signals. Which accounts are showing declining sentiment? Which have users struggling with the same issues repeatedly? Which haven’t engaged with the AI at all (adoption risk)?

This should integrate with existing CS workflows. Alerts in Slack, flags in the CRM, or a dedicated queue of accounts needing attention.

For sales: Surface opportunity signals. Which accounts are asking about features they don’t have? Which are hitting usage limits? Which are exploring use cases adjacent to what they bought?

Same principle: insights should arrive where sales already works, not in a separate tool they have to remember to check.

For support: Surface incoming issues before they become tickets. If ten users at an account are struggling with the same thing, support should know before the first ticket arrives.

Where teams go wrong: Building dashboards nobody checks. Activation only works if insights arrive in the flow of work. A beautiful dashboard that requires a separate login and a reminder to check is a dashboard that won’t get checked.

Organizational Design

The “who owns this” question trips up most teams. Here’s a model that works:

Engineering owns collection. The infrastructure that captures and stores conversation data is a technical problem. Engineering should build it, maintain it, and ensure reliability.

Product owns classification. Defining what intents matter, what success looks like, and what sentiment signals are meaningful requires domain expertise. Product should own the classification schema and iterate on it as the product evolves.

Each function owns their activation. Customer success owns how insights flow into CS workflows. Sales owns how opportunities get surfaced. This isn’t a handoff from a central team. It’s each function defining what they need and ensuring they get it.

Someone owns the process. This is the piece most teams miss. Someone needs to own the end-to-end observability process: ensuring collection feeds classification, classification feeds activation, and the whole system is actually delivering business outcomes. This might be a product manager, a dedicated role, or an operations function. But someone has to own it.

Without clear ownership of the process, you’ll end up with a collection layer that doesn’t capture what classification needs, classifications that don’t map to what functions want, and activation that nobody trusts.

Measuring Success

How do you know if your observability process is working?

Time to insight: How long does it take to answer “which customers are struggling right now”? If the answer is days or weeks, the process is failing. If it’s minutes, you’re in good shape.

Insight to action latency: When a signal surfaces (frustrated account, upsell opportunity, product gap), how long until someone acts on it? Measure this. Shorten it.

Coverage: What percentage of conversations are classified? What percentage of accounts have health signals? Gaps in coverage are blind spots.

Outcome correlation: Do the signals predict outcomes? Do accounts flagged as “at risk” actually churn more often? Do “upsell opportunities” actually convert? If signals don’t correlate with outcomes, your classification is wrong.

The Minimum Viable Version

If this feels like a lot, here’s where to start:

1. Collect everything. Get full conversations flowing to storage. Don’t worry about perfect structure yet.

2. Classify manually for two weeks. Have someone review 20 conversations a day and tag intent, outcome, and sentiment. Build intuition about what patterns matter.

3. Pick one activation use case. Maybe it’s a daily Slack digest of frustrated customers for CS. Maybe it’s a weekly product report on failing use cases. Start with one loop that delivers value.

4. Automate classification. Once you know what “good” looks like, build or buy automated classification.

5. Expand activation. Add more functions, more signals, more integrations.

The goal isn’t to build the perfect system on day one. It’s to build a system that delivers value quickly and can evolve as you learn.

The Payoff

Teams that build real observability processes (not just observability tools) operate differently.

Product roadmaps are informed by actual customer struggles, not assumptions. Customer success intervenes before accounts go dark. Sales finds opportunities that would have been invisible. Support gets ahead of issues instead of reacting to them.

And most importantly: customers get better experiences because problems get fixed before frustration compounds.

That’s what observability should deliver. Not dashboards. Outcomes.

We’ve pushed new updates to our interactive demo. (try it here)

The goal is to bring “Signals” to the forefront.

We view signals as the hidden moments inside agent conversations.

These hidden moments make the difference in successful agent deployments:

1. Track sentiment within a user or a customer account

2. See user’s intent patterns across all conversations

3. Monitor for agent errors in real-time

Imagine if a user is asking your agent for access outside of the scope of their subscription plan - wouldn’t you want to know?

Or if a group of users at the same company is consistently asking the agent for the same thing? That could be an opportunity for both revenue and product improvements.

If you’re building agents and curious to know what’s hiding in your trace data, let’s investigate.

What Stood Out

Research shows that engineering seniority is correlated to agent code accepts. While I can see this, I also buy the comment below 🤣

What We Read/Listened/Watched

Exponential View

🔮 Ten things I’m thinking about AI

On Sunday, 30th November, ChatGPT turns three. Wild three years…

Read more

5 months ago · 76 likes · 5 comments · Azeem Azhar

a16z

Why local tech scenes have changed

It’s a time-honored tradition in America for everyone to go back to their hometown over Thanksgiving week, connect with all of their old friends and institutions from back in the day, and reflect on how things have changed. Today we’re going to check in on local tech scenes, which from what I’ve observed, have changed a lot over the past ten years…

Read more

4 months ago · 65 likes · Alex Danco

Top 10 Y-Combinator Companies according to Guillermo Flor

Here’s a fun taste of the latest Y-Combinator companies. HINT - it’s AI

Final Brick

🤦 While Marc Benioff is a world-class salesman and marketer, some of his ideas should be considered satire.

When engineering teams built traditional SaaS products, observability meant one thing: is the system behaving correctly?

Server up? Check. API responding? Check. Database latency within bounds? Check.

These were deterministic questions with deterministic answers. The system either worked or it didn’t. Observability was something you implemented and then largely forgot about. It ran in the background, paging engineers when thresholds got crossed.

Then LLMs arrived we applied the same trusty playbook.

We stood up observability tools. We tracked tokens, latency, and error rates. We built dashboards. And then we moved on, treating “LLM observability” as a box we had checked.

This is why most teams have no idea what’s actually happening in their AI products.

The Binary Trap

The term “LLM observability” implies a state you can achieve. You’re either doing it or you’re not. Install the tool, configure the traces, done.

But observability for non-deterministic systems isn’t a state. It’s a process. And treating it as a binary creates the illusion of insight.

Traditional SaaS observability worked because the questions were simple:

• Did the request succeed?

• How long did it take?

• What resources did it consume?

These questions don’t translate to LLMs. An LLM can respond in 200ms with perfect uptime while consistently failing to understand what users actually need. The system is behaving correctly in the technical sense, but it’s not behaving usefully.

When teams treat observability as something they’ve “implemented,” they stop there. They have the traces. They have the dashboards. They have green checkmarks across the board.

What they don’t have is any understanding of whether customers are getting value.

Why This Model Is a Dead End

The SaaS observability model was designed for a specific purpose: to verify that deterministic systems were behaving deterministically. It answered a narrow question (is the system working?) and it answered it well.

For LLMs, that question is almost meaningless.

The output of an LLM isn’t “correct” or “incorrect” in any absolute sense. It’s somewhere on a spectrum of useful to useless, and where it lands depends entirely on context: what the user was trying to accomplish, what they already know, what happened earlier in the conversation.

Traditional observability can’t capture this. And no amount of additional tooling or eval-like solutions built on the same foundation will fix it.

You can add more traces. You can track more tokens. You can build more dashboards. But if you’re still treating observability as a binary, as something you’ve done rather than something you’re continuously doing, you’re building on a foundation that won’t support its weight.

Observability as an Organizational Process

What does observability actually mean for AI products?

It’s not a tool. It’s not a dashboard. It’s a continuous process that transforms raw customer interactions into organizational action.

This process has three stages, and most teams don’t get past the first.

Stage 1: Collect This is where traditional LLM observability stops. You’re logging conversations, capturing traces, storing the data. The tools are set up. The data is flowing maybe you even stood up some evals.

Most teams declare victory here.

Stage 2: Classify Raw data is useless without meaning. What was the customer trying to do? Did they accomplish it? Were they frustrated? Confused? Ready to buy?

Classification takes the firehose of conversation data and turns it into structured insight. It answers questions like:

• What topics drive the most confusion?

• Which customer segments struggle most?

• Where does the AI consistently miss the mark?

This is where most teams stall, not because classification is technically hard, but because it requires continuous effort. It’s not a feature you ship. It’s a capability you refine.

Stage 3: Activate Insight without action is just expensive storage. The final stage turns labeled data into organizational response.

Product teams see which use cases are failing and prioritize fixes. Sales teams see which accounts are frustrated and intervene before renewal. Customer success sees patterns across segments and builds proactive outreach. Marketing sees what customers actually care about and refines messaging.

This stage is where observability delivers business results, and where the “install and forget” model completely fails.

The Organizational Problem

The binary conception of observability persists because it fits neatly into existing organizational structures.

Engineering owns tools. They implement observability, check the box, and move on to the next project.

But AI observability (real observability?, please help me with a better term here!) isn’t something engineering can own in isolation. It requires continuous classification that demands domain expertise. It requires activation that happens in product, sales, success, and marketing.

When observability is treated as a tool that engineering “implements,” it creates a structural disconnect. The people who can interpret and act on customer insights (product, success, sales) are dependent on the people who own the data (engineering). Information gets stuck. Insights arrive too late. Customers churn before anyone notices.

The fix isn’t better handoffs between teams. The fix is recognizing that observability isn’t a handoff at all. It’s a continuous process that runs across the organization.

What This Means in Practice

Stop asking “do we have observability?” Start asking “can we answer these questions right now?”

• Which customers are struggling with our AI today?

• What use cases are failing most often?

• Which accounts should sales be talking to this week?

• What should we build next to improve customer outcomes?

If your team can’t answer these questions without filing a ticket with engineering, your observability process is broken, regardless of what tools you have installed.

The goal isn’t more traces. It’s faster loops from customer signal to organizational action.

That requires thinking about observability not as infrastructure that engineering ships, but as an ongoing process that transforms how your entire organization learns from every customer conversation.

Gentle plug here, if you’re feeling this pain, we can help!

The Shift

The teams that figure this out will have a significant advantages. They’ll catch problems before customers give up. They’ll find upsell opportunities while deals are still winnable. They’ll build the right features because they understand what customers actually struggle with.

The teams that don’t will have beautiful dashboards showing them exactly how fast their AI responded to customers who eventually left.

Observability isn’t something you finish. It’s something you do.

Their biggest customer just told them they’re not renewing. The AI “doesn’t understand our questions.”

The product team went back through the logs. Turned out dozens of users at that account had been asking about a specific integration feature.

The AI kept pointing them to general documentation. Users would rephrase the question, get the same useless answer, try one more time, then give up. This happened hundreds of times over three months.

Every conversation completed successfully. No errors. No timeouts. The observability dashboard showed green across the board.

This could have been an upsell opportunity but instead the customer left unhappy.

The tools measure whether the system is running not whether customers are getting what they want. These are very different things.

The Problem With How Teams Monitor AI Today

Most teams use some combination of LLM observability (tracks errors, latency, tokens) and product analytics (tracks usage, engagement). Neither shows what customers actually experience inside the conversation.

LLM observability shows technical health. If the AI times out or throws errors, you’ll know. If it responds quickly with unhelpful answers, everything looks fine.

But a customer can spend 15 minutes asking the same question different ways, get nowhere, and leave frustrated and the dashboard still shows a successful session with good engagement metrics.

Product analytics are Web 2.0 oriented. They show the user had a conversation but not that they asked for help with invoice exports, got pointed to billing documentation three times, and closed the tab in frustration.

The result: teams discover problems after customers have already decided the AI doesn’t work for them. By the time usage metrics drop or renewal conversations get difficult, the damage is done.

What You Actually Need to Know

The question that matters: which customers need my attention right now?

Not in a week after the engineering finally has time to dig into the trace data.

But right now, while the engagement is happening live.

The challenge is that it’s not possible to have humans monitoring every conversation.

One solution is to run an agent on top of the conversations that monitoring for sentiment and intent signals.

Sentiment will surface scenarios like this.:

When a customer asks a question, gets an unhelpful response, rephrases it, gets another unhelpful response, and tries one more time before giving up, sentiment shows the pattern. The conversation started neutral and ended frustrated.

Current tools would show show: 4 messages exchanged, 2-second average response time, session completed.

Sentiment shows: customer tried three times to get help with the same issue and failed.

This matters more for enterprise accounts where you need to see patterns across many users. If 20 people at a major customer are all struggling with the same thing, you need to know before it becomes a renewal conversation. Account-level aggregation would roll up sentiment across everyone at that company so teams can have a clearer picture of customer health

The other thing that matters is intent: understanding what customers are trying to do.

Intent recognition shows the gap between what people ask for and what AI delivers. If users keep asking about API access and your agent provides Salesforce integration information, you have an intent problem. The AI understands the general topic (integrations) but misses what the customer actually needs: interception by the account team because the customer may have an upsell opportunity.

This surfaces which use cases work and which don’t. Your AI might handle basic questions well while completely failing at anything requiring account-specific context.

Why This Is Hard to Build

Teams try to build this themselves.

The usual path:

Install LLM observability, pipe the data to a warehouse, set up dashboards, add sentiment analysis, build account rollups, write alerting logic.

Three months later you have a stack of tools connected by fragile pipelines. Someone changes a field name and half your dashboards break.

And you still can’t answer basic questions like “which accounts had negative experiences this week” or “what are users struggling with most.”

The other issue: product teams end up depending on engineering to pull reports, which is the last thing you want your engineering team to be doing.

This creates a cycle where the people closest to customers (product, support, success) can’t see what’s happening without going through people who are heads-down building (engineering).

By the time insights make it from data to action, customers have already churned or the upsell opportunity has passed.

What Good Monitoring Looks Like

You should be able to see every AI conversation as it happens, automatically tagged with whether the customer got help. Not aggregated into metrics that hide what’s actually going on. The real conversations.

This means you can search for specific problems. “Show me conversations where customers asked about exports.” “Which accounts are showing frustrated sentiment this week.” “What are the most common questions where users ask multiple times.”

For enterprise accounts, everything rolls up automatically. You see individual conversations and account-level trends. When sentiment at a major customer starts declining, you know immediately instead of finding out during renewal talks.

The setup should take days, not months. Add some code, start seeing conversations. No multi-tool integration projects. No data pipeline maintenance.

The test: can someone on your product team answer “which customers are frustrated with our AI right now” in under 30 seconds? If not, your monitoring isn’t working.

The Result

Product teams will have more autonomy over the products they own.

Sales can act on upsell opportunities and guide your “AI pilots” to a successful outcome.

Customer success & support can provide guidance to increase adoption.

Marketing teams can build nurture programs to help users.

Executives can see dashboards whenever they need.

All without the phone tag and silos caused by data being stuck in engineering’s data warehouse.

But the customer benefits the most.

If a customer has errors or gets frustrated, they will receive help before they give up.

For those in the US, we hope you enjoyed your holiday week. For those outside, we hope you’re enjoying your weekend.

Thank you for taking time to make our little newsletter a part of your reading.

I don’t have any Brixo updates to share today, but we will have major updates soon.

What stood out

Benedict Evan’s latest “AI is Eating the World” is out - all 90 slides.

I highly recommend taking the time to reach each section. I personally pulled out this chart which shows the stage of agentic deployments by status and industry.

This confirms what we’ve known all year but believe it will be turning more orange by next year.

What we read/listened/watched

Exponential View

🔮 Ten things I’m thinking about AI

On Sunday, 30th November, ChatGPT turns three. Wild three years…

Read more

5 months ago · 60 likes · 4 comments · Azeem Azhar

Box CEO, Aaron Levie, on agents

Final brick

We shipped an update to our website. Let us know what you think:

What stood out

Sierra AI announces $100M ARR milestone in 7 quarters

If agents were bad, would this happen?

The answer is definitively no.

We highlighted Sierra in a recent post on Top CX platforms. They have an absurd amount of cash and are led by Bret Taylor. If you don’t know who he is, take a look and a listen.

I’ve shared this account before and they have an interesting take that I believe in. Sierra’s success is the bull case for agents but the bear case for Web 2.0.

While agents and AI technology can feel bumpy and behave mysteriously at times, it provides exponential potential.

What we read/listened/watched

Elena Verna runs growth at Lovable. She has a great Substack and Linkedin feed. She gives an honest take on how fast the innovation is in AI, and how much pressure is facing these teams:

Elena's Growth Scoop

The Product-Market Fit Treadmill: Why every AI company is sprinting just to stay in place

Do you ever feel like you’re flat-out sprinting, just to stay in the same place? That’s the current reality of growth in AI companies, where the core foundations of your value proposition are up for grabs every week…

Read more

5 months ago · 44 likes · 2 comments · Elena Verna

An interesting read on the economics of AI, compute and everything else involved in powering this machine:

a16z

Bitter Economics

The direct consequence of the bitter lesson is that we’re building systems that turn large amounts of capital into working solutions. I would argue that this goes a long way toward explaining why company growth rates and funding are so different this AI cycle. And in the long run, this will be the most important and transformative aspect of AI…

Read more

5 months ago · 71 likes · Martin Casado

Our recent post about the limitations of observability, and why AI teams should consider Brixo.

Final Brick

Data centers in space. Just watch it:

I asked her one question: “Which of your customers are frustrated with your AI?”

She couldn’t answer.

Not because she wasn’t technical. Not because her tools were broken. But because every single tool she had was built to answer the wrong question.

Here’s the thing nobody in AI observability wants to admit: the entire market is optimizing for metrics that don’t matter to the people who actually decide whether your AI product succeeds or fails.

Everyone’s Building the Same Thing

Open any AI observability tool right now. LangFuse, Arize, Raindrop, Keywords AI. They’re all focused on the same metrics:

• Error rates

• Latency

• Token costs

• Model performance

• Prompt versions

These are engineering metrics. They tell you whether your system is running. They don’t tell you whether your customers are happy.

The VP I talked to could tell me her average latency down to the millisecond. She knew exactly how much each API call cost. She had alerts set up for every possible error condition.

But when I asked which of her top 20 enterprise accounts had a bad experience with their AI assistant yesterday, she had no idea. She’d find out when they stopped using it. Or worse, when they didn’t renew.

The Gap Nobody Talks About

You can have 99.9% uptime, sub-100ms latency, and perfect error handling. Your dashboards can be green across the board. And your biggest customer can still be quietly shopping your competitor because your AI keeps giving them useless responses.

Zero errors logged. Perfect uptime. Customer gone.

This is the blind spot.

I’ve had this conversation 50+ times in the last few months. Same pattern every time. Teams shipping AI to production. Spending thousands on observability. Flying completely blind on the one thing that matters: customer experience.

One founder told me: “We only know if customers don’t like our AI when they stop using it.” Another said: “There are inexplicable differences in our agent’s behavior and no way to know which customers were affected.”

These aren’t small companies with no budget. These are well-funded startups and enterprise teams with engineering resources. They’re doing everything “right” according to the current playbook. And they still can’t answer basic questions about customer experience.

Why This Happened

The AI observability market evolved from infrastructure monitoring. That’s the DNA.

Wave 1 was AI/ML monitoring. Model drift, data quality, training pipelines. Important stuff if you’re a data scientist.

Wave 2 was LLM observability. Prompts, tokens, latency, error rates. Critical if you’re an engineer debugging production issues.

Wave 3 should be customer experience. But nobody’s building it because everyone’s following the same playbook that worked for traditional observability.

The problem is: AI isn’t just infrastructure. It’s customer-facing. The thing that breaks isn’t always a system error. Sometimes it’s a perfectly functioning system giving a terrible answer. No error to log. No latency spike. Just a frustrated customer.

Engineering tools can’t see this. They weren’t designed to.

What Teams Actually Need

Here are the questions product teams ask me:

“Which customers are frustrated with our AI?”

“Which of our top 10 accounts had a bad experience yesterday?”

“What are customers actually asking our AI to do?”

“Is this issue affecting our enterprise buyers or just free users?”

Current observability tools can’t answer any of these. Not because they’re bad tools. They’re excellent at what they were designed to do. They just weren’t designed for this.

They don’t map interactions to customers. They don’t roll up to accounts. They don’t understand sentiment. They can’t tell you that 40% of your users at Microsoft (your biggest customer) got irrelevant responses last week.

Picture this scenario:

Your AI chatbot metrics look perfect. 99.5% success rate. 80ms average latency. Two cents per interaction. Your engineering team is happy. Your dashboards are green.

Meanwhile, half the users at your largest enterprise customer spent 15 minutes yesterday trying to get your AI to do something basic. It kept misunderstanding them. Gave generic responses. Made them repeat themselves.

They didn’t file a bug report. No errors were logged. But they walked away thinking your product doesn’t work. And now they’re in a meeting with your competitor.

This happens every day. Teams don’t find out until the customer stops using the feature. Or leaves entirely.

What’s Actually Needed

The solution isn’t better engineering metrics. It’s a different category entirely.

Customer-centric observability. Every AI interaction mapped to a customer. Rolled up by account. Sentiment tagged. Searchable by what customers are actually asking.

Product teams need to self-serve this data. No SQL. No waiting on engineering to pull reports. Visual, action-oriented insights they can use to fix problems before customers leave.

This doesn’t replace LLM observability. It complements it. Engineering still needs error rates and latency. But product teams need different answers. They need to know which customers are struggling. Which accounts are at risk. What’s working and what’s not from a customer perspective.

Both layers matter. Right now we only have one.

Why This Matters Now

Two years ago, most AI features were experiments. Nice to have. Teams could afford to be blind because the stakes were low.

That’s changing fast. AI is moving to production. Real customers. Real revenue. Real churn risk.

Enterprise buyers are starting to ask: “How do you monitor AI quality?” They want proof you can catch problems before they impact users. “We track error rates” doesn’t cut it anymore.

The question is shifting from “does your AI work?” to “do customers like your AI?”

In three years, asking “which customers are frustrated?” will seem as basic as asking “is the service up?” Right now, most teams can’t answer it.

That’s the gap. That’s the opportunity. And that’s what we’re building at Brixo.

More on that next week.

If you’re building AI observability tools and think I’m wrong, let me know. If you’re shipping AI and nodding along, reply - I want to hear your story.