Mixpanel Agent Intelligence: Finally Answering "Is My Agent Worth It?"

Mixpanel introduced Agent Intelligence, a product built around a frustratingly simple question that most teams shipping agents could not answer: is this thing actually worth it? Teams were not short on data - they could tell you how many conversations happened, what each cost and how long responses took. What they could not tell you was what happened to their customers at scale. Agent Intelligence connects what the agent did in a conversation to what the customer did afterward, and whether that drove a business outcome. The design decisions behind it are worth studying even if you never buy the product.

Conversations land on the same user as the rest of your product

The problem: two tools and a lot of duct tape

The clearest signal Mixpanel found was not in a benchmark but in a workaround. Most teams measured their agent in a separate observability tool - sensible, since you need something to trace requests and debug failures - but that tool sits apart from the rest of the product data. So teams exported AI events into Mixpanel, built cohorts by hand, and stitched the agent back into the customer experience themselves. One customer put it plainly: when most of the experience happens inside a conversation, how are you supposed to connect that back to everything else you measure? The workaround was the product brief.

# The core idea: conversations are the unit a PM thinks in,
# not spans and traces. Agent Intelligence is built on OpenTelemetry.

INGEST = {
    'protocol': 'OpenTelemetry (OTLP)',
    'sdk_needed': False,
    'join_key': 'user_id (ties each conversation to the person)',
    'unit_of_analysis': 'conversation -> turn -> span -> tool call',
}

Decision one: conversations come first

Mixpanel made a choice that shapes almost everything else: the conversation is the primary unit you see, not the span or the trace. Spans and tool calls are the right view for an engineer debugging a single request, but they are not how a PM thinks about a customer. A PM wants to know what the customer asked, how the agent responded, and whether they got what they needed or had to ask three times. So the conversation is the top-level object, and everything underneath - each turn, each span, each tool call and what it returned - is nested one click down. Depth is preserved; the entry point is human.

# Send the traces you already collect; just add a user_id so each
# conversation connects to the rest of the customer journey.

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

exporter = OTLPSpanExporter(endpoint='https://api.mixpanel.com/v1/otel')

tracer = trace.get_tracer('my-agent')
with tracer.start_as_current_span('conversation') as span:
    span.set_attribute('user_id', current_user.id)
    span.set_attribute('conversation_id', conv.id)

Use the funnels, cohorts and retention reports you already have

Decision two: build on OpenTelemetry

The second decision was to build on OpenTelemetry rather than invent a proprietary ingest. That matters because most teams building agents already emit OTel traces, so there is no new SDK to add. You send the traces you already collect, alongside whatever other tools you use, and include a user ID so each conversation ties to the person behind it. This is the quiet enabler of the whole product: because agent events land on the same user as the rest of your analytics, the agent stops being a black box off to the side and becomes one step in a journey you can actually follow. Standards, it turns out, are the cheapest integration you will ever ship.

# Ask the question that engagement charts cannot answer:
# was a long conversation deeper, or just the same question rephrased?

def quality_signal(conversation):
    # turns alone are ambiguous; outcomes disambiguate
    return {
        'turns': conversation.turns,
        'resolved': conversation.ended_with_goal_met,
        'user_returned_7d': conversation.user.retained(7),
        'tool_failures': [t for t in conversation.tools if not t.ok],
    }

What you can finally answer

With conversations and customer behavior in one place, the questions change. You can tell a good conversation from a bad one without building a dashboard: cost, latency, error and volume metrics appear as soon as traces land. That matters because engagement alone misleads - a user sending six prompts might be going deeper, or might be rephrasing the same question because the agent keeps getting it wrong. On a usage chart those look identical; once you can open the conversation and see what happened, they do not. And because token spend and retention now live together, you can analyze them with the funnels, cohorts and retention reports you already use.

# Now you can use the reports you already know - funnels, cohorts,
# retention - because agent events land on the same user.

funnel = [
    'opened_agent',
    'conversation_started',
    'goal_achieved',
    'returned_next_week',
]

cohort = users.where(
    lambda u: u.used_agent and not u.returned(days=7)
)

One platform for the agent and the customer journey

Closing the loop

The point is not a prettier dashboard; it is fixing the agent. Mixpanel describes a representative loop: you spot a cohort of users who are not coming back after using the agent, dig into their conversations, and find they were all trying to do the same thing - and that the same tool call failed every time. You fix it, ship the new version behind a feature flag, and run it as an experiment to see whether retention improves. You go from noticing a problem to proving the fix without leaving the platform. As their customer at Sprout put it, knowing an agent responded does not tell you whether the customer achieved their goal, and that is what matters most.

# Close the loop: find the failing tool call, fix it, ship behind a
# feature flag, and run it as an experiment. All in one place.

STEPS = [
    'spot a cohort that does not return after using the agent',
    'open their conversations; find the same failing tool call',
    'fix it; ship behind a feature flag',
    'run an experiment; check whether retention improves',
]

How to wire it up

Setup is deliberately light: because it is built on OpenTelemetry, you export traces to Mixpanel with a user ID attribute and you are largely done. The work that remains is conceptual, not mechanical. Decide what a successful conversation means for your product, then instrument that outcome - a conversion, a resolved request, a return visit - so it becomes a first-class event rather than something you stitch together later. Pick one cohort that is not returning, open their conversations, and read what actually happened. If the first thing you learn is uncomfortable, the product is already working. A missing user ID is the one mistake that breaks the whole join.