UX research

UX research

Analytics is not research

Analytics is not research

Stephanie smiling.
Stephanie smiling.

Stéphanie Dorval

Experience Design

Most B2B product teams can tell you how many clients asked for a feature. Far fewer can describe, in any detail, what a user was trying to get done in the minute before they gave up.

It isn't for lack of data: funnels, heatmaps and session replays are everywhere. But being data-literate isn't the same as knowing the people you design for.

At EDL, we gathered input from designers, researchers and product leaders at 60+ Danish B2B SaaS companies. I expected to find teams that had stopped talking to users and started watching dashboards instead. The data told a less comfortable story.

Analytics tells you where to look, not what you're looking at

Take a funnel showing 35% of admins dropping out at step three of a permissions setup. Most teams would read that as a usability problem and simplify the form. But the number only tells you where admins leave. It says nothing about why. Those admins may have left to ask IT which role to assign, or to rebuild the setup in a spreadsheet they trust more. The workaround is the finding, and it lives entirely outside your event stream.

Even the most heavily instrumented team in our sample knows this. A designer at a fintech that tracks everything still credits in-person merchant visits with surfacing "what kind of hacks or workarounds the users of the solutions come up with, things we would never have imagined."

Most teams we spoke to use analytics accordingly: as an alarm, not a compass. A UX director at an enterprise SaaS company described it as checking whether features "cry for an update". The trouble starts when the alarm gets mistaken for an explanation.

The limit is structural. Behavioural data only describes what happens inside your product. It can't see the job the user is trying to do, the workarounds in spreadsheets and email, the people who never used the feature, or the context around the click. More tracking sharpens the what. The why stays where it was.

Teams don't measure too much. They're too far away.

Analytics ranks fifth of six decision drivers


The usual version of this argument is that product teams stopped talking to users and started measuring them. Our data does not support it. Only 40% of respondents name analytics as something that drives their decisions, the second-lowest of the six drivers we asked about.

Asked whether "we make most of our product decisions based on data, not conversations", 69% disagreed.

Direct conversations do rank high (62%), but most teams have them monthly or less. What's always on is secondhand: 88% name at least one of leadership, gut feeling or feedback from sales and customer success. The drift isn't from talking to users towards measuring them. It's from talking to users occasionally towards hearing about them constantly, through someone else. Analytics isn't the only indirect signal standing in for research. It's just the most visible one.

The cost: we know the signals better than the people

Product teams read funnels, follow the customer success (CS) channel and know which client asked for what, but they rarely watch anyone work. A designer at a mid-size SaaS company said it plainly: "most of the time what we build does end up being the correct solution. But it's never built around, hey, let's try and get some people in, see if it's true."

Most teams don’t hear from users every week


User contact is rare. 38% of respondents hear from users weekly. The majority, 53%, only monthly or a few times a year, and 9% never. Contact usually happens around a big release, and research is the first thing to go when a deadline gets close.

Secondhand feedback never stops: 60% name sales and CS feedback as a driver. A product designer at a marketplace company, where sales and support share customer feedback every day, summed it up: "even though it's not firsthand experience, it's, in a way, secondhand experience." It's useful, but it's filtered through people whose job is to sell or keep accounts.

It also favours the loudest voices. A problem often only counts once several clients have raised it, and in B2B the easiest people to hear are the ones who pay. At one enterprise platform, subcontractors have to use the product but don't buy it, and the designer doubted their feedback ever reaches the team.

Analytics, when it's used, usually comes last: design it, test with a few people, launch, watch the numbers. A senior designer at a scale-up described skipping research on a heavily upvoted feature; the plan was to "release and test basically and see if we have some tracking on it." By then, the direction is already set.

This isn't negligence. 93% of respondents name a concrete barrier to doing more research: time, access and buy-in. Those are structural problems, and more training won't solve them.

Getting closer to users

Let the numbers send you to the users


None of this means less data. It means using data as a reason to go and talk to users, so that a metric opens a conversation instead of closing one. Five places to start:

  1. Make anomalies a research trigger. The fintech above tracks everything, and when a release behaves differently than expected, that becomes "a trigger that forces us to go out there in the specific market and investigate." It was the clearest case in our sample of analytics working as a question rather than a verdict.

  2. Treat secondhand signal as a lead, not evidence. Log sales and CS requests in your research repository. Five requests from one segment is a hypothesis to test, not a roadmap item.

  3. Recruit the user, not just the account. Map who buys, who uses and who is forced to use, and budget time to recruit that last group. No request channel will bring them to you.

  4. Give research an owner and a deadline. Research loses to delivery because only delivery has a date attached. A named owner and a slot in planning do more than any new tool.

  5. Decide how you'll judge a decision before you ship it. 20% of respondents can't tell whether a past decision was right. Write down the metric and the behaviour you expect, then check both: your analytics for the first, users for the second.

AI will widen the gap, unless we close it on purpose

AI is speeding up the build side: faster synthesis, drafted specs, prototypes in an afternoon. But faster building doesn't bring more understanding. It just means choosing between more options with the same thin picture of the user.

Faster building doesn’t bring more understanding


One team tried interviewing AI-simulated users instead of real ones. The answers simply confirmed what the team already believed. Like a dashboard, it gave a neat answer without the surprises a real person brings.

The takeaway

The problem was never analytics. It’s that almost every signal teams rely on, from dashboards to client requests, describes users from a distance.

Analytics tells you where to look, but only users tell you what you’re looking at


Analytics is not research, and neither is any other indirect signal. They tell you where to look. Only time with real users tells you what you're looking at, and as building gets cheaper, that time becomes the scarcest thing a product team has.

A simple test for your next roadmap decision: who on the team has actually watched a user deal with this problem? If the answer is nobody, you don't have research yet. You have signals.

Stephanie smiling.

stephanie-dorval.jpg

Stéphanie Dorval

Experience Design

Stéphanie combines holistic product understanding with solid design craftsmanship. She plays a key role in working with user flows, documentation, and handover to development.

Newsletter

Sign up to our newsletter and be the first informed

By submitting the form, you agree to our privacy policy.

Newsletter

Sign up to our newsletter and be the first informed

By submitting the form, you agree to our privacy policy.

Newsletter

Sign up to our newsletter and be the first informed

By submitting the form, you agree to our privacy policy.

Newsletter

Sign up to our newsletter and be the first informed

By submitting the form, you agree to our privacy policy.