Marketers needed custom reports to answer specific questions about their data—questions the Marketing Dashboard couldn't answer.

Dashboards show how marketing is doing generally, over time. Reports answer specific questions on fixed time periods for recording and communicating outside the system. Example questions:

• “How did our July 4th promotion perform compared to Memorial Day?”
• “Which audience segments drove the most revenue last quarter?”
• “What were total fulfilled orders by email name for Q3?”

Users needed a way to build custom queries, export the results, and share them with stakeholders. The dashboard wasn't built for this—reports were.

Building a visual query builder is straightforward from a UI perspective. The hard part is whether users understand how to use it: metrics, groupings, filters, and predicting what a report will look like before running it.

Through discussions with engineers, I learned reports could take 2+ minutes to run—sometimes much longer under load. Users couldn't afford trial-and-error. If they built the wrong query, they'd waste minutes waiting, then start over.

The critical question: can users translate their reporting needs into query logic? If not, we'd build a feature that frustrated users and wasted engineering time.

A Figma click-through wouldn't answer that question. Query builders require users to think—to translate reporting needs into metrics, groupings, and filters. I needed to watch them struggle, succeed, and reveal mental models in real time.

So I built a functional prototype in CodePen using HTML, CSS, and jQuery. Users could type, select, and manipulate real elements. localStorage persisted reports so they could create, save, and return to their work—without a backend or production codebase baggage.

I collaborated with a design manager to set up unmoderated testing on UserTesting.com. We recruited 10 users matching Mailchimp's sophisticated marketer ICP. Task-based scenarios (“Build a report showing revenue by audience”) plus open-ended qualitative questions at the end—users interacted with the live CodePen prototype.

Testing validated the visual query builder approach. Users understood the mental model and could successfully build reports. But we found one critical issue: the report preview was hidden behind a “Show layout preview” toggle.

My hypothesis was that users would want to preview their query before committing to a 2-minute wait. The hypothesis was correct—but the preview was too hidden. Users didn't know the feature existed, so they weren't using it to validate queries.

The change: show the preview by default. Don't make users discover it. Given the wait time, every user needed to see the structure before committing.

Interactive recreation: /experiments/preview-by-default

The final design gave users a visual query builder with instant feedback:

1. Report setup — name, audience (subscriber list), date range
2. Build the query — metrics, group by, optional filters
3. Preview & run — table structure with column headers and empty rows, then “Run report”
4. Report library — save, re-run on different date ranges, access from a library

Custom Reports shipped in early 2023, nearly pixel-perfect to the CodePen prototype. I handed off in December 2022; the team shipped shortly after. One notable change from prototype to production: the “Report type” dropdown was removed—likely a scope reduction or data architecture decision made after I left.

Marketing Dashboard·Creative Assistant