Discrepancy tracer for two dashboards reporting different numbers
Problem
When two departments report different numbers for the same fixture, something like attendance, revenue, or a conversion rate, the first response is a Slack thread asking which figure is correct, followed by an analyst opening both dashboards' underlying queries side by side. Finding the divergence means reading two hundred lines of SQL, checking whether one query filters on a different date boundary, joins a different lookup table, or excludes cancelled transactions the other includes. This can take the better part of a day, and it usually happens right before a board meeting, when the wrong number has already been quoted somewhere. Nobody keeps a record of what caused the last three discrepancies, so the same class of mismatch resurfaces every reporting cycle without anyone noticing the pattern.
Product idea
A diagnostic tool built for the moment two numbers disagree. Point it at two report definitions, a pair of SQL queries, two dbt model names, or two BI fields, and it walks each one back through its source tables, filters, joins and date ranges, then reports the exact step where they diverge: a different cancellation filter, a join that drops walk-up sales, a date boundary set to kick-off rather than gate-open. The output is a plain-English explanation with the row counts and conditions at each step as evidence, not just a diff of raw SQL. It does not decide which number is correct, and it does not change the metric definition. That decision stays with whoever owns the metric; this tool exists to make the argument evidence-based instead of a guess.
Who it is for
Analytics engineers and data engineers investigating a reporting mismatch, and the insight and BI analysts who get asked to explain it, sponsored by the head of data who owns the eventual fix.
Possible first version
The smallest useful version accepts two pasted SQL queries, or two model names from an uploaded dbt manifest file, runs them against a connected warehouse, and produces a step-by-step comparison of source tables, filters, joins and date ranges, ending in a plain-English divergence report. Out of scope for version one: any direct connection to a BI tool's API to pull dashboard definitions automatically, lineage tracing across ELT and BI tool boundaries, and any automatic correction of the metric. Report definitions are pasted or uploaded by hand until the pattern of use justifies a live connection.
- Build classification
- Micro-tool
- Rough effort
- 10-day prototype
- Roles involved
- Analytics engineer, Data engineer, Insight and BI analyst, Head of data
- Relevant to
- Professional club, League office, Federation / governing body, Collegiate athletics
- Systems in play
- Cloud data warehouse, dbt (or equivalent transformation framework), Business intelligence and visualisation tools
- Product framing
- Diagnose
Questions we get asked
What do you need from us to get this working on day one?
Read access to the warehouse and either the raw SQL behind each report or, if you run dbt, the project's manifest file. The tool does not need write access anywhere, and it does not need to see every table in the warehouse, only the ones the two disputed queries actually touch. A first useful run is normally possible against one real discrepancy rather than a synthetic example, because the whole point is showing the actual divergence, not a demonstration of the concept.
We already have a data catalogue with lineage graphs. Doesn't that already show this?
A catalogue's lineage graph shows what feeds what at the table level, which is necessary but not sufficient here. It will tell you both dashboards eventually read from the same fact table; it will not tell you that one query filters out refunded tickets and the other does not. This tool sits downstream of the catalogue and answers the narrower question a lineage graph is not built to answer: where, specifically, did these two numbers part ways.
Our analysts already do this by reading the SQL side by side. Why do we need a tool for it?
They do, and that is exactly why it takes most of a day each time: it is a manual read of two long queries under time pressure, redone from scratch on every occurrence because nothing from the last one was kept. The tool does not replace the analyst's judgement about which query is right; it removes the hours spent finding where they differ, and it leaves a record behind so the next occurrence of the same mismatch is recognised in minutes rather than investigated again from zero.
What happens once it finds the divergence? Does it fix the report?
No. It produces an explanation and the evidence behind it, and stops there deliberately. Changing a query or a metric definition based on that explanation is a decision for whoever owns the metric, and it should go through whatever sign-off your organisation already requires for a definition change. This tool is built to make that conversation shorter, not to make the decision for you.
Is this your workflow?
Tell us one sports workflow that still runs on paper, spreadsheets, WhatsApp or an outdated system. We will map it and show you what a simpler product looks like.
Tell us about itMore in Data platform & engineering
- Access permission drift monitor for warehouse and reporting toolsA monitor that compares entity access rules across the identity provider, warehouse and reporting tool, and alerts a named owner when they drift out of sync.
- Attendance reconciliation report across ticketing and till systemsA reconciliation report that turns a ticketing export and a till export for the same fixtures into one defensible attendance and spend figure, with the variance between them shown and explained.
- Chat intake assistant that triages ad hoc data requestsA chat assistant that answers a data request instantly when the metric already exists, and turns anything new into a structured, tracked ticket instead of a message lost in a chat thread.