Root-cause finder for recurring pipeline and report failures
Problem
A scheduled report fails, or a number in it looks wrong, and it is not the first time. The data engineer who gets pulled in opens the orchestration tool's run history, scrolls through transformation job logs, checks whether a vendor's export arrived late again, and searches the team's messaging channel for the thread where this happened last time, assuming anyone wrote it down. Most of the time nobody did, so the same investigation gets repeated from scratch by whoever is on call that week. The fix gets applied, the report goes out, and the pattern is never recorded anywhere a colleague could search, so the next occurrence starts the whole search again.
Product idea
An assistant that takes a recurring failure or a flagged number and pulls the job run history, schema change log and vendor feed timing around the failure window, then compares it against every past occurrence of the same failure it has seen. It proposes the likeliest cause ranked by how well the pattern matches history, for example a vendor export consistently landing three hours late on the days a downstream join breaks, and links directly to the run logs and schema diffs it used as evidence. Every diagnosis is saved, so the next occurrence starts from what is already known rather than from zero. It does not rerun pipelines or edit transformations. A person still decides and applies the fix.
Who it is for
Data engineers and analytics engineers who get pulled into repeat investigations, sponsored by the head of data who is tired of paying for the same diagnosis twice.
Possible first version
A first version that ingests exported logs rather than connecting live: CSV or JSON exports from the orchestration tool's run history and a manually maintained log of known schema changes, covering a single recurring failure the team nominates. It matches the failure window against past entries, ranks candidate causes and shows the evidence behind each one, with a simple field to record what the actual cause turned out to be. Out of scope for version one: any live connection to the orchestration tool or warehouse, automatic pipeline reruns, and coverage of more than one failure type.
- Build classification
- Micro-tool
- Rough effort
- 10-day prototype
- Roles involved
- Data engineer, Analytics engineer, Head of data
- Relevant to
- Professional club, League office, Federation / governing body, Collegiate athletics
- Systems in play
- Pipeline orchestration and scheduling, Transformation and modelling frameworks, Data catalogues and quality monitoring, Messaging apps
- Product framing
- Diagnose
Questions we get asked
What do we need to hand over before this can find anything useful?
An export of the orchestration tool's run history for the pipeline in question, and whatever record exists of schema changes, even an informal one. If nobody has been logging schema changes, the first few diagnoses will have less pattern to work with, but each confirmed diagnosis adds to what it can match against next time, so the second and third occurrence of the same failure get easier to explain, not harder.
We already run a data quality tool. Doesn't it already tell us this?
A data quality tool such as Monte Carlo or Soda is good at telling you a table looks wrong. It generally does not tell you why, or whether this is the third time the same vendor export caused it. This sits downstream of that alert: it takes the failure the quality tool already flagged and does the correlation work an engineer would otherwise do by hand, reading from your quality tool's alerts rather than replacing them.
Our engineers already know these pipelines. Why would they trust a tool's guess over their own judgement?
It isn't meant to be trusted over their judgement, it's meant to save them the search. The output is a ranked list of candidate causes with the actual logs and diffs behind each one, which the engineer reviews and can dismiss outright. The value shows up most for whoever is on call for a pipeline someone else built, where there is no personal memory of the last time this happened to draw on.
Who has to keep this fed once it's running?
Whoever resolves the incident needs to record what the actual cause turned out to be, which takes a couple of minutes and is usually the same engineer who was already writing up the fix. If that step gets skipped consistently, the pattern library stops improving and the tool slowly degrades back into a search tool with no memory of its own.
Is this your workflow?
Tell us one sports workflow that still runs on paper, spreadsheets, WhatsApp or an outdated system. We will map it and show you what a simpler product looks like.
Tell us about itMore in Data platform & engineering
- Access permission drift monitor for warehouse and reporting toolsA monitor that compares entity access rules across the identity provider, warehouse and reporting tool, and alerts a named owner when they drift out of sync.
- Attendance reconciliation report across ticketing and till systemsA reconciliation report that turns a ticketing export and a till export for the same fixtures into one defensible attendance and spend figure, with the variance between them shown and explained.
- Chat intake assistant that triages ad hoc data requestsA chat assistant that answers a data request instantly when the metric already exists, and turns anything new into a structured, tracked ticket instead of a message lost in a chat thread.