Freshness monitor for vendor data pipelines and exports
Problem
An analyst pulls CSVs from roughly thirty vendor systems every Monday morning and only then finds out a feed broke on Thursday, because nothing tells anyone earlier. Pipelines a contractor built two years ago run on a schedule nobody reviews, and when one fails silently, the gap only surfaces when a board paper cites a number nobody can trace. Vendor schema changes are worse: a GPS provider renames a column or a ticketing platform adds a field mid-season, the transformation still runs, and the season-over-season comparison quietly breaks. There is no single place that shows, for any given source, whether today's load actually happened, whether the row count looks normal, or whether the shape of the data changed since last week.
Product idea
A single board listing every vendor feed and warehouse load next to its last successful run time, its row count against a rolling seven-day average, and a fingerprint of its column structure compared with the last known-good shape. Each source shows green, amber or red against thresholds set per feed, not one blanket rule, because a low-traffic sponsorship feed and a matchday ticketing feed do not fail the same way. Crossing a threshold sends an alert to the named owner of that source before the Monday pull, not after. It does not fix a pipeline, diagnose why a schema changed, or route the fix to anyone: it only tells the right person, early, that something needs looking at.
Who it is for
Data engineers who own the pipelines, analytics engineers who build on top of them, and the Head of data who sponsors it and needs to trust a number before it reaches a board paper.
Possible first version
A single dashboard reading load timestamps and row counts from one ingestion tool and the warehouse's own metadata tables, covering a defined list of critical feeds rather than all thirty at once. Thresholds are configurable per source, and a breach sends a Slack or email alert to a named owner. Out of scope for v1: automatic remediation, root-cause diagnosis of schema changes, and any ownership or escalation workflow beyond the single alert, since routing and follow-up belong to a separate tracker once one exists.
- Build classification
- Micro-tool
- Rough effort
- 10-day prototype
- Roles involved
- Data engineer, Analytics engineer, Head of data
- Relevant to
- Professional club, League office, Federation / governing body, Venue & stadium operator
- Systems in play
- Cloud data warehouses and lakehouses, Ingestion and ELT tools, Pipeline orchestration and scheduling, Messaging apps
- Product framing
- Monitor
Questions we get asked
What do we need to have in place before this shows us anything real?
Run logs or load timestamps from your ingestion tool, and row counts either from those logs or from the warehouse tables themselves, for the specific feeds you put in scope. Without that it has nothing to read. This is not a tool that runs on typed-in sample data, because the whole point is watching a real, live state, so a demonstration only means something once it is pointed at an actual source.
We already get failure emails from our orchestration tool. Why do we need another alert?
Orchestration failure emails fire when a job errors out. They say nothing when a job finishes cleanly but pulls in half the expected rows, or when a vendor renames a column and the load still succeeds against the new shape. That silent case is the one doing the damage to season comparisons. This watches the data behind a successful run, not just whether the job exited without error.
Does this replace a pipeline failure ownership tracker if we already have one?
No, it sits upstream of it. This decides whether a flag should be raised at all. An ownership tracker decides who is responsible for fixing it and whether it got fixed. Where both exist, the sensible wiring is for an alert from this board to open the ticket there rather than land in a separate inbox nobody checks.
What about a feed that legitimately runs late, like an off-season sponsorship export?
Thresholds are set per source rather than as one global rule, so a feed known to run weekly or only in season gets a longer allowed gap instead of being flagged stale every quiet week. Getting those thresholds wrong, in either direction, is the fastest way a tool like this loses trust, so they need setting by whoever actually knows that feed's real cadence, not left on a default.
Is this your workflow?
Tell us one sports workflow that still runs on paper, spreadsheets, WhatsApp or an outdated system. We will map it and show you what a simpler product looks like.
Tell us about itMore in Data platform & engineering
- Access permission drift monitor for warehouse and reporting toolsA monitor that compares entity access rules across the identity provider, warehouse and reporting tool, and alerts a named owner when they drift out of sync.
- Attendance reconciliation report across ticketing and till systemsA reconciliation report that turns a ticketing export and a till export for the same fixtures into one defensible attendance and spend figure, with the variance between them shown and explained.
- Chat intake assistant that triages ad hoc data requestsA chat assistant that answers a data request instantly when the metric already exists, and turns anything new into a structured, tracked ticket instead of a message lost in a chat thread.