Data retention obligation register for warehouse tables
Problem
Legal or compliance keeps the data retention schedule in a policy document or spreadsheet: fan marketing data held for a set period, injury and medical records held longer under a different rule, a vendor's raw wearable feed with no agreed retention at all. Nobody has connected that document to what actually sits in the warehouse. A promotion table from three seasons ago, a medical note copied into a training log, a churned member's contact record, all sit untouched because the person who wrote the policy does not own the tables and the engineer who owns the tables has not read the policy. When an audit or a subject access request asks for proof that data past its retention date was deleted, the answer gets assembled from memory and old email threads, and it is often wrong.
Product idea
A register of data categories, each mapped to the tables and feeds that hold it, a retention period sourced from the policy, and a named owner. As a category approaches its deadline the register flags it amber, then red, to the owner, not to legal. The owner records what happened: deleted, anonymised, or extended with a written reason and an expiry for that extension. Nothing is deleted automatically. Every action is timestamped and kept as an audit trail, so a subject access request or an external audit is answered by exporting a log rather than reconstructing one from memory. Categories with no assigned owner sit in a visible unowned queue rather than silently ageing. It does not interpret the law: the retention period for each category is entered by whoever already owns that decision.
Who it is for
Data engineers and analytics engineers who own the tables, the head of data who is accountable for the register overall, and compliance staff who set retention periods but rarely see what is actually stored.
Possible first version
A web form for entering data categories, their retention period, mapped tables and an owner, an email alert when a category enters its warning window, a status view showing upcoming, due and overdue categories, and a log of actions taken with dates and reasons. Retention periods and table mappings are entered manually in version one. There is no automatic scan of warehouse schemas or vendor feeds to detect new tables, and no connection to a data catalogue. Escalation is a second email to the head of data after a set number of days, not a ticketing integration.
- Build classification
- Workflow application
- Rough effort
- 5-6 week build
- Roles involved
- Head of data, Data engineer, Analytics engineer, CRM and CDP manager
- Relevant to
- Professional club, League office, Federation / governing body, Venue & stadium operator
- Systems in play
- Cloud data warehouse, Spreadsheets, Data catalogue and governance tools
- Product framing
- Manage compliance
Questions we get asked
What do we need to have decided before we can set this up?
The retention period for each data category has to already be agreed by compliance or legal. This tool records that decision and tracks against it, it does not derive the period itself. If categories are not yet defined, a workable starting set is the handful that actually came up in the last audit or subject access request, rather than trying to enumerate every table in the warehouse on day one.
We already run a data catalogue. Doesn't that already cover this?
A catalogue documents what a table is, where it came from and who owns it. Few catalogues track when the data inside it has to be deleted or reviewed, and fewer still escalate to a named person when that date passes. This sits alongside the catalogue rather than duplicating it: the catalogue answers what the data is, the register answers when it has to go.
Compliance already keeps a spreadsheet of retention rules. Why replace it?
It probably shouldn't be replaced outright, it should be imported. The spreadsheet states the rule; it does not know which tables the rule applies to, does not flag when a table crosses the line, and produces no exportable record for an audit. Importing the existing spreadsheet as the starting register is the intended path, not writing the rules again from nothing.
Who ends up owning this once it exists?
Typically the head of data owns the register and decides what happens when a category goes unowned or overdue, while individual data engineers own the categories mapped to their own tables. The ongoing time cost is small if categories stay current, and grows quickly if new tables and feeds are never added, which is the most likely way this quietly stops being trustworthy.
Is this your workflow?
Tell us one sports workflow that still runs on paper, spreadsheets, WhatsApp or an outdated system. We will map it and show you what a simpler product looks like.
Tell us about itMore in Data platform & engineering
- Access permission drift monitor for warehouse and reporting toolsA monitor that compares entity access rules across the identity provider, warehouse and reporting tool, and alerts a named owner when they drift out of sync.
- Attendance reconciliation report across ticketing and till systemsA reconciliation report that turns a ticketing export and a till export for the same fixtures into one defensible attendance and spend figure, with the variance between them shown and explained.
- Chat intake assistant that triages ad hoc data requestsA chat assistant that answers a data request instantly when the metric already exists, and turns anything new into a structured, tracked ticket instead of a message lost in a chat thread.