But there is a lot more to build.
Here is what is on the roadmap.
Right now the gold layer exposes raw dimensional tables and fact tables. The dashboard queries them directly with hand-written SQL. This works, but it means business logic lives in two places — the transformation layer and the dashboard queries.
The dbt Semantic Layer would centralise all metric definitions in one place. total_goals, win_rate, xg_overperformance — defined once in dbt, queryable everywhere. The dashboard would consume metrics rather than writing joins. No more drift between how a metric is calculated in one page versus another.
The pipeline runs nightly and the dashboard is public. If bad data makes it through, real users see wrong numbers — and there is currently no automated check stopping that from happening.
dbt has a built-in testing framework that fits naturally into the existing setup. Tests live alongside the models and run as part of the same pipeline. The basics are straightforward: uniqueness and not-null constraints on keys, accepted value checks on categorical columns, referential integrity between the fact table and every dimension. These catch the obvious failures — a venue ID that resolves to nothing, a match result outside the expected set, a duplicate surrogate key.
Beyond the built-in tests, the dbt-expectations package brings a richer set of statistical checks: row count thresholds, value range assertions, column distribution checks. These are useful for catching subtler issues — a round where suspiciously few goals were recorded, a team with negative possession, a season where no matches were flagged as complete.
The goal is for every nightly run to either produce correct data or fail loudly. Silent corruption is the worst outcome in a pipeline like this.
The bronze layer already ingests player-level data — appearances, goals, assists, shots, passes, cards, ratings — for every fixture. None of it surfaces in the dashboard yet.
The plan is to build a full player analytics layer on top of what is already there: top scorers, top assisters, player form over time, contribution per 90 minutes. A player profile page in the dashboard. Head-to-head comparisons.
The data is sitting in the warehouse. It just needs to be modelled and served.
Right now the pipeline only ingests Superligaen — the Danish top division. But the same API covers the full Danish football pyramid: the 1st Division (second tier), the 2nd Division, and the DBU Pokalen cup competition.
The plan is to extend ingestion to cover all of these, model them through the same bronze → silver → gold pipeline, and build dedicated dashboard pages for each competition. Teams moving up and down between divisions, cup upsets, cross-division comparisons — all of it becomes possible once the data is flowing.
This is the most experimental idea on the list.
The concept: a page in the dashboard where the data is not just displayed but discussed. Different analytical personalities — a statistician who trusts only the numbers, a football traditionalist who distrusts xG, a fan who reads into every result — analyse the same data and reach different conclusions.
The personas would be generated by a language model, grounded in the actual data from the warehouse, and updated each matchday. It would make the dashboard less of a static report and more of a living conversation about the season.
Whether this is useful or just a novelty is an open question. But it is worth finding out.
The current dashboard tells you what happened. The next step is to tell you what it means — and to do that, the visualisations need to work harder.
Right now most charts are single-metric bar charts or line charts. They are readable, but they leave a lot of the data on the table. The plan is to move toward techniques that surface relationships and context that are invisible in a single-axis view.
Scatter plots comparing attacking output versus defensive solidity across teams. Radar charts that give a full performance fingerprint for a team or player in a single glance. Rolling averages that separate a genuine form run from a single good result. These are standard tools in professional football analytics — and the data to drive all of them is already in the warehouse.
On the benchmarking side, the dashboard currently shows a team’s numbers in isolation. A win rate of 60% means something very different depending on whether the league average is 40% or 55%. The plan is to add contextual benchmarks throughout: league averages as reference lines on charts, percentile rankings alongside raw values, and head-to-head comparisons that anchor a team’s performance relative to its peers.
The goal is a dashboard where a casual fan understands the story at a glance, and an analyst can find genuine signal without exporting to a spreadsheet.
The original goal was to learn by building something real. That goal was met. But the more interesting discovery is that a project like this does not have a natural end — it just has the next thing to build.
The data keeps arriving. The season keeps moving. The dashboard keeps growing.
]]>The live dashboard is at superligaanalytics.vercel.app.
The final state of the project at launch:
fct_match_resultssuperligaen_dev and superligaen databases, dev and prod dbt targetsThis project went from initial commit to live dashboard in approximately ten days of active development. That is fast enough that almost every architectural choice was made under time pressure, with incomplete information, and revised at least once.
The tools that delivered exactly what they promised: MotherDuck, DuckDB, dbt, GitHub Actions. No surprises, no unexplained failures.
The tools that required more navigation: Netlify (build limits), Cloudflare Pages (file size limits), Evidence.dev (underdocumented behaviour around layouts and template syntax).
The choices I would make the same way: MotherDuck as the warehouse, dbt for transformations, Evidence.dev for the dashboard, Vercel for hosting, the Kimball star schema, the dev/prod database separation.
The choices I would make differently: evaluate hosting platforms against build frequency before committing; add dbt tests from the beginning rather than deferring them; set up dbt documentation from the start.
The ambitions that did not make it in: multi-league support (blocked by API quota), dbt tests, dbt semantic layer, dbt documentation, real-time match events (requires a paid API tier).
The project is now in maintenance mode. The nightly pipeline runs, the data updates, and the dashboard reflects last night’s results every morning. For a project built entirely on free tiers in ten days, that is a good place to be.
]]>The options were straightforward: Vercel Analytics (built into the hosting platform), Cloudflare Web Analytics (a separate free service), or Google Analytics (the industry default but heavier and requiring a cookie consent banner under GDPR).
Google Analytics was ruled out immediately. GDPR cookie banners are user-hostile and unnecessary for a project where we genuinely do not need detailed personal data — we just want page view counts and visitor numbers.
Both Vercel Analytics and Cloudflare Web Analytics are cookieless and privacy-first. They count visits using aggregated signals rather than tracking individuals. No consent banner required.
We decided to use both — not because we needed redundancy, but because each gives you a slightly different view of traffic data, and running both costs nothing.
Adding Vercel Analytics was not as simple as enabling a toggle in the Vercel dashboard. For SvelteKit apps (which Evidence.dev is built on), you need to install the @vercel/analytics npm package and call inject() somewhere in your app.
The natural place for a call that should run on every page is a layout component. Evidence.dev supports a pages/+layout.svelte file. I created one:
<script>
import { onMount } from 'svelte';
import { inject } from '@vercel/analytics';
onMount(() => inject());
</script>
<slot />
This broke the site completely. Every page lost its navigation, sidebar, theming, and layout chrome.
The reason: Evidence.dev has its own built-in +layout.svelte that imports its stylesheet, loads its default layout component (EvidenceDefaultLayout), and handles the app shell. When you create a pages/+layout.svelte, Evidence copies your file into its template directory, overwriting its own layout. My file only had <slot /> — which rendered the page content but none of Evidence’s surrounding UI.
The fix required knowing what Evidence’s own layout looked like. Once that was clear, the correct version wraps Evidence’s layout rather than replacing it:
<script>
import '@evidence-dev/tailwind/fonts.css';
import '../app.css';
import { EvidenceDefaultLayout } from '@evidence-dev/core-components';
import { onMount } from 'svelte';
import { inject } from '@vercel/analytics';
export let data;
onMount(() => inject());
</script>
<EvidenceDefaultLayout {data}>
<slot slot="content" />
</EvidenceDefaultLayout>
This correctly extends Evidence’s layout rather than replacing it.
Adding the Cloudflare beacon alongside Vercel Analytics required one more trick. The Cloudflare script tag looks like this in standard HTML:
<script defer src="https://static.cloudflareinsights.com/beacon.min.js"
data-cf-beacon='{"token": "your-token"}'></script>
The { and } characters in the data-cf-beacon attribute value are Svelte template delimiters. If you put this tag inside a <svelte:head> block, Svelte’s compiler tries to parse {"token": "..."} as a template expression and fails with a parse error.
The workaround: inject the script using document.createElement inside the onMount callback, where Svelte’s template compiler does not process the string:
onMount(() => {
inject(); // Vercel Analytics
const script = document.createElement('script');
script.defer = true;
script.src = 'https://static.cloudflareinsights.com/beacon.min.js';
script.dataset.cfBeacon = JSON.stringify({ token: 'your-token' });
document.head.appendChild(script);
});
This is less elegant than a <script> tag in the HTML but works correctly and is straightforward to understand.
run_silver.py and run_gold.py. Each script would read a directory of .sql files, connect to MotherDuck, and execute them in a specific order. It worked. The data was correct. But as the number of models grew and the logic became more complex, the cracks in the approach started to show.
Order dependency was manual. If dim_team needed to run before fct_match_results, you had to remember that and name or number the files accordingly. When we added a new model, figuring out where it slotted into the execution order was entirely up to the developer.
No incremental logic. Every run was a full rebuild. For silver models that flatten tens of thousands of fixture records, this was slow and unnecessary. There was no way to say “only process records that arrived since the last run” without writing custom Python logic around each SQL file.
No compilation validation. SQL syntax errors only appeared at runtime. There was no way to check whether a model was valid without actually running it against the database.
No lineage. There was no documentation of which model depended on what. Understanding the pipeline required reading the code, not querying a manifest.
Parameterisation was awkward. The nightly pipeline uses a rolling date window (last 5 days). The full-refresh pipeline uses a full season reload. Passing different variables to the same SQL file required string interpolation in Python, which is fragile and hard to read.
All of these problems have well-known solutions in the data engineering world. They are solved by dbt.
dbt (data build tool) is a transformation framework that sits on top of your data warehouse and manages SQL models. You write SQL SELECT statements, and dbt handles the CREATE TABLE AS, dependency ordering, incremental logic, and documentation.
The key features we needed:
Dependency resolution — dbt builds a DAG (directed acyclic graph) of your models by analysing which model references which. You write {{ ref('silver_fixtures') }} instead of a table name, and dbt knows to run silver_fixtures before whatever model references it.
Incremental models — dbt’s incremental materialisation allows a model to process only new or updated records on each run, using a configurable filter. For silver models that process fixture data, this means a nightly run that touches only the last 5 days of records rather than reprocessing all 200+ fixtures from every season.
Macros — dbt allows you to write Jinja macros for reusable SQL logic. We created three: fixture_filter() for filtering by date window, season_filter() for full-season reloads, and gold_incremental_filter() for the gold layer’s incremental logic. These macros mean the filter logic lives in one place and is consistent across all models.
CI validation — dbt’s compile command parses all SQL and resolves all references without executing anything. Adding dbt compile --target dev to the CI workflow (run on every pull request) means SQL syntax errors and broken references are caught before they ever reach main.
Schema management — dbt’s generate_schema_name macro controls the schema prefix applied to model outputs. We use this to ensure models land in exactly the right schema (silver or gold) regardless of the dbt target, without Cloudflare or Vercel prefixes being appended.
The migration itself took one day. The process:
dbt/ directory with dbt_project.yml and profiles.yml.{{ ref() }} calls where models depend on each other.dbt run --select silver.* and dbt run --select gold.* instead of python run_silver.py.Step 6 was satisfying. Deleting code that is no longer needed is one of the better feelings in software development.
There was one technical issue during the migration: DuckDB’s dialect handles some SQL constructs differently from other databases, and certain patterns that work in standard SQL do not work in DuckDB’s dbt adapter. The main culprit was date arithmetic — DuckDB uses INTERVAL syntax and epoch() for timestamp operations, which dbt’s generic date macros do not always handle correctly. We needed to write DuckDB-specific macro implementations rather than relying on dbt’s cross-database abstractions.
The dbt-duckdb adapter version also needed to match the DuckDB version that MotherDuck was running (1.5.1 at the time). Version mismatches between the adapter, the dbt-core version, and the DuckDB version produced cryptic errors that took a while to resolve. Once the versions were pinned and aligned, everything worked cleanly.
The dbt profiles configure two targets:
dev — points to superligaen_dev on MotherDuck. Used for local development and for CI.prod — points to superligaen. Used by the nightly GitHub Actions pipeline.The MotherDuck token is passed via the MOTHERDUCK_TOKEN environment variable rather than being stored in profiles.yml. This keeps credentials out of version control and makes rotating the token a matter of updating a GitHub Secret rather than committing a change.
dbt has a testing framework (dbt test) that lets you define assertions about your data — things like “this column has no null values”, “this foreign key always joins successfully”, “this value is always positive”. We did not implement any tests during this project. The right time to add them would be now that the models are stable and the architecture is not changing frequently. Tests would catch data quality regressions from the API before they reach the dashboard.
dbt also has documentation generation (dbt docs generate) that produces a browsable data catalog with descriptions, lineage graphs, and column-level documentation. Another thing worth adding.
The third item on the list is the dbt semantic layer. Rather than defining metrics like “goals per game” or “win rate” directly in dashboard SQL queries, the semantic layer lets you define them once in dbt and expose them as reusable, consistent metrics across any downstream tool. For a project where the same aggregations appear in multiple dashboard pages, this would eliminate duplication and make the numbers trustworthy by construction.
Next: a look at the API limits and infrastructure constraints that shape what this project can and cannot become.
]]>I asked Gemini for a recommendation on where to host an Evidence.dev dashboard. The answer was Netlify. Netlify is a well-established static site hosting platform with a generous free tier, good documentation, and a GitHub integration that makes deployment trivially easy — push to main, Netlify rebuilds and deploys automatically.
I set it up. The first build worked. The dashboard was live. Everything looked fine.
The problem appeared five days later.
Netlify’s free tier includes 300 build minutes per month. An Evidence.dev build — installing npm dependencies, running evidence sources to query MotherDuck, compiling the dashboard — takes roughly 4 to 5 minutes. If the nightly pipeline triggers a rebuild every night, that is 35 minutes a week, 150 minutes a month. Still within the limit.
Except: during active development, every push to the main branch triggered a rebuild. In those five days, between debugging pipeline issues, iterating on dashboard pages, and fixing configuration problems, I triggered somewhere around 60 to 70 builds. That was essentially the entire monthly quota. On day five, Netlify suspended the site’s builds until the next billing cycle.
I could not deploy. The site was frozen. I could have paid to upgrade, but paying for hosting on a side project did not feel right when the entire rest of the stack was on free tiers.
Netlify’s build limit is reasonable for a normal static marketing site that deploys a few times a week. It is not designed for a project in active development or for a pipeline that rebuilds nightly with data freshness as a feature. In hindsight, the right approach would have been to build Evidence.dev in GitHub Actions and upload the output to Netlify using the CLI, bypassing Netlify’s build system entirely. We eventually did implement this — but by then, there were other reasons to switch.
There were also token handling issues. Evidence.dev requires the MotherDuck token to be base64-encoded and written to a connection.options.yaml file before the build runs. Getting that into Netlify’s build environment in a way that survived across the npm run sources step required several attempts and a dedicated CI step to write the file.
Cloudflare’s market position is well-known. It runs a significant portion of the internet’s DNS and CDN infrastructure. Cloudflare Pages is their static site hosting product, and it is genuinely fast — assets are served from Cloudflare’s edge network, which means low latency everywhere. The free tier has 500 builds per month, which solved the quota problem immediately.
I migrated. The site was up on Cloudflare Pages. The pipeline was working. Things were good.
Then the data grew.
Evidence.dev bundles the query results as Parquet files into the static build output. As we added more dashboard pages — match results going back to 2020, player stats, referee data — the build output got larger. Cloudflare Pages has a 25 MB file size limit for individual deployable assets.
The bundled Parquet data from the Evidence.dev build was a few megabytes over that limit. Cloudflare would not deploy it. We tried compression options, tried splitting queries to reduce individual file sizes, tried serving some data lazily — nothing moved the needle enough without significantly compromising the dashboard’s performance.
This was a hard limit that Cloudflare was not going to lift on the free tier.
At this point I sat down and compared all three platforms properly against what this project actually needs:
| Requirement | Netlify | Cloudflare Pages | Vercel |
|---|---|---|---|
| Builds per month | 300 | 500 | Unlimited (hobby) |
| Max file size | 100 MB total | 25 MB per file | 100 MB per file |
| Build timeout | 15 min | 20 min | 45 min |
| GitHub integration | ✓ | ✓ | ✓ |
| Cost | Free tier | Free tier | Free tier |
Vercel’s hobby tier has no monthly build limit and a 100 MB per file limit. Both problems solved.
The migration itself was straightforward — connect the GitHub repository, configure the build command (npm run build) and output directory (build), add the MotherDuck token as an environment variable. First build succeeded.
The one complication with Vercel was controlling when it deploys. By default, Vercel rebuilds on every push to main. For this project, that would mean rebuilding every time a code change is merged — including changes that have nothing to do with the dashboard data. We only want to rebuild when the nightly pipeline finishes, or when manually triggered.
The initial approach was a Vercel deploy hook — a unique URL you POST to, and Vercel queues a build. The GitHub Actions pipeline would curl that URL at the end of the gold step.
This seemed simple. It was not.
The curl call was returning 200 but the build was not triggering. We added verbose logging to the curl command. The response body was correct. We added a separate test workflow that did nothing but curl the hook and report the response. It worked in isolation. When called from the main pipeline, it did not.
The exact reason was never fully isolated. There were several issues layered on top of each other: the Vercel deploy hook behaves differently when called from a GitHub Actions runner on certain network configurations, there were permission issues with the GITHUB_TOKEN in the workflow, and at one point a test commit was made to check whether Vercel was even watching the right branch.
We eventually abandoned the deploy hook approach entirely and replaced it with a dedicated deployment branch — publish_dashboard/vercel. The nightly pipeline’s final step makes an empty commit to that branch, and Vercel is configured to only watch that branch for deployments. The GitHub Actions step needs contents: write permission to push to a branch, which was another discovery made after the fact.
- name: Push to Vercel deploy branch
run: |
git config user.email "github-actions@github.com"
git config user.name "GitHub Actions"
git checkout -B publish_dashboard/vercel origin/main
git commit --allow-empty -m "chore: nightly data refresh $(date -u '+%Y-%m-%d %H:%M UTC')"
git push origin publish_dashboard/vercel --force
This approach is more reliable than a webhook because it uses standard git push semantics, which GitHub Actions handles correctly, and Vercel’s Git integration handles correctly. The empty commit triggers the deploy without cluttering the main branch history.
If I were starting over, I would evaluate hosting platforms against the specific build profile of the project — build frequency, output size, build time — before committing to anything. The differences between free tiers are not academic; they determine whether your project actually works at the end.
For Evidence.dev specifically, Vercel is the right choice as of today. Unlimited builds, large file support, straightforward integration. The deploy hook is unreliable and the branch-push approach is better.
Next: one of the most impactful refactors in the project — migrating the transformation layer to dbt.
]]>Tableau and Power BI were immediately out — they are expensive, and the free tiers do not allow public sharing without embedding tricks.
Grafana is excellent for operational metrics but not designed for product analytics dashboards. The user experience for building and sharing analytical views is awkward.
Metabase was a serious contender. It is open source, has a clean UI, and connects to DuckDB. But self-hosting Metabase adds infrastructure overhead, and the cloud version has a cost.
Streamlit would have worked, but it requires you to write Python to build UI, and the resulting dashboards do not look polished without significant effort.
Superset — similar story to Metabase: powerful, but infrastructure-heavy.
Evidence.dev is a different paradigm entirely. You write dashboard pages in Markdown. SQL queries go in fenced code blocks directly in the page. The output of each query is available as a variable in the same file. Charts, tables, and filters are Svelte components that you use inline in the Markdown. The whole thing compiles to a static site — no server, no runtime, just HTML, CSS, and JavaScript.
```sql matches
select match_date, home_team, away_team, score
from superligaen.match_results_by_match
order by match_date desc
limit 10
` ` `
<DataTable data={matches} />
That is the entire workflow. Write a SQL query, reference its name in a component, and you have a table on your dashboard. It is the fastest path from data to UI that I have found.
Evidence.dev connects to MotherDuck natively via the @evidence-dev/motherduck plugin. At build time, it runs the SQL queries against MotherDuck and bundles the results as Parquet files into the static build output. The deployed site loads these Parquet files at runtime using DuckDB-WASM — meaning the dashboard runs analytical queries entirely in the browser, with no server. It is genuinely impressive engineering.
We built seven pages:
Home — a hero banner with the league logo and flag, four KPI tiles (current leader, team count, matches played, goals scored this season), and a navigation grid linking to every other page.
Standings — three separate tables covering the Championship Round, Relegation Round, and Regular Season standings, with a season selector to browse historical tables.
Match Results — a filterable table of all historical results with a Goals vs xG chart by round that shows which rounds were overperforming or underperforming expected goals.
Upcoming Fixtures — a table of the next fixtures sorted by date and kick-off time, plus a match analysis section: select any upcoming fixture from the dropdown and see the head-to-head history between those two clubs and the last five results for each team.
League Analytics — cross-team benchmarks: top scorers, most disciplined teams, possession rankings, and season-level trends. This is the “zoom out” page.
Team Analytics — deep dive into a single team: select a team and see their season KPIs, recent form, shooting accuracy, possession stats, and discipline record. This page was the most technically interesting to build because it required combining multiple gold views with different grains.
Referee Analytics — cards issued, fouls per match, team exposure (which referees are most frequently assigned to which clubs), and a match log for each referee. This one came from a genuine curiosity: in a small league with a small pool of referees, team-referee familiarity is a real thing.
Evidence.dev is a static site generator. That means the data is frozen at build time. There is no live query against MotherDuck when a user loads the page — they are loading the data that was baked in during the last build. For a nightly pipeline that updates at midnight, this is perfectly fine. The dashboard is always at most 24 hours stale, which is acceptable for a league that plays two or three times a week.
The implication is that to refresh the data you need to trigger a new build. The nightly GitHub Actions pipeline does this automatically: bronze ingestion → silver dbt run → gold dbt run → trigger Vercel deploy. The deploy takes about two minutes and the site is updated.
A few things about Evidence.dev that were not obvious:
The % sign in SQL — Evidence.dev uses % as a template delimiter internally, which conflicts with the SQL LIKE operator pattern '%value%'. We ran into this when formatting percentage values. The fix was to use the pct0 format specifier that Evidence provides for formatting numbers as percentages, rather than concatenating a % symbol in SQL.
Sidebar and TOC — By default, Evidence.dev renders a table of contents and a sidebar on every page. For a dashboard that is supposed to look like a product, these are in the way. The sidebar: never and hide_toc: true frontmatter options suppress them. This was not in the main documentation but buried in a GitHub issue.
Mobile responsiveness — The default Evidence.dev layout is desktop-first. Getting the home page hero banner and the KPI grid to look right on mobile required custom CSS in the Markdown using Tailwind utility classes (which Evidence.dev ships with). Nothing dramatic, but it needed explicit work.
The evidence.config.yaml layout key — An early version of the config file had an invalid layout: key that was silently causing a build warning. We removed it when it started being noisy.
Next: the part of the project that took the most calendar time relative to its apparent simplicity — getting the dashboard deployed.
]]>The silver layer’s job is to take each bronze table and turn it into a proper relational table — extract columns from the JSON, cast types, handle nulls, and normalise nested structures. Every bronze endpoint gets a corresponding silver model.
DuckDB’s JSON handling is one of its best features. You can navigate nested JSON using arrow operators and the ->> syntax, and UNNEST is available for arrays. For a fixture statistics row that looks like this in bronze:
{
"fixture": {"id": 12345},
"statistics": [
{"type": "Shots on Goal", "value": 4},
{"type": "Ball Possession", "value": "55%"}
]
}
The silver transformation pivots this into a proper row with typed columns — shots_on_goal INTEGER, ball_possession_pct DECIMAL, and so on.
One rule we followed throughout: keep all columns in silver. When in doubt, keep it. Storage is cheap and you can always choose not to expose a column in gold or in the dashboard, but you cannot recreate data you threw away. Silver models kept logos, flags, URLs, internal IDs, everything — even things that looked useless at the time.
The fixture_players silver model was the most complex one to write. Each fixture returns a deeply nested JSON structure: a list of teams, each containing a list of players, each containing a list of statistics. Getting all of that into a flat table required multiple levels of UNNEST.
The initial version used nested UNNESTs — unnesting teams, then unnesting players within the same query, then unnesting statistics:
SELECT ...
FROM bronze.fixture_players,
UNNEST(response->'response') AS t(team),
UNNEST(t.team->'players') AS p(player),
UNNEST(p.player->'statistics') AS s(stat)
This worked fine in development on a local DuckDB instance with no memory constraints. When we ran it in production on MotherDuck’s free tier, it hit the 953 MB memory cap on the Pulse compute plan and crashed.
The fix was to stop doing all the unnesting in a single query and instead use sequential CTEs — unpack one level per CTE, materialise it, then unpack the next level from that:
WITH teams AS (
SELECT fixture_id, UNNEST(response->'response') AS team
FROM bronze.fixture_players
),
players AS (
SELECT fixture_id, team->>'id' AS team_id, UNNEST(team->'players') AS player
FROM teams
),
stats AS (
SELECT fixture_id, team_id, player->>'id' AS player_id, UNNEST(player->'statistics') AS stat
FROM players
)
SELECT ...
FROM stats
Each CTE processes a smaller set of intermediate results rather than holding the entire explosion in memory at once. After this refactor the query ran cleanly within the memory budget.
This was one of the more interesting problems in the whole project — not because the fix was complicated, but because the failure mode was invisible in development. Local DuckDB has no memory cap. The bug only appeared in production, and the error message (Out of Memory: cannot allocate) was not immediately helpful in pointing to nested UNNESTs as the culprit.
Once silver tables were clean and stable, I built the gold layer as a Kimball star schema. The fact grain is one row per team per match — meaning each fixture produces two rows in the fact table, one for the home team and one for the away team. This grain was chosen because most analytical questions in football are team-centric: how many goals has this team scored? What is their xG differential at home?
The fact table, fct_match_results, contains all the measurable numeric values — goals, shots, possession, passes, fouls, cards, expected goals, and points earned. Everything else is pushed into dimensions.
We ended up with 10 dimension tables:
| Dimension | What it represents |
|---|---|
dim_date |
Calendar attributes of the match date |
dim_time |
Hour of kick-off and period of day (Morning / Afternoon / Evening / Night) |
dim_team |
Club identity — name, code, country, logo |
dim_opponent_team |
Role-playing dimension — same structure as dim_team, aliased to represent the opposing club |
dim_match |
Match metadata — round, season, names, status |
dim_league |
League identity — name, country, logo, flag |
dim_stadium |
Venue — name, city, capacity, surface |
dim_referee |
Referee name |
dim_team_side |
Home or Away |
dim_match_result |
Win, Draw, or Loss |
Having dim_team and dim_opponent_team as separate dimensions makes self-join queries much cleaner. A query like “show me all home results where the opponent was a top-four side” is a simple join rather than a correlated subquery.
Every dimension uses an integer surrogate key as its primary key — team_sk, match_sk, and so on. These are stable across runs: when a new referee appears, they get a new SK, and existing referees keep theirs. This is the standard Kimball pattern.
Each dimension also has two sentinel rows:
-1 Unknown [Attribute] — for records where the value exists but is genuinely unknown-2 Not Applicable [Attribute] — for records where the dimension does not applyThese sentinel rows mean the fact table can always have a valid foreign key, even for fixtures that have no referee assigned yet (common for upcoming matches) or for venues that are listed as TBD. Dashboard queries never need a LEFT JOIN or a null check — every fact row joins cleanly.
One early version of the sentinel rows had generic labels like -1 Unknown and -2 Not Applicable. We later updated them to be attribute-specific: -1 Unknown Referee, -2 Not Applicable Stadium, and so on. This makes them instantly readable in query results without having to check which dimension you are looking at.
The star schema centres on a single fact table joined to ten dimensions. The grain is one row per team per match — each fixture produces two rows, one for the home team and one for the away team.
Next: building the dashboard on top of this model.
]]>The first version of the ingestion code was a single Python script that did everything: built URLs, called the API, handled pagination, and wrote to MotherDuck. It worked, but by the time it covered three or four endpoints it was already hard to follow. The first significant refactor split it into focused modules:
api.py — the HTTP client, rate limiting, retry/backoffdb.py — the MotherDuck connectionconfig.py — endpoint configuration, environment variablesingest_*.py — one file per logical group of endpointsThat structure stayed for the rest of the project.
api-football.com allows 10 requests per minute. The first version of the code just did time.sleep(6) between calls — six seconds per request keeps you just under the limit if every call takes zero time, which of course they do not. On fast calls the sleep is too short; on slow calls it is wasted time.
The proper solution is retry with exponential backoff: make the call, and if you get a 429, wait and try again. The wait doubles each retry with a small random jitter to avoid thundering herd problems. Here is the core of what we ended up with:
def api_get(url, params, retries=5):
for attempt in range(retries):
response = requests.get(url, headers=HEADERS, params=params)
if response.status_code == 200:
return response.json()
if response.status_code == 429:
wait = (2 ** attempt) + random.uniform(0, 1)
time.sleep(wait)
raise Exception(f"API call failed after {retries} retries: {url}")
This was more correct than sleep-based limiting and also faster on days when the API was responding quickly.
One of the more important early decisions was how to handle re-runs. If the nightly pipeline fails halfway through and you re-run it, you do not want to double-insert yesterday’s data. The pattern we chose was delete-before-insert: before inserting any records for a given date window, delete everything for that date window first. If the insert then succeeds, you have exactly one copy of the data. If it fails, the next run will delete and re-insert cleanly.
For full loads the pattern is a full table truncate before reloading. For incremental runs it is a targeted delete by date range — typically a rolling window of recent days to catch any late-arriving corrections from the API.
Getting this right took several iterations. One early bug was that the teams endpoint returns a JSON array of team objects, and the initial code was inserting the whole array as a single row rather than unnesting it first. Another was that the venues endpoint needed a season parameter that was not being passed, so it silently returned empty results for several seasons.
The final bronze layer covers 21 endpoints from the api-football.com free tier:
| Group | Endpoints |
|---|---|
| League | leagues, seasons, rounds, standings |
| Match | fixtures, fixture events, fixture statistics, fixture lineups, fixture player stats |
| Team | teams, venues |
| Player | players, top scorers, top assisters, top yellow cards, top red cards |
| Prediction | fixture predictions, fixture odds |
| Other | injuries, sidelined, trophies, coaches |
Each endpoint has its own ingestion script because the API parameters, pagination behaviour, and response shapes vary significantly. Some endpoints require a league ID and season. Some require a fixture ID and can only be fetched one fixture at a time (which is why fixture statistics alone accounts for a large chunk of the daily API call budget). Some, like odds and predictions, return data that changes right up until kick-off, so they need to be re-fetched regularly.
The ingestion runner supports two modes, controlled by a command-line flag:
--full-load — truncate and reload everything from 2020 to the current season. Used for initial bootstrap and occasional corrective runs.The --lookback parameter controls how many days back the incremental run looks. Setting it to 5 rather than 1 gives a buffer for late-arriving data and ensures that matches played over the weekend are picked up reliably even if the cron runs only once per day.
The nightly schedule runs at 23:00 UTC, which is midnight Danish time. That is late enough to catch the result of any evening match from the same day.
One subtlety with football APIs: the “current season” is not always obvious. Superligaen runs on a split-season calendar — the 2025/26 season starts in mid-2025 and finishes in mid-2026. Whether you are in the “2025” season or the “2026” season depends on the calendar and the convention used by the API.
The first version of the code hardcoded CURRENT_SEASON = 2024, which was wrong. The second version tried to derive it from today’s date using a heuristic (if month >= 7, season = year; else season = year - 1), which was better but still not correct around season transitions. The final version queries the leagues endpoint directly to find which season is currently active. The API knows — you should ask it.
Every bronze table has a consistent shape: the raw API response JSON is stored in a response column of type JSON, alongside metadata columns like inserted_at, season, and sometimes fixture_id or league_id depending on the endpoint. No type casting, no column extraction — just the raw blob.
This means bronze is not useful for direct querying, but it is a perfect foundation for silver. If we ever decide that a silver transformation was wrong, we can drop the silver table and rerun the transformation against the unchanged bronze data. That is the point.
Next: turning the raw JSON blobs into structured, typed tables — and then into a Kimball dimensional model.
]]>The first thing I did was look for a football API. Two candidates came up immediately.
football-data.org was the first I tried. The documentation is clean, the free tier is usable, and the community around it is solid. I set up an account, read through the available endpoints, and then hit the first wall: the Danish Superligaen is not included in the free tier. The free tier covers the top five European leagues — Premier League, La Liga, Bundesliga, Serie A, and Ligue 1 — plus a handful of cup competitions. Superligaen requires a paid plan. That was the end of that.
api-football.com was the second option, and it did have Superligaen in the free tier. The free tier gives you access to all leagues but caps you at 100 API calls per day. That sounds like a lot until you start mapping out what you need to fetch.
Each match needs: fixture metadata, statistics, events (goals, cards), lineups, player stats, and predictions. For a season with roughly 200 matches across multiple years, a full historical load requires thousands of calls just for fixtures. Add standings, top scorers, venues, referees, injuries, and odds, and a full bootstrap run across all available seasons (2020–2025) adds up to something in the hundreds of thousands of calls — spread over many days with careful throttling.
The 100-call-per-day limit was also one of the reasons I couldn’t expand the project beyond Superligaen. I wanted to include the Danish Cup, the Danish second division, or even add a comparison league from another country — Turkish Süper Lig would have been interesting. The architecture we built is genuinely scalable: adding a new league is just a config change. But doing so would immediately exceed the daily quota. That is a ceiling I am still bumping against.
There is also a rate limit within each day: 10 requests per minute. Exceeding it returns a 429, and the API does not give you a Retry-After header — you just have to know the limit and respect it. Early versions of the ingestion code used a naive sleep(6) between calls, which worked but was fragile. We later replaced it with a retry-with-backoff strategy that is both more correct and more efficient.
Once I knew the data source, I needed a place to store it. The options I considered were:
duckdb with a md: prefix on the connection string.MotherDuck won immediately. The developer experience is exceptional: you connect with a token, you write standard DuckDB SQL, and your data persists in the cloud. There is a web UI, a CLI, and it integrates natively with Evidence.dev (which I will get to later). For a side project, it removes every infrastructure concern that would otherwise become a time sink.
The one thing MotherDuck does not tell you upfront is that the free plan runs on a Pulse tier compute node with a 953 MB memory cap. That limit would come back to bite us several weeks later in a way that was not obvious at all, but more on that in a later post.
One decision I made early that saved a lot of grief: set up two separate databases on MotherDuck — superligaen for production and superligaen_dev for development. Every pipeline run, every dbt model, every SQL change would first be tested against superligaen_dev before being pointed at prod. This is standard practice in professional data engineering but easy to skip on a side project. I am glad I did not skip it.
The GitHub Actions workflows have a target_db parameter so you can point any run at either database. The dbt profiles have explicit dev and prod targets. This separation meant I could break things in superligaen_dev freely — and I broke things constantly — without ever risking the production data.
At this point the stack was: api-football.com as the source, MotherDuck as the warehouse, Python for ingestion, and some form of transformation and dashboard yet to be decided. The architecture I had in mind was a medallion architecture: three layers.
That design held throughout the project. The tools used to implement each layer changed significantly, but the three-layer architecture never did.
Next: building the bronze layer.
]]>I wanted to build an end-to-end data engineering project. Not a tutorial, not a sandbox — something real, with actual data, running in production. The full stack: ingestion, transformation, storage, a live dashboard.
The constraint I gave myself was simple: free, open source tools only. I wasn’t going to spend money on a personal project just to prove I could build something. If the tools couldn’t hold their own without a credit card, I’d find different tools.
The harder constraint was the data. I needed something real. Not synthetic, not historical-only, not something I’d lose interest in halfway through. Weather data felt generic. Titanic and movies felt like tutorial territory. I wanted data I’d actually want to look at after the build was done.
Then I remembered that I love football.
Football data from a major European league would mean: a live API with ongoing updates, a season structure that maps cleanly to a data model, and enough statistical depth to make analytics interesting. Fixtures, results, lineups, referee assignments, standings — the shape of the data is almost purpose-built for a star schema.
The league was the next question. I’d recently moved to Denmark. I follow the big European leagues like most football fans, but I realised I knew almost nothing about Danish football — the players, the clubs, the rivalries, how the season works.
That’s the moment the project clicked into focus. I wasn’t just going to build a data engineering showcase. I was going to build something I’d actually use: an analytics product for Superligaen, the Danish premier football league, aimed at people like me who want to understand it better.
That’s how this started. A love of data engineering, a constraint around cost, and a new country I wanted to understand through the sport I already loved.
The rest is what went wrong — and eventually right.
]]>