Dutch Bike Weather Pipeline
Weather data, made useful.
The starting point.
Weather arrives as nested API responses. Useful cycling advice requires validated records, consistent transformations, and storage that handles repeated runs safely.
What I built.
I built Open-Meteo ingestion, Pydantic validation, pandas cycling scores, PostgreSQL upserts and raw JSON archival in Azure Blob Storage. I added Docker packaging, model validation tests, Ruff checks and CI image publishing to Azure Container Registry.
Follow the flow.
Fetch one day of hourly forecasts for Amsterdam, Rotterdam, Utrecht, Den Haag, and Eindhoven.
The pipeline covers Amsterdam, Rotterdam, Utrecht, Den Haag and Eindhoven. A synthetic offline preview demonstrates validation and scoring without credentials; the cloud workflow checks the source and publishes a container image.
Decisions that matter.
- 01
Keep raw evidence and useful rows.
Archive the fetched forecast records as JSON in Azure Blob Storage while PostgreSQL holds the scored output. The two stores support inspection and downstream use.
- 02
Make a repeat run safe.
A unique city and forecast-time key lets PostgreSQL upsert existing observations instead of creating duplicates.
- 03
Test before the image ships.
Ruff and pytest validate the source and input model before CI builds a commit-tagged image. Image publishing and runtime scheduling are separate steps.
The work is open.
Implementation, decisions, and review history.
Built with
Python / Pydantic / pandas / Azure Blob Storage / PostgreSQL / Docker / ACR / GitHub Actions / pytest / Ruff