NYC Taxi Databricks Lakehouse
Explore, model and merge.
The starting point.
Taxi records need explicit model grain and loading boundaries before an incremental analytics table can be trusted. The lab connects exploration, transformation and scheduled-job configuration in Databricks.
What I built.
Using the course-provided yellow-taxi tables, I explored borough and payment patterns with PySpark, built dbt staging views and a daily borough mart, and configured a Git-backed Databricks Job to run dbt.
Follow the flow.
Read the course-provided raw trips and zones; join borough information and inspect counts and payment-type averages.
The repository preserves the completed notebook, Delta model and a successful manual job run from July 2026. The saved run records three passing data tests and one warning-level check; its job trigger was paused.
Decisions that matter.
- 01
Make the daily grain explicit.
The inherited model name is fct_trips, but each row represents a borough and date. The merge key and uniqueness checks use those two columns together.
- 02
Understand the incremental boundary.
The current model only selects newer dates. Corrections to earlier days, late arrivals and additional records on the latest loaded day require a full refresh; a lookback window is a future improvement.
- 03
Keep cloud runs inspectable.
Notebook outputs, Delta history and job screenshots preserve what ran during the coursework. Local parsing checks the dbt structure, while executing the models needs access to the course source tables in Databricks.
The work is open.
Implementation, decisions, and review history.
Built with
Databricks / PySpark / SQL / dbt / Delta Lake / Databricks Jobs