Community Challenge: Data Engineering
Build a pipeline that turns raw e-commerce files into a daily sales report.
Join the ChallengeBuild a Dockerized pipeline on a real Brazilian e-commerce dataset (Olist). Load several related CSV files into a Postgres database, clean them with decisions you can defend, and model them with dbt until an analyst could query something useful. Over seven days you'll finish with a public GitHub repo that anyone can run with two commands.
The week at a glance
Day 1 · Thu, Oct 22
Git and Postgres in Docker
Skim Docker Compose (20 min max). Then clone the template repo, start a Postgres database with docker compose up -d, and connect to it. Personalise your README and commit. By tonight: Postgres running and your first personal commit. If Docker fails for more than 20 minutes, ask in the group.
Day 2 · Fri, Oct 23
README, folders, and how raw data will look
Skim how a dbt project is organised (20 min max). Then fill in your README skeleton with the question, dataset credit and architecture, and confirm your folder layout. Rebuild the database once to prove it works, then merge a pull request. By tonight: a README skeleton and a written plan for raw data.
Day 3 · Sat, Oct 24
Extract and a short inventory
Learn how to load files into a database (15 min max). Then finish src/extract.py to load the five CSVs into Postgres. It must be re-runnable, so running it twice doesn't double the rows. Run six quick checks on the data and record the numbers in your README. Don't clean anything yet. By tonight: five tables loaded and a data quality inventory.
Day 4 · Sun, Oct 25
Transform with prescribed defaults
Read about keeping raw data untouched (20 min max). Then finish src/transform.py to create four cleaned tables, using the defaults: only delivered orders count as trusted, and unique customers are counted by customer_unique_id. Write a Cleaning decisions table in your README. By tonight: four cleaned tables and your documented decisions.
Day 5 · Mon, Oct 26
dbt staging and key tests
Learn dbt basics and tests (25 min max). Then connect dbt to your database, build four staging models, and add the required tests (unique, not_null, relationships). Fix any failures properly. By tonight: dbt debug passes, four staging models run, and the key tests pass.
Day 6 · Tue, Oct 27
Marts that answer the question
Learn star schemas (15 min max). Then build two tables: fct_order_items (one row per trusted order item) and rpt_daily_sales (orders, sales value and unique customers per day). Connect everything into one command, python src/run_pipeline.py, and spot-check one day by hand. By tonight: the pipeline runs end-to-end and answers the question.
Day 7 · Wed, Oct 28
Finish and share
Polish and catch-up day. Finish your README (architecture, decisions, how to run, limitations) and confirm your .env file and CSVs are not committed. Rebuild from scratch to prove it works, then post your repo link to the community. By tonight: a repo a stranger can run and understand.