Back to Work
Independent projectWorking prototypeData engineering

Xetra ETL Pipeline

A tested, containerized ETL pipeline for public trading data

2024Oyindamola Victor

The problem

The problem

Deutsche Börse publishes a non-commercial dataset of minute-level Xetra and Eurex trading data on a public AWS S3 bucket, through AWS's Open Data program, but it arrives as raw daily CSVs rather than anything shaped for analysis.

The approach

A Python pipeline reads the relevant files directly from the public deutsche-boerse-xetra-pds S3 bucket, tracks what's already been processed through a metadata step, and runs the data through transform and report stages before writing results back out to S3. Each stage was built and unit-tested incrementally, commit by commit, with the full pipeline wired behind a single entrypoint, profiled for memory use, linted, and packaged into a Docker image.

Implementation

Implemented

  • CSV extraction from Deutsche Börse's public Xetra S3 bucket
  • An update_meta_process step to track what's already been processed
  • Extract, transform_report1 and etl_report1 pipeline stages, each with unit tests
  • A single entrypoint wired up with Pipenv for dependency management
  • Code profiling with memory_profiler and linting with pylint
  • Dockerized for packaged execution
PythonpandasAWS S3DockerPipenvpylint

Results and status

Results and status

An independent practice pipeline, not a production system or client engagement — no business outcome is claimed here. The repository's own README documents nothing beyond the project name, so this case study is deliberately limited to what the commit history and file structure confirm: 14 commits building the pipeline incrementally between August 1-20, 2024, each implementation step paired with its own unit tests, and a working, Dockerized entrypoint.

Contributors & attribution

Contributors and attribution

Built by Oyindamola Victor. Works against Deutsche Börse's public, non-commercial Xetra trading dataset on AWS Open Data — a widely used public dataset for data-engineering practice, not a commercial or client dataset.

Have a similar operational challenge?

Tell us what's happening and we'll tell you honestly whether it's something we can help with.

Let's talk
Data Fellows

An independent automation studio solving recurring operational problems with software, data and AI, grown from a community of data learners and builders that's still going strong.

Partner network

We co-build with accelerators, training partners, and allied communities who keep Data Fellows grounded in real outcomes.

AWS StartupsMicrosoft for StartupsDataCamp ClassroomZummit AfricaVatebra AcademyCommunitechAccelerator CentreTBDC
© 2026 Data Fellows Inc. All rights reserved.Built by Data Fellows Inc.