Data Engineering

Duration: 3 Days

Description

Every AI system is only as good as the data feeding it, and data engineering is the discipline that makes that data trustworthy, current, and available at scale. This course gives participants a practical foundation in building data pipelines: ingesting data from varied sources, transforming and validating it, storing it appropriately, and orchestrating workflows that run dependably even as volume and complexity grow. Coverage extends into the data patterns increasingly used to feed AI systems directly, including preparing and chunking data for retrieval pipelines. The emphasis is on concepts and patterns that apply across tools, with hands-on exercises that build real pipeline skills relevant to AI and analytics work.
Class time is divided between instructor-led explanation (40 percent) and hands-on exercises (60 percent).

Audience

Developers, data practitioners, and aspiring data engineers who want to build robust data pipelines for analytics, ML, and AI, including pipelines that feed retrieval and agentic systems. Assumes basic programming ability.

Objectives

• Explain the role and scope of data engineering
• Ingest data from files, databases, and APIs
• Transform, clean, and validate data reliably
• Choose appropriate storage for different needs
• Orchestrate and schedule data workflows
• Apply principles of data quality, reliability, and scale

Prerequisites

Participants should be comfortable writing code. The following will help:

• Working knowledge of Python (see Python Foundations)
• Basic familiarity with databases and data formats

Related AI Courses

See the full AI training roadmap and course directory for how this fits into a broader learning path.

Pair this course with Vector Databases and MLOps & LLMOps to see how well-engineered data pipelines feed retrieval and production AI systems.

Course Outline

• Module 1 – Course Introduction
  • Welcome and objectives
  • What data engineering is
  • The modern data stack overview
• Module 2 – Data Ingestion
  • Sources: files, databases, and APIs
  • Batch versus streaming ingestion
  • Connecting to and extracting data
  • Hands-on exercise: ingest from multiple sources
• Module 3 – Transformation and Validation
  • Cleaning and shaping data
  • Transformation patterns (ETL and ELT)
  • Data validation and quality checks
  • Hands-on exercise: build a transformation step
• Module 4 – Storage
  • Databases, warehouses, and lakes
  • Choosing storage for the use case
  • Schemas, partitioning, and formats
  • Storage for ML and AI workloads
• Module 5 – Orchestration
  • Workflows and dependencies
  • Scheduling and automation
  • Monitoring and failure handling
  • Hands-on exercise: orchestrate a pipeline
• Module 6 – Reliability and Scale
  • Data quality and observability
  • Idempotency and reprocessing
  • Performance and cost at scale
  • Pipelines that feed ML and AI
• Module 7 – Course Wrap-Up
  • Key takeaways
  • Resources for continued learning
  • Next steps