Kafka Connect - Building Data Pipelines With Kafka

Learn how to use Kafka Connect to move data between Kafka and external systems.

What you'll learn

What Kafka Connect is and how to use it to move between Kafka and external systems

Download and install Kafka Connect Connectors

Run Kafka Connect in Standalone mode

Run Kafka Connect in Distributed mode

Run Kafka Connect in Docker containers

Manage Kafka Connect cluster using REST API

Requirements

Basic knowledge of Apache Kafka

Docker Desktop application installed on your computer

Java 17 or newer should be installed on your computer

Description

This video course is about Kafka Connect and how to use it to move data between Apache Kafka and external systems.You will start by learning what Kafka Connect is and how its core components work. I will explain the role of connectors, tasks, workers, and how Kafka Connect stores data in Kafka topics.Since Kafka Connect needs a running Kafka cluster, you will also learn how to run Apache Kafka in Docker containers. I will show how to use basic Kafka CLI tools to create topics, list them, and consume messages.Then we will focus on Kafka Connect.You will learn how to:Download and install Kafka Connect,Run Kafka Connect in standalone mode using Kafka CLI and Docker,Run Kafka Connect in distributed mode using Kafka CLI and Docker.I will also show how to manage Kafka Connect clusters using its REST API. You will learn how to check connector status, restart connectors, and remove them using simple HTTP requests.We will use Kafka Connect to:Move data from MySQL to Kafka using a source connector,Move data from Kafka to PostgreSQL using a sink connector.So, I will also show how to run MySQL and PostgreSQL databases in Docker. You’ll be able to test everything on your machine without installing anything manually.This course is beginner-friendly. You do not need to have any prior knowledge of Kafka Connect. By the end of this course, you will have a working data pipeline that streams data from a source database to Kafka and then from Kafka to a destination database.

Overview

Section 1: Introduction

Lecture 1 What is Kafka Connect?

Lecture 2 When and Why to use Kafka Connect?

Lecture 3 Kafka Connect Core Components: Workers, Connectors, and Tasks

Lecture 4 Kafka Connect Key Components: Transforms, Converters

Section 2: Software

Lecture 5 Download Apache Kafka

Lecture 6 Install Docker

Lecture 7 Run MySQL in Docker Container

Lecture 8 MySQL: Create database, database table and insert a new record.

Lecture 9 Run PostgreSQL in Docker Container

Lecture 10 PostgreSQL: Create database, database table, and insert a new record

Section 3: Apache Kafka Cluster

Lecture 11 Docker compose file for Apache Kafka cluster

Lecture 12 Starting Apache Kafka Cluster in Docker on Mac

Lecture 13 Starting Apache Kafka Cluster in Docker on Windows

Section 4: Kafka Connect Connectors

Lecture 14 Introduction to Kafka Connect Connectors

Lecture 15 Download Confluent JDBC Connector

Lecture 16 Download MySQL Connector

Lecture 17 Download PostgreSQL connector

Section 5: Kafka Connect Standalone mode - Using CLI scripts

Lecture 18 Introduction to running Kafka Connect in a standalone mode

Lecture 19 Kafka Connect worker configuration: Standalone mode

Lecture 20 Prepare MySQL Source Connector Configuration

Lecture 21 Run Kafka Connect in a Standalone mode

Lecture 22 Trying how Kafka Connect source connector works in a standalone mode

Lecture 23 Will it work if I delete a record from a source database?

Lecture 24 Managing Large Data Loads with Kafka Connect

Lecture 25 Sink Connector review

Lecture 26 Sink Connector configuration - PostgreSQL

Lecture 27 Source Connector & Sink Connector - Trying how it works

Section 6: Kafka Connect Standalone mode - In Docker container

Lecture 28 Kafka Connect Docker Compose file

Lecture 29 Kafka Connect in Docker Container: Worker Configuration

Lecture 30 Kafka Connect in Docker Container: JDBC Connector Configuration

Lecture 31 Kafka Connect with Docker Compose: Environment variables

Lecture 32 Starting Kafka Connect in standalone mode in Docker container

Section 7: Kafka Connect Cluster in Distributed Mode using CLI

Lecture 33 Introduction to Kafka Connect Distributed mode

Lecture 34 Kafka Connect Internal Topics in Distributed Mode

Lecture 35 Worker 1 - Creating new configuration file

Lecture 36 Worker 1 - Bootstrap Servers configuration

Lecture 37 Worker 1 - Group ID configuration

Lecture 38 Worker 1 - Key and Value Converters

Lecture 39 Worker 1 - Kafka internal topics settings

Lecture 40 Worker 1 - Kafka Connect REST API configuration

Lecture 41 Worker 1 - Plugins path configuration

Lecture 42 Creating Configuration file for Worker 2

Lecture 43 Running Kafka Connect workers in distributed mode using CLI

Lecture 44 Review Source Connector for Distributed mode

Lecture 45 Deploying Source Connector using REST API

Lecture 46 Review Sink Connector for Distributed Mode

Lecture 47 Deploying Sink Connector using REST API

Lecture 48 Kafka Connect Distributed mode - Trying how it works

Section 8: Managing Kafka Connect Cluster running in Distributed mode with REST API

Lecture 49 List Deployed Connectors with the Kafka Connect /connectors Endpoint

Lecture 50 Get Connector Configuration Details

Lecture 51 Checking Kafka Connect Connector status

Lecture 52 Pause or Resume Kafka Connect Connector

Lecture 53 Updating Kafka Connect Connector configuration

Lecture 54 Deleting Kafka Connect Connector using REST API

Section 9: Kafka Connect Cluster in Distributed Mode with Docker

Lecture 55 Introduction to running Kafka Connect in distributed mode with Docker

Lecture 56 Introduction to Docker File and Docker Image

Lecture 57 Creating Docker file for Kafka Connect

Lecture 58 Building Kafka Connect Docker image

Lecture 59 [Optional] Push Docker Image to Docker Hub

Lecture 60 An Environment file for Kafka Connect distributed mode

Lecture 61 Run Kafka Connect worker 1 in Docker container

Lecture 62 Run Kafka Connect worker 2 in Docker container

Lecture 63 Deploy Kafka Connector to a group of workers running in distributed mode

Java developers interested in Apache Kafka and Kafka Connect