English | MP4 | AVC 1280×720 | AAC 44KHz 2ch | 4h 14m | 1.24 GB
Kafka Connect – Learn How to Source Twitter Data, Store in Apache Kafka Topics and Sink in ElasticSearch and PostgreSQL
A comprehensive and new course for learning the Apache Kafka Connect framework with hands-on Training.Kafka Connect is a tool for scalable and reliable streaming data between Apache Kafka and other data systems. Apache Kafka Connect is a common framework for Apache Kafka producers and consumers.Apache Kafka Connect offers an API, runtime, and REST service to enable developers to define connectors that move large data sets into and out of Apache Kafka in real time. It inherits strong concepts such as fault-tolerance and elasticity thanks to being an extension of Apache Kafka. Kafka Connect can ingest entire databases, collect metrics, and gather logs from all your application servers into Apache Kafka topics, making the data available for stream processing with low latency. Kafka Connect standardises the integration of other data systems with Apache Kafka, simplifying connector development, deployment, and management. In this course, we are going to learn Kafka connector deployment, configuration, and management with hands-on exercises. We are also going to see the distributed and standalone modes to scale up to a large, centrally-managed service supporting an entire organisation or scale down to development, testing, and small production deployments. The REST interface is used to submit and manage connectors to your Kafka Connect cluster via easy to use REST API’s.
Kafka Connect standardises the integration of other data systems with Apache Kafka, simplifying connector development, deployment, and management.
What You Will Learn
- Configure and run Apache Kafka source and sink connectors
- Learn concepts behind Kafka Connect and the Kafka Connect architecture
- Launch a Kafka Connect cluster using Docker Compose
- Deploy Kafka connectors in standalone and distributed modes
- Write your own Kafka connector
Table of Contents
01 Important Pre-Requisites
02 Course Objectives
03 Course Structure
04 About Your Instructor
05 What is Kafka Connect
06 Kafka Connect Architecture Design
07 Connectors, Configuration, Tasks, Workers
08 Standalone vs Distributed Mode
09 Distributed Architecture in Details
10 Docker on Mac (recent versions)
11 Docker Toolbox on Mac (older versions)
12 Docker on Linux (Ubuntu as an example)
13 Docker on Windows 10 64bit
14 Docker Toolbox on Windows (older versions)
15 Starting Kafka Connect Cluster using Docker Compose
16 It’s not working! What to do
17 Where to view logs
18 Kafka Connect Source Architecture Design
19 FileStream Source Connector – Standalone Mode – Part 1
20 FileStream Source Connector – Standalone Mode – Part 2
21 FileStream Source Connector – Distributed Mode
22 List of Available Connectors
23 Twitter Source Connector – Distributed Mode – Part 1
24 Twitter Source Connector – Distributed Mode – Part 2
25 Section Summary
26 Kafka Connect Sink Architecture Design
27 ElasticSearch Sink Connector – Distributed Mode – Part 1
28 ElasticSearch Sink Connector – Distributed Mode – Part 2
29 Kafka Connect REST API
30 JDBC Sink Connector – Distributed Mode
31 Goal of the section – GitHubSourceConnector
32 Finding the code and installing required software
33 Description of the GitHub Issues API
34 Using the Maven Archetype to get started
35 Config Definitions
36 Connector Class
37 Writing a schema
38 Data Model for our Objects
39 Writing our GitHub API HTTP Client
40 Source Partition & Source Offsets
41 Source Task
42 Building and running a Connector in Standalone Mode
43 Deploying our Connector on the Landoop cluster
44 More Resources for Developers
45 Setting up Kafka Connect in Production (1_2)
46 Setting up Kafka Connect in Production (2_2)
47 What’s next
Resolve the captcha to access the links!