Kafka Connect Flashcards

What is Kafka Connect?

Answer: A framework for reliably streaming data between Kafka and external systems using connectors

Kafka Connect is a scalable, fault-tolerant framework that simplifies integrating Kafka with databases, storage systems, and other data sources/sinks.

What is the difference between a 'source connector' and a 'sink connector' in Kafka Connect?

Answer: Source connectors ingest data into Kafka; sink connectors export data from Kafka to external systems

Source connectors pull data from external systems (like databases) into Kafka topics, while sink connectors push data from Kafka topics to external systems.

What mode allows Kafka Connect to run as part of a distributed cluster for fault tolerance?

Answer: Distributed mode

In distributed mode, multiple Connect workers form a group, share connector tasks, and automatically rebalance when a worker joins or leaves.

How do you deploy a connector in Kafka Connect distributed mode?

Answer: By posting a JSON configuration to the Connect REST API

You deploy connectors in distributed mode by sending a POST request with connector configuration JSON to the Connect workers' REST API endpoint.

What is a 'task' in Kafka Connect?

Answer: The unit of work that implements the actual data copying, with multiple tasks enabling parallelism

A task is an implementation of the connector's copy logic; a connector can split work into multiple tasks running in parallel across Connect workers.

What is the purpose of Single Message Transforms (SMTs) in Kafka Connect?

Answer: Apply lightweight transformations to individual records as they pass through the connector pipeline

SMTs are simple, chainable transformations applied to each record in-flight within a connector, allowing field renaming, filtering, routing, and other mutations.