Apache Kafka Kafka Connect 1 — Questions and Answers
Question 1: What is Kafka Connect?
- A framework for reliably streaming data between Kafka and external systems using connectors (Correct answer)
- A protocol for connecting two Kafka clusters
- A GUI for managing Kafka topics
- A CLI for producing test messages
Correct answer: A framework for reliably streaming data between Kafka and external systems using connectors
Kafka Connect is a scalable, fault-tolerant framework that simplifies integrating Kafka with databases, storage systems, and other data sources/sinks.
Question 2: What is the difference between a 'source connector' and a 'sink connector' in Kafka Connect?
- Source connectors ingest data into Kafka; sink connectors export data from Kafka to external systems (Correct answer)
- Source reads from Kafka; sink writes to Kafka
- Source is for streams; sink is for batch
- Source handles keys; sink handles values
Correct answer: Source connectors ingest data into Kafka; sink connectors export data from Kafka to external systems
Source connectors pull data from external systems (like databases) into Kafka topics, while sink connectors push data from Kafka topics to external systems.
Question 3: What mode allows Kafka Connect to run as part of a distributed cluster for fault tolerance?
- Distributed mode (Correct answer)
- Standalone mode
- Cluster mode
- Replicated mode
Correct answer: Distributed mode
In distributed mode, multiple Connect workers form a group, share connector tasks, and automatically rebalance when a worker joins or leaves.
Question 4: How do you deploy a connector in Kafka Connect distributed mode?
- By posting a JSON configuration to the Connect REST API (Correct answer)
- By editing server.properties
- By running a CLI command on each broker
- By writing a custom Kafka producer
Correct answer: By posting a JSON configuration to the Connect REST API
You deploy connectors in distributed mode by sending a POST request with connector configuration JSON to the Connect workers' REST API endpoint.
Question 5: What is a 'task' in Kafka Connect?
- The unit of work that implements the actual data copying, with multiple tasks enabling parallelism (Correct answer)
- A scheduled job in ZooKeeper
- A Kafka consumer group assigned to one partition
- A schema version in the Schema Registry
Correct answer: The unit of work that implements the actual data copying, with multiple tasks enabling parallelism
A task is an implementation of the connector's copy logic; a connector can split work into multiple tasks running in parallel across Connect workers.
Question 6: What is the purpose of Single Message Transforms (SMTs) in Kafka Connect?
- Apply lightweight transformations to individual records as they pass through the connector pipeline (Correct answer)
- Transform entire topic schemas
- Compress messages at the broker
- Convert between Avro and JSON formats in the Schema Registry
Correct answer: Apply lightweight transformations to individual records as they pass through the connector pipeline
SMTs are simple, chainable transformations applied to each record in-flight within a connector, allowing field renaming, filtering, routing, and other mutations.
Apache Kafka Kafka Connect 1
What is Kafka Connect?