At the beginning of all this my aim was to learn something new (Go), and use it to write a version of a utility that I’d previously hacked together in Python that checks your Apache Kafka broker configuration for possible problems with the infamous advertised.listeners setting. Check out a blog that I wrote which explains all about Apache Kafka and listener configuration.

You can find the code at https://github.com/rmoff/kafka-listeners

It’s been a fun journey, and now I am pleased to be able to show the results of it.

cap01

You need Confluent’s Golang Client for Apache Kafka™️, which you can install thus:

go get gopkg.in/confluentinc/confluent-kafka-go.v1/kafka

You can run the code directly with go:

➜ go run .
(A single commandline argument should be used to specify the broker. Defaulting to localhost:9092)
Using Broker: localhost:9092
--------------------------

✔️ Created AdminClient
✔️ Metadata - Originating broker [i.e. the broker to which you're connected here]
	[ID 1] localhost:9092/1
…

And you can also build it to create a single executable

go build
➜ ./listener_test asgard05.moffatt.me:9092
Using Broker: asgard05.moffatt.me:9092
--------------------------

✔️ Created AdminClient
✔️ Metadata - Originating broker [i.e. the broker to which you're connected here]
	[ID 1] asgard05.moffatt.me:9092/1
✔️ Metadata - Brokers [i.e. the advertised listeners of the brokers in the cluster]
	[ID 2] asgard05.moffatt.me:19092
	[ID 3] asgard05.moffatt.me:29092
	[ID 1] asgard05.moffatt.me:9092

✔️ ClusterID: Vf5L7L88QC66PLYDjN5Qvg
✔️ ControllerID: 3
✅ AdminClient worked
…

The rest of this article gives you some sample outputs for given situations.

Broker up but advertised.listeners is misconfigured

(A single commandline argument should be used to specify the broker. Defaulting to localhost:9092)
Using Broker: localhost:9092
--------------------------

✔️ Created AdminClient
✔️ Metadata - Originating broker [i.e. the broker to which you're connected here]
        [ID -1] localhost:9092/bootstrap
✔️ Metadata - Brokers [i.e. the advertised listeners of the brokers in the cluster]
        [ID 1] foobar:9092

😱 😱 😱 😱 😱 😱 😱 😱
🛑 None of the advertised listeners on the cluster match the broker (localhost:9092) to which you're connecting.

You're gonna have a bad time trying to produce or consumer with the config like this.

🔗 Check out https://www.confluent.io/blog/kafka-client-cannot-connect-to-broker-on-aws-on-docker-etc to understand more
✔️ ClusterID: vF4bIJeiQFiS7abLjBZ1tA
✔️ ControllerID: 1
✅ AdminClient worked
--------------------------

❌There was a problem calling the producer:


☠️ Uh oh, caught an error:
        foobar:9092/1: Failed to resolve 'foobar:9092': nodename nor servname provided, or not known (after 2ms in state CONNECT)
⚠️ Failed to flush all messages after 5000 milliseconds. 1 message(s) remain

Everything a-ok 👌

Using Broker: localhost:9092
--------------------------

✔️ Created AdminClient
✔️ Metadata - Originating broker [i.e. the broker to which you're connected here]
        [ID 1] localhost:9092/1
✔️ Metadata - Brokers [i.e. the advertised listeners of the brokers in the cluster]
        [ID 1] localhost:9092

✔️ ClusterID: KJeOfk4KTjegF1oH-UOwjg
✔️ ControllerID: 1
✅ AdminClient worked
--------------------------

✔️ Message 'foo / Fri, 17 Jul 2020 15:43:30 +0100' delivered to topic 'rmoff_test_00' (partition 0 at offset 0)
✨ All messages flushed from the queue
✅ Producer worked
--------------------------

Starting consumer, 👀 looking for specific message:
        foo / Fri, 17 Jul 2020 15:43:30 +0100

Subscribed to topic rmoff_test_00……
……
……
……
……
……
……
……
Partition 0 position unset
✅ Message 'foo / Fri, 17 Jul 2020 15:43:30 +0100' received from topic 'rmoff_test_00' (partition 0 at offset 0)
✔️ Read the message we were waiting for. Closing the consumer.
✅ Consumer worked
--------------------------

Broker down

Using Broker: localhost:9092
--------------------------

✔️ Created AdminClient
❌There was a problem calling the Admin Client:
😢 Error (Local: Broker transport failure) getting cluster Metadata. Is localhost:9092 valid broker and reachable from the machine on which this is running?

📺 More Episodes…