Kafka Connect Change Log Level and Write Log to File

By default Kafka Connect sends its output to stdout, so you’ll see it on the console, Docker logs, or wherever. Sometimes you might want to route it to file, and you can do this by reconfiguring log4j. You can also change the configuration to get more (or less) detail in the logs by changing the log level.

Finding the log configuration file

The configuration file is called connect-log4j.properties and usually found in etc/kafka/connect-log4j.properties.

To find out the config file location simply inspect the running Kafka Connect process, or the existing stdout, and look for the location of connect-log4j.properties in -Dlog4j.configuration:

[2019-01-15 10:34:22,755] INFO WorkerInfo values:
        jvm.args = -Xms256M, -Xmx2G, -XX:+UseG1GC, -XX:MaxGCPauseMillis=20, -XX:InitiatingHeapOccupancyPercent=35, -XX:+ExplicitGCInvokesConcurrent, -Djava.awt.headless=true, -Dcom.sun.management.jmxremote, -Dcom.sun.management.jmxremote.authenticate=false, -Dcom.sun.management.jmxremote.ssl=false, -Dkafka.logs.dir=/var/folders/q9/2tg_lt9j6nx29rvr5r5jn_bw0000gp/T/confluent.tcrfo4zH/connect/logs, -Dlog4j.configuration=file:/Users/Robin/cp/confluent-5.1.0/bin/../etc/kafka/connect-log4j.properties

By default the file looks like this:

log4j.rootLogger=INFO, stdout

log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=[%d] %p %m (%c:%L)%n

log4j.logger.org.apache.zookeeper=ERROR
log4j.logger.org.I0Itec.zkclient=ERROR
log4j.logger.org.reflections=ERROR

Changing the log level

In troubleshooting Kafka Connect it can be useful to increase the log level to DEBUG (or even TRACE). To do this simply change the config file’s rootlogger:

#log4j.rootLogger=INFO, stdout
log4j.rootLogger=DEBUG, stdout

Sending logs to file

To send the output to file instead, change it to this:

log4j.rootLogger=INFO, logFile

log4j.appender.logFile=org.apache.log4j.DailyRollingFileAppender
log4j.appender.logFile.DatePattern='.'yyyy-MM-dd-HH
log4j.appender.logFile.File=/tmp/connect-worker.log
log4j.appender.logFile.layout=org.apache.log4j.PatternLayout
log4j.appender.logFile.layout.ConversionPattern=[%d] %p %m (%c)%n

log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=[%d] %p %m (%c:%L)%n

log4j.logger.org.apache.zookeeper=ERROR
log4j.logger.org.I0Itec.zkclient=ERROR
log4j.logger.org.reflections=ERROR

This will write it to /tmp/connect-worker.log.

Sending logs to both `stdout` and file

You can get fancy and send TRACE to file whilst retaining INFO (and above) on stdout, using the Threshold option for the stdout appender:

log4j.rootLogger=TRACE, traceFile, stdout

log4j.appender.traceFile=org.apache.log4j.DailyRollingFileAppender
log4j.appender.traceFile.DatePattern='.'yyyy-MM-dd-HH
log4j.appender.traceFile.File=/tmp/connect-worker-trace.log
log4j.appender.traceFile.layout=org.apache.log4j.PatternLayout
log4j.appender.traceFile.layout.ConversionPattern=[%d] %p %m (%c)%n

log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.Threshold=INFO
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=[%d] %p %m (%c:%L)%n

log4j.logger.org.apache.zookeeper=ERROR
log4j.logger.org.I0Itec.zkclient=ERROR
log4j.logger.org.reflections=ERROR

Finding the log configuration file

Changing the log level

Sending logs to file

Sending logs to both stdout and file

Sending logs to both `stdout` and file