Dears,
Today morning I changed the log level from DEBUG to INFO in log4j.properties file and did restart nodes one by one.
Now I have problems with my kafka cluster. One node is unstable, Kafka service restarts every few minutes.
I log file there is many lines like this one:
[2024-04-04 08:24:16,019] WARN [Log partition=__consumer_offsets-26, dir=/kafka_data] Non-monotonic update of high watermark from (offset=986277 segment=[986274:554]) to (offset=986274 segment=[986274:0]) (kafka.log.Log)
and:
[2024-04-04 06:47:57,451] ERROR Error while creating ephemeral at /brokers/ids/2, node already exists and owner '216214412779454482' does not match current session '72057609474473995' (kafka.zk.KafkaZkClient$CheckedEphemeral)
[2024-04-04 06:47:57,451] ERROR Error while creating ephemeral at /brokers/ids/2, node already exists and owner '216214412779454482' does not match current session '72057609474473995' (kafka.zk.KafkaZkClient$CheckedEphemeral)
[2024-04-04 06:47:57,457] ERROR [KafkaServer id=2] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists
at org.apache.zookeeper.KeeperException.create(KeeperException.java:126)
at kafka.zk.KafkaZkClient$CheckedEphemeral.getAfterNodeExists(KafkaZkClient.scala:1904)
at kafka.zk.KafkaZkClient$CheckedEphemeral.create(KafkaZkClient.scala:1842)
at kafka.zk.KafkaZkClient.checkedEphemeralCreate(KafkaZkClient.scala:1809)
at kafka.zk.KafkaZkClient.registerBroker(KafkaZkClient.scala:96)
at kafka.server.KafkaServer.startup(KafkaServer.scala:308)
at kafka.Kafka$.main(Kafka.scala:109)
at kafka.Kafka.main(Kafka.scala)
[2024-04-04 06:47:57,457] ERROR [KafkaServer id=2] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists
at org.apache.zookeeper.KeeperException.create(KeeperException.java:126)
at kafka.zk.KafkaZkClient$CheckedEphemeral.getAfterNodeExists(KafkaZkClient.scala:1904)
at kafka.zk.KafkaZkClient$CheckedEphemeral.create(KafkaZkClient.scala:1842)
at kafka.zk.KafkaZkClient.checkedEphemeralCreate(KafkaZkClient.scala:1809)
at kafka.zk.KafkaZkClient.registerBroker(KafkaZkClient.scala:96)
at kafka.server.KafkaServer.startup(KafkaServer.scala:308)
at kafka.Kafka$.main(Kafka.scala:109)
at kafka.Kafka.main(Kafka.scala)
How to solve this problem?
Best regards,
Dan
More from logs:
[2024-04-04 08:37:30,150] ERROR Error while truncating log to offset 986274 for __consumer_offsets-26 in dir /kafka_data (kafka.server.LogDirFailureChannel)
java.nio.file.NoSuchFileException: /kafka_data/__consumer_offsets-26/00000000000000986274.snapshot
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
...
at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:123)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
[2024-04-04 08:37:30,150] ERROR Error while truncating log to offset 986274 for __consumer_offsets-26 in dir /kafka_data (kafka.server.LogDirFailureChannel)
java.nio.file.NoSuchFileException: /kafka_data/__consumer_offsets-26/00000000000000986274.snapshot
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
...
at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:123)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
[2024-04-04 08:37:30,153] WARN [ReplicaManager broker=2] Stopping serving replicas in dir /kafka_data (kafka.server.ReplicaManager)
[2024-04-04 08:37:30,153] WARN [ReplicaManager broker=2] Stopping serving replicas in dir /kafka_data (kafka.server.ReplicaManager)
[2024-04-04 08:37:30,154] ERROR [ReplicaFetcher replicaId=2, leaderId=3, fetcherId=0] Failed to truncate __consumer_offsets-26 at offset 986274 (kafka.server.ReplicaFetcherThread)
org.apache.kafka.common.errors.KafkaStorageException: Error while truncating log to offset 986274 for __consumer_offsets-26 in dir /kafka_data
Caused by: java.nio.file.NoSuchFileException: /kafka_data/__consumer_offsets-26/00000000000000986274.snapshot
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
...
at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:123)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
[2024-04-04 08:37:30,154] ERROR [ReplicaFetcher replicaId=2, leaderId=3, fetcherId=0] Failed to truncate __consumer_offsets-26 at offset 986274 (kafka.server.ReplicaFetcherThread)
org.apache.kafka.common.errors.KafkaStorageException: Error while truncating log to offset 986274 for __consumer_offsets-26 in dir /kafka_data
Caused by: java.nio.file.NoSuchFileException: /kafka_data/__consumer_offsets-26/00000000000000986274.snapshot
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
..
at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:123)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
[2024-04-04 08:37:30,156] WARN [ReplicaFetcher replicaId=2, leaderId=3, fetcherId=0] Partition __consumer_offsets-26 marked as failed (kafka.server.ReplicaFetcherThread)
[2024-04-04 08:37:30,156] WARN [ReplicaFetcher replicaId=2, leaderId=3, fetcherId=0] Partition __consumer_offsets-26 marked as failed (kafka.server.ReplicaFetcherThread)
[2024-04-04 08:37:30,162] ERROR Uncaught exception in scheduled task 'delete-file' (kafka.utils.KafkaScheduler)
org.apache.kafka.common.errors.KafkaStorageException: The log dir /kafka_data is offline due to a previous IO exception.
[2024-04-04 08:37:30,162] ERROR Uncaught exception in scheduled task 'delete-file' (kafka.utils.KafkaScheduler)
org.apache.kafka.common.errors.KafkaStorageException: The log dir /kafka_data is offline due to a previous IO exception.
[2024-04-04 08:37:30,175] ERROR Uncaught exception in scheduled task 'delete-file' (kafka.utils.KafkaScheduler)
org.apache.kafka.common.errors.KafkaStorageException: The log dir /kafka_data is offline due to a previous IO exception.
[2024-04-04 08:37:30,175] ERROR Uncaught exception in scheduled task 'delete-file' (kafka.utils.KafkaScheduler)
org.apache.kafka.common.errors.KafkaStorageException: The log dir /kafka_data is offline due to a previous IO exception.
[2024-04-04 08:37:30,188] ERROR Uncaught exception in scheduled task 'delete-file' (kafka.utils.KafkaScheduler)
org.apache.kafka.common.errors.KafkaStorageException: The log dir /kafka_data is offline due to a previous IO exception.
[2024-04-04 08:37:30,188] ERROR Uncaught exception in scheduled task 'delete-file' (kafka.utils.KafkaScheduler)
org.apache.kafka.common.errors.KafkaStorageException: The log dir /kafka_data is offline due to a previous IO exception.
[2024-04-04 08:37:30,201] ERROR Uncaught exception in scheduled task 'delete-file' (kafka.utils.KafkaScheduler)
org.apache.kafka.common.errors.KafkaStorageException: The log dir /kafka_data is offline due to a previous IO exception.
[2024-04-04 08:37:30,201] ERROR Uncaught exception in scheduled task 'delete-file' (kafka.utils.KafkaScheduler)
org.apache.kafka.common.errors.KafkaStorageException: The log dir /kafka_data is offline due to a previous IO exception.
[2024-04-04 08:37:30,202] WARN [ReplicaManager broker=2] Broker 2 stopped fetcher for partitions __consumer_offsets-22,com.visiona.ecommerce.merchant-order.payment-1,com.visiona.ecommerce.blik.alias-0,__consumer_offsets-30,__consumer_o ffsets-8,__consumer_offsets-21,ecommerce-backend-prod-status-processing-terminal-payment-status-store-changelog-1,com.visiona.ecommerce.payment.refund-1,__consumer_offsets-4,ecommerce-backend-prod-technical-reversal-technical-reversal- task-store-changelog-1,com.visiona.ecommerce.hi.status-0,__consumer_offsets-27,__consumer_offsets-7,__consumer_offsets-9,__consumer_offsets-46,com.visiona.ecommerce.hi.status-1,ecommerce-backend-prod-refund-notification-processing-pend ing-refund-store-changelog-1,ecommerce-backend-prod-status-processing-cancelled-blik-payment-store-changelog-2,ecommerce-backend-prod-technical-reversal-KSTREAM-AGGREGATE-STATE-STORE-0000000002-changelog-1,__consumer_offsets-25,com.vis iona.babbler.consume-0,__consumer_offsets-35,com.visiona.ecommerce.payment.info-2,com.visiona.ecommerce.technical-reversal-task.store-1,__consumer_offsets-41,__consumer_offsets-33,__consumer_offsets-23,__consumer_offsets-49,__consumer_ offsets-47,__consumer_offsets-16,__consumer_offsets-28,com.visiona.ecommerce.payment.refund-2,__consumer_offsets-31,__consumer_offsets-36,__consumer_offsets-42,com.visiona.ecommerce.blik.alias-1,com.visiona.ecommerce.technical-reversal -task.store-2,__consumer_offsets-3,ecommerce-backend-prod-status-processing-pending-payment-status-store-changelog-1,__consumer_offsets-18,com.visiona.ecommerce.processing.error-0,ecommerce-backend-prod-status-processing-new-payment-st atus-store-changelog-0,__consumer_offsets-37,com.visiona.ecommerce.technical-reversal-task-count.store-0,__consumer_offsets-15,__consumer_offsets-24,ecommerce-backend-prod-notification-processing-payment-status-change-store-changelog-2 ,com.visiona.ecommerce.payment.flow-0,com.visiona.ecommerce.technical-reversal-task.store-0,com.visiona.ecommerce.payment.flow-1,com.visiona.babbler.consume-2,__consumer_offsets-38,__consumer_offsets-17,__consumer_offsets-48,com.vision a.ecommerce.payment.cleared-0,com.visiona.ecommerce.payment.cleared-1,com.visiona.ecommerce.merchant-order.payment-0,ecommerce-backend-prod-status-processing-pre-auth-cancel-task-store-changelog-0,__consumer_offsets-19,com.visiona.ecom merce.hi.status-2,__consumer_offsets-11,com.visiona.ecommerce.processing.error-1,com.visiona.ecommerce.blik.alias-2,com.visiona.ecommerce.payment.info-0,__consumer_offsets-13,__consumer_offsets-2,com.visiona.ecommerce.merchant-order.pa yment-2,__consumer_offsets-43,__consumer_offsets-6,__consumer_offsets-14,com.visiona.ecommerce.technical-reversal-task-count.store-1,com.visiona.babbler.consume-1,com.visiona.ecommerce.technical-reversal-task-count.store-2,com.visiona. ecommerce.payment.flow-2,com.visiona.ecommerce.processing.error-2,__consumer_offsets-20,__consumer_offsets-0,__consumer_offsets-44,com.visiona.ecommerce.payment.cleared-2,__consumer_offsets-39,__consumer_offsets-12,com.visiona.ecommerc e.payment.refund-0,__consumer_offsets-45,__consumer_offsets-1,__consumer_offsets-5,__consumer_offsets-26,__consumer_offsets-29,__consumer_offsets-34,__consumer_offsets-10,__consumer_offsets-32,__consumer_offsets-40,com.visiona.ecommerc e.payment.info-1 and stopped moving logs for partitions because they are in the failed log directory /kafka_data. (kafka.server.ReplicaManager)
[2024-04-04 08:37:30,202] WARN [ReplicaManager broker=2] Broker 2 stopped fetcher for partitions __consumer_offsets-22,com.visiona.ecommerce.merchant-order.payment-1,com.visiona.ecommerce.blik.alias-0,__consumer_offsets-30,__consumer_o ffsets-8,__consumer_offsets-21,ecommerce-backend-prod-status-processing-terminal-payment-status-store-changelog-1,com.visiona.ecommerce.payment.refund-1,__consumer_offsets-4,ecommerce-backend-prod-technical-reversal-technical-reversal- task-store-changelog-1,com.visiona.ecommerce.hi.status-0,__consumer_offsets-27,__consumer_offsets-7,__consumer_offsets-9,__consumer_offsets-46,com.visiona.ecommerce.hi.status-1,ecommerce-backend-prod-refund-notification-processing-pend ing-refund-store-changelog-1,ecommerce-backend-prod-status-processing-cancelled-blik-payment-store-changelog-2,ecommerce-backend-prod-technical-reversal-KSTREAM-AGGREGATE-STATE-STORE-0000000002-changelog-1,__consumer_offsets-25,com.vis iona.babbler.consume-0,__consumer_offsets-35,com.visiona.ecommerce.payment.info-2,com.visiona.ecommerce.technical-reversal-task.store-1,__consumer_offsets-41,__consumer_offsets-33,__consumer_offsets-23,__consumer_offsets-49,__consumer_ offsets-47,__consumer_offsets-16,__consumer_offsets-28,com.visiona.ecommerce.payment.refund-2,__consumer_offsets-31,__consumer_offsets-36,__consumer_offsets-42,com.visiona.ecommerce.blik.alias-1,com.visiona.ecommerce.technical-reversal -task.store-2,__consumer_offsets-3,ecommerce-backend-prod-status-processing-pending-payment-status-store-changelog-1,__consumer_offsets-18,com.visiona.ecommerce.processing.error-0,ecommerce-backend-prod-status-processing-new-payment-st atus-store-changelog-0,__consumer_offsets-37,com.visiona.ecommerce.technical-reversal-task-count.store-0,__consumer_offsets-15,__consumer_offsets-24,ecommerce-backend-prod-notification-processing-payment-status-change-store-changelog-2 ,com.visiona.ecommerce.payment.flow-0,com.visiona.ecommerce.technical-reversal-task.store-0,com.visiona.ecommerce.payment.flow-1,com.visiona.babbler.consume-2,__consumer_offsets-38,__consumer_offsets-17,__consumer_offsets-48,com.vision a.ecommerce.payment.cleared-0,com.visiona.ecommerce.payment.cleared-1,com.visiona.ecommerce.merchant-order.payment-0,ecommerce-backend-prod-status-processing-pre-auth-cancel-task-store-changelog-0,__consumer_offsets-19,com.visiona.ecom merce.hi.status-2,__consumer_offsets-11,com.visiona.ecommerce.processing.error-1,com.visiona.ecommerce.blik.alias-2,com.visiona.ecommerce.payment.info-0,__consumer_offsets-13,__consumer_offsets-2,com.visiona.ecommerce.merchant-order.pa yment-2,__consumer_offsets-43,__consumer_offsets-6,__consumer_offsets-14,com.visiona.ecommerce.technical-reversal-task-count.store-1,com.visiona.babbler.consume-1,com.visiona.ecommerce.technical-reversal-task-count.store-2,com.visiona. ecommerce.payment.flow-2,com.visiona.ecommerce.processing.error-2,__consumer_offsets-20,__consumer_offsets-0,__consumer_offsets-44,com.visiona.ecommerce.payment.cleared-2,__consumer_offsets-39,__consumer_offsets-12,com.visiona.ecommerc e.payment.refund-0,__consumer_offsets-45,__consumer_offsets-1,__consumer_offsets-5,__consumer_offsets-26,__consumer_offsets-29,__consumer_offsets-34,__consumer_offsets-10,__consumer_offsets-32,__consumer_offsets-40,com.visiona.ecommerc e.payment.info-1 and stopped moving logs for partitions because they are in the failed log directory /kafka_data. (kafka.server.ReplicaManager)
[2024-04-04 08:37:30,203] WARN Stopping serving logs in dir /kafka_data (kafka.log.LogManager)
[2024-04-04 08:37:30,203] WARN Stopping serving logs in dir /kafka_data (kafka.log.LogManager)
[2024-04-04 08:37:30,205] ERROR Shutdown broker because all log dirs in /kafka_data have failed (kafka.log.LogManager)
[2024-04-04 08:37:30,205] ERROR Shutdown broker because all log dirs in /kafka_data have failed (kafka.log.LogManager)
[2024-04-04 08:37:32,358] INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util)
[2024-04-04 08:37:32,462] INFO Registered signal handlers for TERM, INT, HUP (org.apache.kafka.common.utils.LoggingSignalHandler)
[2024-04-04 08:37:32,489] INFO Client environment:zookeeper.version=3.5.9-83df9301aa5c2a5d284a9940177808c01bc35cef, built on 01/06/2021 20:03 GMT (org.apache.zookeeper.ZooKeeper)
[2024-04-04 08:37:32,489] INFO Client environment:host.name=kafka-vimoni-prod-n2.kafka.prod.oraclevcn.com (org.apache.zookeeper.ZooKeeper)
[2024-04-04 08:37:32,489] INFO Client environment:java.version=11.0.22 (org.apache.zookeeper.ZooKeeper)
[2024-04-04 08:37:32,489] INFO Client environment:java.vendor=Red Hat, Inc. (org.apache.zookeeper.ZooKeeper)
[2024-04-04 08:37:32,490] INFO Client environment:java.home=/usr/lib/jvm/java-11-openjdk-11.0.22.0.7-1.0.1.el7_9.x86_64 (org.apache.zookeeper.ZooKeeper)
[2024-04-04 08:37:32,490] INFO Client environment:java.class.path=/opt/kafka/bin/../libs/activation-1.1.1.jar:/opt/kafka/bin/../libs/aopalliance-repackaged-2.6.1.jar:/opt/kafka/bin/../libs/argparse4j-0.7.0.jar:/opt/kafka/bin/../libs/au dience-annotations-0.5.0.jar:/opt/kafka/bin/../libs/commons-cli-1.4.jar:/opt/kafka/bin/../libs/commons-lang3-3.8.1.jar:/opt/kafka/bin/../libs/connect-api-2.8.2.jar:/opt/kafka/bin/../libs/connect-basic-auth-extension-2.8.2.jar:/opt/kafk a/bin/../libs/connect-file-2.8.2.jar:/opt/kafka/bin/../libs/connect-json-2.8.2.jar:/opt/kafka/bin/../libs/connect-mirror-2.8.2.jar:/opt/kafka/bin/../libs/connect-mirror-client-2.8.2.jar:/opt/kafka/bin/../libs/connect-runtime-2.8.2.jar: /opt/kafka/bin/../libs/connect-transforms-2.8.2.jar:/opt/kafka/bin/../libs/hk2-api-2.6.1.jar:/opt/kafka/bin/../libs/hk2-locator-2.6.1.jar:/opt/kafka/bin/../libs/hk2-utils-2.6.1.jar:/opt/kafka/bin/../libs/jackson-annotations-2.10.5.jar: /opt/kafka/bin/../libs/jackson-core-2.10.5.jar:/opt/kafka/bin/../libs/jackson-databind-2.10.5.1.jar:/opt/kafka/bin/../libs/jackson-dataformat-csv-2.10.5.jar:/opt/kafka/bin/../libs/jackson-datatype-jdk8-2.10.5.jar:/opt/kafka/bin/../libs /jackson-jaxrs-base-2.10.5.jar:/opt/kafka/bin/../libs/jackson-jaxrs-json-provider-2.10.5.jar:/opt/kafka/bin/../libs/jackson-module-jaxb-annotations-2.10.5.jar:/opt/kafka/bin/../libs/jackson-module-paranamer-2.10.5.jar:/opt/kafka/bin/.. /libs/jackson-module-scala_2.12-2.10.5.jar:/opt/kafka/bin/../libs/jakarta.activation-api-1.2.1.jar:/opt/kafka/bin/../libs/jakarta.annotation-api-1.3.5.jar:/opt/kafka/bin/../libs/jakarta.inject-2.6.1.jar:/opt/kafka/bin/../libs/jakarta.v alidation-api-2.0.2.jar:/opt/kafka/bin/../libs/jakarta.ws.rs-api-2.1.6.jar:/opt/kafka/bin/../libs/jakarta.xml.bind-api-2.3.2.jar:/opt/kafka/bin/../libs/javassist-3.27.0-GA.jar:/opt/kafka/bin/../libs/javax.servlet-api-3.1.0.jar:/opt/kaf ka/bin/../libs/javax.ws.rs-api-2.1.1.jar:/opt/kafka/bin/../libs/jaxb-api-2.3.0.jar:/opt/kafka/bin/../libs/jersey-client-2.34.jar:/opt/kafka/bin/../libs/jersey-common-2.34.jar:/opt/kafka/bin/../libs/jersey-container-servlet-2.34.jar:/op t/kafka/bin/../libs/jersey-container-servlet-core-2.34.jar:/opt/kafka/bin/../libs/jersey-hk2-2.34.jar:/opt/kafka/bin/../libs/jersey-server-2.34.jar:/opt/kafka/bin/../libs/jetty-client-9.4.48.v20220622.jar:/opt/kafka/bin/../libs/jetty-c ontinuation-9.4.48.v20220622.jar:/opt/kafka/bin/../libs/jetty-http-9.4.48.v20220622.jar:/opt/kafka/bin/../libs/jetty-io-9.4.48.v20220622.jar:/opt/kafka/bin/../libs/jetty-security-9.4.48.v20220622.jar:/opt/kafka/bin/../libs/jetty-server -9.4.48.v20220622.jar:/opt/kafka/bin/../libs/jetty-servlet-9.4.48.v20220622.jar:/opt/kafka/bin/../libs/jetty-servlets-9.4.48.v20220622.jar:/opt/kafka/bin/../libs/jetty-util-9.4.48.v20220622.jar:/opt/kafka/bin/../libs/jetty-util-ajax-9. 4.48.v20220622.jar:/opt/kafka/bin/../libs/jline-3.12.1.jar:/opt/kafka/bin/../libs/jopt-simple-5.0.4.jar:/opt/kafka/bin/../libs/kafka_2.12-2.8.2.jar:/opt/kafka/bin/../libs/kafka_2.12-2.8.2-sources.jar:/opt/kafka/bin/../libs/kafka-client s-2.8.2.jar:/opt/kafka/bin/../libs/kafka-log4j-appender-2.8.2.jar:/opt/kafka/bin/../libs/kafka-metadata-2.8.2.jar:/opt/kafka/bin/../libs/kafka-raft-2.8.2.jar:/opt/kafka/bin/../libs/kafka-shell-2.8.2.jar:/opt/kafka/bin/../libs/kafka-str eams-2.8.2.jar:/opt/kafka/bin/../libs/kafka-streams-examples-2.8.2.jar:/opt/kafka/bin/../libs/kafka-streams-scala_2.12-2.8.2.jar:/opt/kafka/bin/../libs/kafka-streams-test-utils-2.8.2.jar:/opt/kafka/bin/../libs/kafka-tools-2.8.2.jar:/op t/kafka/bin/../libs/log4j-1.2.17.jar:/opt/kafka/bin/../libs/lz4-java-1.7.1.jar:/opt/kafka/bin/../libs/maven-artifact-3.8.1.jar:/opt/kafka/bin/../libs/metrics-core-2.2.0.jar:/opt/kafka/bin/../libs/netty-buffer-4.1.73.Final.jar:/opt/kafk a/bin/../libs/netty-codec-4.1.73.Final.jar:/opt/kafka/bin/../libs/netty-common-4.1.73.Final.jar:/opt/kafka/bin/../libs/netty-handler-4.1.73.Final.jar:/opt/kafka/bin/../libs/netty-resolver-4.1.73.Final.jar:/opt/kafka/bin/../libs/netty-t cnative-classes-2.0.46.Final.jar:/opt/kafka/bin/../libs/netty-transport-4.1.73.Final.jar:/opt/kafka/bin/../libs/netty-transport-classes-epoll-4.1.73.Final.jar:/opt/kafka/bin/../libs/netty-transport-native-epoll-4.1.73.Final.jar:/opt/ka fka/bin/../libs/netty-transport-native-unix-common-4.1.73.Final.jar:/opt/kafka/bin/../libs/osgi-resource-locator-1.0.3.jar:/opt/kafka/bin/../libs/paranamer-2.8.jar:/opt/kafka/bin/../libs/plexus-utils-3.2.1.jar:/opt/kafka/bin/../libs/re flections-0.9.12.jar:/opt/kafka/bin/../libs/rocksdbjni-5.18.4.jar:/opt/kafka/bin/../libs/scala-collection-compat_2.12-2.3.0.jar:/opt/kafka/bin/../libs/scala-java8-compat_2.12-0.9.1.jar:/opt/kafka/bin/../libs/scala-library-2.12.13.jar:/ opt/kafka/bin/../libs/scala-logging_2.12-3.9.2.jar:/opt/kafka/bin/../libs/scala-reflect-2.12.13.jar:/opt/kafka/bin/../libs/slf4j-api-1.7.30.jar:/opt/kafka/bin/../libs/slf4j-log4j12-1.7.30.jar:/opt/kafka/bin/../libs/snappy-java-1.1.8.1. jar:/opt/kafka/bin/../libs/zookeeper-3.5.9.jar:/opt/kafka/bin/../libs/zookeeper-jute-3.5.9.jar:/opt/kafka/bin/../libs/zstd-jni-1.4.9-1.jar (org.apache.zookeeper.ZooKeeper)
[2024-04-04 08:37:32,490] INFO Client environment:java.library.path=/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper)
[2024-04-04 08:37:32,490] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper)
[2024-04-04 08:37:32,490] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)
[2024-04-04 08:37:32,490] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper)
[2024-04-04 08:37:32,490] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper)
[2024-04-04 08:37:32,490] INFO Client environment:os.version=5.4.17-2136.329.3.1.el7uek.x86_64 (org.apache.zookeeper.ZooKeeper)
[2024-04-04 08:37:32,490] INFO Client environment:user.name=kafka (org.apache.zookeeper.ZooKeeper)
[2024-04-04 08:37:32,490] INFO Client environment:user.home=/home/kafka (org.apache.zookeeper.ZooKeeper)
[2024-04-04 08:37:32,490] INFO Client environment:user.dir=/ (org.apache.zookeeper.ZooKeeper)
[2024-04-04 08:37:32,490] INFO Client environment:os.memory.free=4036MB (org.apache.zookeeper.ZooKeeper)
[2024-04-04 08:37:32,490] INFO Client environment:os.memory.max=4096MB (org.apache.zookeeper.ZooKeeper)
[2024-04-04 08:37:32,491] INFO Client environment:os.memory.total=4096MB (org.apache.zookeeper.ZooKeeper)
[2024-04-04 08:37:32,493] INFO Initiating client connection, connectString=10.21.1.212:2181,10.21.1.222:2181,10.21.1.220:2181 sessionTimeout=18000 watcher=kafka.zookeeper.ZooKeeperClient$ZooKeeperClientWatcher$@373ebf74 (org.apache.zoo keeper.ZooKeeper)
[2024-04-04 08:37:32,495] INFO jute.maxbuffer value is 4194304 Bytes (org.apache.zookeeper.ClientCnxnSocket)
[2024-04-04 08:37:32,508] INFO zookeeper.request.timeout value is 0. feature enabled= (org.apache.zookeeper.ClientCnxn)
[2024-04-04 08:37:32,530] INFO Client successfully logged in. (org.apache.zookeeper.Login)
[2024-04-04 08:37:32,532] INFO Client will use DIGEST-MD5 as SASL mechanism. (org.apache.zookeeper.client.ZooKeeperSaslClient)
[2024-04-04 08:37:32,561] INFO Opening socket connection to server kafka-vimoni-prod-n2.kafka.prod.oraclevcn.com/10.21.1.222:2181. Will attempt to SASL-authenticate using Login Context section 'Client' (org.apache.zookeeper.ClientCnxn)
[2024-04-04 08:37:32,564] INFO Socket connection established, initiating session, client: /10.21.1.222:50414, server: kafka-vimoni-prod-n2.kafka.prod.oraclevcn.com/10.21.1.222:2181 (org.apache.zookeeper.ClientCnxn)
[2024-04-04 08:37:32,571] INFO Session establishment complete on server kafka-vimoni-prod-n2.kafka.prod.oraclevcn.com/10.21.1.222:2181, sessionid = 0x2000051ef8f0004, negotiated timeout = 18000 (org.apache.zookeeper.ClientCnxn)
[2024-04-04 08:37:35,510] INFO Successfully logged in. (org.apache.kafka.common.security.authenticator.AbstractLogin)
[2024-04-04 08:37:36,482] ERROR Error while creating ephemeral at /brokers/ids/2, node already exists and owner '144115539987333123' does not match current session '144115539987333124' (kafka.zk.KafkaZkClient$CheckedEphemeral)
[2024-04-04 08:37:36,482] ERROR Error while creating ephemeral at /brokers/ids/2, node already exists and owner '144115539987333123' does not match current session '144115539987333124' (kafka.zk.KafkaZkClient$CheckedEphemeral)
[2024-04-04 08:37:36,489] ERROR [KafkaServer id=2] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists
at org.apache.zookeeper.KeeperException.create(KeeperException.java:126)
at kafka.zk.KafkaZkClient$CheckedEphemeral.getAfterNodeExists(KafkaZkClient.scala:1904)
at kafka.zk.KafkaZkClient$CheckedEphemeral.create(KafkaZkClient.scala:1842)
at kafka.zk.KafkaZkClient.checkedEphemeralCreate(KafkaZkClient.scala:1809)
at kafka.zk.KafkaZkClient.registerBroker(KafkaZkClient.scala:96)
at kafka.server.KafkaServer.startup(KafkaServer.scala:308)
at kafka.Kafka$.main(Kafka.scala:109)
at kafka.Kafka.main(Kafka.scala)
[2024-04-04 08:37:36,489] ERROR [KafkaServer id=2] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists
at org.apache.zookeeper.KeeperException.create(KeeperException.java:126)
at kafka.zk.KafkaZkClient$CheckedEphemeral.getAfterNodeExists(KafkaZkClient.scala:1904)
at kafka.zk.KafkaZkClient$CheckedEphemeral.create(KafkaZkClient.scala:1842)
at kafka.zk.KafkaZkClient.checkedEphemeralCreate(KafkaZkClient.scala:1809)
at kafka.zk.KafkaZkClient.registerBroker(KafkaZkClient.scala:96)
at kafka.server.KafkaServer.startup(KafkaServer.scala:308)
at kafka.Kafka$.main(Kafka.scala:109)
at kafka.Kafka.main(Kafka.scala)
[2024-04-04 08:37:37,181] INFO EventThread shut down for session: 0x2000051ef8f0004 (org.apache.zookeeper.ClientCnxn)