Apache Kafka : Installing and Running on Windows

This notes will provide a step by step guide to run Apache Kafka on a windows OS. This guide will also provide instructions to setup Java & zookeeper. Apache kafka is a fast & scalable messaging queue, capable of handling real heavy loads in context of read & write. You can find more about kafka on kafka. Apache kafka requires a zookeeper instance running which is used for reliable distributed coordination. Please find more about Zookeeper on zookeeper.

Downloading the Required Files

• Download Server JRE according to your OS & CPU architecture from http://www.oracle.com/technetwork/java/javase/downloads/jre8-downloads-2133155.html
• Download and install 7-zip from http://www.7-zip.org/download.html
• Download & extract Zookeeper using 7-zip from http://zookeeper.apache.org/releases.html
• Download & extract Kafka using 7-zip from https://www.apache.org/dyn/closer.cgi?path=/kafka/1.0.0/kafka_2.11-1.0.0.tgz

Installation

1. Install & Setup Java Runtime Environment (JRE)

• Follow this Step by step notes : Download and Install Java Development Kit (JDK) on Windows Steps by step

2. Installing & Running Zookeeper

Kafka uses ZooKeeper so you need to first start a ZooKeeper server if you don't already have one. Otherwise Install ZooKeeper by following below steps.

• Goto your zookeeper config directory. For me its C:\zookeeper-<version>\conf
• Rename file “zoo_sample.cfg” to “zoo.cfg”
• Open zoo.cfg in any text editor like notepad but I’ll prefer notepad++.
• Find & edit dataDir=/tmp/zookeeper to dataDir=C:\zookeeper-<version>\data
• Add entry in System Environment Variables as we did for java
• Add in System Variables ZOOKEEPER_HOME = C:\zookeeper-<version>
• Edit System Variable named “Path” add ;%ZOOKEEPER_HOME%\bin;
• You can change the default zookeeper port in zoo.cfg file (Default port 2181).
• Run zookeeper by opening a new cmd & type zkserver.

We can start zookeeper server from here or we can start from Kafka installation folder by using C:\kafka_2.11-0.9.0.0\config\zookeeper.properties.

3. Setting Up Kafka

• Go to your Kafka config directory. For me its C:\kafka_2.11-0.9.0.0\config
• Edit file “server.properties”
• Find & edit line “log.dirs=/tmp/kafka-logs” to “log.dir= C:\kafka_2.11-0.9.0.0\kafka-logs”.
• If your zookeeper is running on some other machine or cluster you can edit “zookeeper.connect=localhost:2181” to your custom IP & port. For this demo we are using same machine so no need to change. Also Kafka port & broker.id are configurable in this file. Leave other settings as it is.
• Your Kafka will run on default port 9092 & connect to zookeeper’s default port which is 2181. 

4. Running Kafka Server

Please ensure that your zookeeper is up & running before starting Kafka server.

• Go to your kafka installation directory C:\kafka_2.11-0.9.0.0\
• Open command line in your Kafka installation folder
• Launch Zookeeper with .\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties
• Open a second command line in your Kafka installation folder
• Launch single Kafka broker: .\bin\windows\kafka-server-start.bat .\config\server.properties

Now your Kafka is up & running, so now you can create topics & produce or consume data from java/scala code & command prompt also.

5. Creating Kafka Topic

• Now create a topic with name “test” & replication factor 1 as we have only one Kafka server running. If you have a cluster with more than 1 Kafka servers running, you can increase the replication-factor accordingly which will increase the data availability & act like a fault-tolerant system.
• Open a new command prompt in the location C:\kafka_2.11-0.9.0.0\bin\windows
• Type following command and hit enter
• kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

6. Creating a producer & consumer to test server

• Open a new command prompt in the location C:\kafka_2.11-0.9.0.0\bin\windows
• To start a producer type command “kafka-console-producer.bat --broker-list localhost:9092 --topic test”.
• Again open a new command prompt in the same location as C:\kafka_2.11-0.9.0.0\bin\windows
• Now start a consumer by typing command “kafka-console-consumer.bat --zookeeper localhost:2181 --topic test”.

Now you will have two command prompts like image below...

• Now type anything in the producer command prompt & press enter and you should be able to see the message in the other consumer command prompt.
• If you are able to push & see your messages in consumer side, your Kafka setup is ready.

If you think our posts are helpful, then do not forget to share our posts with your friends because your support makes better future. Register here for more updates ☆

Learn ~ Share ~ Grow ☺ Thanks & Happy Reading ✍