Kafka useful commands on Macbook or linux

-
  • Start up the Zookeeper.
./zookeeper-server-start.sh ../config/zookeeper.properties
  • Add the below properties in the server.properties
listeners=PLAINTEXT://localhost:9092auto.create.topics.enable=false
  • Start up the Kafka Broker
./kafka-server-start.sh ../config/server.properties

How to create a topic ?

./kafka-topics.sh –create –topic test-topic –bootstrap-server localhost:9092 –replication-factor 1 –partitions 4

Output: Created topic test-topic.

Describe Topic command

./kafka-topics.sh –describe –topic test-topic –bootstrap-server localhost:9092

output:

Topic: test-topic TopicId: O-uBj0D_R6aMhKMsTUgqhg PartitionCount: 4 ReplicationFactor: 1 Configs: segment.bytes=1073741824

Topic: test-topic. Partition: 0   Leader: 0. Replicas: 0. Isr:   0

Topic: test-topic. Partition: 1. Leader: 0. Replicas: 0. Isr:   0

Topic: test-topic. Partition: 2. Leader: 0. Replicas: 0. Isr:   0

Topic: test-topic. Partition: 3. Leader: 0. Replicas: 0. Isr:   0

How to instantiate a Console Producer?

Without Key

./kafka-console-producer.sh --broker-list localhost:9092 --topic test-topic

With Key

./kafka-console-producer.sh --broker-list localhost:9092 --topic test-topic --property "key.separator=-" --property "parse.key=true"

How to instantiate a Console Consumer?

Without Key

./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test-topic --from-beginning

With Key

./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test-topic --from-beginning -property "key.separator= - " --property "print.key=true"

With Consumer Group

./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test-topic --group <group-name>

To list down all topics

./kafka-topics.sh –bootstrap-server localhost:9092 –list

To know consumer groups:

./kafka-consumer-groups.sh –bootstrap-server localhost:9092 –list

List the topics in a cluster

./kafka-topics.sh --bootstrap-server localhost:9092 --list

Describe topic

  • The below command can be used to describe all the topics.
./kafka-topics.sh --bootstrap-server localhost:9092 --describe
  • The below command can be used to describe a specific topic.
./kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic <topic-name>

Alter the min insync replica

./kafka-configs.sh  --bootstrap-server localhost:9092 --entity-type topics --entity-name library-events --alter --add-config min.insync.replicas=2

Delete a topic

./kafka-topics.sh --bootstrap-server localhost:9092 --delete --topic test-topic

How to view consumer groups

./kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list

Consumer Groups and their Offset

./kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group console-consumer-27773

Viewing the Commit Log

./kafka-run-class.sh kafka.tools.DumpLogSegments --deep-iteration --files /tmp/kafka-logs/test-topic-0/00000000000000000000.log

Setting the Minimum Insync Replica

./kafka-configs.sh --alter --bootstrap-server localhost:9092 --entity-type topics --entity-name test-topic --add-config min.insync.replicas=2

Popular posts from this blog

What is Garbage collection in Spark and its impact and resolution

-
Image
Garbage Collection Spark runs on the Java Virtual Machine (JVM). Because Spark can store large amounts of data in memory, it has a major reliance on Java’s memory management and garbage collection (GC). Therefore, garbage collection  (GC) can be a major issue that can affect many Spark applications. Common symptoms of excessive GC in Spark are: Slowness of application Executor heartbeat timeout GC overhead limit exceeded error Spark’s memory-centric approach and data-intensive applications make it a more common issue than other Java applications. Thankfully, it’s easy to diagnose if your Spark application is suffering from a GC problem. The Spark UI marks executors in red if they have spent too much time doing GC. Spark executors are spending a significant amount of CPU cycles performing garbage collection. This can be determined by looking at the “Executors” tab in the Spark application UI. Spark will mark an executor in  red  if the executor has spent more than 10% of the time in gar...
Read more

How to change column name in Dataframe and selection of few columns in Dataframe using Pyspark with example

-
Input details: #● File has json records #● Each record has fields: #○ user_id #○ card_num #○ merchant #○ category #○ amount #○ ts ### Below analysis to be done Sample data: +------+--------+---------+--------+----------+-------+|amount|card_num| category|merchant| ts|user_id|+------+--------+---------+--------+----------+-------+| 243| C_108| food| M_102|1579532902| U_104|| 699| C_106|cosmetics| M_103|1581759040| U_103|| 228| C_104| children| M_110|1584161986| U_103| Solution: from pyspark.sql.functions import col # this can be done without using window function creditCardData = spark.read.json(“card_transactions.json”) useridMaxSpendDF = df.groupby(‘user_id’).max(‘amount’) useridMaxSpendDF=useridMaxSpendDF.withColumnRenamed(“max(amount)”,”max_amount”) useridMaxSpendDF=useridMaxSpendDF.withColumnRenamed(“user_id”,”m_user_id”) cond = [creditCardData.user_id == useridMaxSpen...
Read more

Window function in PySpark with Joins example using 2 Dataframes (inner join)

-
Input details: #● File has json records #● Each record has fields: #○ user_id #○ card_num #○ merchant #○ category #○ amount #○ ts ### Below analysis to be done Sample data: +------+--------+---------+--------+----------+-------+|amount|card_num| category|merchant| ts|user_id|+------+--------+---------+--------+----------+-------+| 243| C_108| food| M_102|1579532902| U_104|| 699| C_106|cosmetics| M_103|1581759040| U_103|| 228| C_104| children| M_110|1584161986| U_103| Solution: from pyspark.sql.functions import col # this can be done without using window function creditCardData = spark.read.json(“card_transactions.json”) useridMaxSpendDF = df.groupby(‘user_id’).max(‘amount’) useridMaxSpendDF=useridMaxSpendDF.withColumnRenamed(“max(amount)”,”max_amount”) useridMaxSpendDF=useridMaxSpendDF.withColumnRenamed(“user_id”,”m_user_id”) cond = [creditCardData.user_id == useridMaxSpen...
Read more