Apache Kafka
Kafka for Data Engineers
Kafka is the prominent queuing system which is the most used technology in all streaming solutions. In most of my streaming use cases at Walmart, I use Kafka either to consume records from it or produce to it. But, What makes Kafka most...
Intro to Kafka Security for Data Engineers - Part 1
I have a story about Kafka and Data Engineers that I'd like to share. In the world of Data Engineering, there are two types of engineers: those who have experience in real-time streaming pipelines and those who have only learned the basics.When both types of engineers....
Disaster Recovery in Kafka Servers
I recently tried onboarding disaster management for our streaming pipeline which involves Kafka, Spark Streaming and MongoDb in one of our use cases at Walmart Global Tech India. The last few weeks were a good learning curve for....
Spark Streaming
Spark Streaming - Part 1
A few months back, I was given a codebase that used Spark Streaming and it was written in scala. We were supposed to make major changes and go for the new version of the project. I had worked on Spark Structure Streaming before but not in....
Spark Streaming: Session 2
Spark Streaming mostly provides two types or categories of built-in streaming sources.
- Basic Source - Socket connection, File system
- Advance Source - Kafka, Pub/Sub, Kinesis
Basic source like socket connection & File System comes in very handy when you are learning....
Stateful transformations in Spark Streaming - Part 1
In this session, we will mainly focus on the following stateful transformations.
- updateStateByKey (This Article)
- reduceByKeyAndWindow (Next Article)
These functions will help you to understand the stateful transformation in...
Apache Spark
Syntactic Sugar in Spark Scala Codebase - Part 1
Have you heard of syntactic sugar in the codebase? What does it mean & moreover why I am focusing it on the Spark Scala codebase? In a handful of words, syntactic sugar is a syntax that allows developers to write code easier in a “sweet”, “concise” & “clear” way. Most of the code review....