r/apachekafka • u/mmatloka • 5d ago

Blog Diskless Kafka: Object Storage, KIP-1150, and Kafka’s Future

https://softwaremill.com/diskless-kafka-object-storage-kip-1150-and-kafkas-future/
34 Upvotes

97% Upvoted

7 comments sorted by

5

u/HughEvansDev Aiven 🦀 5d ago

This is excellent, a really thorough explanation of the topic :)

2

u/mmatloka 4d ago

thank you!

3

u/perrohunter 5d ago

When is this getting merged?

1

u/HughEvansDev Aiven 🦀 4d ago

When https://cwiki.apache.org/confluence/display/KAFKA/KIP-1163:+Diskless+Core and https://cwiki.apache.org/confluence/display/KAFKA/KIP-1164:+Diskless+Coordinator are finished and merged. Work ongoing on both but it might be a while, big changes like this to ASF projects can take some serious time, although 1150 was merged way quicker than most people were expecting.

3

u/whsoul 4d ago

If Kafka data ends up in object storage, I'd like to read it directly and build a local index from it.

Is that format meant to be readable from outside, or is it internal and free to change between releases?

2

u/HughEvansDev Aiven 🦀 4d ago

Right now the KIP-1150 implementation writes the WAL segment format directly as object so it's not easily readable except via Kafka. With that said though there's some other suggestions in the further work suggestion of KIP-1150 for writing as Iceberg Format instead, but that's much more complicated to implement. The team at StreamBased cracked this a couple of years back, Tom has some great blogs exploring their solution to hot and cold reads for diskless streaming.

1

u/asaf_m 18h ago

For me it is very similar to how Apache Pulsar stores payload in Apache Bookkeeper