A collection of 12 posts tagged with "Apache Kafka"
Built a portable Java data pipeline using MinIO's AIStor and Kafka that scales from Mac to Kubernetes. Containerized stack (Kafka, AIStor, Prometheus, Grafana) processes millions of events, preserving raw data for analysis while delivering real-time dashboard summaries.
Read more
AIStor is a foundational component for creating and executing complex data workflows. At the core of this event-driven functionality is MinIO bucket notifications using Kafka.
Read more
AutoMQ enhances Kafka's architecture by using MinIO's object storage, cutting costs, and boosting elasticity while keeping Kafka API compatibility. This combo offers scalable, secure, and efficient data streaming, ideal for diverse cloud environments.
Read more
Build a streaming Change Data Capture (CDC) pipeline with Redpanda and MinIO into Snowflake. This solution simplifies data migration and analytics, with Redpanda offering scalability, MinIO as efficient storage, and Snowflake as a cloud-native analytics engine.
Read more
Confluent, Intel and MinIO conducted benchmarking and certification testing for MinIO Tiered Object Storage for Kafka storage. This blog post describes the observations and results of testing MinIO object storage as a backend for the tiered storage feature of Confluent Platform 7.1.0 on servers equipped with third generation Intel Xeon Scalable processors. The scope of these tests was
Read more
Explore the next generation of data streaming with WarpStream and MinIO! While Apache Kafka has been the standard for streaming data, it may be time to consider a simpler, more cost-effective, and cloud-native solution.
Read more
Apache Kafka and Apache Spark are two leading technologies used to build the streaming data pipelines that feed data lakes and lake houses. At a really high level, Kafka streams messages to Spark where they are transformed into a format that can be read in by applications and saved to storage.
Read more
Kafka and Spark Structured Streaming are used together to build data lakes/lake houses fed by streaming data and provide real time business insights.
Read more
Make you Kafka topics performant and efficient with Kafka Schema Registry.
Read more
Apache Kafka is an open-source distributed event streaming platform that is used for building real-time data pipelines and streaming applications. It was originally developed by LinkedIn and is now maintained by the Apache Software Foundation. Kafka is designed to handle high volume, high throughput, and low latency data streams, making it a popular choice for building scalable and reliable data
Read more
Streaming data is a core component of the modern object storage stack. Whether the source of that data is an edge device or an application running in the datacenter, streaming data is quickly outpacing traditional batch processing frameworks. Streaming data includes everything from log files (think Splunk SmartStore), web or mobile applications, autonomous vehicles, social networks and, of course financial
Read more
In the first part of this two post series, we’ll take a look at how object storage is different from other storage approaches and why it makes sense to leverage object storage like Minio for data lakes.
Read more
