A collection of 69 posts tagged with "Performance"
Introduction JSON has established itself as the "lingua franca" of the web. As such the parsing performance of JSON is hugely important for many applications. Despite the simple and human-friendly nature of JSON, it is not a technically trivial format to parse at high speeds. Recently some new designs have been presented one of which is simdjson by
Read more
With another Strata in the rearview mirror, it is time to reflect on what we saw and heard during the week. Strata is clearly a data science show at this point but data science is broad topic. Our perspective, as a provider of high performance object storage, is framed accordingly and we focus on the data stack more than we
Read more
The demise of Hadoop is probably overblown. It will not suddenly disappear from the enterprise landscape - there are simply too many clients, too much sunk investment for it to vanish into the night. What is not overblown is the fact that Hadoop, like countless technologies before it, is in secular, irreversible decline. There are a number of reasons but
Read more
Few would argue with the statement that Hadoop HDFS is in decline. In fact, the HDFS part of the Hadoop ecosystem is in more than just decline - it is in freefall. At the time of its inception, it had a meaningful role to play as a high-throughput, fault-tolerant distributed file system. The secret sauce was data locality. By co-locating
Read more
The power of scale is well-documented in the world of business. Cloud providers - Amazon in particular - have amassed extraordinary scale in a very short period of time. The cloud providers are now using this scale to rearchitect how enterprises interact with their data. They are remaking the enterprise data landscape with two primary levers: price and performance. Let’
Read more
Apache Spark is a framework for distributed computing. It provides one of the best mechanisms for distributing data across multiple machines in a cluster and performing computations on it. Spark achieves this by constructing data structures called RDDs (Resilient Distributed Datasets). RDDs allow data to be broken into disparate chunks and processed independently of one another. The individual chunks can
Read more
The growth of Presto in the enterprise is a function of its speed, SQL compatibility, extensibility and enterprise feature set. While initially designed to speed up Hadoop, the success of the project has led to much broader adoption - on S3, Cassandra, MySQL, and more. Presto allows for data queries that traverse data stores and locations - a big plus
Read more
Well written software is fast software. When MinIO was conceived it was designed from scratch to be simple, to scale (because simple things scale better) and to be fast. Simplicity and scale have their own subjective and objective measures - but fast is generally a numbers game. When you take well-written, fast software and pair it with fast hardware the
Read more
High performance object storage is one of the hotter topics in the enterprise today. On the one hand, object storage has become an indispensable part of the enterprise storage strategy (public or private cloud) - carrying the vast, vast majority of the enterprise burden when measured in TBs or PBs. On the other hand, object storage has traditionally served a
Read more
Software isn't usually described as bombproof. Particularly the type of software that is responsible for large analytic jobs or machine learning workloads. The words “finicky”, “complex” or in the case of good marketing “professional grade” (meaning you need years of study and multiple certifications) are more common. Bombproof software, however, is one of the many benefits associated with
Read more
When one looks at the amazing roster of talks for the Spark + AI Summit, what you don’t see is a lot of discussion on how to leverage object storage. On some level you would expect to —
Read more
One of the key challenges in any digitization journey is the adoption of machine learning techniques. Given the explosion of tools and frameworks, it can be difficult
Read more
When early object storage APIs were developed they focused on the efficient storage and retrieval of objects. Amazon’s success with S3 and its implementation of the robust S3 API quickly became the de facto standard for object storage in the cloud. MinIO, recognizing this, invested heavily in creating the most compliant implementation of the S3 API outside of Amazon.
Read more
How to use Apache Flink to build a private cloud data pipeline for a variety of use cases.
Read more
In this post we’ll learn more about object storage, specifically Minio and then see how to connect Minio with tools like Apache Spark and Presto for analytics workloads.
Read more
In the first part of this two post series, we’ll take a look at how object storage is different from other storage approaches and why it makes sense to leverage object storage like Minio for data lakes.
Read more
In this post, we learn about Pivotal Container Service deployment and how to use the pks command line tool to create and manage Kubernetes clusters. We also saw how to deploy Minio once your PKS Kubernetes cluster is set up and running.
Read more
In this post, we learn about why and how Presto is becoming the tool of choice when querying large datasets from platforms like MinIO. We then learn the steps to setup and deploy Presto on private infrastructure.
Read more
Minio now has built-in support for exporting Prometheus compatible data on an unauthenticated endpoint. This enables Prometheus monitoring for Minio server deployments without sharing server credentials and eliminates the need to run an external Prometheus exporter.
Read moreThere has been a lot going on MinIO server development front. We recently added support for * Disk Caching [https://docs.minio.io/docs/minio-disk-cache-guide] * Large/Petascale buckets [https://docs.minio.io/docs/minio-large-bucket-support-quickstart-guide] * Storage Classes [https://github.com/minio/minio/tree/master/docs/erasure/storage-class] While features like MinIO bucket federation [https://github.com/minio/minio/pull/5501], revamped MinIO
Read more