Data is crucial. It’s the fuel that powers today’s transformative business intelligence and analytics applications, and provides the raw material for cutting-edge AI tools. Yet while we think about the data we gather, how we protect it and what we do with it, we don’t always understand the critical importance of how and where it’s stored. The sheer quantities of unstructured data captured by our business applications can create obstacles to using it, while fragmentation is becoming a real issue, as data becomes scattered and siloed across multiple locations, networks and clouds. As data grows, so can storage costs, and so can the challenges of using it effectively.
This is why data architecture matters, and why many organisations are moving on from old file-and-block storage architectures, and towards more efficient object storage. To understand why, we need to take a deeper look at object storage and the key advantages it holds over file and block.
The limits of file-and-block storage
It’s easy to get your head around file storage, because it’s the traditional architecture that most of us still use every day. It’s a hierarchical system based on a metaphor of files and folders, where data is stored in files, named and stored in folders or directories, which may themselves be nested within other folders or directories. By tracing a path through these folders, applications can navigate to the data that they need. To make it easier to find the files through searching, they can also be tagged with metadata, so that you can track them down by the date they were created or modified, their file type, relevant keywords or other useful information.
We still use this storage architecture because it works across many scenarios. It’s fine for collections of small documents. It’s well established, flexible, easy to understand and very efficient when you’re working on a small scale. However, once you start dealing with larger datasets – particularly datasets that are updating in real-time or masses of unstructured data – you hit the limits of file storage fast. The larger and more complex your architecture grows, the harder it is to find a specific file or pull the necessary data from it. What’s more, as your storage structures grow, so does the metadata, meaning files and the data contained within take longer to access and query.
Block storage improves on this by chopping up large chunks of data and storing it within fixed-size blocks or storage volumes, to be accessed through storage protocols by operating systems or applications. It’s normally used in storage area networks or in cloud-based storage environments. Each block has a unique identifier, and the SAN or cloud platform can place each block in the most suitable environment and location, making it easier to optimise block storage for cost or performance. Then, when an application requests data from the storage system, it uses the identifiers to locate the relevant blocks, reassemble them and locate the required data within them.
Block storage still uses metadata, but it’s stored separately from the file itself. What’s more, that metadata is limited to basic file attributes, usually set up when the storage architecture is defined. Like file storage, block storage has its strengths. It’s a fast, effective way to handle vast datasets with low latency, and if you need to change a file you only need to change the block affected by the changes. However, it’s constrained by the limited amounts of metadata, which can slow down search and retrieval operations. What’s more, as the metadata repository grows, it can also grow unwieldy. Most of all, whether it’s stored in a SAN or on cloud-based infrastructure, block storage can become a source of cost creep as the volumes of the data being stored expand.
The object storage difference
Object storage does things differently, bundling file data and metadata together in a single container – the object – with its own unique global identifier. All the objects can be kept in a single, flat warehouse of storage, rather than being held within a nested structure of directories. Each object is self-contained, and the metadata becomes the key to locating and retrieving data. That metadata can be customised for different objects, to cover everything from when the object was created, to its age, to any privacy and security details, any access restrictions and a wide range of other, more specific information. For instance, a video object may include metadata covering which camera and lens was used, the location, any settings used, the people shooting the video, any people in the video and any subjects covered. All this makes it much easier and faster to retrieve the right data on demand, using modern REST APIs.
This flexibility is a key strength of object storage. It scales well, but as datasets grow the metadata doesn’t grow unwieldy. It’s easy to manage, copy and sync data across multiple hardware and cloud-based environments. Object storage is also great for handling large quantities of unstructured data, including image, audio and video data, or data coming in from sensors and Internet of Things (IoT) devices. That makes it ideal for new applications in AI, machine learning or real-time analytics, as well as data backup and archiving solutions.
Of course, object storage has some downsides; you can’t modify an object once created, only recreate it as a new object, and it’s not always as effective at block storage for running more traditional databases.
Object storage and Hot Cloud Storage
Object storage also plays directly into the strengths of cloud-based storage services, including Wasabi Hot Cloud Storage. Hot Cloud Storage was designed around high-performance object storage from the ground up, both to optimise how data is stored on Wasabi’s infrastructure, and to optimise how it can be retrieved from large arrays of drives at lightning speed. It’s part of the secret of Wasabi’s unlimited scalability, and its ability to provide high-performance storage without separate hot, cool and cold tiers. The flat, metadata-driven structure means that all stored data can be treated in the same way, no matter which applications or solutions it serves and no matter how it’s classified. It’s also perfect for applications that need to work with high quantities of unstructured data. You can store it where it’s most cost-effective, on the cloud, but without hitting performance or latency barriers that would make AI, ML and analytics applications unworkable.
Using an object storage architecture also helps Wasabi deliver a service without complex price points and SLAs, access constraints or hidden costs. It provides high levels of resilience, as objects can be distributed across multiple storage nodes and rebuilt, if needed, on demand. The flat structure helps combat fragmentation and keeps data accessible and outside of silos. What’s more, it enables data immutability through an Object Lock function, protecting data at the object level so that it can’t be altered or deleted until a set period has expired. That’s ideal for data with compliance requirements, but also any data under threat from ransomware.
Object storage is the ideal storage architecture for next-gen cloud storage services, and Wasabi has the technology to put it to great use.
