Home » Metadata: Adding context adds value and improves useability

Metadata: Adding context adds value and improves useability

While we were at Storage Visions a few weeks ago, organizer Tom Coughlin restated a prediction he made at Storage Visions back in 2007: “By 2015, there will be a terabyte of data in your pocket, a petabyte in your home, exabytes in data centers, and zetabytes in the world.” Though there don’t appear to be readily available statistics on how close we’re coming to meeting this prediction, friends in the data center business confirm what we all know to be true; the digitization of business and peoples’ lives is causing data to grow at an exponential rate.

Which leads Datalight CEO Roy Sherrill to ask the question; “With all that information, how on earth are we going to find anything without a better system for metadata?”

Metadata is the context behind the information that’s being stored. A simple example is the time and date a photo was taken – not typically an option in the embedded world (as I learned the hard way, the ‘file created’ stamp of the file on my desktop has nothing to do with when I took the picture with my digital camera), instead we rely on our own memories, or cumbersome filing systems to find what we’re looking for. Metadata could also be an event’s name, GPS location, peoples’ names, audio from the event, text/captions, or any other information that tells why the data is important.

So how do they create context in the enterprise market, where petabytes, soon to be exabytes of information is being stored? With a value of $80 billion, at least four times the embedded storage market, enterprise has had to get very good at dealing with data on a massive scale. And if the embedded storage market is really going to get there, what can we learn from enterprise? The differences in metadata management are dramatic between enterprise and embedded. While storage in embedded devices is often based on a 1980’s-era file system (FAT), with reliability, storage growth, and metadata as afterthoughts, enterprise storage uses file systems like NTFS, CIFS, and others that emphasize reliability, have plenty of capacity for storage growth, and support a sophisticated system for managing metadata.

So what can the embedded industry do to keep up with rising demands for data storage? It seems obvious that we need a system for metadata that causes it to be created automatically and stored with the file at the time of creation. We in the embedded market have a unique advantage over the enterprise market in that the devices we build to create the data -- be it a cellphone, a factory controller or an automobile -- often include a multitude of sensors that capture context.

Having context stored with the data brings value to the information, and enables the information to be found in new and flexible ways over time. The more embedded device users have context for data that they create, the more of it they will keep. And the more if it they keep, the more storage they will need.

Using metadata must be intuitive and easy. File systems must automatically store metadata with the file. File systems must be flexible, reliable, and high performance. Metadata must be portable between systems. Applications across the storage universe must be ‘context aware.’ This need for better metadata management is one thing that led us to develop Reliance Nitro, which has OEM attributes, customizable metadata fields that allow information like time, data, GPS coordinates or other critical information to be automatically stored with the file and used as needed.

To learn more about Datalight File Systems visit our Flash File Systems page