From chaos to order: Archiving data on tape with Software-defined Tape

Software-defined Tape is a fast and ultra-secure way to archive your object storage on tape that acts as an S3 bridge between an object based disk product and tape technology.

Software-defined Tape was developed to address the increasing use of object storage and the fact that accumulating data only on SSD and disk is very expensive. Object storage, the act of using a solution that is not a file system, is growing as it offers a linear organization for a large volume of unstructured data by using metadata, as key words, to facilitate the search for files.

Today, there is 1.04 zettabytes of data stored in Europe and by 2025, this is expected to more than double to 2.3 ZB. You might think that the biggest challenge is how to fit all of this somewhere, but actually, it is more than just reducing floorspace requirements. This kind of data volume also generates a huge amount of files, which must be stored, accessible and readable for years to come.

Many end users asked us to develop a system that would allow them to store object storage on tape and factors we were asked to address include creating a system that is scalable, open and allows small files to be saved faster, that is suitable for all types of industries and can offer all of the security benefits that come with tape technology at a cost effective price. 

Knowing that tape is the safest and best value storage product available, we chose to use the advantages of tape technology, but to refine this product by combining it with features that were designed specifically for object storage. 

Fujifilm created the OTFormat in order to address the issue of saving small files and to create a system that is more efficient than LTFS whilst having all of the benefits of an open format. In fact, the similarities with LTFS are only in the structure – both use two partitions to create an index style system in order to locate your data.

With LTFS, the complete index is overwritten after every writing operation and then re-written in full again in a second partition where the data is stored. Whereas, on OTFormat, the index is saved on an incremental basis.

OTFormat is a format specifically designed for objects, which uses metadata to locate the file. It systematically saves four copies of the metadata on the tape. The great advantage of saving metadata is really that it is the ultimate in security.

Software-defined Tape and OTFormat offer all of the benefits of tape technology, such as the protection against viruses and hackers, an archive life of over thirty years and a data integrity that is 10,000 times greater than disk, with an added bonus that the metadata is saved on the tape four times.

As anyone working with small file sizes will be aware, often the transfer rate may not be the one you will actually reach. For example, in tests carried out we found that in the case of very small files of 1 MB or less the speed of the tape drive often fell to only 15 MB/s and was even lower, around 4-5 MB/s, when using hard disk.

Thus, the accumulation of lots of files in an unstructured environment can have serious detrimental effects on the speed. The risk is that when files are unsorted they can lead to backhitches and the shoe-shining effect. These two factors can result in a reduced speed and will ultimately reduce the lifespan of your hardware. In fact, there is so much to say about speed and its impact - we could talk about this all night long!

To counteract this, Software-defined Tape packs the files into larger 10 GB packs that are sent to the tape library. By packing into one large file of 10 GB, the OTFormat is able to reach faster and more predictable write speeds. 

Software-defined Tape is all about openness and this was key in the design of this new technology. Just to give you a few facts about this:

• The specifications of the OTFormat have been published.
• By using the OTFormat Reader, a free software tool, provided by Fujifilm, any user will be able to read their data without a vendor lock in. Therefore, it offers an easy exit strategy.
• Fujifilm chose to use S3 compatible APIs, as it is open and easy to use, and enables a seamless integration with object storage systems including Cloudian, Netapp and Caringo.
• A single GUI gives direct online access to your archive from compatible object storage solutions.
• It can also be used alongside a datamover, which makes it compatible with many different types of systems.
• Software-defined Tape works with most brands of tape hardware including IBM, Quantum, Spectralogic, Qualstar, HPE and Overland.

Software-defined Tape is a scalable system that can store up to 128 billion objects. This expandability means that it is valid from 1 PB to 200 PB with a unique library system.

 Based on a subscription model, this service allows for future planning as well as any unexpected data growth. Set over a five-year period and structured according to the volume of data from 1-10 PB, 11-50 PB and over 50 PB, the more data stored the cheaper the cost per terabyte. For example, for a user with 20 PB this solution would cost £656,667 over a five-year period including 3,600 LTO8 tapes.

Software-defined Tape also makes a simultaneous copy ensuring that two copies of your data are made immediately, saving time and making more efficient use of the hardware. This second copy is included in the license fee, so when you purchase 20 PB you actually would write 40 PB of capacity on tape.

With this solution, it is possible to address the chaotic challenges thrown by an unstructured environment and allow them to be put into an ordered library system that is quickly accessible due to its advanced metadata.