A new report has shed light on the scale of the dilemma facing organizations as demand for data storage capacity continues to soar.
Published jointly by Fujitsu and Twist Bioscience, both of which operate in the archival storage market, The report (opens in a new tab) predicts that the gap between available storage capacity and demand will exceed 7.8 million petabytes by 2030.
In this scenario, companies will have no choice but to delete vast amounts of old data to make way for new, which is enough to send a shiver down the spine of any company with aspirations in areas such as than artificial intelligence.
As the volume of data produced by Internet activity, digital devices and IoT sensors continues to grow at an aggressive rate, companies are running out of time to solve a critical problem: where to put it all.
Whereas HDDs (hard drives) and solid state hard drives (SSD) do an excellent job of retaining and delivering the amounts of data that waiters and the client devices must work, neither are suitable for storing information in bulk and for long periods of time.
With regard to the storage of archives, Open Linear Tape (LTO) magnetic tape reigns supreme, with the lowest cost per capacity of any technology. The current generation of tape, LTO-9, has a native capacity of 18TB and can be purchased for as little as $150 (or about $8.30/TB).
According to the report, large enterprises will need to invest heavily in tape and other archival media as the volume of data produced by business operations continues to grow. The alternative would be to reject the old data, but that would mean losing their potential value as a source of information; the most advanced AI products are typically informed by the largest and most comprehensive pools of data.
“We believe that most of this [new] enterprise data will be unstructured, ‘cold’, rarely accessed, and will need to be maintained at minimal cost,” said report author Jon Monroe, who says the allocation of storage spending will need to reflect this fact.
However, while profitable, the tape also has its weaknesses; data is only accessible in serial form, making it difficult to locate specific files, and companies must also migrate to new tapes on a semi-regular basis to avoid data loss.
In light of these issues, researchers are looking for new ultra-dense, ultra-durable storage technologies. A few different candidates have emerged, but one concept seems particularly promising: DNA.
DNA, the basic material of living organisms, is made up of four molecular building blocks: adenine (A), guanine (G), cytosine (C) and thymine (T). These compounds connect in pairs (AT & GC) to form the rungs of the famous double helix ladder.
This structure can be used as an extremely dense and durable form of data storage, by converting binary 1s and 0s into the four-letter genetic alphabet. A single gram of DNA turned out to be able to store 215PB (220,000 TB) of data.
“DNA delivers on the promise of delivering the three magic ingredients of storage: ultra-high density, reasonable cost, and durability,” said Emily Leproust, CEO and co-founder of Twist Bioscience, which is investing heavily to bring the technology to fruition.
“We predict that new media will be needed to meet the more than $7 billion in unmet storage demand predicted in the coming years.”
As it stands, the technology remains unusable at scale, due to the time it takes to write data to DNA and various other challenges. Naturally, the report should also be taken with a grain of salt, produced as it was by two organizations with vested interests in increased spending on archival storage.
However, it is undeniable that the increase in capacity of traditional data storage technologies is failing to keep pace with data production, which means that a reorientation of the storage stack is inevitable.
“The data centers of the future will need whatever the SSD, HDD, and tape industries can manufacture and supply, along with new DNA and optics and perhaps other enterprise storage technologies, to cost-effectively and reliably preserve the priceless artifacts of our staff, corporate and cultural history,” Monroe added.
“Availability and durability challenges, combined with the costs of managing our multi-million petabyte dataverse over increasingly long periods of time, will create new use cases for legacy storage technologies and require the creation new, more cost-effective and energy-efficient storage technologies.
Going through Blocks and files (opens in a new tab)