Beyond MySpace Complaints: An Updated Forecast of Worldwide Information Growth

The Diverse and Exploding Digital Universe, An Updated Forecast of Worldwide Information Growth Through 2011 (pdf) updates IDC’s inaugural forecast published in March 2007. Sponsored by storage giant EMC, key findings of this white paper indicate that it is time to upgrade the 1990's dot.com bubble's financed Internet infrastructure. Why? Well, it's not just because of MySpace users! [JH]

Key findings and recommendations from this EMC-sponsored IDC white paper include

• The digital universe in 2007 — at 2.25 x 1021 bits (281 exabytes or 281 billion gigabytes) — was 10% bigger than we thought. The resizing comes as a result of faster growth in cameras, digital TV shipments, and better understanding of information replication. By 2011, the digital universe will be 10 times the size it was in 2006.
• As forecast, the amount of information created, captured, or replicated exceeded available storage for the first time in 2007. Not all information created and transmitted gets stored, but by 2011, almost half of the digital universe will not have a permanent home.
• Fast-growing corners of the digital universe include those related to digital TV, surveillance cameras, Internet access in emerging countries, sensor-based applications, datacenters supporting “cloud computing,” and social networks.
• The diversity of the digital universe can be seen in the variability of file sizes, from 6 gigabyte movies on DVD to 128-bit signals from RFID tags. Because of the growth of VoIP, sensors, and RFID, the number of electronic information “containers” — files, images, packets, tag contents — is growing 50% faster than the number of gigabytes. The information created in 2011 will be contained in more than 20 quadrillion — 20 million billion — of such containers, a tremendous management challenge for both businesses and consumers.
• Of that portion of the digital universe created by individuals, less than half can be accounted for by user activities — pictures taken, phone calls made, emails sent — while the rest constitutes a digital “shadow” — surveillance photos, Web search histories, financial transaction journals, mailing lists, and so on.
• The enterprise share of the digital universe is widely skewed by industry, having little relationship to GDP or IT spending. The finance industry, for instance, accounts for almost 20% of worldwide IT spending but only 6% of the digital universe. Meanwhile, media, entertainment, and communications industries will account for 10 times their share of the digital universe in 2011 as their share of worldwide gross economic output.
• The picture related to the source and governance of digital information remains intact: Approximately 70% of the digital universe is created by individuals, but enterprises are responsible for the security, privacy, reliability, and compliance of 85%.

To deal with this explosion of the digital universe in size and complexity, IT organizations will face three main imperatives:

One. They will need to transform their existing relationships with the business units. It will take all competent hands in an organization to deal with information creation, storage, management, security, retention, and disposal in an enterprise. Dealing with the digital universe is not a technical problem alone.

Two. They will need to spearhead the development of organizationwide policies for information governance: information security, information retention, data access, and compliance.

Three. They will need to rush new tools and standards into the organization, from storage optimization, unstructured data search, and database analytics to resource pooling (virtualization) and management and security tools. All will be required to make the information infrastructure as flexible, adaptable, and scalable as possible.