Sunday, June 01, 2014

Historical External Disk Storage Data: IDC Worldwide tracking report

Data from IDC's Quarterly Worldwide External Disk Storage Systems Factory Revenues series (Press Releases). Multiply quarterly values by 4 for an approx yearly value. Full data not available prior to 2011.
For 2013: US$24.4 billion and 34.6PB.

The naively calculated cost per Gigabyte of shipped "External Disk Storage", excluding NAS, iSCSI, "open SAN" and internal Direct Attached Storage (DAS).

This is not the "Total Disk" market for Enterprise storage, but roughly Enterprise RAID arrays.

For the Enterprise Storage reported by IDC, there are no small, cheap systems sold: all systems sold will have both some room for growth and be large enough to amortise the highest cost items across many drives. The calculated Price/GB isn't inflated by small systems.

Enterprises are sophisticated purchasers of I.T. - they spend considerable time & resources on purchase and upgrade decisions, sifting through many options to select optimal price/performance for each application. They are also very aware of the vendor product offerings and their price points, opting for low-priced priced solutions when they are applicable. This direct-substitution effect sets the caps the price vendors can charge. As mid-range systems grow in features, performance and capacity due to technology progression, they poach sales from high-end systems. In the same way that lower-level platforms,s mini-computers, then PC's + LANs and now tablets/portables with "Cloud" Apps, have steadily eroded the markets for high-end computers, mainframe and supercomputer.

One of the factors not widely discussed is the purchasing policy of Enterprises is directed by their existing equipment. Technically, there is both a Barrier to Entry and a Barrier to Exit.

There are sound commercial and managerial reasons to limit the number of vendors on-site and the degree of change. It begins with access to trained staff and the learning curve involved in maximising systems throughput ('tuning'), learning troubleshooting and monitoring tools & techniques and the administration of ancillary tasks, such as snapshots, backup/restore and archives. Migrating the complete Storage pool of an enterprise from old to new equipment is a critical task, one that cannot be allowed to fail, that cannot be allowed to corrupt any data and cannot exceed the planned window. It also cannot result in any 'surprises', like incompatible block-sizes blowing out filesystem sizes or renamed/missing 'extents' creating problems for databases and filesystems.

Enterprises only change major I.T. vendors with extreme reluctance. Hitachi Data Systems, HDS, had a detailed study of the costs of Data Migration: it was between 5-10 times the cost of the Storage.

A single vendor, even if migrating some data to a lower-cost product, is both experienced in the technique and motivated to get it right, first time. Large customers will flee dysfunctional and poorly performing vendors, albeit slowly. The most obvious example is Unisys. Since the early 1980's merger of Sperry-Univac and Burroughs, its whole business has shrunk 10-fold with mainframe sales by much more. I was once hired into a Unisys site where the customer was unhappy to the point of tearing up all contracts.

The EMC/IDC "Digital Universe" report estimates in 2013 there were 4.4 Zettabytes (4,400 Exabytes) of installed Storage Capacity (Disk, Flash, etc) at an estimated 75% utilisation.

The data illustrates the "Storage Paradox", revenue decreases whilst capacity increases, noted in the "Digital Universe" report. There are at least three effects in operation:
  • The per-unit cost of 'capacity optimised' Enterprise drives from vendors is continuing to fall,
  • For high-performance Storage, many types of Flash memory are displacing premium-priced smaller, high-RPM drives, and
  • competition between vendors and from their own lower-cost "substitutes" is forcing Enterprise pricing down.
During the 2008 Great Recession/Global Financial Crisis, sales of Enterprise storage first increased as large corporates invested more in Information Technology, then in late 2009, suffered a reversal when they reduced spending to achieve savings. They are presumed to have bought cheaper disk solutions, such as workgroup servers and smaller NAS appliances from the same vendors.

Qtr$ per GBRevenue
(US billion)
Average installed Cost of Enterprise Storage

(US billion)
PB Delta
(year over year)

4Q132.4%$6.910,20026.2%March 72014
3Q13-3.5%$5.78,40016.1%December 62013
2Q13-0.8%$5.98,20021.5%September 62013
1Q13-0.9%$5.97,80026.4%June 72013

4Q122.3%$6.78,00025.3%March 82013
3Q123.3%$5.97,10424.4%December 72012

4Q117.7%$6.66,27922.4%March 52012
3Q1110.8%$5.85,42930.7%December 22011
2Q1112.2%$5.6??September 22011
1Q1113.2%$5.64,95646.3%June 32011

4Q1016.2%$6.15,12755.7%March 42011
3Q1019.0%$5.24,29965.2%December 32010
2Q1020.4%$5.03,64554.6%September 32010
1Q1017.1%$5.03,39755.2%June 42010

1Q09-13.6%$5.6?14.8%June 52009

2Q0816.7%$5.11,77743.7%September 52008

3Q075.1%$4.41,30049.4% December 102007
1Q07?$4.31,000?July 32007

4Q0517.9%?65054.6%March 52006
1Q056.7%$3.8??March 52005

4Q038.4%$3.7??December 52003
IDC Worldwide External Disk Storage

No comments: