SteveJ's lab-notes

Sunday, June 01, 2014

Historical External Disk Storage Data: IDC Worldwide tracking report

Data from IDC's Quarterly Worldwide External Disk Storage Systems Factory Revenues series (Press Releases). Multiply quarterly values by 4 for an approx yearly value. Full data not available prior to 2011.
For 2013: US$24.4 billion and 34.6PB.

"MAID" using 2.5 in drives

What would a current attempt at MAID look like with 2.5" drives?

"MAID", Massive Array of Idle Disks, was an attempt by Copan Systems (bought by SGI in 2009) at near-line Bulk Storage. It had a novel design innovation, mounting drives vertically back-to-back in slide-out canisters (patented), and was based on an interesting design principle: off-line storage can mostly be powered down.

It was a credible attempt, coming out of The Internet Archive, and their "Petabox" (a more technical view and on Wikipedia). At 24 x 3.5" drives per 4RU, they contain around half the 45 drives of the Backblaze 4.0 Storage Pod. The Petabox has 10Gbps uplinks and much beefier CPU's and more DRAM.

The Xyratex ClusterStor (now Seagate) offers another benchmark: their Scalable Storage Unit (SSU) stores 3 rows of 14 drives in 2.5RU x 450mm slide-out draws, allowing hot-plug access to all drives. Two SSU's comprise a single 5RU unit of 84 drives, with up to 14 SSU's per rack for 1176 drives per rack, an average of 28 x 3.5" drives per Rack Unit.

RAID-0 and RAID-3/4 Spares

This piece is not based on an exhaustive search of the literature. It addresses a problem that doesn't seem to have been addressed as RAID-0 and the related RAID-3/4, a single parity drive.

Single parity drives seem to be deemed early on to be impractical because it apparently comprises a deliberate system bottleneck. RAID-3/4 has no bottleneck for streaming reads/writes and for writes, performance becomes, not approaches, the raw write performance of the array is available, identical to RAID-0 (stripe). For random writes, the 100-150 times speed differential between sequential and random access of modern drives can be leveraged with a suitable buffer to remove the bottleneck. The larger the buffer, the more likely the pre-read of data, to save to calculate the new parity, won't be needed. This triples the array throughput by avoiding the full revolution forced by the read/write-back cycle.

Multiple copies of the parity drive (RAID-1) can be kept to mitigate against the very costly failure of a parity drive: all blocks on every drive must be reread to recreate a failed parity drive. For large RAID groups and the very low price of small drives, this is not expensive.

With the availability of affordable, large SSD's, naive management of a single parity drive also removes the bottleneck for quite large RAID groups. The SSD can be backed by a log-structured recovery drive, trading on-line random IO performance for rebuild time.

Designing Local and/or Global Spares for large (N=64..512) RAID sets is necessary to reduce overhead, improve reconstruction times and avoid unnecessary partitioning, limiting recovery options and causing avoidable data loss events.

Comparing consumer drives in small-systems RAID

This stems from an email conversation with a friend: why would he be interested in using 2.5" drives in RAID, not 3.5"?

There are two key questions for Admins at this scale, and my friend was exceedingly sceptical of my suggestion:

Cost/GB
'performance' of 2.5" 5400RPM drives vs 7200RPM drives.

I've used retail pricing for the comparisons. Pricelist and sorted pricelist.

Retail Disk Prices, as printed

Table of current retail prices for various types of disk with cost-per-GB.

Disclaimer: This table is for my own point-in-time reference, does not carry any implicit or explicit recommendations or endorsement for the retailer, vendor or technologies.

Retail disk prices, sorted.

Table of current retail prices for various types of disk, sorted on cost-per-GB.
Disclaimer: This table is for my own point-in-time reference, does not carry any implicit or explicit recommendations or endorsement for the retailer, vendor or technologies.

3.5" Internal drives are the cheapest $/GB, ranging from 4.3 cents/GB to 10-11 cents/GB. Generally, larger drives have cheaper $/GB. Higher spec drives, suitable for high duty-cycle applications, are more expensive. This retailer doesn't sell 10K or SAS drives.

It's not possible to track 3.5" drives from Internal to External to arrive at a cost of packaging.

2.5" Internal drives range 8 to 16.5 cents/GB, generally higher than 3.5" drive costs. There seems to be little extra cost of packaging for external drives. There is a small premium in consumer drives for 7200RPM. This retailer only sells 2TB drives (15mm vs 9.5mm?) as external drives.

There was no information in the retailers rather compact format on the thickness (5mm, 7mm, 9.5mm, 12.5mm, 15mm) of 2.5" drives.

Solid State Disks are 5+ times more expensive than Hard Disk Drives, at 59 cents/GB to $1.37/GB.
The smaller mSATA drives start at 72.8 cents/GB.
No supplier information on SSD specs are included: SLC/MLC, transfer rates, IO/sec and number of write cycles. SSD's are very sensitive to wear and device selection requires very careful reading of device specifications.

**Retail Disk Prices**
					01-May-2014
			http://www.msy.com.au/Parts/PARTS.pdf

F-Fac	Typ	Dsk	Conn	RPM	Brand/Model	Cap	Cost	$/GB	GB

3.5"	Int	HDD	SATA3	7200?	WD Green EZRX	3TB	129	0.0430	3000GB
3.5"	Int	HDD	SATA3	7200?	Seagate	3TB	129	0.0430	3000GB
3.5"	Int	HDD	SATA3	7200?	WD Green EZRX	4TB	185	0.0462	4000GB
3.5"	Int	HDD	SATA3	7200?	Seagate	4TB	189	0.0473	4000GB
3.5"	Int	HDD	SATA3	7200?	WD Green EZRX	2TB	95	0.0475	2000GB
3.5"	Int	HDD	SATA3	7200?	Seagate	2TB	95	0.0475	2000GB
3.5"	Int	HDD	SATA3	7200?	Seagate NAS	3TB	160	0.0533	3000GB
3.5"	Int	HDD	SATA3	7200?	WD Red NAS EFRX	3TB	165	0.0550	3000GB
3.5"	Int	HDD	SATA3	7200?	Seagate NAS	4TB	229	0.0573	4000GB
3.5"	Int	HDD	SATA3	7200?	WD Red NAS EFRX	4TB	235	0.0587	4000GB
3.5"	Int	HDD	SATA3	7200?	WD Purple PURX Surveillance	3TB	179	0.0597	3000GB
3.5"	Int	HDD	SATA?	7200?	Hitachi HGST NAS	3TB	179	0.0597	3000GB
3.5"	Int	HDD	SATA?	7200?	Hitachi HGST NAS	4TB	249	0.0622	4000GB
3.5"	Int	HDD	SATA3	7200?	Seagate NAS	2TB	125	0.0625	2000GB
3.5"	Int	HDD	SATA3	7200?	WD Green EZRX	1TB	64	0.0640	1000GB
3.5"	Int	HDD	SATA3	7200?	WD Red NAS EFRX	2TB	129	0.0645	2000GB
3.5"	Int	HDD	SATA3	7200?	WD Purple PURX Surveillance	4TB	259	0.0648	4000GB
3.5"	Int	HDD	SATA3	7200?	Seagate	1TB	65	0.0650	1000GB
3.5"	Int	HDD	SATA3	7200?	WD Purple PURX Surveillance	2TB	135	0.0675	2000GB
3.5"	Int	HDD	SATA3	7200?	WD Red NAS EFRX	1TB	89	0.0890	1000GB
3.5"	Int	HDD	SATA2	7200?	Hitachi HGST UltraStar	1TB	89	0.0890	1000GB
3.5"	Int	HDD	SATA3	7200?	Hitachi HGST	3TB	270	0.0900	3000GB
3.5"	Int	HDD	SATA3	7200?	Hitachi HGST	4TB	365	0.0912	4000GB
3.5"	Int	HDD	SATA3	7200?	Hitachi HGST	2TB	185	0.0925	2000GB
3.5"	Int	HDD	SATA3	7200?	WD Purple PURX Surveillance	1TB	95	0.0950	1000GB
3.5"	Int	HDD	SATA3	7200?	Seagate	500G	55	0.1100	500GB

3.5"	Ext	HDD	USB3.0	7200?	WD Element	3TB	129	0.0430	3000GB
3.5"	Ext	HDD	USB3.0	7200?	Seagate Expansion	3TB	139	0.0463	3000GB
3.5"	Ext	HDD	USB3.0	7200?	Seagate Expansion	2TB	95	0.0475	2000GB
3.5"	Ext	HDD	USB3.0	7200?	WD Mybook Essential	3TB	149	0.0497	3000GB
3.5"	Ext	HDD	USB3.0	7200?	WD Mybook Essential	4TB	209	0.0522	4000GB
3.5"	Ext	HDD	USB3.0	7200?	Seagate BackUp Plus	3TB	159	0.0530	3000GB
3.5"	Ext	HDD	USB3.0	7200?	Seagate BackUp Plus	2TB	115	0.0575	2000GB
3.5"	Ext	HDD	USB3.0	7200?	WD Mybook Essential	2TB	139	0.0695	2000GB

2.5"	Int	HDD	SATA?	5400	Hitachi HGST	1TB	80	0.0800	1000GB
2.5"	Int	HDD	SATA?	5400	WD JPVX	1TB	83	0.0830	1000GB
2.5"	Int	HDD	SATA?	5400	WD BPVX	750G	64	0.0853	750GB
2.5"	Int	HDD	SATA?	5400	Hitachi HGST	750G	66	0.0880	750GB
2.5"	Int	HDD	SATA?	5400	Hitachi HGST	1.5TB	139	0.0927	1500GB
2.5"	Int	HDD	SATA?	7200	Hitachi HGST	1TB	93	0.0930	1000GB
2.5"	Int	HDD	SATA?	7200	WD BPKX	750G	78	0.1040	750GB
2.5"	Int	HDD	SATA?	5400	Hitachi HGST	500G	55	0.1100	500GB
2.5"	Int	HDD	SATA?	5400	Seagate	500G	56	0.1120	500GB
2.5"	Int	HDD	SATA?	7200	Hitachi HGST	750G	85	0.1133	750GB
2.5"	Int	HDD	SATA?	5400	WD LPVX	500G	57	0.1140	500GB
2.5"	Int	HDD	SATA?	7200	Hitachi HGST	500G	64	0.1280	500GB
2.5"	Int	HDD	SATA?	7200	Seagate	500G	64	0.1280	500GB
2.5"	Int	HDD	SATA?	7200	WD BPKX	500G	67	0.1340	500GB
2.5"	Int	HDD	SATA?	5400	Hitachi HGST	320G	53	0.1656	320GB
2.5"	Int	HDD	SATA?	5400	WD LPVX	320G	53	0.1656	320GB
2.5"	Int	HDD	SATA?	5400	Seagate	320G	53	0.1656	320GB

2.5"	Ext	HDD	USB3.0	5400?	WD Element	2TB	149	0.0745	2000GB
2.5"	Ext	HDD	USB3.0	5400?	Samsung	2TB	149	0.0745	2000GB
2.5"	Ext	HDD	USB3.0	5400?	Samsung	1.5TB	115	0.0767	1500GB
2.5"	Ext	HDD	USB3.0	5400?	WD Passport	2TB	159	0.0795	2000GB
2.5"	Ext	HDD	USB?.0	5400?	Hitachi HGST Touro Mobile	1TB	80	0.0800	1000GB
2.5"	Ext	HDD	USB3.0	5400?	WD Passport Ultra	2TB	165	0.0825	2000GB
2.5"	Ext	HDD	USB3.0	5400?	WD Passport	1.5TB	129	0.0860	1500GB
2.5"	Ext	HDD	USB3.0	5400?	Samsung	1TB	86	0.0860	1000GB
2.5"	Ext	HDD	USB3.0	5400?	WD Element	1TB	89	0.0890	1000GB
2.5"	Ext	HDD	USB3.0	5400?	WD Passport	1TB	89	0.0890	1000GB
2.5"	Ext	HDD	USB?.0	5400?	Hitachi HGST Touro Pro	1TB	92	0.0920	1000GB
2.5"	Ext	HDD	USB3.0	5400?	Seagate BackUp Plus	1TB	99	0.0990	1000GB
2.5"	Ext	HDD	USB3.0	5400?	WD Passport Ultra	1TB	104	0.1040	1000GB
2.5"	Ext	HDD	USB?.0	5400?	Hitachi HGST Touro Mobile	500G	56	0.1120	500GB
2.5"	Ext	HDD	USB3.0	5400?	Samsung	500G	62	0.1240	500GB
2.5"	Ext	HDD	USB3.0	5400?	Seagate Expansion	500G	69	0.1380	500GB
2.5"	Ext	HDD	USB3.0	5400?	WD Passport Ultra	500G	74	0.1480	500GB
2.5"	Ext	HDD	USB3.0	5400?	Seagate BackUp Plus	500G	88	0.1760	500GB

2.5"	Int	SSD	SATA3	-	Samsung 840 EVO	1TB	589	0.5890	1000GB
2.5"	Int	SSD	SATA?	-	SanDisk Ultra Plus	256G	157	0.6133	256GB
2.5"	Int	SSD	SATA?	-	Seagate 600	480G	299	0.6229	480GB
2.5"	Int	SSD	SATA?	-	Plextor M5-PRO	512G	329	0.6426	512GB
2.5"	Int	SSD	SATA?	-	Plextor M5S	256G	168	0.6562	256GB
2.5"	Int	SSD	SATA3	-	Samsung 840 EVO	500G	329	0.6580	500GB
2.5"	Int	SSD	SATA?	-	Kingston V300	240G	159	0.6625	240GB
2.5"	Int	SSD	SATA?	-	Seagate 600	240G	159	0.6625	240GB
2.5"	Int	SSD	SATA?	-	Fujitsu	256G	170	0.6641	256GB
2.5"	Int	SSD	SATA3	-	Samsung 840 EVO	250G	170	0.6800	250GB
2.5"	Int	SSD	SATA?	-	Kingston V300	480G	329	0.6854	480GB
2.5"	Int	SSD	SATA?	-	SanDisk Ultra Plus	128G	89	0.6953	128GB
2.5"	Int	SSD	SATA?	-	Plextor M5-PRO	256G	179	0.6992	256GB
2.5"	Int	SSD	mSATA3	-	Samsung 840 EVO	250G	182	0.7280	250GB
2.5"	Int	SSD	SATA?	-	Kingston V300	120G	88	0.7333	120GB
2.5"	Int	SSD	SATA?	-	Fujitsu	512G	383	0.7480	512GB
2.5"	Int	SSD	SATA?	-	OCZ Vertec 450	128G	97	0.7578	128GB
2.5"	Int	SSD	SATA?	-	SanDisk Extreme	240G	185	0.7708	240GB
2.5"	Int	SSD	SATA?	-	Fujitsu	128G	99	0.7734	128GB
2.5"	Int	SSD	SATA?	-	SanDisk Extreme II	480G	379	0.7896	480GB
2.5"	Int	SSD	SATA3	-	Samsung 840 EVO	120G	95	0.7917	120GB
2.5"	Int	SSD	SATA?	-	SanDisk Extreme II	240G	195	0.8125	240GB
2.5"	Int	SSD	SATA?	-	Seagate 600	120G	99	0.8250	120GB
2.5"	Int	SSD	SATA?	-	Kingston HyperX	240G	199	0.8292	240GB
2.5"	Int	SSD	SATA3	-	Samsung 840 PRO	512G	439	0.8574	512GB
2.5"	Int	SSD	SATA?	-	Intel 520	120G	104	0.8667	120GB
2.5"	Int	SSD	SATA?	-	Intel 530	240G	209	0.8708	240GB
2.5"	Int	SSD	SATA?	-	Kingston HyperX	120G	105	0.8750	120GB
2.5"	Int	SSD	mSATA3	-	Samsung 840 EVO	120G	105	0.8750	120GB
2.5"	Int	SSD	SATA3	-	Samsung 840 PRO	256G	232	0.9062	256GB
2.5"	Int	SSD	SATA?	-	Intel 530	120G	115	0.9583	120GB
2.5"	Int	SSD	SATA?	-	SanDisk Extreme II	120G	118	0.9833	120GB
2.5"	Int	SSD	SATA?	-	Kingston SMS200s3	120G	119	0.9917	120GB
2.5"	Int	SSD	mSATA3	-	Intel 530	240G	242	1.0083	240GB
2.5"	Int	SSD	SATA?	-	Intel 530	180G	184	1.0222	180GB
2.5"	Int	SSD	SATA?	-	Plextor M5-PRO	128G	135	1.0547	128GB
2.5"	Int	SSD	SATA?	-	Fujitsu	64G	69	1.0781	64GB
2.5"	Int	SSD	SATA3	-	Samsung 840 PRO	128G	138	1.0781	128GB
2.5"	Int	SSD	SATA?	-	Kingston V300	60G	65	1.0833	60GB
2.5"	Int	SSD	mSATA3	-	Intel 530	120G	130	1.0833	120GB
2.5"	Int	SSD	mSATA3	-	Intel 525	120G	139	1.1583	120GB
2.5"	Int	SSD	SATA?	-	SanDisk Ultra Plus	64G	75	1.1719	64GB
2.5"	Int	SSD	SATA?	-	Intel 730	240G	285	1.1875	240GB
2.5"	Int	SSD	SATA?	-	Intel S3500	240G	288	1.2000	240GB
2.5"	Int	SSD	SATA?	-	Kingston SMS200s3	60G	76	1.2667	60GB
2.5"	Int	SSD	SATA?	-	Intel S3500	160G	209	1.3062	160GB
2.5"	Int	SSD	SATA?	-	Intel S3500	120G	164	1.3667	120GB

2.5"	Int	SSHD	SATA?	5400?	Seagate	1TB	129	0.1290	1000GB
2.5"	Int	SSHD	SATA?	5400?	Seagate	500G	85	0.1700	500GB

Tuesday, April 22, 2014

RAID++: RAID-0+ECC

Current RAID schemes, and going back to the 1987/8 Patterson, Gibson, Katz RAID paper, make no distinction between transient and permanent failures: errors or dropouts versus failure.

Storage: Spares and Parity in large disk collections

What approaches are available to deal with spare drives and RAID parity for 300-1,000 drives in a single box?
Will existing models scale well?
Do other technologies fill any gaps?

Storage: First look at Hardware block diagram

Stuffing 500-1000 2.5" drives in an enclosure is just the start of a design adventure.

The simplest being choosing fixed or hot-plug drive mounting. There's a neat slide-out tray system for 3.5" drives that allows hot-plug access for densely vertically packed drives that could be adapted to 2.5" drives.

Storage: Challenges of high-count disk enclosures

Stuffing 500-1,000 2.5" drives in a single enclosure may be technically possible, but how do you make those drives do anything useful?

Increasing drives per enclosure from 15-45 for 3.5" drives to 1,000 requires a deep rethink of target market, goals and design.

Not the least is dealing drive failures. With an Annualised Failure Rate (AFR) of 0.4%-0.75% now quoted by Drive Vendors, dealing with 5-15 drive failures per unit, per year is a given. In practice, failure rates are at least twice the Vendor quoted AFR not the least because in systems, conditions can be harsh and other components/connectors also fail, not just drives. Drives have a design life of 5 years, with an expected duty-cycle. Consumer-grade drives aren't expected to run 24/7 like the more expensive enterprise drives. Fail Rates over time, when measured on large fleets in service, increase over time and considerably towards end of life.

It's isn't enough to say "we're trying to minimise per unit costs", all designs do that, but for different criteria.
What matters is the constraints you're working against or parameters being optimised.

Storage: How many drives can be stuffed in a Box?

How many 2.5" drives can be stuffed into a single enclosure, allowing space for power, cooling, wiring and a single motherboard? Short answer: ~500-1000.

Storage: more capacity calculations

Following on from the previous post on Efficiency and Capacity, baselining "A pile of Disks" as "100% efficient".

Some additional considerations:

Storage: Efficiency measures

In 2020 we can expect bigger disk drives and hence Petabyte stores. Price per bit will come at a premium, it won't track capacity as it does now: larger capacity drives will cost more per unit.

What are the theoretical limits on which Storage solution "efficiency" can be judged?

We're slowly approaching what could be the last factor-10 improvement, to 10Tbits/in², in rotational 2-D magnetic recording technologies of Hard Disk Drives. Jim Gray (~2000) and Mark Kryder (2009) suggested 7TB/platter for 2.5" disk drives by 2020, assuming a 40%/yr capacity growth.

Rosenthal et al (2012) suggest that, like CPU-speed "Moore's Law", disk capacity growth rates have slowed, suggesting 100Tbits/in² may be possible in the far future. They predict 1.8 Tbits/in² commercially available in 2020, vs 0-6-0.7Tb/in² currently.

Reasonably Trustworthy Messaging (RTM)

PJ of Groklaw got spooked a Lavabit founder responding to PRISM by saying "if you knew what I did, you wouldn't use the Internet/email".

This is the start of a design for a reasonably trustworthy messaging system, in the same way that PGP was only pretty good privacy.

I'd like to combine 3 tools/concepts on top of the obvious measures.

SFTP as a file delivery mechanism.

ACSnet (later MHSnet) was a Store & Forward system that separated content and control while passing files to a content-handler on the far end.

Bespoke classified message systems contain two useful concepts:

Messages have an urgency and separate security classification.

these are used in routing & queuing decisions.

Every message is tracked & acquitted via multiple sequence numbers:

Per-link or channel sequence number
end-end sequence numbers

Obvious measures:

File splitting

Everything is a 2Kb or 4Kb block
I'd prefer an "M of N" redundancy system to allow some data to be lost.

Parchive & friends do this

There's a "batch reassembly" file implied by this.
SFTP needs to have some means of grouping batched files together.

Either a named control file, or
a directory name, a sequence number in clear or encrypted.

PGP/GPG encryption

Encrypt everything
Compress first, for text files especially.

Ideally, force different coding tables per file.

Content-based file naming

Files only referred to by content: its hash-sum, SHA-1 or MD5 (old)
Queued blocks can be jumbled and transmitted in any order, being reassembled into correct order later.

This implies "message headers" contain the hash-keys of contents.

There are old programs, like "fetchmail" (or 'fdm' now), that know how to pull mail from many sources and present to the Mail User Agent in a form it can handle, like the traditional Unix mailbox format.

The ACSnet software was able to accept emails from "sendmail", so a simple SMTP daemon is also needed here to accept outbound messages from Mail Clients, allowing

An extra layer of obfuscation and aggregation, like ToR or VPN's, makes the task of matching inbound/outbound packets harder.

One of the central notions is no system, apart from the end-points, ever has the unencrypted files/messages.

Messages are encrypted with the public-key of the next-hop destination before sending.

Messages queued to a link/channel should be kept as-received (encrypted for current system), only being decrypted before transmission.

Small blocks mean encryption can be performed in-memory, without any intermediate results touching a disk. Prevent swapping and VM page-write may be a challenge, but only one layer is lost.

SFTP/SSH use per-session keys to encrypt transfers.

With content re-encrypted per hop, there is no common plain-text allowing keys to be compromised.

Solution

That's what I'm aiming for, a low user-intervention system that users can reasonably trust:

Messages and files sent, when delivered can be shown to intact and complete.

No messages/files can be read "in clear" on the wire or on any intermediate system.

Messages/files only appear unencrypted on the source and destination system.
Simple tools, like USB drives, can be used to transfer files to/from off-line systems.

Users can get confirmations of delivery and/or acceptance back from each step along the way.

Data is sufficiently obscured so traffic analysis yield minimal useful metadata:

Using SFP in a Store+Forward mode removes the normal headers
Any SFP service can be used as a relay, if the per-hop encryption can't be organised on the server and exposing one-layer of encryption an acceptable risk.

Use Case

Alice and Bob want to exchange Company-Confidential information.

They both have access to the same SFTP server, or preferably a service that also supports re-encryption.

Or they each have access to SFTP servers hosted by co-operating trustworthy providers.

Alice and Bob both have off-line computers that they load/download encrypted files onto.

Alice and Bob both have internet-connected computers with the RTM App, PGP/GPG & SFTP installed, plus the transfer host PGP/GPG private keys.

Alice and Bob have exchanged their email Public PGP/GPG keys, both only have their private email keys on their off-line system.

Alice and Bob have exchanged their internet-host PGP/GPG public keys with their upstream server.

Alice, on her off-line system, writes one or more messages to Bob (and others) from her usual Mail Client and possibly queues some file transfers with another App.

The RTM software creates a directory of encrypted files which Alice then copies to a USB drive.

On her on-line system, Alice loads the USB drive and starts the RTM software than uses SFTP to transfer files to their shared server. The RTM software creates a control file for the batch and encrypts all files with the public key of the next hop.

The next-hop system receives the files and decrypts the control file, queuing blocks for whichever next hop link is to be used and creates & encrypts control files for batches.

After zero of more hops, Bob's SFTP server receives blocks batched for him from Alice and others and queues them for Bob, ready to encrypt them with his transfer host public key.

Bob, in his own time, starts RTM on his transfer system and downloads the blocks queued for him, decrypting them to copy to his USB drive. These blocks were encrypted for Bob by Alice and others using the email public key he shared with them. They cannot be decrypted on the transfer host.

Bob takes his USB drive to his off-line system and starts RTM to upload the encrypted blocks, decrypt them, checks hash-keys, reassembles the sent files and then delivers to the appropriate 'handler', email, file transfer or other specific tool.

Bob can then check email on his off-line machine using his favourite Mail Client. He can choose to save/distribute the decrypted files transferred by whatever means he needs.

Alice and Bob do not have to use off-line systems with air-gaps. If they accept the risk, they can run both the transfer host and "in-clear" system as Virtual Machines in the same system. A private, internal network can be used to transfer encrypted blocks between the two VM's.

If the system hosting the two VM's is compromised, the most an attacker can do is monitor the display.

I haven't discussed the various passwords/pass-phrases that would be needed in operation.

The system should be simple to install and configure and be mostly self-administering.

Monday, February 04, 2013

Storage: New Era Taxonomies

There are 3 distinct consumer facing market segments that must integrate seamlessly:

portable/mobile devices: "all flash".
Desktop (laptop) and Workstations
Servers and associated Storage Arrays.

We're heading into new territory in Storage:

"everything" is migrating on-line. Photos, Documents, Videos and messages.

but we don't yet have archival-grade digital storage media.

Write to a portable drive and data retention is 5 years: probable, 10 years: unlikely. That's a guess on my part, real-life may well be much worse.

Currently householders don't understand the problem.

Flash drives are not (nearly) permanent or error-free.
Most people have yet to experience catastrophic loss of data.
"Free" cloud storage may only be worth what you pay for it.

Disk Storage (magnetic HDD's) is entering its last factor-10 increase.

We should expect 5-10TB/platter for 2.5" drives as an upper bound.
Unsurprisingly, the rate of change has gone from "doubling every year" to 35%/year to 14%/year.
As engineers approach hard limits, the rate of improvement is slower and side-effects increase.
Do we build the first maximum-capacity HDD in 2020 or a bit later?

Flash memory is getting larger, cheaper and faster to access, but itself is entering an end-game.

but retention is declining, whilst wear issues may have been addressed, at least for now.
PCI-attached Flash, the minimum latency config, is set to become standard on workstations and servers.

How do we use it and adequately deal with errors?

Operating Systems, General Business servers and Internet-scale Datacentres and Storage Services have yet to embrace and support these new devices and constraints.

David Patterson, author with Gibson & Katz of the 1989 landmark paper on "RAID", noted that every storage layer trades cost-per-byte with throughput/latency.
When a layer is no longer cheaper than a faster layer, consumers discard it. Tapes were once the only high-capacity long-term storage option

My view of FileSystems and Storage:

high-frquency, low-latency: PCI-flash.
high-throughput, large-capacity: read/write HDD.
Create-Once Read-Maybe snapshot and archival: non-update-in-place HDD.

'Create' not 'Write'-Once. Because latent errors can only be discovered actively, one of the tasks of Archival Storage systems is regularly reading and rewriting all data.

What size HDD will become the norm for read/write and Create-once HDD's?
I suspect 2.5" because:

Watts-per-byte are low because aerodynamic drags increases near the fifth power of platter diameter and around cube of rotational speed.

A 3.5" disk has to spin at around 1700rpm to match a 5400rpm 2.5" drive in power/byte, and ~1950rpm to match a 7200rpm 2.5" drive.
All drives will use ~2.4 times the power to spin at 7200rpm vs 5400rpm.
Four 2.5" drives provide around the same capacity as a single 3.5" drive

Area of 2.5" platters are half a 3.5" platters.
2.5" drives are half the thickness as 3.5" drives (25.4mm)
3.5" drives may squeeze 5 platters, 25% better than 2.5".

Drives are cheap, but four smaller drives will always be more expensive than a single larger drive.

Four sets of heads will always provide:

higher aggregate throughput
lower-latency
more diversity, hence more resilience and recovery options
"fewer eggs in one basket". Impact of failures are limited to a single drive.

In raw terms, the cheapest, slowest, most error-prone storage will always be 3.5" drives. But admins build protected storage, not raw.

With 4TB 3.5" drives, 6 drives will provide 16TB in a RAID-6 config.

Note the lack of hot-spares.

With 1TB 2.5" drives, RAID-5 is still viable.

24 drives as two sets of 11 drives + hot-spare, provide 20TB.
For protected storage, 3.5" drives only offer at best 3.2 times the density and many-fold less throughput and latency.

Here are some of my take-aways from LCA 2013 in Canberra.

* We're moving towards 1-10TB of PCI-Flash or other Storage Class Memory being affordable and should expect it to be a 'normal' part of desktop & server systems. (Fusion IO now 'high-end')

- Flash isn't that persistent, does fade (is that with power on?).
- How can that be managed to give decade long storage?
- PCI-Flash/SCM could be organised as one or all of these:
- direct slow-access memory [need a block-oriented write model]
- fast VM page store
- File System. Either as
- 'tmpfs' style separate file system
- seamlessly integrated & auto-managed, like AAPL's Fusion LVM
- massive write-through cache (more block-driver?)

- there was a talk on Checkpoint/Restart in the Kernel, especially for VirtMach, it allows live migration and the potential for live kernel upgrades of some sort...
- we might start seeing 4,000 days uptime.
- PCI-Flash/SCM would be the obvious place to store CR's and as source/destination for copies
- nobody is talking about error-detection and data preservation for this new era: essential to explicitly detect/correct and auto-manage.

- But handling read-errors and memory corruption wasn't talked about..
- ECC won't be enough to *detect* let alone correct large block errors.
- Long up-times means we'll want H/A CPU's as well to detect compute errors.
Eg. triplicated CPU paths with result voting.

- the 'new era' approach to resilience/persistence has been whole-system replication and 'network' (ethernet/LAN) connection, and away from expensive internal replication for H/A.

==> As we require more and more of 'normal' systems, they start to need more and more Real-time and H/A components.

==> For "whole-system" replication, end-end error detection of all storage transfers starts to look necessary. i.e. an MD5 or other checksum generated by the drive or Object store and passed along the
chain into PCI-Flash and RAM: and maybe kept for rechecking.

==> With multiple levels of storage with latency between them and very high compute rates in CPU's, we're heading into the same territory that Databases addressed (in the 80's?) with Transactions and ACID.

* Log-structured File Systems are perfect fit for large Flash-based datastores.
- but log-structured FS may also be perfect for:
- write-once data, like long-term archives (eg. git repos)
- shingled write (no update-in-place) disks, effectively WORM.

==> I think we need an explicit Storage Error Detect/Correct layer between disks and other storage to increase BER from 10^14 or 10^16 to more like 10^25 or 10^30. [I need to calculate what numbers are actually needed.] Especially are everything gets stored digitally and people expect digital archives to be like paper and "just work" over many decades.

Thursday, January 17, 2013

Storage: FileSystems, Block/Object Storage and Physical Disk Management in 21st Century Systems

The central social contract filesystem and storage layers have with users is:

Don't lose data
Make it easy to get data in and out, preferably verifiably correct.
Performance is nice, but can never talks precedence over preserving data and replaying it correctly.

The approaches and paradigms that worked for Unix in 1970 won't work now. Its was a world of 5-10MB drives @ 1-10Mbps, 1MHz CPU's without cache and "off-line" storage was 6250bpi 9-track 0.5in tape (2400' ~120Mb).

Even nearly 20 years later in 1988, the year of the Patterson/Gibson/Katz RAID paper, streaming the full contents of a drive for a rebuild (100MB 5.25" SCSI drives) was ~100 seconds and ~1000 seconds for 1GB 8" Fujitsu Eagle drives preferred by the first Storage Arrays.

What's changed is the relative capacity and speeds of storage devices, the demands of "average users" and some additional layers of storage, like cache and Flash memory.

The old approaches are creaking and becoming more & more complex in attempts to handle performance (rate), volume and size. One "fast" filesystem, ReiserFS, was popular for a time but notorious with users for corrupting disks and losing data. Breaking the contract loses users...

The 10 TB/platter 2.5" drives expected by 2020 will only read 2-3 times faster than current 1TB drives (250-400MB/sec). That's 40,000 seconds to stream the whole drive: 10-12 hours. Increasingly, Jim Grays' millennial observation, "Tape is Dead, Disk is the new Tape" (meaning disks are good at streaming, poor at random I/O), is driving Storage designs. Enterprise Class Storage Arrays cannot compete with Flash memory for random I/O and to cover need increasingly long drive rebuild times (4-150 hours) have adopted slower, more inefficient/complex parity schemes.

We now have chips with 3 levels of cache, soon with on-chip DRAM, on-board DDR3 DRAM, PCIe Flash, SATA/SAS Flash and HDD drives and soon "no update-in-place" Shingled-Write drives.
SCM, Storage Class Memories, like Flash are hoped to provide the path to higher capacity devices, but to date, their are no obvious commercial technologies.

This in the context of at least 4 types of compute devices, each with different demands for Storage and data recovery and protections.

Mobile: smartphones and tablets. Not usually "content creators" but "viewers". Software from Firmware and vendor App Stores. Auto-sync config and data to "Cloud" or desktop.
Laptop/low-end Desktop: Limited "content creation". Restores via vendor products, erratic/random backups and data protection.
"Power user" Workstations: Professional platforms for content creation. Dedicated Storage Appliances, with problematic & erratic data protection.
Servers:

SOHO/SME, small ISP: single servers or small farms. Nil or problematic data protection.
SMP servers, business server farms: SAN's + Storage Arrays, H/A, multiple-sites, fail-over, ...
Clusters and large arrays: special filesystems, lots of storage, fast networks.
Internet-scale Data Centres: purpose built hardware and storage solutions.

On my average Mac desktop there are over 2M files after 5years, a scale not anticipated by the original Unix Directory-Inode-Links-Blocks design.

It's now possible and feasible for individuals to follow Gordon Bell and digitally record their entire lives. This is more than storing random snaps from smartphones, but creating a usable, accessible store.
In 10M recording seconds per year, individuals can create 100k files/year, 1TB at low data rates (100KB/sec) and view 1-10M files/web-pages.

This load for even the current 1-2B smartphone users (not the 6B cell phone services), whilst potentially being a boon for Network Operators & Storage vendors, requires new services and new approaches. Especially:

Strong User Identification with many roles per individual, for work, interests and personal life.
Single Federated views of individual-Identity storage.
New Search, Indexing, tagging and annotation tools.
Integrated "point-in-time" file browsing and scanning.
Internet-scale data de-duplication and peer-peer Storage.

We already have definitive solutions for "point-in-time" recovery of:

Text files via Version Control Systems like SVN, CVS, RCS, ...
Relational Databases with full-DB snapshot and "roll-forward" transaction logs,
but other important binary data types, {DB's, images, videos, sound, PDF-docs, geo-data, machine control, ...}, aren't born with verifiable digital signatures, nor their own change logs.

Metadata, both system generated like timestamps, Geo-locn and user GUID, and user-supplied data, like tags and text, are as important as the data changes.

Backups and Version Control Systems typically offer 3 sorts of versioning. A combination of these methodologies will be used at various levels:

Full Backup. 100% replication of all bits.
Incremental: Store only the bits changed since last Incremental.

Notionally, the minimum storage required.
Slowest to recover: all Incrementals must be applied sequentially, in order.
Most prone to error and data loss.

If one delta-file is deleted or corrupted, the entire set is useless.

Differential: Store all bits changed since last Full Backup.

Each differential is larger than the last, potentially up to the size of a Full Backup.
Fastest to recover.
Simplest to manage
Robust against errors and deletions, if the dataset was stored.

Work on non-Relational Databases is occurring, but there are important challenges for relational Databases a continuous-timeline view of storage, more than the current transactional/data-wharehouse duality/conversion:

limited data storage formats can be supported, "importing and conversion"
indexing of data is a separate activity and stored/accessed differently.
Schemas and Database names have to survive changes.
Semantics of individual fields are as important as

The Wayback Machine, a.k.a. The Internet Archive, gives us a working model and informs us that people can tolerate a) retrieval delays, b) some datasets unavailable and c) some data loss.

It costs a lot less for "Best Effort" rather than "Guaranteed" storage services, suggesting multiple approaches, cost structures and service offerings in the marketplace. Hopefully consumers won't be inveigled to over-pay or complacently rely on inappropriate low-cost providers.

Will current Consumer Protection laws need to be extended to this area??
If you share data within a group (Family and Friends) and some people don't maintain their part of the archive - losing data for people that rely on them, do current laws apply or will new law be needed?

Will this lead to new businesses of "Archive Auditing"?

There are currently three "drop-dead" problems for these services, ignoring the current "unsupported file format" and "ancient system & run-time" issues:

Currently, there is no archival quality digital media.
Hard Disks, Flash memory and CD/DVD's have limited lifetimes. They cannot be left on a shelf and be expected to work a couple of decades on... Data must be constantly scanned, rebuilt and migrated to new storage systems.

Acid-free paper and microforms will store documents for over 100 years.
Colour film is still the only archival media for movies and still images.
No good magnetic media exist for medium-long term storage of sound recordings.

Vendor longevity and professional misconduct or negligence, even systemic corruption.

When an Archival Storage Service goes bust, how do the owners of the data recover their data? Not over network links and if the facilities are locked and powered-off by administrators or sheriffs, not physically either.
There are around the world, just a few Telcos or Power Utilities that are 100 years old. Can we really expected profitable Storage to start now and last 5 times longer than Google without any commercial upsets? I'd argue "no".
Rogue admins and managers are the least of the problem, though they'll exist and cause problems.
Expecting ordinary, fallible owners, workers and managers to always resist temptation, bribery and sloth/negligence is more than naive and simplistic. Mistakes will happen, security breaches will occur and ordinary folk doing boring jobs will take shortcuts.
Valuable resources will always attract those wishing to steal it. These sorts of facilities must begin by never storing anything of value. Organised crime's only access must be via the individual users' system/device, not in a single, centralised resource.

Legal access issues: a whole new area of lucrative International Law awaits us...

Who has the right to look at data?
Can data "in default" (unpaid fees) be sold? To whom? At what price?
Can a Vendor move data from the Jurisdiction of origin, with or without permission?
Can Vendors share data across facilities in different Jurisdictions?
Can Storage custodians be forced to grant local Law Enforcement Offices access to individual or bulk data?

There are now three distinct views of the filesystem provided in the abstract model for user applications:

Current files
snapshots
archives

The O/S has to provide these services for each of those dimensional slices through the storage:

map names (paths) to inodes. Subsumes a "mount device/mount-point" model.
inodes (the immutable file, with metadata)
datablock link map, which reduces to start/end for contiguous allocation.
data blocks and free block list
Physical drive management, like LVM.

Systems have to address four different aspects of real-world storage access:

availability and connection paths
errors and rereads
erasures and failures
durability and longevity of data sets (protection and archive)

Overlaid on this are 4-5 distinct access patterns, similar to a metal working "temperatures":

"white-hot" region: read/write access on-board (RAM and PCIe Flash)
"red-hot" region: read/write access to direct-connect updatable HDD's
cool region: write once access to Big, Slow HDD's, probably non-update-in-place.
"blue" (cold) region: write once, seldom read HDD's. No update-in-place, append-only.
"black" (frozen) region: remote and archival storage. Rarely Accessed, Critical when needed.

There is a direct correspondence between different temperature regions and the filesystem abstraction they are providing.

Archives are read-only and live only in cool, cold and frozen regions.
Snapshots may be in a "red-hot" region, but otherwise in cool and cold regions.

Files are ever only moved to Archive from the Snapshot areas.

Current files will be migrated from, or cached into, the high-speed read/write regions on demand.

The link between Snapshots and Current files is: Snapshot[0] == Current filesystem.

My thesis is that the traditional Unix filesystem and O/S structure of Directories-Inodes-Block_maps-Data_blocks cannot serve all these demands well, but that we already have very good tools to handle them.

Schemes to handle inodes, Block_Maps & Linking and Block access for each "temperature" storage can be designed well for the specific trade-offs and performance expectations.

The major problem appears to me to be mapping File Names to Inodes:

It either requires very high performance and low-latency for the hottest I/O region, or
requires very large namespaces for snapshots and archives.
Indexes for Current & Snapshot views may be stored in low-latency storage, but the volume of names stored in long-term Archives means they cannot.

Neither of which is well served by the traditional "directory in a block", backed by O/S cache model.

But both are robustly handled by Database systems, albeit differently organised, indexed and tuned.

What is missing in normal systems is:

Filesystem or storage layer of "What's Changed?" (Deltas) via md5sums or change messages.
Swapping snapshot views between "Delta"and "Full" filesystem views:

'rsync' identifies changed files, but users have to create full filesystem images themselves.
Apple's TimeMachine creates a full filesystem image at a point-in-time, but provides no "Delta" interface beyond a single file or directory.

There are two implications that fall out of this analysis:

Consumers will demand "Open" storage standards allowing them to swap devices, systems and Storage Vendors, not be locked into Proprietary standards, especially single-vendor solutions, and
a software solution model based on the Apache web-server or Linux kernel: co-operative Open Source backed by the GNU license. This allows all vendors to avoid license and patent issues, share work, leverage prior work, support and develop common standards, whilst also allowing market-differentiation by offering specific tools or hardware/software combinations.

The current Unix-like approaches of filesystems, O/S supported directory scanning (name to inode mapping), LVM handling {data protection, logical and physical volumes}, independent snapshot/archive facilities, independent hot-plug media and manual setup and operation of Archival stores cannot provide an Identity-keyed Federated Storage & Archive system.

Not all data stores or vendors will provide the same grade of service. Features that can be borrowed from:

NTP (Network Time Protocol): stratum level of server. Just how good are they?
IP Routing: "cost of routes". Preferentially chose the faster, cheaper services.

The main features required in an Identity-keyed Federated Storage & Archive system are:

Data access limited by Identity (data privacy as part of "Security")

Multiple Identities per user, based on role or use.
Multiple Users and Identities per Device.
Master Identity access to specified data, for work and families.

Automatic implementation of Policies
Addition and management of user-managed hot-plug media
Automatic integration across all single-Identity devices of local disk, local network storage, peer storage and multiple Vendor services
Policies set as targets:

Cost
Maximum size of store
Maximum data recovery time
Minimum and Maximum times between recovery points:

every minute for the last 36 hours
every hour for the last fortnight
every day for the last year
every week for the last decade
every month after that

normal performance: access rate, I/O per sec
By datatype, Data Resilience and Longevity (Probability Data Loss per period, Maximum data loss event size)
Warnings, Alerts and Alarms.
Default and specified Data Destruction dates

Wednesday, January 02, 2013

Storage: Specifying Data Resilience and Data Protection

In Communications theory, there are two distinct concepts:

Errors [you get a signal, but noise has induced one or more symbol errors], and
Erasures [you lost the signal for a time or it was swamped by noise]

Erasures are often in "bursts", so techniques are needed to not just recover/correct a small number of symbols, but

This is the theory behind the Reed-Solomon [Galois Field] encoding for CD's and later DVD's.
It uses redundant symbols to recreate the data, needing twice as many symbols to correct errors as recreate erasures. A [24,28] RS code encodes 24 symbols into 28, with 5 symbols/bytes of redundancy. This can be used to correct up to 2 errors (2*2 symbols used) plus 1 erasure.

The innovation in CD's was applying 2 R-S codes successively, but between them using Cross Interleaving to handle burst errors by spreading a single L1 [24,28] frame across a whole 2352 sector [86?,98]. Only 1 byte of an erased L1 frame would appear in any single L2 sector.

DVD's use a related but different form of combining two R-S codes: Internal/External Parity.

CR-ROM's apply an L3 R-S Product Code on top of the L1&L2 RS codes + CIRC to get more acceptable Bit Error Rates (BER's) of ~10^15, vs 10^9. Data per frame goes down to 2048by (2Kb) fro 2352by.

With Hard Disks, and Storage in general, the last two big advances were:

RAID [1988/9, Patterson, Gibson, Katz]
Snapshots [Network Appliance, early 1990's]

RAID-3/4/5 was notionally about catering for erasures caused by the failure of a whole drive or component, such as a controller or cable.
This was done with low overhead by using the computationally cheap and fast XOR operation to calculate a single parity block.

But in use, the ability to correct both errors and erasures with parity blocks has been conflated...

RAID-3/4/5 is now generally though to be about Error Correction, not Failure Protection.

The usual metrics quoted for HDD's & SSD's are:
- MTBF (~1M hours) or Annualised Failure Rate (AFR) 0.6-0.7%
- BER (unrecoverable Bit Error Rate) 1 in 10^15
- Size, Avg seek time, max/sustained transfer rate.

Operational Questions, Drive Reliability:

- For a fleet, per 1000 drives, average drives fail per year?
[1 year = ~8700 hrs, = ~8.5M hours/year/1000 drives = 8.5 drive
fails/year]
Alternatively, AFR: 0.6-0.7% * 1000, = 6-7 drives/1000/year

- What's the minimum wall-clock time to rebuild a full drive?
[Size / sustained transfer rate: 4Tb @ 150MB/sec write = 7.5Hrs ]

- what's the likelihood of a drive fail during a rebuild?
7.5 hrs / 1M hrs = 0.001% [???] per drive.
- for RAID-set of 10, (7.5/1M)/10 = 0.01%

- probability data loss in rebuild (N = 10):
Transfer / BER = 4TB * 10 = 32 * 10* 10^12 bits =
3.2 * 10^14 / 10^15
= .32 = 32% [suggests further protection is needed against data loss]

Data Protection questions. I don't know how to address these...

- If we store data in RAID-6/7 units of 10-drive-equivalents
with a lifetime of 5 years per set:
- In a "lifetime" (60 year = 12 sets),
what's the probability of Data Loss?

- How many geographically separated replicas do we need to
store data 100 years?

I think I know how to specify Data Protection: the same way (%) as AFR.

What you have to build for is Mean-Years-Between-Dataloss
and I guess that implies the degree of Dataloss: 1-by, 1-block (4Kb), 1MB?
And well as complete failure of a dataset-copy.

Typical AFR's are 0.4%-0.7%, as quoted by drive manufacturers based on
accelerated testing.

We know from those 2008(?) studies of large cohorts of drives, this is
optimistic by an order of magnitude...

An AFR of 1 in 10^6 results in a 99.99% 100YR-F-R.
(1 - .0000010) ^ 100

AFR of 1 in 10^5 is 99.9% 100YR-FR (CFR? Century Failure Rate)

AFR of 1 in 10^4 is 99.0% CFR.

So we have to estimate a few more probabilities:
- site suffering natural disaster or fire etc.
- site suffering war damage or intentional attack
- country or economy crumbling [ every 40-50 yrs a depression ]
- company surviving (Kodak lated 100yrs
- admins doing their job competently and fully.
- managers not scamming (selling disks, not provide service)

Are there more??

Monday, December 17, 2012

Storage: Active spares in RAID volumes

If you have a spare HDD in a chassis powered-up and spinning, then the best use of the power you're burning is to use the drive.

Sunday, November 18, 2012

Cross compiling i386-ELF Linux kernel on OS/X, Snow Leopard

This is NOT a tutorial on 'C', Software Development, the GNU toolchains, including 'make', or building a linux kernel. That is assumed knowledge.

There's already good documentation on the Web for building Android (ARM) kernels on OS/X, and some tantalising, though incomplete, comments on building i386 kernels: "it took a while to setup, then was OK"...

Although OS/X runs on x86 (and x86_64), it won't build an x86 Linux kernel. You still need to cross-compile because Linux uses ELF (extensible loader format) and OS/X uses its own multi-CPU format, "mach-o".

This environment variable that must be set:

CROSS_COMPILE=i386-elf- [or the full path to your gcc tools, but only the common prefix]

Optionally, you can set (32-bit):

ARCH=x86

As well, I added these directories to the beginning of my PATH to catch gcc (HOSTCC) and i386-elf-gcc for the cross-compiler:

PATH=/opt/local/bin:/opt/local/sbin:/opt/local/i386-elf/bin:/opt/local/libexec/gcc/i386-elf/4.3.2:$PATH

The Linux kernel Makefile uses some trickery to have verbose, quiet and silent modes, default is "quiet". If you need to see for debugging, the commands issued, set this additional environment variable:

 KBUILD_VERBOSE=1

I chose to use the Macports native 'gcc', not the OS/X supplied compiler. Because the Macport i386-elf version of gcc has incorrect paths compiled in, I needed the 2 additional i386-elf directories.

Note: I had to make a symbolic link for i386-elf-gcc. The port "i386-elf-gcc @4.3.2_1" installed the full set of tools (as, ld, nm, strip, ...) into /opt/local/bin, but didn't install the shortname ('gcc'), only the long version name: i386-elf-gcc-4.3.2, which the Linux kernel Makefile doesn't cater for.

The 'Macports' project provides many GNU tools pre-built, with source. Generally, it's a good first thing to try. I reported multiple faults and found them unresponsive and less than helpful. YMMV.

The command is 'port', after the BSD tool of the same name. BSD delivered pure-source bundles, Macports do not. While Macports notionally updates itself, I had trouble with a major upgrade, initially available only as Source, now available as a binary upgrade.

There seems to be a bias towards newer OS/X environments. "Snow Leopard", Darwin 10.8.0, is now old. "Lion" and "Mountain Lion" have replaced it...

Ben Collins, 2010, has good notes, a working elf.h, and gcc-4.3.3 and binutils from Ubuntu Jaunty.
A comment suggests GNU sed is necessary, not the standard OS/X sed. I made this change.

From 2010, using 'ports' to cross-compile to ARM by Plattan Mattan is useful. Uses OS/X gcc as HOSTCC.
The page suggests installing ports: install libelf git-core
Then using git to clone the kernel source.

I installed ports:

gcc43 @4.3.6_7 (active)
i386-elf-binutils @2.20_0 (active)
i386-elf-gcc @4.3.2_1 (active)
libelf @0.8.13_2 (active)

Other useful pages:
Building GCC toolchain for ARM on Snow Leopard (using Macports)
Building i386-elf cross compiler and binutils on OS/X (from source).

I got a necessary hint from Alan Modra about i386-elf-as (the assembler) processing "/" as comments, not "divide" in macro expansions, as expected in the kernel source:

For compatibility with other assemblers, '/' starts a comment on the i386-elf target. So you can't use division. If you configure for i386-linux (or any of the bsds, or netware), you won't have this problem.

Do NOT in your .config file select "a.out" as an executable fileformat. The i386 processor isn't defined, so compile fails with "SEGMNT_SIZE" not defined.

I went to kernel.org and downloaded a bzipped tar file of linux-2.6.34.13 ("Full Source") for my testing. Always a good idea to check the MD5 of any download, if available.
I wanted a stable, older kernel to test with.Your needs will vary.

I didn't run into the "malloc.h" problem noted by Plattan, it seemed to come with libelf.

I made three sets of changes (changes in red) to the standard linux Makefile (can apply as a patch):

mini-too:linux-2.6.34.13 steve$ diff -u ../saved/Makefile.dist Makefile
--- ../saved/Makefile.dist 2012-08-21 04:45:22.000000000 +1000
+++ Makefile 2012-11-20 14:10:46.000000000 +1100
@@ -231,7 +231,7 @@
 
 HOSTCC       = gcc
 HOSTCXX      = g++
-HOSTCFLAGS   = -Wall -Wmissing-prototypes -Wstrict-prototypes -O2 -fomit-frame-pointer
+HOSTCFLAGS   = -Wall -Wmissing-prototypes -Wstrict-prototypes -O2 -fomit-frame-pointer -idirafter /opt/local/include 
 HOSTCXXFLAGS = -O2
 
 # Decide whether to build built-in, modular, or both.
@@ -335,10 +335,10 @@
     -Wbitwise -Wno-return-void $(CF)
 MODFLAGS = -DMODULE
 CFLAGS_MODULE   = $(MODFLAGS)
-AFLAGS_MODULE   = $(MODFLAGS)
+AFLAGS_MODULE   = $(MODFLAGS)  -Wa,--divide 
 LDFLAGS_MODULE  = -T $(srctree)/scripts/module-common.lds
 CFLAGS_KERNEL =
-AFLAGS_KERNEL =
+AFLAGS_KERNEL =  -Wa,--divide 
 CFLAGS_GCOV = -fprofile-arcs -ftest-coverage
 
 
@@ -354,6 +354,7 @@
      -fno-strict-aliasing -fno-common \
      -Werror-implicit-function-declaration \
      -Wno-format-security \
+     -isystem /opt/local/i386-elf/include -idirafter /opt/local/lib/gcc/i386-elf/4.3.2/include/ -idirafter /usr/include -idirafter /usr/include/i386 \
      -fno-delete-null-pointer-checks
 KBUILD_AFLAGS   := -D__ASSEMBLY__

I created the required "elf.h", not supplied in port libelf, in /opt/local/include, specified above in HOSTCFLAGS:

mini-too:linux-2.6.34.13 steve$ cat /opt/local/include/elf.h 
/* @(#) $Id: $ */

#ifndef _ELF_H
#define _ELF_H
#include <libelf/gelf.h>
/* http://plattanimattan.blogspot.com.au/2010/04/cross-compiling-linux-on-mac-osx.html */

#define R_ARM_NONE        0
#define R_ARM_PC24        1
#define R_ARM_ABS32       2
#define R_MIPS_NONE       0
#define R_MIPS_16         1
#define R_MIPS_32         2
#define R_MIPS_REL32      3
#define R_MIPS_26         4
#define R_MIPS_HI16       5
#define R_MIPS_LO16       6

/* from /opt/local/libexec/llvm-3.1/include/llvm/Support/ELF.h */
/* or http://www.swissdisk.com/~bcollins/macosx/elf.h */

#define R_386_NONE      0
#define R_386_32        1
#define R_386_PC32      2
#define R_386_GOT32     3
#define R_386_PLT32     4
#define R_386_COPY      5
#define R_386_GLOB_DAT  6
#define R_386_JMP_SLOT  7 /* was R_386_JUMP_SLOT */
#define R_386_RELATIVE  8
#define R_386_GOTOFF    9  
#define R_386_GOTPC     10 
#define R_386_32PLT     11 
#define R_386_TLS_TPOFF 14 
#define R_386_TLS_IE    15 
#define R_386_TLS_GOTIE 16 
#define R_386_TLS_LE    17 
#define R_386_TLS_GD    18 
#define R_386_TLS_LDM   19 
#define R_386_16        20 
#define R_386_PC16      21 
#define R_386_8 22
#define R_386_PC8       23 
#define R_386_TLS_GD_32 24 
#define R_386_TLS_GD_PUSH       25         
#define R_386_TLS_GD_CALL       26      
#define R_386_TLS_GD_POP        27         
#define R_386_TLS_LDM_32        28      
#define R_386_TLS_LDM_PUSH      29         
#define R_386_TLS_LDM_CALL      30      
#define R_386_TLS_LDM_POP       31         
#define R_386_TLS_LDO_32        32      
#define R_386_TLS_IE_32 33                 
#define R_386_TLS_LE_32 34                 
#define R_386_TLS_DTPMOD32      35
#define R_386_TLS_DTPOFF32      36
#define R_386_TLS_TPOFF32       37
#define R_386_TLS_GOTDESC       39
#define R_386_TLS_DESC_CALL     40      
#define R_386_TLS_DESC  41                 
#define R_386_IRELATIVE 42 
#define R_386_NUM       43 

#endif /* _ELF_H */

I didn't try specifying all variables on the command-line when invoking make. This might work, though incomplete (only 2 of the 3 changes):

$ make ARCH=x86 CROSS_COMPILE=i386-elf- HOSTCFLAGS="-idirafter /Users/steve/src/linux/linux-2.6.34.13/include/linux" AFLAGS_KERNEL="-Wa,--divide"

It took me sometime to figure out how to browse on-line the kernel.org git repository for my specific kernel, to investigate the change history of a specific file. It's worth taking the time to learn this.

I was not able to figure out how to get the GNU make in OS/X to show me the full commands it was about to execute. "make -d" spits out a mountain of stuff on what it is doing, but not the commands. "make -n" is for dry-runs and prints commands, though whether or not that's what is run later, I'm not sure. See KBUILD_VERBOSE.

It also seems impossible to ask gcc to tell you what directory/ies it's using for the system include files.
Whilst the Macports gcc works, it uses /usr/include, the default OS/X gcc directory and creates some additional header files.

Current status: ~~19-Nov-2012. failing in drivers/gpu with include files missing. Which "CC"?~~

  CC      drivers/gpu/drm/drm_auth.o
In file included from include/drm/drmP.h:75,
                 from drivers/gpu/drm/drm_auth.c:36:
include/drm/drm.h:47:24: error: sys/ioccom.h: No such file or directory
include/drm/drm.h:48:23: error: sys/types.h: No such file or directory

Final status: 20-Nov-2012. Untested - booting kernel.

  BUILD   arch/x86/boot/bzImage
Root device is (14, 1)
Setup is 12076 bytes (padded to 12288 bytes).
System is 3763 kB
CRC d747d6db
Kernel: arch/x86/boot/bzImage is ready  (#1)
  Building modules, stage 2.
  MODPOST 2 modules
  CC      arch/x86/kernel/test_nx.mod.o
  LD [M]  arch/x86/kernel/test_nx.ko
  CC      drivers/scsi/scsi_wait_scan.mod.o
  LD [M]  drivers/scsi/scsi_wait_scan.ko

real 11m5.195s
user 8m31.722s
sys 1m50.243s

A number of header errors (missing or duplicates & incompatible decls) were not solved, but sidestepped by unselected the problem areas in the .config file (details below).

Additional changes:

gsed: sudo port install gsed; (cd /opt/local/bin; sudo ln -s ./gsed sed)
Download byteswap.h from GNUlib on git.savannah.gnu.org to /opt/local/include/. gcc-4.3 may have obsoleted the need for this.
sudo ln -s /usr/include/i386/endian.h /opt/local/include/
Download www.opensource.apple.com/release/mac-os-x-1068/llvmgcc42-2118/fixincludes/tests/base/sys/sysmacros.h, copy to /opt/local/include/sys/; needed mkdir /opt/local/include/sys

Creating config file.

make defconfig

producing a .config that can be edited or patched.

Summary of subsystem items unselected in .config afterwards:

# CONFIG_SUSPEND is not set
# CONFIG_HIBERNATION is not set
# CONFIG_ACPI is not set
# CONFIG_CORE_DUMP_DEFAULT_ELF_HEADERS is not set
# CONFIG_INET_LRO is not set
# CONFIG_NETFILTER_XT_TARGET_CONNSECMARK is not set
# CONFIG_NETFILTER_XT_TARGET_MARK is not set
# CONFIG_NETFILTER_XT_TARGET_NFLOG is not set
# CONFIG_NETFILTER_XT_TARGET_SECMARK is not set
# CONFIG_NETFILTER_XT_TARGET_TCPMSS is not set
# CONFIG_AGP is not set
# CONFIG_DRM is not set
# CONFIG_FB_CFB_FILLRECT is not set
# CONFIG_FB_CFB_COPYAREA is not set
# CONFIG_FB_CFB_IMAGEBLIT is not set

You might try saving the patches below (a 'diff -u' of the above defconfig result to mine) and apply to the .config file as a patch: patch .config defconfig.patch

--- ../saved/defconfig 2012-11-19 23:20:04.000000000 +1100
+++ .config 2012-11-19 23:32:49.000000000 +1100
@@ -1,7 +1,7 @@
 #
 # Automatically generated make config: don't edit
 # Linux kernel version: 2.6.34.13
-# Mon Nov 19 18:57:25 2012
+# Mon Nov 19 15:53:57 2012
 #
 # CONFIG_64BIT is not set
 CONFIG_X86_32=y
@@ -390,7 +390,6 @@
 CONFIG_X86_PAT=y
 CONFIG_ARCH_USES_PG_UNCACHED=y
 CONFIG_ARCH_RANDOM=y
-CONFIG_EFI=y
 CONFIG_SECCOMP=y
 # CONFIG_CC_STACKPROTECTOR is not set
 # CONFIG_HZ_100 is not set
@@ -401,7 +400,6 @@
 CONFIG_SCHED_HRTICK=y
 CONFIG_KEXEC=y
 CONFIG_CRASH_DUMP=y
-# CONFIG_KEXEC_JUMP is not set
 CONFIG_PHYSICAL_START=0x1000000
 CONFIG_RELOCATABLE=y
 CONFIG_X86_NEED_RELOCS=y
@@ -418,45 +416,11 @@
 CONFIG_PM_DEBUG=y
 # CONFIG_PM_ADVANCED_DEBUG is not set
 # CONFIG_PM_VERBOSE is not set
-CONFIG_CAN_PM_TRACE=y
-CONFIG_PM_TRACE=y
-CONFIG_PM_TRACE_RTC=y
-CONFIG_PM_SLEEP_SMP=y
-CONFIG_PM_SLEEP=y
-CONFIG_SUSPEND=y
-# CONFIG_PM_TEST_SUSPEND is not set
-CONFIG_SUSPEND_FREEZER=y
-CONFIG_HIBERNATION_NVS=y
-CONFIG_HIBERNATION=y
-CONFIG_PM_STD_PARTITION=""
+# CONFIG_SUSPEND is not set
+# CONFIG_HIBERNATION is not set
 # CONFIG_PM_RUNTIME is not set
-CONFIG_PM_OPS=y
-CONFIG_ACPI=y
-CONFIG_ACPI_SLEEP=y
-CONFIG_ACPI_PROCFS=y
-CONFIG_ACPI_PROCFS_POWER=y
-# CONFIG_ACPI_POWER_METER is not set
-CONFIG_ACPI_SYSFS_POWER=y
-CONFIG_ACPI_PROC_EVENT=y
-CONFIG_ACPI_AC=y
-CONFIG_ACPI_BATTERY=y
-CONFIG_ACPI_BUTTON=y
-CONFIG_ACPI_VIDEO=y
-CONFIG_ACPI_FAN=y
-CONFIG_ACPI_DOCK=y
-CONFIG_ACPI_PROCESSOR=y
-CONFIG_ACPI_HOTPLUG_CPU=y
-# CONFIG_ACPI_PROCESSOR_AGGREGATOR is not set
-CONFIG_ACPI_THERMAL=y
-# CONFIG_ACPI_CUSTOM_DSDT is not set
-CONFIG_ACPI_BLACKLIST_YEAR=0
-# CONFIG_ACPI_DEBUG is not set
-# CONFIG_ACPI_PCI_SLOT is not set
-CONFIG_X86_PM_TIMER=y
-CONFIG_ACPI_CONTAINER=y
-# CONFIG_ACPI_SBS is not set
+# CONFIG_ACPI is not set
 # CONFIG_SFI is not set
-# CONFIG_APM is not set
 
 #
 # CPU Frequency scaling
@@ -479,11 +443,8 @@
 #
 # CPUFreq processor drivers
 #
-# CONFIG_X86_PCC_CPUFREQ is not set
-CONFIG_X86_ACPI_CPUFREQ=y
 # CONFIG_X86_POWERNOW_K6 is not set
 # CONFIG_X86_POWERNOW_K7 is not set
-# CONFIG_X86_POWERNOW_K8 is not set
 # CONFIG_X86_GX_SUSPMOD is not set
 # CONFIG_X86_SPEEDSTEP_CENTRINO is not set
 # CONFIG_X86_SPEEDSTEP_ICH is not set
@@ -491,7 +452,6 @@
 # CONFIG_X86_P4_CLOCKMOD is not set
 # CONFIG_X86_CPUFREQ_NFORCE2 is not set
 # CONFIG_X86_LONGRUN is not set
-# CONFIG_X86_LONGHAUL is not set
 # CONFIG_X86_E_POWERSAVER is not set
 
 #
@@ -513,9 +473,7 @@
 CONFIG_PCI_GOANY=y
 CONFIG_PCI_BIOS=y
 CONFIG_PCI_DIRECT=y
-CONFIG_PCI_MMCONFIG=y
 CONFIG_PCI_DOMAINS=y
-# CONFIG_DMAR is not set
 CONFIG_PCIEPORTBUS=y
 # CONFIG_HOTPLUG_PCI_PCIE is not set
 CONFIG_PCIEAER=y
@@ -528,7 +486,6 @@
 # CONFIG_PCI_STUB is not set
 CONFIG_HT_IRQ=y
 # CONFIG_PCI_IOV is not set
-CONFIG_PCI_IOAPIC=y
 CONFIG_ISA_DMA_API=y
 # CONFIG_ISA is not set
 # CONFIG_MCA is not set
@@ -556,7 +513,6 @@
 # CONFIG_HOTPLUG_PCI_FAKE is not set
 # CONFIG_HOTPLUG_PCI_COMPAQ is not set
 # CONFIG_HOTPLUG_PCI_IBM is not set
-# CONFIG_HOTPLUG_PCI_ACPI is not set
 # CONFIG_HOTPLUG_PCI_CPCI is not set
 # CONFIG_HOTPLUG_PCI_SHPC is not set
 
@@ -564,7 +520,7 @@
 # Executable file formats / Emulations
 #
 CONFIG_BINFMT_ELF=y
-CONFIG_CORE_DUMP_DEFAULT_ELF_HEADERS=y
+# CONFIG_CORE_DUMP_DEFAULT_ELF_HEADERS is not set
 CONFIG_HAVE_AOUT=y
 # CONFIG_BINFMT_AOUT is not set
 CONFIG_BINFMT_MISC=y
@@ -610,7 +566,7 @@
 # CONFIG_INET_XFRM_MODE_TRANSPORT is not set
 # CONFIG_INET_XFRM_MODE_TUNNEL is not set
 # CONFIG_INET_XFRM_MODE_BEET is not set
-CONFIG_INET_LRO=y
+# CONFIG_INET_LRO is not set
 # CONFIG_INET_DIAG is not set
 CONFIG_TCP_CONG_ADVANCED=y
 # CONFIG_TCP_CONG_BIC is not set
@@ -671,11 +627,11 @@
 CONFIG_NF_CONNTRACK_SIP=y
 CONFIG_NF_CT_NETLINK=y
 CONFIG_NETFILTER_XTABLES=y
-CONFIG_NETFILTER_XT_TARGET_CONNSECMARK=y
-CONFIG_NETFILTER_XT_TARGET_MARK=y
-CONFIG_NETFILTER_XT_TARGET_NFLOG=y
-CONFIG_NETFILTER_XT_TARGET_SECMARK=y
-CONFIG_NETFILTER_XT_TARGET_TCPMSS=y
+# CONFIG_NETFILTER_XT_TARGET_CONNSECMARK is not set
+# CONFIG_NETFILTER_XT_TARGET_MARK is not set
+# CONFIG_NETFILTER_XT_TARGET_NFLOG is not set
+# CONFIG_NETFILTER_XT_TARGET_SECMARK is not set
+# CONFIG_NETFILTER_XT_TARGET_TCPMSS is not set
 CONFIG_NETFILTER_XT_MATCH_CONNTRACK=y
 CONFIG_NETFILTER_XT_MATCH_MARK=y
 CONFIG_NETFILTER_XT_MATCH_POLICY=y
@@ -853,13 +809,6 @@
 CONFIG_PROC_EVENTS=y
 # CONFIG_MTD is not set
 # CONFIG_PARPORT is not set
-CONFIG_PNP=y
-CONFIG_PNP_DEBUG_MESSAGES=y
-
-#
-# Protocols
-#
-CONFIG_PNPACPI=y
 CONFIG_BLK_DEV=y
 # CONFIG_BLK_DEV_FD is not set
 # CONFIG_BLK_CPQ_DA is not set
@@ -950,7 +899,6 @@
 CONFIG_ATA=y
 # CONFIG_ATA_NONSTANDARD is not set
 CONFIG_ATA_VERBOSE_ERROR=y
-CONFIG_ATA_ACPI=y
 CONFIG_SATA_PMP=y
 CONFIG_SATA_AHCI=y
 # CONFIG_SATA_SIL24 is not set
@@ -969,7 +917,6 @@
 # CONFIG_SATA_VIA is not set
 # CONFIG_SATA_VITESSE is not set
 # CONFIG_SATA_INIC162X is not set
-# CONFIG_PATA_ACPI is not set
 # CONFIG_PATA_ALI is not set
 CONFIG_PATA_AMD=y
 # CONFIG_PATA_ARTOP is not set
@@ -1062,7 +1009,6 @@
 # CONFIG_EQUALIZER is not set
 # CONFIG_TUN is not set
 # CONFIG_VETH is not set
-# CONFIG_NET_SB1000 is not set
 # CONFIG_ARCNET is not set
 CONFIG_PHYLIB=y
 
@@ -1364,7 +1310,6 @@
 # CONFIG_INPUT_PCSPKR is not set
 # CONFIG_INPUT_APANEL is not set
 # CONFIG_INPUT_WISTRON_BTNS is not set
-# CONFIG_INPUT_ATLAS_BTNS is not set
 # CONFIG_INPUT_ATI_REMOTE is not set
 # CONFIG_INPUT_ATI_REMOTE2 is not set
 # CONFIG_INPUT_KEYSPAN_REMOTE is not set
@@ -1372,7 +1317,6 @@
 # CONFIG_INPUT_YEALINK is not set
 # CONFIG_INPUT_CM109 is not set
 # CONFIG_INPUT_UINPUT is not set
-# CONFIG_INPUT_WINBOND_CIR is not set
 
 #
 # Hardware I/O ports
@@ -1420,7 +1364,6 @@
 CONFIG_SERIAL_8250_CONSOLE=y
 CONFIG_FIX_EARLYCON_MEM=y
 CONFIG_SERIAL_8250_PCI=y
-CONFIG_SERIAL_8250_PNP=y
 # CONFIG_SERIAL_8250_CS is not set
 CONFIG_SERIAL_8250_NR_UARTS=32
 CONFIG_SERIAL_8250_RUNTIME_UARTS=4
@@ -1464,8 +1407,6 @@
 # CONFIG_NSC_GPIO is not set
 # CONFIG_CS5535_GPIO is not set
 # CONFIG_RAW_DRIVER is not set
-CONFIG_HPET=y
-# CONFIG_HPET_MMAP is not set
 # CONFIG_HANGCHECK_TIMER is not set
 # CONFIG_TCG_TPM is not set
 # CONFIG_TELCLOCK is not set
@@ -1475,7 +1416,6 @@
 CONFIG_I2C_COMPAT=y
 # CONFIG_I2C_CHARDEV is not set
 CONFIG_I2C_HELPER_AUTO=y
-CONFIG_I2C_ALGOBIT=y
 
 #
 # I2C Hardware Bus support
@@ -1500,11 +1440,6 @@
 # CONFIG_I2C_VIAPRO is not set
 
 #
-# ACPI drivers
-#
-# CONFIG_I2C_SCMI is not set
-
-#
 # I2C system bus drivers (mostly embedded / system-on-chip)
 #
 # CONFIG_I2C_OCORES is not set
@@ -1625,12 +1560,6 @@
 # CONFIG_SENSORS_HDAPS is not set
 # CONFIG_SENSORS_LIS3_I2C is not set
 # CONFIG_SENSORS_APPLESMC is not set
-
-#
-# ACPI drivers
-#
-# CONFIG_SENSORS_ATK0110 is not set
-# CONFIG_SENSORS_LIS3LV02D is not set
 CONFIG_THERMAL=y
 # CONFIG_THERMAL_HWMON is not set
 CONFIG_WATCHDOG=y
@@ -1713,42 +1642,19 @@
 #
 # Graphics support
 #
-CONFIG_AGP=y
-# CONFIG_AGP_ALI is not set
-# CONFIG_AGP_ATI is not set
-# CONFIG_AGP_AMD is not set
-CONFIG_AGP_AMD64=y
-CONFIG_AGP_INTEL=y
-# CONFIG_AGP_NVIDIA is not set
-# CONFIG_AGP_SIS is not set
-# CONFIG_AGP_SWORKS is not set
-# CONFIG_AGP_VIA is not set
-# CONFIG_AGP_EFFICEON is not set
+# CONFIG_AGP is not set
 CONFIG_VGA_ARB=y
 CONFIG_VGA_ARB_MAX_GPUS=16
-# CONFIG_VGA_SWITCHEROO is not set
-CONFIG_DRM=y
-CONFIG_DRM_KMS_HELPER=y
-# CONFIG_DRM_TDFX is not set
-# CONFIG_DRM_R128 is not set
-# CONFIG_DRM_RADEON is not set
-# CONFIG_DRM_I810 is not set
-# CONFIG_DRM_I830 is not set
-CONFIG_DRM_I915=y
-# CONFIG_DRM_I915_KMS is not set
-# CONFIG_DRM_MGA is not set
-# CONFIG_DRM_SIS is not set
-# CONFIG_DRM_VIA is not set
-# CONFIG_DRM_SAVAGE is not set
+# CONFIG_DRM is not set
 # CONFIG_VGASTATE is not set
 CONFIG_VIDEO_OUTPUT_CONTROL=y
 CONFIG_FB=y
 # CONFIG_FIRMWARE_EDID is not set
 # CONFIG_FB_DDC is not set
 # CONFIG_FB_BOOT_VESA_SUPPORT is not set
-CONFIG_FB_CFB_FILLRECT=y
-CONFIG_FB_CFB_COPYAREA=y
-CONFIG_FB_CFB_IMAGEBLIT=y
+# CONFIG_FB_CFB_FILLRECT is not set
+# CONFIG_FB_CFB_COPYAREA is not set
+# CONFIG_FB_CFB_IMAGEBLIT is not set
 # CONFIG_FB_CFB_REV_PIXELS_IN_BYTE is not set
 # CONFIG_FB_SYS_FILLRECT is not set
 # CONFIG_FB_SYS_COPYAREA is not set
@@ -1773,13 +1679,11 @@
 # CONFIG_FB_VGA16 is not set
 # CONFIG_FB_UVESA is not set
 # CONFIG_FB_VESA is not set
-CONFIG_FB_EFI=y
 # CONFIG_FB_N411 is not set
 # CONFIG_FB_HGA is not set
 # CONFIG_FB_S1D13XXX is not set
 # CONFIG_FB_NVIDIA is not set
 # CONFIG_FB_RIVA is not set
-# CONFIG_FB_I810 is not set
 # CONFIG_FB_LE80578 is not set
 # CONFIG_FB_MATROX is not set
 # CONFIG_FB_RADEON is not set
@@ -2241,30 +2145,12 @@
 #
 # CONFIG_STAGING is not set
 CONFIG_X86_PLATFORM_DEVICES=y
-# CONFIG_ACER_WMI is not set
-# CONFIG_ASUS_LAPTOP is not set
-# CONFIG_FUJITSU_LAPTOP is not set
-# CONFIG_TC1100_WMI is not set
-# CONFIG_MSI_LAPTOP is not set
-# CONFIG_PANASONIC_LAPTOP is not set
-# CONFIG_COMPAL_LAPTOP is not set
-# CONFIG_SONY_LAPTOP is not set
-# CONFIG_THINKPAD_ACPI is not set
-# CONFIG_INTEL_MENLOW is not set
-CONFIG_EEEPC_LAPTOP=y
-# CONFIG_ACPI_WMI is not set
-# CONFIG_ACPI_ASUS is not set
-# CONFIG_TOPSTAR_LAPTOP is not set
-# CONFIG_ACPI_TOSHIBA is not set
-# CONFIG_TOSHIBA_BT_RFKILL is not set
-# CONFIG_ACPI_CMPC is not set
 
 #
 # Firmware Drivers
 #
 # CONFIG_EDD is not set
 CONFIG_FIRMWARE_MEMMAP=y
-CONFIG_EFI_VARS=y
 # CONFIG_DELL_RBU is not set
 # CONFIG_DCDBAS is not set
 CONFIG_DMIID=y
@@ -2604,7 +2490,6 @@
 # CONFIG_SECURITY_SELINUX_POLICYDB_VERSION_MAX is not set
 # CONFIG_SECURITY_SMACK is not set
 # CONFIG_SECURITY_TOMOYO is not set
-# CONFIG_IMA is not set
 CONFIG_DEFAULT_SECURITY_SELINUX=y
 # CONFIG_DEFAULT_SECURITY_SMACK is not set
 # CONFIG_DEFAULT_SECURITY_TOMOYO is not set

Saturday, August 11, 2012

Man-in-the-Middle Port Protection

I'm wondering if this idea could work as a generic solution to the "OMG, the printer's been hacked' problem".

There are a very large classes of important legacy IP devices that can be easily compromised and either used as "zombies" in a bot-net, as relay devices or like Infrastructure controllers, be high-value targets for disruptive hackers.

What they share in common is:

their software can't be fixed or upgraded to current "hardness" levels and/or support current security protocols, like 802.1x, and
full replacement, or "fork-lift upgrade", is deemed unwarranted or infeasible.

The "Man-in-the Middle" (MitM) single ethernet port protection solution comes in two parts:

a dongle or wall-wart to connect in front of the printer, perhaps with Power-over-Ethernet (PoE) and
a central server/filter/firewall that the dongle(s) connect back to.

The dongle needs two ethernet connections and the software has to learn two things:

up-stream/down-stream traffic flows, which side is the printer, which is the network, and
the IP number + MAC address of the device.

It's job is to transparently take over the IP number and MAC address of the printer.

First it must gain itself a second IP number for its work and authenticate itself to the network if 802.1x is in use.

Having done that, it establishes a secure tunnel, via SSL or SSH, back to the central security device where the real security measures are taken.
The central security device can be implemented as a cloud of local devices, centrally managed.

The device can be mimiced either locally or centrally, depending on network configuration, latency and traffic volume concerns and devices available. Note that the protected device is behind a dongle, it will never be seen 'bare' on the network again, so conflicts will not arise. To the printer or device-under-control, no change in the network or environment will be discernible. To all other devices on the network, the printer or device-under-control will appear to have had a firmware upgrade, but otherwise be identical.

First option is for all network traffic destined for the printer has to be shunted back to the central secure device, modulo trivial ipfilter rules. This includes preventing any unauthorised outbound traffic, or even directing all outbound packets for analysis through the central security device for Intrusion Detection analysis.

Once the traffic is back at the central secure device, it can be properly inspected and cleaned, then turned around on the same SSL/SSH tunnel.

Second option, the central secure device assumes both IP number and MAC address of the device-under-control by advertising it's IP and the same MAC addrs. It can also provide 802.1x-client facilities.

The central secure device then forwards only valid traffic to the dongle at the printer over SSL/SSH, and response traffic is tunnelled back and inspected over the same path.

The difference is where the impersonated IP number/MAC address now appears in the network:

either exactly where it always has been, or
in a central location.

A 2-port version of the Raspberry Pi would be able to do this, plus some tinkering in a firewall appliance.

If very short UTP leads are used on the MitM dongle, it will be very difficult to remove from the printer or device under control.

The Use Case is simple:

get new secure server
install 802.1x certificates for printers/devices-under-control in the secure server
install 802.1x certs in dongles, setup in DHCP, don't have to be static IP numbers.
egister SSH or SSL keys of dongle to secure server
check locally the dongle works and correctly controls a test device
go to printer, install dongle, check it works. Requires normal traffic and a scan for vulnerabilities.
there might be a some outage as the double shuffle happens, the MAC address may now appears elsewhere in the network
not sure how real-time swap might affects existing IP connections. If the disruption in traffic flow is under the TCP disconnect window, it won't be noticed. It's not an option to leave existing connections uninspected, they could Botnet control channel.

Do these things already exist? I don't know..
I can't believe something this simple hasn't already been built.

There are 2½ variants that might also be interesting.
For variant #2, I'm not sure what can already be done in switch software.

1. Plug 2-port MitM dongle directly into switch, capture all packets at the switch, not at the printer.

1A. Plug a multi-port MitM dongle into switch so that it acts like a switch itself, having one up-stream link via the switch, and provides multiple ports for controlled devices to be connected to. Needs the host switch to allow multiple MAC's per port.

Problem with remote dongle:

Printer/device-under-control can be relocated (physically or connection) and lose protection/filtering.

2. High-end Switches have "Port mirroring" software, can that be used or modified for the MitM packet redirection?

Port-mirroring sends a copies of ingress and egress packets to another port, even on another switch.
The remote Filter/firewall (FF) needs two ports, A & B, one ingress, one egress..

Network egress traffic is redirected to Port-A of the FF, even on another VLAN.
Network Ingress traffic is instead received from Port-A of the FF.
Port-B traffic of the FF is sent back as egress traffic of the controlled port, and
ingress traffic of the controlled port is sent to Port-B traffic of the FF.

This achieves logically what the physical dongle+physical patch leads achieved, passing all traffic via an MitM Filter/Firewall.

2A. Multi-port filter/firewall.

Can switch software multiplex multiple MAC addresses onto two ports, serving multiple devices under control with the same 2-port hardware, or
does it need a pair of ports on the filter/firewall for each device under control, or
1 upstream link to capture all redirected traffic, and 1 ethernet port for each device under control.

Sunday, June 01, 2014

Tuesday, May 27, 2014

Sunday, May 04, 2014

Saturday, May 03, 2014

Tuesday, April 22, 2014

Monday, April 21, 2014

Sunday, April 20, 2014

Sunday, March 23, 2014

Thursday, March 20, 2014

Sunday, September 15, 2013

Monday, February 04, 2013

Thursday, January 17, 2013

Wednesday, January 02, 2013

Monday, December 17, 2012

Sunday, November 18, 2012

Saturday, August 11, 2012

Index

Blog Archive

About Me

Labels

MathJax