Optical Disks as Dense near-line storage?

A follow-up to an earlier post [search for 'Optical']:

Could Optical Disks be a viable near-line Datastore?
Use a robot arm to pick and load individual disks from 'tubes' into multiple drives.
Something the size of a single filing cabinet drawer would be both cheap and easily contain a few thousand disks. That's gotta be interesting!

Short answer, no...

A 3.5" drive has a form-factor of:  4 in x 5.75 in x 1in. Cubic capacity: 23 in³
A 'tube' of 100 optical disks: 5.5in x 5.5in x 6.5in Cubic capacity: 160-200 in³ [footprint or packed]

A 'tube' of 100, minus all the supporting infrastructure to select a disk and read it, is 7-9 times the volume of a 3.5in hard disk drive, or each Optical Disk must contain at least 7-9% of a HDD to be competitive.

To replace a 1Tb HDD, optical disks must be at least 7% of 1,000Gb, or 70Gb. Larger than even Blu-ray and 15-20 times larger than Single layer DVD's (4.7Gb).

Current per Gb price of 3.5" HDD's is around $0.05-$0.10/Gb,  squeezing 4.7Gb DVD's on price as well.

2Tb drives are common, 3Tb are becoming available now (2011). Plus it gets worse.

There are estimate is of maximum possible 3.5" HDD size of 20-40Tb.
To be competitive, Optical disks would need to get up around 1Tb in size and cost under $1.

Around 2005, when 20-40Gb drives reigned, there was a time when 4.7Gb DVD's were both the densest and cheapest storage available. Kryders' Law, a doubling of HDD capacity every 1-2 years, has seen the end of that.

Journaled changes: One solution to RAID-1 on Flash Memory

As I've posited before, simple-minded mirroring (RAID-1) of Flash Memory devices is not only a poor implementation, but worst-case.

My reasoning is: Flash wears-out and putting identical loads on identical devices will result in maximum wear-rate of all bits, which is bad but not catastrophic. It also creates a potential for simultaneous failures where a common weakness fails in two devices at the one time.

The solution is to not put the same write load on the two  devices, but still have two exact copies.
This problem would be an especial concern for PCI-SSD devices internal to a system. The devices can't normally can't be hot-plugged, though there are hot-plug standards for PCI devices (e.g. Thunderbolt and ExpressCard), they are not usually options for servers and may be limited performance.

One solution, I'm not sure if it's optimal or not, but it is 'sufficient', is to write blocks as normal to the primary device and maintain the secondary device as snapshot + (compressed) journal entries. When the journal space hits a high-water mark the snapshot is made an exact copy (e.g. bring the snapshot up-to-date when a timer expires (hourly, 6-hourly, daily, ...) or when the momentum of changes will fill the journal to 95% before the snapshot could be updated).

If the journal fills, the mirror is invalidated and either changes must be halted or the devices go into unprotected operation. Both not desirable operational outcomes. A temporary, though unprotected, work-around is to write the on-going journal either to the primary device or into memory.

Flash Memory: will filesystems become the CPU bottleneck?

Flash memory with 50+k IO/sec may be too fast for Operating Systems (like Linux) with file-system operations consuming more CPU, even saturating it. They are on the way to becoming the system rate-limiting factor, otherwise known as a bottleneck.

What you can get away with at 20-100 IO/sec, i.e. consumes 1-2% of CPU, will be a CPU hog at 50k-500k IO/sec, a 5,000-50,000 times speed up.

The effect is the reverse of the way Amdahl speed-up is explained.

Amdahl throughput scaling is usually explained like this:

If your workload has 2 parts (A is single-threaded, B can be parallelised), when you decrease the time-taken for  'B' by adding parallel compute-units, the workload becomes dominated by the single-threaded part, 'A'. If you half the time it takes to run 'B', it doesn't halve the total run time. If 'A' and 'B' parts take equal time (4 units each, total 8), then a 2-times speed up of 'B' (4 units to 2) results in a 25% reduction in run-time (8 units to 6). Speeding 'B' up 4-times is a 37% reduction (8 to 5).
This creates a limit to the speed-up possible: If 'B' reduces to 0 units, it still takes the same time to run all the single-threaded parts, 'A'. (4 units here)

A corollary of this: the rate-of-improvement for each doubling of cost nears zero, if not well chosen.

The filesystem bottleneck is the reverse of this:

If your workload has an in-memory part (X) and wait-for-I/O part (W) both of which consume CPU, if you reduce the I/O wait to zero without reducing the CPU overhead of 'W', then the proportion of useful work done in 'X' decreases. In the limit, the system throughput is constrained by CPU expended on I/O overhead in 'W'.

The faster random I/O of Flash Memory will reduce application execution time, but at the expense of increasing % system CPU time. For a single process, the proportion and total CPU-effort of I/O overhead remains the same. For the whole system, more useful work is being done (it's noticeably "faster"), but because the CPU didn't get faster too, it needs to spend a lot more time on the FileSystem.

Jim Gary observed that:

• CPU's are now mainly idle, i.e. waiting on RAM or I/O.
  Level-1 cache is roughly the same speed as the CPU, everything else is much slower and must be waited for.
• The time taken to scan a 20Tb disk using random I/O will be measured in days whilst  a sequential scan ("streaming") will take hours.

Reading a "Linux Storage and Filesystem Workshop" (LSF) confrence report, I was struck by comments that:

• linux file systems can consume large amount of CPU doing their work, not just fsck, but handling directories, file metadata, free block chains, inode block chains, block and file checksums, ...

There's a very simple demonstration of this: optical disk (CD-ROM or DVD) performance.

• A block-by-block copy (dd) of a CD-ROM at "32x", or approx 3Mb/sec, will copy a full 650Mb in 3-4 minutes. Wikipedia states a 4.7Gb DVD takes 7 minutes (5.5Mb/sec) at "4x".
• Mounting a CD or DVD then doing a file-by-file copy takes 5-10 times as long.
• Installing or upgrading system software from the same CD/DVD is usually measured in hours.

The fastest way to upgrade software from CD/DVD is to copy an image with dd to hard-disk, then mount that image. The difference is the random I/O (seek) performance of the underlying media, not the FileSystem. [Haven't tested times or speedup with a fast Flash drive.]

This performance limit may have been something that the original Plan 9 writers knew and understood:

• P9 didn't 'format' media for a filesystem: initialised a little and just started writing blocks.
• didn't have fsck on client machines, only the fileserver.
• the fileserver wrote to three levels of storage: RAM, disk, Optical disk.
  RAM and disk were treated as cache, not permanent storage.
  Files were pushed to Optical disk daily, creating a daily snapshot of the filesystem at the time. Like Apple's TimeMachine, files that hadn't changed were 'hard-linked' to the new directory tree.
• The fileserver had operator activities like backup and restore. The design had no super-user with absolute access rights, so avoided many of the usual admin-related security issues.
• Invented 'overlay mounts', managed at user not kernel level, to combine the disparate file-services available and allow users to define their own semantics.

Filesystems have never, until now, focussed on CPU performance, rather the opposite, they've traded CPU and RAM to reduce I/O latency, historically improving system throughput, sometimes by orders-of-magnitude.
Early examples were O/S buffers/caching (e.g. Unix) and the  'elevator algorithmn' to optimally reorder writes to match disk characteristics.

 This 'burn the CPU' trade-off shows up with fsck as well. An older LSF piece suggested that fsck runs slowly because it doesn't do a single pass of the disk, effectively forced into near worst-case unoptimised random I/O.

On my little Mac Mini with a 300Gb disk, there's 225Gb used. Almost all of which, especially the system files, is unchanging. Most of the writing to disk is "append mode" - music, email, downloads - either blocks-to-a-file or file-to-directory. With transactional Databases, it's a different story.

The filesystem treats the whole disk as if every byte could be changed in the next second - and I pay a penalty for that in complexity and CPU cycles. Seeing my little Mac or an older Linux desktop do a filesystem check after a power fail is disheartening...

I suggest future O/S's will have to contend with:

• Flash or SCM with close to RAM performance 'near' the CPU(s)  (on the PCI bus, no SCSI controller)
• near-infinite disk ("disk is tape", Jim Gray) that you'll only want to access as "seek and stream". It will also take "near infinite" time to scan with random I/O. [another Jim Gray observation]

And what are the new rules for filesystems in this environment?:

• two sorts of filesystems that need to interwork:
• read/write that needs fsck to properly recover after a failure and
• append-only that doesn't need checking once "imaged", like ISO 9660 on optical disks.
• ''Flash" file-system organised to minimise CPU and RAM use. High performance/low CPU use will become as important as managing "wear" for very fast PCI Flash drives.
• 'hard disk' filesystem with on-the-fly append/change of media and 'clone disk' rather than 'repair f/sys'.
• O/S must seamlessly/transparently:
  1. present a single file-tree view of the two f/sys
  2. like Virtual Memory, safely and silently migrate data/files from fast to slow storage.

I saw a quote from Ric Wheeler (EMC) from LSF-07 [my formatting]:

 the basic contract that storage systems make with the user
 is to guarantee that:

•  the complete set of data will be stored,
•  bytes are correct and
•  in order, and
•  raw capacity is utilized as completely as possible.

I disagree nowdays with his maximal space-utilisation clause for disk. When 2Tb costs $150 (7.5c/Gb) you can afford to waste a little here and there to optimise other factors.
With Flash Memory at $2-$5/Gb, you don't want to go wasting much of that space.

Jim Gray (again!) early on formulated "the 5-minute rule" which needs rethinking, especially with cheap Flash Memory redefining the underlying Engineering factors/ratios. These sorts of explicit engineering trade-off calculations have to be done for the current disruptive changes in technology.

• Gray, J., Putzolu, G.R. 1987. The 5-minute rule for trading memory for disk accesses and the 10-byte rule for trading memory for CPU time. SIGMOD Record 16(3): 395-398.
• Gray, J., Graefe, G. 1997. The five-minute rule ten years later, and other computer storage rules of thumb. SIGMOD Record 26(4): 63-68.

I think Wheeler's Storage Contract also needs to say something about 'preserving the data written', i.e. the durability and dependability of the storage system.
For how long? what what latency? How to express that? I don't know...

There is  also a matter of "storage precision", already catered for with CD's and CD-ROM, Wikipedia states:

The difference between sector size and data content are the header information and the error-correcting codes, that are big for data (high precision required), small for VCD (standard for video) and none for audio. Note that all of these, including audio, still benefit from a lower layer of error correction at a sub-sector level.

Again, I don't know how to express this, implement it nor a good user-interface. What is very clear to me is:

• Not all data needs to come back bit-perfect, though it is always nice when it does.
• Some data we would rather not have, in whole or part, than come back corrupted.
• There are many data-dependent ways to achieve Good Enough replay when that's acceptable.

First, the aspects of Durability and Precision need to be defined and refined, then a common File-system interface created and finally, like Virtual Memory, automated and executed without thought or human interaction.

This piece describes FileSystems, not Tabular Databases nor other types of Datastore.
The same disruptive technology problems need to be addressed within these realms.
Of course, it'd be nicer/easier if other Datastores were able to efficiently map to a common interface or representation shared with FileSystems and all the work/decisions happened in Just One Place.

Will that happen in my lifetime? Hmmmm....

Building a RAID disk array circa 1988

In "A Case for Redundant Arrays of Inexpensive Disks (RAID)" [1988], Patterson et al of University of California Berkeley started a revolution in Disk Storage still going today. Within 3 years, IBM had released the last of their monolithic disk drives, the 3390 Model K, with the line being discontinued and replaced with IBM's own Disk Array.

The 1988 paper has a number of tables where it compares Cost/Capacity, Cost/Performance and Reliability/Performance of IBM Large Drives, large SCSI drives and 3½in SCSI drives.

The prices ($$/MB) cited for the IBM 3380 drives are hard to reconcile with published prices:
 press releases in Computerworld  and IBM Archives for 3380 disks (7.5Gb, 14" platter, 6.5kW) and their controllers suggest $63+/Mb for 'SLED' (Single Large Expensive Disk) rather than the
"$18-10" cited in the Patterson paper.

The prices for the 600MB Fujitsu M2316A ("super eagle") [$20-$17] and 100Mb Conner Peripherals CP-3100 [$10-$7] are in-line with historical prices found on the web.

The last table in the 1988 paper lists projected prices for different proposed RAID configurations:

• $11-$8 for 100 * CP-3100 [10,000MB] and
• $11-$8 for 10 * CP-3100 [1,000MB]

There are no design details given.

1994, Chen et al in "RAID: High-Performance,Reliable Secondary Storage" use two widely sold commercial system as case studies:

• NCR 6298 and
• StorageTek's Iceberg 9200 Disk Array
  1996 Press Release: IBM to exclusively resell Iceberg as RAMAC via Archive.org
  1996 RAID product page on Archive.org.

The (low-end) NCR device was more what we'd call a 'hardware RAID controller' now, ranging from 5 to 25 disks. Pricing $22-102,000. It provided a SCSI interface and didn't buffer. A system diagram was included in the paper.

The StorageTek's Iceberg was high-end device meant for connection to IBM mainframes. Advertised as starting at 100GB (32 drives) for $1.3M, up to 400Gb for $3.6M, It provided multiple (4-16) IBM ESCON 'channels'.

For the NCR, from InfoWorld 1 Oct 1990, p 19 in Google Books

• min config: 5 * 3½in drives, 420MB each.
• $22,000 for 1.05Gb storage
• Add 20*420Mb to 8.4Gb list $102,000. March 1991.
• $4,000/drive + $2,000 controller.
• NCR-designed controller chip + SCSI chip
• 4 RAID implementations: RAID 0,1,3,5.

The StorTek Iceberg was released in late 1992 with projected shipments of 1,000 units in 1993.  It was aimed at replacing IBM 'DASD' (Direct Access Storage Device): exactly the comparison made in the 1988 RAID paper.

The IBM-compatible DASD, which resulted from an investment of $145 million and is technically styled the 9200 disk array subsystem, is priced at $1.3 million for a minimum configuration with 64MB of cache and 100GB of storage capacity provided by 32 Hewlett-Packard 5.25-inch drives.

A maximum configuration, with 512MB of cache and 400GB of storage capacity from 128 disks, will run more than $3.6 million. Those capacity figures include data compression and compaction, which can as much as triple the storage level beyond the actual physical capacity of the subsystem.

Elsewhere in the article more 'flexible pricing' (20-25% discount) is suggested:

with most of the units falling into the 100- to 200GB capacity range, averaging slightly in excess of $1 million apiece.

Whilst no technical reference is easily accessible on-line, more technical details are mentioned in the press release on the 1994 upgrade, the 9220. Chen et al [1994] claim "100,000 lines of code" were written.

More clues come from an feature, "Make Room for DASD" by Kathleen Melymuka  (p62) of CIO magazine, 1st June 1992 [accessed via Google Books, no direct link]:

• 5¼in Hewlett-Packard drives were used. [model number & size not stated]
• The "100Gb" may include compaction and compression. [300% claimed later]
• (32 drives) "arranged in dual redundancy array of 16 disks each (15+1 spare)
• RAID-6 ?
• "from the cache, 14 pathways transfer data to and from the disk arrays, and each path can sustain a 5Mbps transfer rate"

The Chen et al paper (pg 175 of CACM,, Vol 26, No 2) gives this information on the Iceberg/9200:

• it "implements an extended RAID level-5 and level-6 disk array"
  • 16 disks per 'array', 13 usable, 2 Parity (P+Q), 1 hot spare
  •  "data, parity and Reed-Solomon coding are striped across the 15 active drives of an array"
• Maximum of 2 Penguin 'controllers' per unit.
• Each controller is an 8-way processor, handling up to 4 'arrays' each, or 150Gb (raw).
• Implying 2.3-2.5Gb per drive
• The C3010, seemingly the largest HP disk in 1992, was 2.47Gb unformatted and 2Gb formatted (512by sectors), [notionally 595by unformatted sectors]
• The C3010 specs included:
• MTBF: 300,000 hrs
• Unrecoverable Error Rate (UER): 1 in 10^14 bits transferred
• 11.5 msec avg seek, (5.5msec rotational latency, 5400RPM)
• 256Kb cache, 1:1 sector interleave, 1,7 RLL encoding, Reed-Solomon ECC.
• max 43W 'fast-wide' option, 36W running.
• runs up to 8 'channel programs' and independently transfer on 4 channels (to mainframe).
• manages a 64-512Mb battery-backed cache (shared or per controller not stated)
• implements on-the-fly compression, cites maximum doubling capacity.
• and dynamic mapping necessary CKD (count, key, data) for variable-sized IBM blocks onto the fixed blocks internally.
• a extra (local?) 8Mb of non-volatile memory is used to store these tables/maps.
• Uses a "Log-Structured File System" so blocks are not written back to the same place on the disk.
• Not stated if the SCSI buses are one-per-arry or 'orthogonal'. i.e. Redundancy groups are made up from one disk per 'array'.

Elsewhere, Katz, one of the authors, uses a diagram of a generic RAID system not subject to any "Single Point of Failure":

• with dual-controllers and dual channel interfaces.
• Controllers cross-connected to each interface.
• dual-ported disks connected to both controllers.
• This halves the number of unique drives in a system, or doubles the number of SCSI buses/HBA's, but copes with the loss of a controller.
• Implying any battery-backed cache (not in diagram) would need to be shared between controllers.

From this, a reasonable guess at aspects of the design is:

• HP C3010 drives were used, 2Gb formatted. [Unable to find list prices on-line]
• These drives were SCSI-2 (up to 16 devices per bus)
• available as single-ended (5MB/sec) or 'fast' differential (10MB/sec) or 'fast-wide' (16-bit, 20MB/sec). At least 'fast' differential, probably 'fast-wide'.
• "14 pathways" could mean 14 SCSI buses, one per line of disks, but it doesn't match with the claimed 16 disks per array.
• 16 SCSI buses with 16 HBA's per controller matches the design.
• Allows the claimed 4 arrays of 16 drives per controller (64) and 128 max.
• SCSI-2 'fast-wide' allows 16 devices total on a bus, including host initiators. This implies that either more than 16 SCSI
• 5Mbps transfer rate probably means synchronous SCSI-1 rates of 5MB/sec or asynchronous SCSI-2 'fast-wide'.
• It cannot mean the 33.5-42Mbps burst rate of the C3010.
• The C3010 achieved transfer rates of 2.5MB/sec asynchronously in 'fast' mode, or 5MB/sec in 'fast-wide' mode.
• Only the 'fast-wide' SCSI-2 option supported dual-porting.
• The C3010 technical reference states that both powered-on and powered-off disks could be added/removed to/from a SCSI-2 bus without causing a 'glitch'. Hot swapping (failed) drives should've been possible.
• RAID-5/6 groups of 15 with 2 parity/check disk overhead, 26Gb usable per array, max 208Gb.
•  RAID redundancy groups are implied to be per (16-disk) 'array' plus one hot-spare .
• But 'orthogonal' wiring of redundancy groups was probably used, so how many SCSI buses were needed per controller, in both 1 and 2-Controller configurations?
• No two drives in a redundancy group should be connected via the same SCSI HBA, SCSI bus, power-group or cooling-group.
  This allows live hardware maintenance or single failures.
• How were the SCSI buses organised?
  With only 14 devices total per SCSI-2 bus, a max of 7 disks per shared controller was possible.
  The only possibly configurations that allow in-place upgrades are: 4 or 6 drives per bus.
  The 4-drives/bus resolves to "each drive in an array on a separate bus".
• For manufacturing reasons, components need standard configurations.
  It's reasonable to assume that all disk arrays would be wired identically, internally and with common mass terminations on either side, even to the extent of different connectors (male/female) per side.
  This allows simple assembly and expansion, and trivially correct installation of SCSI terminators on a 1-Controller system.
  Only separate-bus-per-drive-in-array (max 4-drives/bus), meets these constraints.
  SCSI required a 'terminator' at each end of the bus. Typically one end was the host initiator. For dual-host buses, one at each host HBA works.
• Max 4-drives per bus results in 16 SCSI buses per Controller (64-disks per side).
  'fast-wide' SCSI-2 must have been used to support dual-porting.
  The 16 SCSI buses, one per slot in the disk arrays, would've continued across all arrays in a fully populated system.
  In a minimum system, 32 drives, would've been only 2 disks per SCSI bus.
• 1 or 2 controllers with a shared 64M-512M cache and 8Mb for dynamic mapping.

This would be a high-performance and highly reliable design with a believable $1-2M price for 64 drives (200Gb notional, 150Gb raw):

• 1 Controllers
• 128Mb RAM
• 8 ESCON channels
• 16 SCSI controllers
• 64 * 2Gb drives as 4*16 arrays, 60 drives active, 52 drive-equivalents after RAID-6 parity.
• cabinets, packaging, fans and power-supplies

From the two price-points, can we tease out a little more of the costs [no allowance for ESCON channel cards]:

• 1 Controller + 32 disks + 64M cache = $1.3M
• 2 Controllers + 128 disks + 512M cache = $3.6M

As a first approximation, assume that 512M RAM costs half as much as 2 Controllers for a 'balanced' system. Giving us a solvable set of simultaneous equations:

• 1.0625 Controllers + 32 disks  = $1.3M
• 2.5 Controllers + 128 disks = $3.6M

roughly:

• $900,000 / Controller [probably $50,000 high]
• $70,000 / 64M cache [probably $50,000 low]
• $330,000 / 32 disks ($10k/drive, or $5/MB)

High-end multi-processor VAX system pricing at the time is in-line with this $900k estimate, but more likely an OEM'd RISC processor (MIPS or SPARC?) was used.
This was a specialist, low-volume device: expected 1st year sales volume was ~1,000.
In 1994, they'd reported sales of 775 units when the upgrade (9220) was released.

Another contemporary computer press article cites the StorageTek Array costing $10/Mb compared to $15/MB for IBM DASD. 100Gb @ $10/Mb  is $1M, so congruent with the claims above.

How do the real-world products in 1992 compare to the 1988 RAID estimates of Patterson et al?

• StorageTek Iceberg: $10/Mb vs $11-$8 projected.
• This was achieved using 2Gb 5¼in drives not the 100Mb 3½in drives modelled
• HP sold a 1Gb 3½in SCSI-2 drive (C2247) in 1992. This may have formed the basis of the upgrade 9220 ~two years later.
• Using the actual, not notional, supplied capacity (243Gb) the Iceberg cost $15/Mb.
• The $15/Mb for IBM DASD compares well to the $18-$10 cited in 1988.
• But IBM, in those intervening 5 years, had halved the per-Mb price of their drives once or twice. The 1988 "list price" from the archives of ~$60/Mb are reasonable.
• In late 1992, 122Mb Conner CP-30104 were advertised for $400, or $3.25/Mb.
  These were IDE drives, though a 120Mb version of the SCSI CP-3100 was sold, price unknown.

The 8.4Gb 25-drive NCR  6298 gave $12.15/Mb, again close to the target zone.
From the Dahlin list, 'street prices' for 420Mb drives at the time, were $1600 for Seagate ST-1480A and $1300 for 425Mb Quantum or $3.05-$3.75/Mb.

The price can't be directly compared to either IBM DASD or StorageTek's Iceberg, because the NCR 6298 only provided a SCSI interface, not an IBM 'channel' interface.

The raw storage costs of the StorageTek Iceberg and NCR are roughly 2.5:1.
Not unexpected due to the extra complexity, size and functionality of the Iceberg.

High Density Racking for 2.5" disks

Datacentre space is expensive: one 2010 artice puts construction at $1200/sq ft and rental at $600/pa/sq ft for an approx design heat load of 10-50W/sq ft. Google is reputed to be spending $3,000/sq ft building datacentre with many times this heat load.

There are 3 different measures of area:

• "gross footprint". The room plus all ancillary equipment and spaces.
• "room". The total area of the room. Each rack, with aisles & work-space uses 16-20 sq ft.
• "equipment footprint". The floor area directly under computing equipment. 24"x40", ~7sq ft.

Presumably the $600/pa rental cost is for "equipment footprint".

The 2½ inch drive form-factor is (100.5 mm x 69.85 mm x 7-15 mm).
Enterprise drives are typically 12.5 or 15mm thick. Vertically stacked, 20-22 removable 2½ inch drives can be fitted across a rack, taking a 2RU space, with around 15mm of 89mm unused.

2½ inch drive don't fit well in standard 19" server racks (17" wide, by 900-1000mm deep, 1RU = 1.75" tall), especially if you want equal access (eg. from the front) to all drives without disturbing any drives. Communications racks are typically 600mm deep, but not used for equipment in datacentres.

With cabling, electronics and power distribution, a depth of 150mm (6") should be sufficient to house 2½ inch drives. Power supply units take additional space.

Usable space inside a server rack, 17" wide and 1000mm deep, would leave 850mm wasted.
Mounting front and back, would still leave 700mm wasted, but create significant heat removal problems, especially in a "hot aisle" facility.

The problems reduces to physical arrangements that maximise exposed area (long and thin rectangles vs the close-to-square 19" Rack, if dual sided, with a chimney) or maximise surface area and minimise floor space - a circle or cylinder.

The "long-thin rectangle" arrangement was popular in mainframe days, often as an "X" or a central-spine with many "wings". It assumes that no other equipment will be sited within the working clearances needed to open doors and remove equipment.

A cylinder, or pipe, must contain a central chimney to remove waste heat. There is also a requirement to plan cabling for power and data. Power supplies can be in the plinth as the central cooling void can't be blocked mid-height and extraction fan(s) need to be mounted at the top.

For a 17" diameter pipe, 70-72 disks can be mounted around the circumference allowing 75 mm height per row,  20-24 rows high allowing for a plinth and normal heights. This leaves a central void of around 7" to handle the ~8kW of power of ~1550 drives.

A 19" pipe would allow 75-80 disks per row and a 9" central void to handle ~9.5kW of ~2000 drives.
Fully populated unit weight would be 350-450kg.

Perhaps one in 8 disks could be removed allowing a cable tray, a not unreasonable loss of space.

These "pipes" could be sited in normal racks either at the end of row requiring one free rack-space beside them, or in a row, taking 3 rack-spaces.

As a series of freestanding units, they could be mounted in a hexagonal pattern (the closest-packing arrangement for circles) with minimum OH&S clearances around them, which may be 600-750mm.

This provides 4-5 times the density of drives over the current usual 22 shelves of 24 drives (480) per rack, with better heat extraction.  At $4-5,000/pa rental per rack-space (or ~$10/drive), it's a useful saving.

With current drive sizes of 600-1000Gb/drive, most organisations would get by with one unit of 1-2Pb.

Update: A semi-circular variant 40inx40in for installation in 3 Rack-widths might work as well. Requires a door to create the chimney space/central void - and it could vent directly into a "hot aisle".

---

Papers:
2008: "Cost Model: Dollars per kW plus Dollars per Square Foot of Computer Floor"

2007: "Data center TCO; a comparison of high-density and low-density spaces"

2006: "Total Cost of Ownership Analysis for Data Center Projects"

2006: "Dollars per kW plus Dollars per Square Foot Are a Better Data Center Cost Model than Dollars per Square Foot Alone"

Questions about SSD / Flash Memory

1. Seagate, in 2010, quote their SSD UER specs as:
   Nonrecoverable read errors, max: 1 LBA per 10^16 bits read
   where a Logical Block Address (LBA) is 512 bytes. Usually called a 'sector'.
   
   But we know that Flash memory is organised as 64Kb blocks (min read/write unit).
   Are Seagate saying that errors will be random localised cells, not "whole block at a time"?
   Of course, the memory controller does Error Correction to pick up the odd dropped bits.
2. Current RAID schemes are antagonistic to Flash Memory:
   The essential problem with NAND Flash (EEPROM) Memory is that it suffers "wear" - after a number of writes and erasures, individual cells (with MLC, cell =/= bit) are no longer programmable. A secondary problem is "Data Retention". With the device powered down, Seagate quote "Data Retention" of 1 year.
   
   Because Flash Memory wears with writes, batches of components will likely have very similar wear characteristics and if multiple SSD's are mirrored/RAIDed in a system they will most likely be from the same batch, evenly spread RAID writes (RAID-5 writes two physical blocks per logical block) will cause a set of SSD's to suffer correlated wear failures. This is not unlike the management of piston engines in multi-engined aircraft: avoid needing to replace more than one at a time. Faults, Failures and Repair/Install Errors often show up in the first trip. Replacing all engines together maximises the risk of total engine failure.
   
   Not only is this "not ideal", it is exactly worst case for current RAID.
   A new Data Protection Scheme is required for SSD's.
1. Update 23-Dec-2011. Jim Handy in "The SSD Guy" blog [Nov-17, 2011] discusses SSD's and RAID volumes:
   

   So far this all sounds good, but in a RAID configuration this can cause trouble.  Here’s why.
   
   RAID is based upon the notion that HDDs fail randomly.  When an HDD fails, a technician replaces the failed drive and issues a rebuild command.  It is enormously unlikely that another disk will fail during a rebuild.  If SSDs replace the HDDs in this system, and if the SSDs all come from the same vendor and from the same manufacturing lot, and if they are all exposed to similar workloads, then they can all be expected to fail at around the same time.
   
   This implies that a RAID that has suffered an SSD failure is very likely to see another failure during a rebuild – a scenario that causes the entire RAID to collapse.

3. What are the drivers for the on-going reduction in prices of Flash Memory?
   Volume? Design? Fabrication method (line width, "high-K", ...)? Chip Density?
   
   The price of SSD's has been roughly halving every 12-18 months for near on a decade, but why?
   Understanding the "why" is necessary to be forewarned of any change to the pattern.
4. How differently are DRAM and EEPROM fabricated?
   Why is there about a 5-fold price difference between them?
   

   Prices (Kingston from same store, http://msy.com.au, November 2011):
   DDR3 6Gb 1333Mhz $41 $7/Gb
   SSD 64Gb  $103    $1.50/Gb
   SSD 128Gb  $187 $1.25/Gb
   

   
   It would be nice to know if there was a structural difference or not for designing "balanced systems", or integrating Flash Memory directly into the memory hierarchy, not as a pretend block device.
5. Main CPU's and O/S's can outperform any embedded hardware controller.
   Why do the PCI SSD's not just present a "big blocks of memory" interface, but insist on running their own controllers?
6. Hybrid SSD-HDD RAID.
   For Linux particularly, is it possible to create multiple partitions per HDD in  a set, then use one HDD-partition to mirror a complete SSD.  The remaining partitions can be setup as RAID volumes in the normal way.
   The read/write characteristics of SSD and HDD are complementary: SSD is blindly fast for random IO/sec, while HDD's currently stream reads/writes at higher sustained writes.
   Specifically, given 1*128Gb SSD and 3*1Tb HDD, create a 128Gb partition on all HDD's. Then mirror (RAID-1), the SSD and one or more of the HDD's (RAID-1 isn't restricted to two copies, but can have as many replicas as desired to increase IO performance or resilience/Data Integrity). Remaining 128Gb HDD partitions can be stripped, mirrored or RAIDed amongst themselves or to other drives. The remaining HDD space can be partitioned and RAIDed to suit demand.
   
   Does it make sense, both performance- and reliability-wise, to mirror SSD and HDD?
   Does the combination yield the best, or worse, of both worlds?
   Is the cost/complexity and extra maintenance worth it?

Enterprise Drives: Moving to 2.5" form factor

Update [23-Dec-2011]: IDC, in 2009, discussed in a HP report, the migration to 2.5 inch drives in Enterprise Storage. Started in 2004, projected to be complete in 2011.

Elsewhere, a summary of IDC's 2009 Worldwide HDD shipments and revenue:

• The transition from 3.5in. to 2.5in. performance-optimized form factor HDDs will be complete by 2012.
• Growing interest in new storage delivery models such as storage as a service, or storage in the cloud is likely to put greater storage capacity growth demands on Internet datacenters.
• The price per gigabyte of performance-optimized HDD storage will continue to decline at a rate of approximately 25% to 30% per year.

---

Types of 'Flash Memory'

"Flash Memory", or EEPROM (Electrically Erasable Programmable Read Only Memory), is at the heart of much of the "silicon revolution" of the last 10 years.

How is it packaged and made available to consumers or system builders/designers?

Mobile appliances are redefining Communications and The Internet, precisely because of the low-power, high-capacity and longevity - and affordable price -  of modern Flash Memory.

There are many different Flash Memory configurations: NAND, NOR, SLC, MLC, ...
This piece isn't about those details/differences but the in how they are packaged and organised.
What technology does what/has what characteristics is out there on the InterWebs.

Most Flash Memory chips are assembled into different packaging for different uses at different price points. Prices are "Retail Price (tax paid)" from a random survey of Internet sites:

• Many appliances use direct-soldered Flash Memory are their primary or sole Persistent Datastore.
  The genesis of this was upgradeable BIOS firmware. Late 1990's?
  Per-Gb pricing not published: approx. derivable from model price differences.
• Commodity 'cards' used in cameras, phones and more: SD-card and friends.
  Mini-CD and Micro-SD cards are special cases and attract a price premium.
  Some 'high-performance' variants for cameras.
  A$2-3/Gb
• USB 'flash' or 'thumb drives':
  A$2-3/Gb.
• High-end camera memory cards: Compact Flash (CF). The oldest mass-use format?
  IDE/ATA compatible interface. Disk drive replacement for embedded systems.
  Fastest cards are 100MB/sec (0.8Gbps). Max is UDMA ATA, 133MB/sec.
  Unpublished Bit Error Rate, Write Endurance, MTBF, Power Cycles, IO/sec.
  A$5-$30/Gb
• SATA 2.5" SSD (Solid State Drives). Mainly 3Gbps and some 6Gbps interfaces.
  MTBF: 1-2M hours,
  Service Life: 3-5 years at 25% capacity written/day.
  IO/sec: 5,000 - 50,000 IO/sec [max seen: 85k IO/sec]
  BER: "1 sector per 10^15-16 bits read"
  sustained read/write speed: 70-400MB/sec . (read often slowest)
  
  Power: 150-2000W active, 75-500mW idle
  32Gb - 256GB @ A$1.50-$2.50/Gb.
• SATA 1.8" SSD. Internal configuration of some 2.5" SSD's.
  Not yet widely available.
• SAS (Serial Attached SCSI) 2.5" drives.
  not researched. high-performance, premium pricing.
• PCI "SSD". PCI card presenting as a Disk Device.
  Multiple vendors, usual prices ~A$3-4/Gb. Sizes 256Gb - 1Tb.
  "Fusion-io" specs quoted by Dell. Est A$20-25/Gb. [vs ~$5/Gb direct]
  

  640GB (Duo)
  NAND Type: MLC (Multi Level Cell)
  Read Bandwidth (64kB): 1.0 GB/s
  Write Bandwidth (64kB): 1.5 GB/s
  Read IOPS (512 Byte): 196,000
  Write IOPS (512 Byte): 285,000
  Mixed IOPS (75/25 r/w): 138,000
  Read Latency (512 Byte): 29 μs
  Write Latency (512 Byte): 2 μs
  Bus Interface: PCI-Express x4 / x8 or PCI Express 2.0 x4

• Mini-PCIe cards, Intel:  40 and 80Gb. A$3/Gb
  

  Intel SSD 310 Series 80GB mini PCIe
  
  * Capacity: 80 GB,
  * Components: Intel NAND Flash Memory Multi-Level Cell (MLC) Technology
  * Form Factor, mini PCIe, mSATA, Interface,
  * Sequential Read - 200 MB/s, Sequential Write - 70 MB/s,
  * Latency - Read - 35 µs, Latency - Write - 65 µs,
  * Lithography - 34 nm
  * Mean Time Between Failures (MTBF) - 1,200,000 Hours

Roughly, prices increase with size and performance.
The highest density chips, or leading edge technology,  cost a premium.
As do high-performance or "specialist" formats:

• micro-SD cards
• CF cards
• SAS and PCI SSD's.

When is an SATA drive not a drive, when it's compact flash

The CompactFlash Association  has released the new generation of the CF specification, CFast™ [from a piece on 100MB/sec CF cards]

10-15 years ago, CF cards were used almost exclusively in all Digital cameras, now they are used only in "high-end" Digital SLR's (DSLR), presumably because of cost/availability compared to alternatives like SDHC cards.

The UDMA based CF standard allows up to 133MB/sec transfer rates.
The new SATA based standard, CFast, allows 3Gbps (~300MB/sec) transfer rates.

In another context, you'd call this an SSD (Solid State Disk), even a "SATA drive".
There are two problems:

• common cameras don't currently support CFast™, the SATA based standard, and
• 'fast' CF cards are slower than most SSD's and attract a price premium of 5-10 times.

I'm not sure what decision camera manufacturers will make for their Next Generation high-end storage interface, they have 3 obvious directions:

• CF-card format (43×36×5 mm), SATA interface, CFast™
• 1.8inch or 2.5inch SSD, SATA interface
• 34mm ExpressCard. PCIe interface.

and the less obvious: somehow adapt commodity SDHC cards to high-performance.
Perhaps in a 'pack' operating like RAID.

The "Next Generation Interface" is a $64 billion question for high-end camera manufactures, the choice will stay with the industry for a very long time, negatively affecting sales if made poorly.

Manufacturers are much better off selecting the same standard (common parts, lower prices for everyone), but need to balance the convenience of "special form factors" with cost. Whilst professional photographers will pay "whatever's needed" for specialist products, their budgets aren't infinite and excessive prices restrict sales to high-end amateurs.

Perhaps the best we'll see is a transition period of dual- or triple-card cameras (SDHC, CF-card and CFast™), with the possibility of an e-SATA connector for "direct drive connection".

Update 04-Nov-2011:
Here's a good overview from the Sandisk site of form-factor candidates to replace the CF card form-factor of (43×36×5 mm):

SanDisk® Solid State Drives for the Client
 "A variety of form factors, supporting multiple OEM design needs."

SanDisk® Solid State Drives for the Client
Product Name	Interface	Form Factor	Measurements
SanDisk U100 SSD	SATA	2.5"	100.5 mm x 69.85 mm x 7 mm std allows 9.5mm, 12.5mm, 15mm
SanDisk U100 SSD	SATA	Half-Slim SATA	54.00 mm x 39.00 mm x 3.08 mm (8-64GB), x 2.88 mm (128-256GB), Connector 4.00 mm
SanDisk U100 SSD	SATA	mSATA	30.00 mm x 50.95 mm x 3.4 mm (8-64GB), x 3.2 mm (128-256GB)
SanDisk U100 SSD	SATA	mSATA mini	26.80 mm x 30 mm x 2.2 mm (8GB), x 3.2mm (16-128GB)
SanDisk iSSD(TM)	SATA	SATA uSSD	16 mm x 20 mm x 1.20 mm (8 GB-32 GB) x 1.40 mm (64 GB) x 1.85 mm (128 GB)
Standard 1.8 in disk	SATA	1.8"	54 mm x 71 mm x 8 mm
Express Card	PCIe	Express Card	34mm x 75 mm x 5mm 54mm x 75 mm x 5mm
Compact Flash	ATA	CF-II	36mm x 43 mm x 5mm

Flash Memory vs 15,000 RPM drives

Some I.T. technologies  have a limited life or "use window". For example:

In 2001, the largest Compact Flash (CF) card for Digital cameras wasn't flash memory, but a hard disk, the 1Gb IBM "microdrive" ($500 claimed price). After Hitachi acquired the business, they produced 4Gb and 6Gb drives,  apparently for sale as late as 2006, with the 4Gb variant used in the now discontinued Apple iPod mini.

Around 2006, the 1" microdrive hard disks were out-competed by flash memory and became yet another defunct technology.

Flash memory storage, either SSD or PCI 'drives', for servers now cost A$2-3/Gb for SATA SSD and $3-4/Gb for PCIe cards [A$3/Gb for Intel mini-PCIe cards].

Currently, Seagate 15k 'Cheetah' drives sell for around $2/Gb, but their 2msec (0.5k IO/sec) performance is no match for the 5KIO/sec of low-end SSD's or the 100-250KIO/sec of PCI flash.

10,000 RPM 'Enterprise' drives cost less, around $1.50/Gb, whilst 1Tb 7200 RPM (Enterprise) drives come in at $0.25/Gb.

The only usual criteria 15,000 RPM drives beat other media on is single-drive transfer rate^*.
Which in an 'Enterprise' RAID environment is not an advantage unless a) you're not paying full price or b) you have very special requirements or constraints, such as limited space.

I'm wondering if 2010 was the year that 15,000 RPM Enterprise drives joined microdrives in the backlot of obsolete technologies - replaced in part by the same thing, Flash Memory.

---

Part of the problem stems from the triple whammy for any specialist technology/device:

• Overheads are maximised: By definition 'extreme' technologies are beyond "cutting-edge", more "bleeding-edge", meaning research costs are very high.
• Per Unit fixed costs are maximised: Sales volumes of specialist or extreme devices are necessarily low, requiring those high research costs to be amortised over just a few sales.
  If the technology ever becomes mainstream, it is no longer 'specialist' and research costs are amortised over very large numbers of parts.
• Highest Margin per Unit: If you want your vendor to stay in business, they have to make suitable profits, both to encourage 'capital' to remain invested in the business and have enough surplus available to fund the next, more expensive, round of research. Profitable businesses can be Low-Volume/High Margin or High-Volume/Low Margin (or Mid/Mid).

Specialised or 'extreme performance' products aren't proportionately more expensive, they are necessarily radically more expensive, compounding the problem. When simple alternatives are available to use commodity/mainstream devices (defined as 'least cost per-perf. unit' or highest volume used), then they are adopted, all other things being equal.

^* There are many characteristics of magnetic media that are desirable or necessary for some situations, such as "infinite write cycles". These may be discussed in detail elsewhere.

RAID, Backups and Recovery

There's an ironclad Law of SysAdmin:

"Nobody ever asks for backups to be done, only ever 'restores'"

Discussing RAID and Reliable Persistent Storage cannot be done without reference to the larger context: Whole System Data Protection and Operations.

"Backups" need more thought and effort than a 1980-style dump-disk-to-tape. Conversely, as various politicians & business criminals have found to their cost with "stored emails", permanent storage of everything is not ideal either, even if you have the systems, space and budget for media and operations.

"Backups" are more than just 'a second copy of your precious data' (more fully, 'of some indeterminate age'), which RAID alone partially provide.

Revisting RAID in 2011: Reappraising the 'Manufactured Block Device' Interface.

Reliable Persistent Storage is fundamental to I.T./Computing, especially in the post-PC age of "Invisible Computers". RAID, as large Disk Arrays backed up by tape libraries, has been a classic solution to this problem for around 25 years.

The techniques to construct "high-performance", "perfect" devices with "infinite" data-life from real-world devices with many failure modes, performance limitations and a limited service life vary with cost constraints, available technologies, processor organisation, demand/load and expectations of "perfect" and "infinite".

RAID Disk Arrays are facing at least 4 significant technology challenges or changes in use as I.T./Computing continues to evolve:

• From SOHO to Enterprise level, removable USB drives are becoming the media of choice for data-exchange,  off-site backups and archives as the price/Gb drives multiples below tape and optical media.
• M.A.I.D. (Massive Array of Idle Disks) is becoming more widely used for archive services.
• Flash memory, at A$2-$4/Gb, packaged as SSD's or direct PCI devices, is replacing hard disks (HDDs) for low-latency random-IO applications. E-bay, for example, has announced a new 'Pod' design based around flash memory, legitimising the approach for Enterprises and providing good "case studies" for vendors to use in marketing and sales.
• As Peta- and Exabyte systems are designed and built, its obvious that the current One-Big-Box model for Enterprise Disk Arrays is insufficient. IBM's General Parallel File System (GPFS) home page notes that large processor arrays are needed to create/consume the I/O load [e.g. 30,000 file creates/sec] the filesystem is designed to provide. The aggregate bandwidth provided is orders of magnitude greater than can be handled by any SAN (Storage Area Network,  sometimes called a 'Storage Fabric').

Internet-scale datacentres, consuming 20-30MW, house ~100,000 servers and potentially 200-500,000 disks, notably not as Disk Arrays. Over a decade ago, Google solved its unique scale-up/scale-out problems with whole-system replication and GFS (Google File System): a 21st century variant of the 1980's research work at Berkeley by Patterson et al called "N.O.W." (Network of Workstations). A portion of which was the concept written up in 1988 as the seminal RAID paper: "Redundant Arrays of Inexpensive Disks".

My Ancient Computer Project

Preamble: If you're really looking for advice on floppy drives: 8", 5¼" and 3½", try these links:
http://www.retrotechnology.com/herbs_stuff/s_drives_howto.html
http://www.classiccmp.org/dunfield/
http://www.deviceside.com/

The good folk at "Device Side Data" sell a USB adaptor for 5¼ inch floppy drives and a range of relevant cables, power-supplies and enclosures.
For about US$100, you can have a working  5¼in USB setup, but it's bring-your-own-drive.
Sourcing a 5¼ inch floppy drive may be tricky: they went out of production 10-15 years ago.
Sourcing 5¼ inch floppy disks, new or 'slightly used' is probably a greater challenge.

---

Some weeks ago a friend (AF) asked if I could copy a 5¼ inch floppy disk for him, leading to this little adventure...

AF had a 3½ inch floppy that he might have copied everything onto and wanted to check.
Without specialist equipment, I knew I wouldn't be able to recover all potentially readable data, but I offered to do what I could with a standard drive.

In the end, AF decided to try something else, so I  didn't get to do his copy.
But it did make me get my old 386-SX properly setup, networked so I could move files to/from it and a way to easily copy 5¼ in and 3½ floppy disks.
Before this, I still booted it occassionally, but the real-time clock battery had died and it had no network card or CD-ROM drive.

This decision led to weeks of farnarkling and some interesting lessons.

While it's unclear if my 386-SX will survive another 2 decades, the software can live on through tools like QEMU, WINE/Crossover and even DOSBOX. So there is some value in recovering the data both on the hard disk and my collection of floppies.
The impetus to recover data from unreadable media, 5¼in floppies, is obvious.
For the readable 3½in floppies, taking a copy now is a good investment of time if I ever want to access the data again: magnetic media does degrade over time. In another 20 years, the coating on those floppies may be flaking off in big lumps.

386-SX Initial Config:

• purchased late 1991, ~$3250
• 386-SX CPU, 20Mhz (selectable to 8Mhz). No FPU. 8Mhz ISA bus, 8-slots.
• Dual floppy drives, 5¼ in [boot] and 3½ in.
• IDE disk. 80Mb, WD AC280.
• single parallel port, dual serial ports, one used for mouse, other for modem.
• 5Mb RAM [max at time]
• Super VGA1024x768 16-col, 640x480, 256 col. 512Kb [K-i-l-o not Mb]
• 33cm display. [fixed scanrate, can be destroyed @ wrong Hz]
• mini-tower. pre-ATX power-supply (no 'soft' power switch or 'halt')
• No sound-card.
• DOS 5.0 and Windows 3.0

Current state-of-play is:

• 386-SX has DOS 6.20, Win 3.11, Networking, CD-ROM and Zip drive all working.
  Hard-drive cloned/backed up and (5) ZIP disks read and copied. [ZIP drive back in its box]
  Single 3½in floppy as A: drive. BIOS only allows booting from first drive.
• 5¼ in drive working as 2nd drive in a Linux machine (2001, Celeron 667) with built-in networking.
• Working through copying all  5¼ in floppies I can find.
  Tally so far [40]: 1 unreadable, 1 with errors.
  Update 04-Nov-2011: 210 floppies read, 40-50 5 in: 8 "no data". 150+  3½in, 5 "no data"
• Now have a USB 3½ in floppy drive, can read those at leisure on newer machines.

The most important lesson for me came about two weeks in:

• I was fixated on doing everything on my 1991 vintage 386-SX.
  At one point I was running through the options of replacing the motherboard and the various costs which weren't attractive given I was 'just playing'.
• Since USB became ubiquitous, finding machines/motherboards with floppy drive controllers is increasingly difficult, which means even embedded boards with FDC's are rare and expensive.
• Then I realised I already had everything I needed in the 2001 vintage Celeron system I had tucked away.
  It's loaded with Fedora Core 3 (support ended in 2004) with a linux 2.4 kernel. Old, but usable.
• It was perfect for what I wanted to do.
  It had a floppy drive controller and I could transplant the cable (with  5¼in 'slot' connectors) from the 386-SX to the Celeron 667.

 I also learnt a little about floppy drive connectors:

• 5¼in drives have a 'slot' (card-edge) connector on the drive and a header not unlike a 40-pin IDE connector (34-pin is used).
• 'Classic' 3½in drives use a socket connector similar to IDE connectors (but 34-pin)
• The $30 USB 3½in drive I bought uses an incompatible tiny connector (two versions, a plug and a socket with a conversion cable). Previously, I didn't know this variant existed. Noted to save other people from popping open put-together-permanently cases. I didn't care about the warranty, but the case needs to be firmly shut or drive operation is affected.

Before starting any work, I had to backup the original 3860-SX disk. My memory was that I'd bought a 30Mb 'RLL' drive (a 20Mb ST-506 drive with a modified controller).
Turns out I really had an 85Mb IDE (now called ATA or PATA), a "WD Caviar® AC280", not only larger, but it would allow me to connect an IDE CD-ROM drive in their as well. My unused hardware pile has any number of CD-ROM drives.

Most importantly, an IDE/ATA drive gave me the option of backing up the drive via an IDE/USB interface... Of which I have a number of versions.

I also had an old system backup of around 30 * 720kb 3½ floppies. DOS 5.0 and Win 3.0.
Using QEMU on a Linux system, I was able to restore this backup to a virtual disk drive.
Shuffling all those disks was painful and slow.

Connecting an early IDE drive to a modern(ish) IDE/USB interface didn't work.
Probably because additional commands were introduced to identify the drive, possibly because this old drive responded to "CHS" (cylinder-head-sector), not LBA (Logical Block Addressing). From the AC280 spec. sheet, the drive electronics did support any reasonable CHS settings, not only the physical layout.

My next preferred method was to connect a second IDE drive as a 'slave' (D: in DOS), fdisk and format it and copy the original drive contents, then connect this drive to a modern system with IDE/USB interface and back it up.

The first 3.5" HDD I tried from my unused pile (1Gb Fujitsu) had errors.
Next drive tried was a 3.5" 4.3Gb. Worked reliably.
In the final config, I replaced that drive with a slower, quieter 2Gb 2.5in drive, cloning it via an IDE/USB interface.

The Phoenix BIOS in the 386-SX is very old. Not only doesn't it support LBA drives, it seemed to limit drives to 1023 cylinders - and 15 heads/63 sectors. Around 470Mb.
A very large fraction of my time was spent fiddling with disk CHS specifications and attempting to get fdisk to ignore the BIOS settings.
[No, I can't update the firmware, the BIOS is pre "Flash-the-BIOS"].

I could setup multiple (large) partitions on the drive with Linux and the IDE/USB interface, but then would run into troubles under DOS and the 386-SX.
I tried 'fdisk' from DOS 5.0, 6.2, 6.22, Win-95 and FreeDos on the 386-SX, but all would only see the 470Mb allowed by the BIOS. Extra partitions would be displayed, but couldn't be changed.

I don't have enough unused 1.44Mb 3½in floppies to back up the entire 85Mb drive, so was very glad my 2nd method worked.

Getting a working ISA-bus (not PCI) network card was simple: I had two in my "unused bits pile".
The one I chose I'd bought new, a Netgear NE2000 clone, but I didn't realise that or find the box (with install floppy) for a while. Relied on Windows NE2000 driver and Internet downloads at first.
The other card didn't have a clear name/identifier, nor did I get a good match from the chip numbers.

This was another 'surprise': how to get any info on installed ISA cards.
I also spent 2 or more days fiddling with the IRQ/DMA settings on the NE2000 card. I'd forgotten the problems that PCI made go away. I ended up with IRQ 5 (COM2) and found an IO address range through trial-and-error. I don't know for sure if its a clash or not.
I couldn't find a tool that would list for me all the cards + settings in the system.
Norton's "SI" provides everything but DMA ports.
MSD (Microsoft Diagnostics) didn't help either.

Getting the CD-ROM working was a good idea, if a little problematic.
Using standard Linux tools, I was able to create an ISO image of the original DOS/Windows system and also add some additional tools.
The problematic part was creating a disk image with Uppercase filenames. Whilst DOS 6.2 (really MSCDEX) reads the root directory correctly, no files or directories can be read/listed.
Perhaps it is the 'Joliet' option I use that causes this... Had the same difficulty with the FreeDos CD.

I managed to get non-DOS booting via 3½in floppies:

• FreeDos [tried last]. Booted, read CD-ROM, not installed on HDD.
• BG-TLB: http://www.giannone.ch/bgtlb/ [Tiny Linux Boot]
• floppyfw - floppy firewall. http://www.zelow.no/floppyfw/

Until I tried it, I wasn't aware that FreeDos uses linux as a base. Uses 'SYSLINUX' as the boot loader and seems to have a kernel.sys.

It took a good deal of searching to find any Linux that would support:

• vanilla 386
• no FPU
• no RAM disk for under 12Mb,.

"Floppyfw" recognised the (single) NE2000-compatible network card, but didn't include a shell.
BG-TLB does include BusyBox, but only support 'plip' networking over the parallel port (not tried).
So while I have seen a linux shell prompt and been able to mount the DOS/FAT filesystem, I haven't been able to dual-boot the 386-SX or run it as a Linux only system.

Whilst FreeDos could read its install CD-ROM, DOS 6.2 was unable to read files/directories contents (the lowercase name problem above).

Another Linux not-tried was tomsrtbt: "Tom's floppy which has a root filesystem and is also bootable." http://www.toms.net/rb/tomsrtbt.FAQ. It advertises itself as "The most GNU/Linux on 1 floppy disk."
Which might have worked, but it formats 3½in floppies at the non-standard 1.7Mb, not 1.44Mb, with the caveat "Doing this may destroy your floppy drive".

One of the 'surprises' I got was being unable to replace the original 3½in floppy drive with a newer drive. The interface and connectors were all the same, but the newer drive wouldn't work in the old system.
Was it me connecting it incorrectly, a faulty drive or something more?
Unable to tell and unwilling to devote a bunch of time testing it.

One of the worst 'surprises' I got was after installing the IOmega (parallel-port) ZIP drive software on the system after a fresh install of Windows 3.11. [Windows 3.11 had decent Networking support and Microsoft still have downloadable a good TCP/IP stack.]
I had it all setup, tested and working and foolishly, in Windows, selected "Optimise settings" and the system hung.
Whereafter, the system couldn't see any Comms ports, serial or parallel. Which was very problematic because the 386-SX didn't come with a mouse port (DIN or PS/2). I used a serial port for the mouse.
Windows hung when it booted, leading me to try to revert the ZIP drive install and later to re-install Windows.
There were countless reboots and 6-8 hours later I gave up and went to bed.
I had realised/diagnosed that when the machine booted, the BIOS reported "0 serial ports" and "0 parallel ports". The BIOS setup screen only allowed me to selected HDD and floppy drive settings.

First thing in the morning, I powered on the machine and it worked perfectly. Including the original copy of Windows that would hang.
All I can think of was a power-cycle (off/on) cleared the fault, whereas a 'cold' or 'warm' reset (reset switch or ctrl-alt-del) didn't. In the many reboots, I hadn't thought to power-cycle the machine. [A note for myself and others experiencing weirdness on old hardware.]

I also have an old Dell Inspiron 7000, dating from 1999. I got it with a removable ZIP drive, figuring I could do backups and bulk-data transfers using it and the parallel-port ZIP drive. While tinkering around, I disassembled the other ZIP drive. It's an IDE/ATAPI drive, but the connectors are non-standard. I was hoping I'd be able to kludge it to work on another system - but not to be.

One of the BIOS limits, noted above, was it will only boot from the first floppy drive. When I'd configured the machine, I'd made the 5¼in drive "A:". Part of my reconfig was to move that drive and make the 3½in drive "A:", so it could boot from it. And most "floppy disk images" on the net are 1.44M for 3½in drives.
Then I executed a perfect "rookie mistake": I forgot to make a bootable 3½in system disk before moving the 5¼in drive. And all my system disks were, of course, 5¼in.

The 386-SX is in a "mini-tower" case with very limited space between the back of devices in the drive bays and the motherboard etc. This makes running cables and changing drives quite time consuming. Especially with older connectors that are loose and can be jiggled off. This was part of the reason to move to a 2½in HDD - much more space. I did need to find a spare 2½in-3½in mounting kit first.
I didn't believe the weight of the whole system, let alone just the removable cover. Meaty!

Early on I replaced the on-board/real-time clock battery so the system would remember the time over reboots. More modern systems use 3V lithium batteries (CR2032) for this. This old motherboard used a 4.5V alkaline battery (mounted off-board with Velcro). About a week in, I used 3 AAA alkaline batteries in a modified carrier (and a cannibalised connector) to craft a replacement. A less pretty way is to load 3 batteries in a cut-to-length tube with wire soldered directly to exposed battery ends. Soldering wires directly onto batteries requires some technique. You may need help if you try it. There could be an explosion risk with alkaline batteries becoming overheated (they are marked "do not dispose of in fire"). Research this properly or get help if you choose to do this.

The system, including DOS, quite happily accepts dates of 2011. No problems there...

One of the problems I haven't addressed yet is: "How do I clean the drive heads?"
Back in the day when I was a sometime 'operator' on mainframes, cleaning tape drive heads was part of the ritual. We used Isopropyl alcohol + cotton swabs - because it didn't leave a residue. The swabs would always come away stained with oxide coating from the tape.
For these old drives, I've two reasons to want to be able to clean the heads:
a) these are old drives and may well have an internal dust build-up, and
b) older disks are likely to shed more of their coating than when new.
Some media formulations from the late 1980's are known to suffer problems. I've heard first-hand accounts of the work needed to recover period audio-tape recordings due to this problem.

• 5¼ inch floppy drives load the heads directly in-line with feed-slot.
  It is possible to get a swab in there, though v. difficult to see what's happening.
• 3½in floppy drives drop the disk to both lock the disk in-place and load the heads, hence the heads, being offset, aren't accessible for swabbing through the feed slot.

Another thing to investigate...

What is an electronic document and why that Matters.

#comment-2157 agimo.govspace.gov.au back-to-the-future-another-chance-to-influence-coe-development

#comment-2158 agimo.govspace.gov.au back-to-the-future-another-chance-to-influence-coe-development

First one says:
 There isn't yet a clear Industry Standard, so do Nothing for a while
longer.

Second says:
 The basis of your problem is you're conflating many problems with
conflicting requirements under the banner of "document exchange".
 And there's *never* a good reason to give non-authors an editable copy.

With a little postscript: FOSS definitively solved the "multiple
concurrent update versioning problem" - 20 years ago?
I didn't say, the 'secret sauce' is Use Simple Text Files!

That's at the heart of all these problems: binary formats.
Should've been banned 30 years ago... But weren't and now we're paying
for it.

Feedback model of Software Production

The bottom part of the diagram is a representation of a single step in the software production cycle.
It consists of two Amplifiers (A1, A2) with gains G1 and G2.
[The top part is an outline of the possible succession of steps in the process. This too has interesting characteristics. Not addressed here.]

Part of the output of A1 leaks back, via A2, into the input of A1.
A classic schematic for a wide-band amplifier or an oscillator.

How much is fed back, and in what time-relationship to the input?
The value "R". (Really it should be "Z" for impedance, and have lead/lag/in-phase components)

The crucial aspect of the diagram is that all inputs are "positive" - if the output directly follows the input, not the inverse (as found in a classic "negative feedback", or correcting, system.)

The model says there are only 3 internal metrics to be measured an one external inflow and two outflows.
The 3 internal metrics:

• G1 - Raw Lines of Code per Day
• R - errors per hundred Lines of Code
• G2 - Time to fix one error. In a normal Project, this can't be less than 1/2 day because of overheads, analysis and diagnosis, testing and change commit.

The external inflow is Errors from steps further along the chain.
I've forgotten to draw its inverse, an outflow to previous steps.

The most obvious outflow is "Delivered Lines of Code".
You can see from the nature of "R", there may be significant 'lag' (time delay) so at notional Project Delivery, Errors may not be fully realised.

There are exactly 3 dynamic states of this system, given that G1, G2 and R are constant.
The "gain around the loop", G = G1 * ( G2 * R ).
[if r = 1/R; then G = G1 * (G2 / r ). Possibly more intuitive.]

The Loop Gain, G, can be greater than, equal to, or less than 1. [mistake in diagram, I wrote '0' (zero)]
The system dynamic are:

• G < 1. System "converges". It heads towards the X-axis asymptotically (?). The smaller G, the faster the convergence.
• G = 1. System is in a "holding pattern". It never moves towards completion, but doesn't move away.
• G > 1. The System "diverges". Every day of effort produces more than one day's extra effort. The classic "Death March".

Having 3 variables, they can be manipulated to push the system in a desired direction. If you were paying for the project and wanted a result, you'd want the smallest values for R and G2.

• Reduce G1. Slow the coding process. Counter-intuitive and probably would cause the Project Managers' head to explode. But its there in the figures. Spend more time fixing errors and less in creating more new code and their concomitant errors.
• Reduce G2. Better people, better systems, better tools, better training to find faults faster and produce better solutions. So do subject-area experts. Lesson: Don't move people away from what they know.
• Reduce R. Better Design, Design and Code Reviews and better people/tools/processes for coding. Create less rework, adopt practices that reduce errors (like pair-programming). And most crucially, don't over-work people. Even a small increase in Error rate can push a project into the Red Zone...

Programming Projects are defined by Rework and its management.

Projects do not "fail/get later one day at a time".  They fail before they start.
Let go, they often gallop away from a solution.
No amount of effort can retrieve them - the more effort expended the more that's needed to complete them. This was the genesis of Microsoft's 2005 "Longhorn Reset". 25,000 person-years were discarded.

Wireless Broadband Plans (Australia)

Talking to a friend about "going broadband" led me to think I should write something down...

If you are "on the road", MacDonalds offer free WiFi hotspots. If that works for you, saves all the hoopla of getting 3G broadband.
'Lifehacker' suggests there is a 50Mb download limit.
maccasfreewifi.com provides a hotspot finder for 706 locations.

Additional Comments [19-Jan-2010].
This writeup of George Bray's is clear and concise.
I think it shows a definitive, cheap, simple solution to Mobile Broadband, plus flags a bunch of caveats.
Stop now and go read his piece if you want or need to actually do something soon.

Here's a writeup and video of my "Broadband from the Beach" equipment.

http://geobray.com/2010/01/19/broadband-3g-from-the-beach/

George
  

Caveat:

Not having had/run a mobile wireless broadband service myself, there'll be stuff I don't know about...
Please contact me by email or "comment" (below) to correct errors/misinformation, provide links or additional war-stories/advice/information/tips/comments.

Mobile Wireless Broadband:

There is some confusion about "Mobile" and "Wireless" Internet/Broadband.
There are mobile-phone 3G networks, long-distance data-over-wireless and 802.11 Wireless Ethernet, or "WiFi". This piece concentrates on 3G-based services. Australia is yet to see broadscale WiMax or 802.11 networks.

Mobile or Wireless Internet/Broadband and Laptops are an extremely powerful combination.
The only caveat is Mobile Internet generally comes from a USB device (or via USB from your phone).
The lack of a hardware/stand-alone firewall means it's difficult to share a connection and you give up some security.

Hand-crafted arrangements can be built:

... in Armidale and the only place I could get reliable 3G link was from my bedroom window.
Nokia N95 in the window with bluetooth link to EeePC 701SD on the desk and from there an Ad-hoc WiFi link providing access to me anywhere in the house.

Whirlpool Forums (a.k.a. "Broadband Choice") are a very good resource: http://forums.whirlpool.net.au/forum/114
Whirlpool have a list of Networks and Resellers, as of Dec-2009.

There are three "3G" networks in Australia offering data (HSDPA: High-Speed Downlink Packet Access) and competing reasonably aggressively:

• Telstra - best coverage, generally best potential speed. Max 42Mbps,
  
• Optus
• Vodafone, 3, Hutchinson

"3G" (3rd Generation) mobile phone services are based on CDMA (Code Division Multiple Access) protocols. This means if you have signal, you can get service (not so with GSM, which is Time Division Multiple Access). It also means you can improve your download speeds and effective coverage by attaching a larger aerial.

Telstra have very few or no resellers of its NextG broadband network.

Optus and Vodafone/Hutchinson have many resellers: folks like Virgin Mobile and Exetel offer great packages using one of these networks.


But there is a nasty 'gotcha': "Roaming Charges".
On some plans, especially low-end, you can be charged (from memory) around $100/Gb for roaming.
This may have changed, but is worth checking for specifically.

"Roaming" is where your provider doesn't have a service, but another provider does and will carry your bits - for a price...

On older 2G ("GSM" or 'digital mobile') networks, the data is called GPRS - General Packet Radio Service and used to be quite expensive and comparatively slow. Some carriers use this as a fall-back when you are 'out of area'.

Some friends did the "Gray Nomad" thing through Western Australia and back to Sydney this year.
They got a Telstra 3G USB modem and a new laptop.
It generally worked very well. They had about a week of 'glitches' at the beginning, but eventually found a helpdesk operator who'd seen the problem and got them on-line.
They also had a problem with the laptop power-supply 'failing'. It ended up being a seed lodged in the connector.

Travelling, they had 3 types of service:

• None.
• Slow, out in farming communities.
  
• Fast, near mining towns and cities.
  

They were able to stay in-touch without on-site support: that's pretty impressive.
George notes on the CLUG list he regularly achieves 4Mbps with a tripod-mounted directional aerial.

Google threw up a few sites offering service comparisons:

• broadbandguide.com.au
  
• comparebroadband.com.au 
• broadbandexpert.com.au (Seems to be limited, perhaps sponsored)

They have different coverage. For example, only 'compare broadband' threw up  Exetel plans for $17.50/month for 1Gb and they only say "speeds vary".

Selecting your plan:
[See also below "Specific advice for selecting a service and setting up your connection"]

• Choose Prepaid or Contract. Prepaid means you can experiment.
• Contracts: You are locked in for 12-24 months with "Early Termination" penalties. Beware of "automatic renewal" at the end of the contract - they may require you to specifically "opt out".
  
• Prepaid: You need to look for $ per Gb and 'Expiry' - how long you've got to use a payment. (Hint: Don't apply payments until close to when you need them of you might lose them.)
• Prepaid 'gotcha's: Recharges must be planned.

1. First problem is to change plan options (phone or internet mobile plans) before applying the 'recharge'.  Payments/recharges cannot be altered once applied. [From my limited experience]
2. It can take 48 hours for a 'recharge' to be applied to your account for some carriers. While you might technically be in credit, your service could stop or be charged at a (much) higher rate until the Telco systems process your payment.
   

• Modems: Are they 'locked' to a network/ISP, or can you use it anywhere? You're also going to be interested if the modem is supported by your Operating System.
  Important user advice: Select your modem/gateway first.
  
• Speed and Download limits: Mostly how plans are differentiated.
  Beware of the costs of downloading e-mail.
• Monitoring Usage: Whether on a pre-paid or contract plan, you will most likely have to monitor the usage seen by your provider. George in his CLUG post, points out that that there is no standard way and potentially is an application tied to a specific operating system.  There is also the issue of stale and incorrect usage data being reported by the tool.
  This is a nasty 'gotcha': If you are paying by the Mb and can't reliably track it, don't fly too close to the limit. At best you'll be suddenly cut-off or 'traffic shaped', at worst get a very nasty bill.
  

Virgin Mobile (and presumably others) also provide a Home Phone package.

For $80/month (Dec-2009) Virgin Mobile give you a box that connects via the Optus 3G network and gives you unlimited VoIP calls within Australia (with a normal dial-in number), 4Gb/mth download and they're a little coy about the speed. But it falls back to 128kbps.

People like Virgin (and Optus?) have a "Fair Use" policy.  If they deem your usage 'excessive', your account may be cancelled or you'll incur fines.
'gotcha': On the CLUG list there was a report of Exetel dumping a customer as they "were unprofitable". You can't avoid this: the service providers supply you their service subject to their Terms and Conditions. You must agree to the "T&C" or there is no contract.

The only 'remedy' is to avoid the problem. If you intend to heavily use a service, sign-up for an appropriate plan.

Great Tips:
Tridge happened to mention at a CLUG meeting that at home he runs both an ADSL link and a back-up Mobile Wireless link. If ADSL goes out, he can still access his home computers from outside.

The normal Unix "route" software for IP doesn't allow for asymmetric links: there can only be one default route.
Packets may find their way to your host via a second route, but must return by the default route.

The details of "asymmetric routing" using "iproute2" are spelled out in this thread on the CLUG list.
Links to iproute2 Project and Linux Advanced Routing and Traffic HOWTO.

Speed and Latency:
"Oils ain't Oils" applies.
 4Mbps on mobile wireless will not provide the same apparent level of service as 4Mbps over fixed-wireless, 802.11 or fixed-line (ADSL or Cable).
Whilst you may have a theoretical download speed of 4Mbps, TCP/IP needs to correct errors and to wait to 'ACK' packets to come back before sending more. The round-trip-time can limit link performance.

Mobile wireless suffers from transmission errors (noise + lost packets), variable data rates (there are other users competing for air-time) and long latency (affects round-trip-time) due to many layers of packetisation, buffering and forwarding.

Long latency only marginally affects big downloads (the TCP/IP 'ACK' problem), but can be a deal-breaker for real-time services (VoIP, Skype, IM, ...) and when downloading many small files - as in normal websurfing and collecting email.

Radio reception is fickle for many reasons - tune in to AM radio at night for an example.
Mobile networks necessarily need to track 'handsets' and SIM cards to route traffic: what towers are you currently near?

Which means mobile networks must have a centralised handset location database and a single or limited number of interconnect points.  This means your phone or USB dongle is many hops from 'central office'. The ISP connects from there.

Telstra trumpet they support 21Mbps (going to 42Mbps?) - but not on all towers and your "actual speed" will vary with aerial, equipment, signal strength, other users and backhaul capacity...
As my Gray-Nomad friends found, Telstra won't invest in unwarranted (unprofitable?) capacity.

Even when you hit a "white spot" (high-speed tower), you have to compete with everyone else in the area for a slice of bandwidth.  May be great in the bush, may be less great in highly used areas - like the city or a rural/regional tower offering the only effective broadband in the area.

Additional Comments:
[30-Dec-2009] Thanks to CLUGers, in no particular order, for their help and comments:

• Alex Satrapa
• Scott Ferguson
• Ian Munsie
• Keith Goggin
• Miles Goodhew
• George Bray

A great piece on using mobile wireless by George Bray sent on the CLUG list.


A note on the CLUG list on setting up Telstra 3G (the MF626) on Linux (Ubuntu 9.04) by Carlo Hamalainen. [19-Feb-2010]

Contract 'gotchas':

"Early Termination Fees" that most providers are charging these days:
Sign up for a two year contract costing $480 all up, they'll sting you for $540 if you terminate the contract before the two years is up (not to mention, they automatically sign you up for a further two years if you don't opt out during the three month grace period at the end of the contract).

The moral of the story is make sure you check the conditions on those contracts, folks.  Early termination, "roaming" (in the case of at least one provider, they will charge you the 'normal rate' for data if you fall off the 3G network onto their GPRS network), "acceptable usage" and other caveats.

General comments on Mobile vs Fixed-line service:

Summary: "it sucks, don't do it."  :P

Seriously, Virgin Mobile offers free tethering with their $54/month iPhone plan - that's $54/month including payments on the phone. I don't have the details on me, and I can't find the plan on the Virgin Mobile website, (the kiosk in the Canberra Centre near Mac1 will be able to help you).
I think it's a 300MB/month data allowance.

On the flip side, I've never had a good experience with 3G broadband except for the situation where you can mount the antenna in just the right spot and leave it there.
In other words, you get the best 3G coverage in the same places you get the best ADSL 2+ coverage.

Optus specific comments:

• provide an internet addressable IP to their mobile broadband customers
• does not charge extra for roaming onto their slower GPRS link
• provides a "usage meter" to help you track your usage (some reports of delays and misreporting)
• Virgin use the Optus network with mostly the same Terms and Conditions, with the additional proviso that peer-to-peer traffic may be capped.

Non-mobile wireless, Canberra only:

Netspeed 'Longreach'  works with Linux (SuSE 11.2) with a 2km range, but "Slows when Foggy". The Netspeed whitepaper says it's a fixed wireless network using a Motorola product with a 10km range. Expect 7Mbps, with a 14Mbps maximum.  Their 'availability' checker lists 3 deployed base stations (Canberra South Side) and 2 towers under deployment (north side).  Good rates if it works for you.

Tethered data on the Hutchison/Three network:

I use my Nokia 6110N paired via Bluetooth to my netbook for tethered Internet access. I pay $8/mo for the privilege of the "Internet" connection (Otherwise you're in a walled garden). They have a $5/mo plan with much higher data cost and lower included data volume. Their roaming data charges used to be astronomical, but I think they changed this recently ...
Generally it's slow, the latency's massive, but it works and it's very convenient for email/web. Also, if you commit to a 12 or 24-month plan they double your included data volume.

Specific advice for selecting a service and setting up your connection:

When shopping for a 3G plan I would recommend selecting a modem first, then finding an ISP that will sell you it - or will sell you a SIM only.

One feature you should look for is an external antennae connection - very useful in some areas. Check to make sure the model fits without interfering with access to other ports, and that it doesn't raise to laptop off the desk.
I'd recommend a 5m USB repeater cable for better reception, and to allow placing the antennae away from the user.

Make sure your machine is capable of supplying more than 500ma to the USB ports. Consider buying a "split-tail" USB lead - one input for data and power, another for extra power from a second USB port if your machine is USB power taxed or lacking. For about $3 from JayCar you can buy a portable USB power supply - it takes 4 AA batteries and reduces the output to 5V, hack on another single AA carrier and it will supply 5V from Ni-MH batteries.

Most of the UMTS (3G) modems I've looked at have the option of selecting HPSDA only (no roaming surcharges). I'd recomment setting your modem to that unless you are going into a bad signal area. Some plans do not include SMS.

Check the modem model numbers - when I was looking a couple of months ago I noticed that Dick Smith in Civic had 4 prepaid Dodo Huawei modems.
I noticed that one of them had an extra alpha in the model number - one sales rep assured me the "G" model was the same as the non-G, another said it was better. A quick search on the internet showed the "G" as a lower powered model - supposedly no longer distributed. The Huawei has an external antennae connection.

You will hear reports of slow connection speeds under Linux with some modems - generally because the user has an old driver module with a very small maxSize (the module takes a vendor and product parameter only).

Gateway, connection sharing and stand-alone firewall:

... have a bigpond 7.2 Home Network Gateway (3G9WB)

• 4 port Ethernet router
• configurable security
• runs off 12V DC with internal voltage regulator more than adequate for car use
• can use high gain antenna for fringe area
• EXPENSIVE $90 per month 5GB shaped
• VOIP works OK mostly @ ~40kms near line of sight to tower (or better

So I bought a new Intel Mac Mini...

In early 2006 I bought a Power PC (PPC, G4, DDR2-SODIM) Mac Mini to use for a Uni course.
Could've gone Intel, but it was new technology, so I avoided it...

Coming up to 4 years on, and I've filled the disk, the CPU fan is a little noisy, new things like 'Google Chrome' are 'intel-only' and OS/X 10.6 ('Snow Leopard') is Intel only... Plus VMware and Parallels can run on Intel platforms.


This time I paid the ~20% extra for a top of the line 2.53Ghz core duo, 4Gb DDR3 RAM, 320Gb 2.5" SATA - which is roughly 4 times the capacity in all dimensions as the PPC version.

First surprise, doesn't come with a DVI-VGA adaptor in the box. BUT, now there are 2 video connectors, 'mini-DVI' and 'miniPort'. The shop only had a 'miniPort' VGA adaptor on the day and I went home with that...

Attach to KVM, boot and - no monitor found. VGA adaptor works fine directly connected.

Second surprise, 800Mhz Firewire is a 9-pin connector not 6-pin (400Mhz on PPC Mac).

Booted new Mac and it says "got an old Mac?" and gives instructions to reboot the old one "holding down the 'T' key" to start a transfer. Which meant closing out all my running Apps and discovering just what they meant.

Which was hold down the 'T' key whilst the machine was shutting down, before the reboot.
It comes up in 'Transfer mode' - displaying a "firewire transfer" symbol. Apparently you can use an 'ethernet' transfer mode... Didn't try that.

After 2-2.5 hours, the transfer completed - 75Gb moved over. Accounts, Preferences, Applications and User Data copied and working.

I've had to reload Adobe Acrobat Reader, apparently it wasn't a 'Universal Binary'.

Even the licensed & registered 3rd-party software I had (MS-Office, NovaMind, OmniGraffle) 'Just worked' on reboot. The transfer was clever enough to not move system binaries. As a bonus, I got a work Perl system back... In the past I'd installed a second, non-Apple, version and managed to zap CPAN.pm on both versions...

Third surprise was initialising "Time Machine". It took 2.5hrs to transfer 75Gb, which is about right - I've timed the old HDD @ ~20Mb/sec sustained raw read rate.

It took a day and a half (!!) for "Time Machine" to initialise... 33-36 hours, Ouch!
The USB drive, a 1Tb Western Digital 'Book for Mac', runs at about 10Mb/sec for raw reads (36Gb/hour).

I can only guess why it ran so slowly... The "Time Machine" is a HFS filesystem.
"Activity Monitor" showed some very large numbers for Gb read from the system Disk, but the amount written to the USB drive matched the progress numbers reported by 'backupd'.

In the middle of the initialise, I rebooted the machine because I could see no evidence of progress.
Rebooted, restarted the initialise and off it went again. Seemed to pick up from where it left off.

There were one or two other times that it took hours, literally, to backup a few Gb.
Now it does 2-3Gb in 5-10 minutes, as you might expect.

What I love about "Time Machine" is this is the first time I've had a good backup system for my home desktop. Seems odd as this is what I do professionally, but there it is.

Another cute thing with "Time Machine" is it can use a Network connected device - like a wireless connected Apple 'Time Capsule' or instructions are out on the web for using a "NAS" - like an old Linux box.

The "Time Machine" interface is well thought out.
You 'enter Time Machine' and the Finder comes 'front and centre', the background changes to 'beginning of the Universe' picture, all the other Apps slide away and the rest of the decorations appear.
Neato!

When you're done, your desktop just slides back into place. Very cute.

There's a major caveat with "Time Machine" - it's granularity is "whole file".
I use 'mailbox' format (a 'folder' contains all messages in a single file) versus 'maildir' format used by Apple Mail (every message is one or more files in a directory). Everytime a new message gets added to one of my 'folders', the entire file is written. I've looked and there are several which as 1+Gb :-(

It's pretty much 'rsync' with a nice GUI and a few tricks in the HFS filesystem.  One of which I read (to confirm) is that it allows directories to be linked.  This was removed from POSIX because you can get complex loops in filesytems...

Things I don't like about "Time Machine", but aren't reasons to bail:

•  I haven't been able to find how to turn on decent logging.
  The default syslog entries written in /var/log/system are high-level (#files, total size).
  Where/How do I get a list of the files it dumped?
  
• I could exclude Thunderbird's mail folders and save on backup time and space.
  What I'd really like if I could tell it that these files grow by concatenation, and to link to just the most recent version... Though data might be lost if file is truncated, but the semantics of that should be simple. Eg. if the current backup version is perfect subset of the new file, just concatenate the new lines. Otherwise, create a copy.
  
• Haven't found, after a quick look, the API. Be nice to be able to integrate with 'rsync'.

'locate' works, but there is some internal magic that seems to exclude 'Desktop' files...
Almost everything is scripts, but I can't find the list of excludes. Perhaps it is just silently failing because of the number of files I have in a directory there...

[30-Dec-2009]
The Western Digital "MyBook" only transfers 10Mb/sec (36Gb/hr). Seems slow.
If "Time Machine" reads or writes every block on the filesystem, that would explain the initial load time and observed volumes of data read.

Mini-display port versus Mini-DVI.
If two VGA adaptors are used, Display-port is the 'primary' (system boot and menus).
If VGA and DVI on Mini-DVI: the Mini-DVI is the 'primary'.
[Don't have enough dongles to test all combos]

Working with a USB/DVI KVM.
The Intel Mac Mini has a DVI-D connector, the PPC Mac Mini has a DVI-I (integrated, allows for DVI-D or DVI-A).
The connectors on the KVM I bought are DVI-I, which won't plug into a DVI-D socket. (A DVI-D cable will plug into either a DVI-I or DVI-D socket).
I won't spring $50 for a separate DVI-D cable  until I talk to the vendor.

Meanwhile, I saw a $25 VGA to DVI-A cable.
Connects to the KVM, but nothing displays. Presumably because the DVI-D connection on the monitor only does digitial. Who'd have thought :-)

The PPC Mac Mini came with a VGA/DVI-I adaptor. It doesn't work in reverse. :-)

The KVM exhibits the usual "doesn't play well with sleeping computers" problems:

• While the KVM does connect speakers/microphone through, there is sporadic hum.
  
• with 2 or 4 ports connected, put a system to sleep via menu and it is 'woken' after a second or two.  Something to do with the USB  interface of the KVM. Execute a delayed command, flip to another input - All Fine.
• Single device connected - system will sleep from menu, but at some random time, it's alive again.
• Waking a system is impossible without unplugging/replugging its USB connection to the KVM
• Not 'sleep' exactly, The keyboard and mouse sometimes become unresponsive. Either reconnect them to the KVM or reset the KVM (remove all powered connections, including video).

This DVI/USB KVM has the habit of complaining about selecting an inactive port. It will lock up (needs 'reset') if moved to an inactive port. Not sure if that's sometimes or if there on the port for more than a few seconds.

I tried turning off the 'beep' from the keyboard - locked up & needed a full 'reset'. Could've been I swapped it to an inactive port.