Gigabit vs Fibre Channel Discussion

My friend Mike has spent a lot of time working on large networks and SAN devices within corporations and government organisations here in Wellington and around New Zealand so when I get curly network questions I usually turn to him. The other day I emailed a really quick question, what provides the most cost-effective high bandwidth network, fibre channel or gigabit?. He soon replied gigabit and then tacked on the end hold on a minute, just writing some notes...

His notes turned out to be a two page series of observations and findings that were so interesting that I asked him if I could put this up on my website so that I could refer others to it. I have edited pieces for clarity and added a bit of background but more or less I have copied things verbatim from his emailed reply....

To start with some people claim that SATA does 1.5 Gb/sec but this is not true as the actual disk throughput is really more like 1.2 Gb/sec. This of course is shared for all calls i.e. operating system, etc, so actual data throughput is more likely 80% of this, at best, 0.96 Gb/sec or 120 MB/sec

A decent gigabit ethernet network does 1 Gb/sec (obviously) however there is a switch limit here and even a decent switch will not be able to switch all ports at 1 Gb/sec at the same time.  For example a 48 port ethernet switch is usually broken into 8 lots of 6 ports and each set of 6 shares a throughput of 1Gb/sec concurrently.  A good Cisco switch will cost about $160 per port, whilst a good Juniper switch would cost less and probably perform better during high volume data transfer.

Both Ethernet and Fibre Channel (FC) allow multiple ports to be used however this is much more expensive when using FC compared to Ethernet. When multiple ports are tied together they can be used to provide fail-over or load sharing (i.e. making two 2Gb/sec Fibre Channel connections into one 4Gb/sec or two 1Gb/sec Ethernet ports into one 2Gb/sec).

A typical Fibre Channel Host Bus Adapter (HBA) will cost approximately $1500 to $3500.  There are some cheaper models available ($700 to $900) but these often do not perform as well as they can introduce errors and reduce throughput.

A good fibre channel switch will switch all ports at full speed concurrently and this is why they are so expensive.  $1000 a port is a pretty normal price for this type of equipment.  However they often cannot handle sustained use on all ports at full speed (20 minutes and above would be a ceiling even on a Brocade Director FC switch). After this amount of time you will start to see errors which causes things to slow down a lot.
Also a FC switch requires transceivers (what you actually plug the cable into on the switch).  It depends on the make, model and supplier as to cost of these.  Sun sell the Brocade Director with a bag of them included in the cost however a HP Brocade Silkworm will be an additional $120 a transceiver from memory.

FC requires a fibre network which is expensive to put in, maintain, protect and is very easy to break.  A piece of dust on a connector will create a lot of errors and drag the speed right down.

FC data blocks are very large (often Megabytes in size) rather the Ethernet packets which are usually measured in kilobytes.  If you are using fibre channel and you are sending small requests (4 KB) all the rest of the packet is wasted space. This makes FC only suitable for large requests (i.e. open that 50MB file, not 500 of those 300KB files).

FC also has very little in the way of control traffic.  This significantly increases the speed of FC however this only works on the principle of minimal errors during data transfer. This is why fibre channel is dependent on high quality gear.

iSCSI does not work very well over FC. No one has quite worked out why yet.  Head technicians at Sun have been working on this issue for the last two years without working out exactly why.

iSCSI handles the small requests very well as it is Ethernet based but it also manages to handles the large data packets very well (FC will beat it but not by a lot). iSCSI also manages to handle the break up of the large SCSI blocks into TCP packets well and resize packets quickly to suit.

Binding TCP (or any protocol based on Ethernet) will be detrimental to speed of a Fibre Channel network. It would not be uncommon to see about a 60% loss of expected speed.

A lot of this stuff is a little irrelevant as all OS's used as storage devices have caches for this exact reason.  A typical NAS will carry 1-2GB's of cache so that it can receive/pre-read the files and pump them across when required. So overall performance often comes down to OS configuration more than network bottlenecks.

For storage the most important thing normally is IOPS (Input/Output per second) which is the number of requests you can handle per second. This is typically what defines how fast the data copy is completed more than the throughput especially in mid-large multi-user scenario's.  A typical SATA drive will give you approximately 120 IOPS where as a SCSI does approximately 250. For a comparison Fibre Channel drives can do as high as 1100 IOPS.

I spent eight months with HP and Sun in labs doing this and found the following (from my notes).  Tests were done with data in memory so there was no disk read/write time, this is based purely on data transmission.

A well setup ethernet network running iSCSI will cost about $310 per machine (single connection, CAT6 incl NIC, cable and switch) and will pass a 300MB file in 0.2 sec either side of 3.4 sec without caching on either device receiving or sending.  If you step up the NIC cost (total cost per machine $860 approximately) you can drop this to about 0.1 either side of 2.2 sec.

A well setup FC network will cost about $3400 to $3800 per machine (single connection, Fibre cable, HBA, switch, HBA software) and will transfer a 300MB file 0.14 sec either side of 1.2 sec without caching on either device receiving or sending

However there was typically additional costs with Fibre such as the need to install a 19" rack as a Brocade FC switch is full depth (approx 800mm depth and 1U high). Fibre also must be properly installed and the ends cooked and polished at approximately $90 per end.  Ethernet switches are more shallow and CAT6 is more robust so installation costs for Ethernet is far lower.  
These costs are also dependent on you buying a number that matches the gear. i.e. if you have 9 machines using FC then you only need HBA's, cables, software and transceivers for 9 however you must still buy a 16 port switch (at a total cost of approximately $16,000, or $1,800 per machine).  FC switches come in 8, 16, 24 and then you typically step up to enterprise gear where you add ports to a switch chassis in sets of 8 or 16.  Ethernet typically comes in 4, 5, 6, 8, 12, 16, 24, 32, 48 ports and then the enterprise gear where you add ports to a switch chassis in sets of 16 or 24.

In short Fibre Channel was made for (and is best for) SAN and backups (using SCSI devices as it is pure SCSI commands over a network).  Ethernet and iSCSI are best for any thing else including Network Attached Storage (NAS) and device to device communication.