I recently upgraded a PC running Ubuntu 11.10 on a ASUS KGPE-D16 motherboard with an Opteron 6128 processor to use a newer Opteron 6274. Most recent OS versions, including Windows 7, can run on the 6200 series processors without requiring a change in configuration or software. But before you go through the whole process, double-check that the OS you have installed does have support for the new processors.

Even when the OS supports the new processor, it may not be optimized for it yet. For instance, Windows 8 is expected to make better use of the dual-core pairs by using different scheduling algorithms than Windows 7 does. So there’s likely some room for improvement as software vendors get their code tuned up for the architectural changes in AMD’s latest processors.

Picking A New AMD Socket G34 CPU

There is a wide variety of 6200 series CPUs. Basically you need to keep in mind four things:

1. Power consumption
2. Core count
3. Clock speed
4. Price

Selecting a processor is a game of trade-offs that depends in large part upon how you use your computer and what you can afford. Absent specific information on your needs and budget, I’d suggest the AMD Opteron 6272 is a good compromise between all the tradeoffs as you can get them in the mid $500 range from SuperBiiz, they should work on pretty much any G34 socket motherboard as they max out at 115W power draw, and the 2.1 GHz cores are still reasonably fast for single threaded or lightly threaded software plus with 16 cores you will be able to get some great multitasking. Others seem to be agreeing as I’ve noticed the 6272 is often sold out more quickly than other 6200 family chips. If that happens, then the 6274 is a good alternative.

But if you’re running only a couple programs that cannot exploit more than about four threads well (many games, old versions of Adobe Photoshop, etc. come to mind), you might be better off with a processor with lower core count and higher clock speed.

If you’re on a tight budget, the AMD Opteron 6212 in the high $200 range is a good choice and will probably perform a bit better than the older AMD Opteron 6128 for most kinds of applications for just slightly more money than roughly $250 that the older Opteron 6128 costs these days.

If you want the absolute fastest G34 processor, be prepared to spend around $1100 for the 6282 SE with its 16 cores running at 2.6 GHz gobbling up 140W maximum power. Some G34 motherboards may not support this processor because of its higher power draw. Others may require a CPU cooler upgrade to handle the higher power dissipation.

Verify You Have Recent BIOS, Upgrade If Necessary

The first thing you need to do to your computer to get ready for the upgrade is to make sure the BIOS on the motherboard supports the newer processors. Reboot the PC and check that the BIOS version is 2005 or newer. If you have an older BIOS such as version 0902 or earlier, you have to upgrade the BIOS to 2005 or 2006 to use the 6200 series processors. To upgrade, download the latest BIOS from the ASUS KGPE-D16 motherboard download site, then extract the BIOS from the ZIP file and then copy it to a USB flash drive. Put the USB flash drive into one of the USB connectors on the PC and then reboot and enter the BIOS setup. Use the Tools menu to select the flash BIOS option and then upgrade the BIOS by picking the right file on the USB flash drive.

If you’re using another motherboard such as from Tyan or Supermicro, similar directions should apply but the BIOS versions and download locations are obviously different than from ASUS.

When buying a new G34 socket motherboard, if you don’t already have a 6100 series processor then you should ask your vendor to verify that the boards have a recent BIOS that supports the 6200 series processors. Otherwise, you are going to have to get your hands on a 6100 series processor to boot the board to be able to upgrade the BIOS. For people buying the ASUS KGPE-D16 motherboards last year, this was certainly a problem for some people at least through December 2011. But as of January 2012, it appears that new motherboards sold through vendors doing some volume of sales do have at least the required 2005 BIOS version.

Replacing the CPU

Next I did the following steps to replace the CPU:

1. Shut down the computer and turn it off at the power supply
2. Pop the computer case top
3. Unclip the two fans on the Noctua NH-U9DO processor cooler
4. Unscrew the NH-U9DO from the mounting point on top of the processor
5. Set the NH-U9DO aside where it won’t get thermal transfer compound on anything else
6. Scrape any excess thermal transfer compound off the metal CPU hold down bracket using a small screw driver
7. Pop the release lever on the CPU and removed the Opteron 6128, setting it aside in a safe place
8. Open up the Opteron 6274 box and place the chip in its socket, matching the positioning triangle on the chip with the triangle on the socket
9. Place the excess thermal compound on the top of the 6274 (the Noctua compound does not harden unlike others do) and squirt a little more on it
10. Put the Noctua cooler back on top of the processor (if you are using a NEW cooler, be sure to remove the plastic protective shield or else you may subject your CPU to 60+ degree Celsius temperatures even just by booting in the BIOS!)
11. First press down the CPU cooler to spread the thermal transfer compound some, then fully screw it into place
12. Clip the fans to the cooler, double-checking the airflow direction so that both fans are blowing the same direction (this is noted by little directional arrows on the fans)
13. Visually recheck everything to be sure the fans are plugged in and everything looks OK
14. Turn on the computer, check that the CPU fans are running. If not, turn off the computer immediately and correct the fan problem.
15. Boot into the BIOS, usually you have to press DEL to select BIOS setup.
16. In the BIOS, check the CPU temperature. If everything is OK, you should be seeing temperatures no higher than around 30 degree Celsius even in a comfortably warm room. (Generally I’m seeing temperatures in the 20s depending upon load and room temperature.)
17. If you saw temperatures much higher than about 30 degrees C, then shut off the computer and go back to step 3 and recheck the thermal transfer compound is OK by again removing the heat sink and verifying that it spread OK. Then follow the rest of the steps again except obviously don’t change out the CPU again.
18. If everything checked out, you can reinstall the case cover and then run longer with a regular software load while watching the CPU temperatures a bit more to verify that all is OK.

Performance and Power Comparison

For typical software that makes use mostly of integer instructions and only lightly uses floating point, the 6272 and 6274 are around twice as fast as the 6128. This is shown by the PassMark CPU testing chart for high-end CPUs where the 6272 is rated at 10,254 and the 6128 is rated at 5105.

Software that uses a lot of floating point instructions may not get as big of a speed boost, particularly if it has not been recompiled with a compiler that optimizes for Bulldozer’s new FPU architecure. One point of discussion about the new Bulldozer processors cores in the 6200 is that each pair of CPU cores shares a FPU. In the 6100 series, each core had its own FPU. The new 6200 FPUs are supposed to be considerably more capable with new 256-bit floating point instruction support and ability to split into executing two 128-bit floating point instructions streams, however fully utilizing these new abilities requires rebuilding code with a new compiler that optimizes for this architecture. If you’re writing your own software, that may be something you can do easily. If not, you may be stuck with older software that is not optimized for the 6200 series processors and will not run as fast as it could. It’s my opinion that this is what causes some of the gap between expected around 100% increase and observed around 67% increase in performance seen in some of the testing done below.

The following is a quick comparison on the performance and power usage between the computer with the 6128 processor and the upgraded version with the 6274. In general, the 6274 draws slightly less power than the 6128 at idle and slightly more at load. Performance goes up mainly due to being able to use more cores. If your typical software mix cannot make use of a lot of cores, you might be better off picking a chip like the 6212 that runs fewer cores at higher speeds.

The particular computer I did this upgrade on has a total of 9 hard disk drives running inside of it along with a AMD/ATI Radeon 6870 graphics card, 32GB of DDR3 ECC 1333MHz memory (four DIMMs), a PC Power and Cooling Silencer 950W Mk2 power supply, a HighPoint USB 3.0 four port RocketU 1144A card, an ASUS PIKE 2008 RAID controller, an ASUS MIO-888 audio card, and a LG WH12LS30 Blu-ray optical drive.

Power monitoring was done by visually watching a Kill-A-Watt power meter into which the computer power supply was the only device plugged in. My guess is there’s about a 1% to 2% margin of error on these measurements between the limitations of the meter and me watching it manually. Of note, I also tried monitoring the power usage with the power meter on a UPS backup but that didn’t seem as consistent.

With the main power off, this computer draws about 5 watts with the IPMI 2.0 hardware (ASUS ASMB4-iKVM) running with either processor. Given that the processors are turned off, that makes sense.

After the BIOS spins up the disks, with the 6128 it draws about 212W and with the 6274 about 210W.

After booting Ubuntu 11.10 and logging in with nothing else running but the OS, the 6128 version draws about 247W and the 6274 about 235W to 238W. What I noticed is that the power draw on the 6274 varies more than I saw when monitoring the 6128.

For software to put load on the CPUs and get some understanding of power consumption and performance differences, I picked the MPrime stress tester and a second program called systester. I monitored these with the Ubuntu System Monitor utility to verify that the expected number of processor cores were at running at 100% load.

For MPrime, I used version 2.66 built as a 64-bit version. I used the torture test, blend model, with 8 threads and for the 6128 saw power usage vary from about 353W to 356W. On the 6274, it varied from about 363W to 365W. But of course this wasn’t fully loaded on the 6274. When I ran it again with 16 threads then the power usage was about 371W to 379W. This program does a good job of burn-in exercising of processors, motherboard, and RAM to verify that everything is working right, so you might want to use it to stress-test a new or upgraded computer for a few hours to a day before putting it into production use.

For systester, I used version 1.4.0 built for x86_64 set to compute Pi to 16 million digits. It’s my understanding that this program using a lot of floating point math which is reputed to be one of the weaker points of the 6200 series processors because a single FPU is shared between pairs of cores whereas on the 6100 series each core had its own FPU.

Here’s a table of the results:

The performance boost shown in the systester output is about a 67% speed bump from the 6128 when all cores are heavily loaded. As shown by the one thread test, each individual core is slightly faster. Each pair of cores on the two thread test seems to be about the same speed. In general on this test, the more threads you have running, the bigger the performance boost.

Overall, upgrading from a 6128 to a 6274 gets you a significant jump in CPU performance with a small bump in power usage at load. Even if you’re just using your computer for typical desktop applications, I think you’ll notice that the additional cores do improve the multitasking performance.

On a 6128, what I’ve noticed for typical (i.e., not benchmark or stress test) use is that you seldom see more than a couple of cores running at 100% but you often do see every one of the cores being used for extended periods of time. With similar workload on the 6274, you very seldom see 100% load on any core and you do see usually one or two cores running at 0% plus a few more at less than 10%. The computer running with the 6274 should be able to take on some more work without much impact on other tasks already running.

Adding A Second CPU

Adding a second CPU is much like the steps above, but you don’t have to remove an existing CPU and watch out for the thermal transfer compound mess on it. If you need another CPU cooler, I’ve found that usually the least expensive source for the Noctua NH-U9DO A3 is AlwaysLowest. Usually they are around $46 per CPU cooler, but they do charge for shipping. Amazon is sometimes competitive with this, but Newegg is usually considerably more expensive.

References

CPU Based Simple System Stability and Benchmark Tester for Ubuntu Linux – systester

Linux Stress Testing and Benchmarking

This creates a new problem, however, as the usual means (e.g., “Disk Utility”) for checking for drive temperatures and errors won’t work. That’s because the OS doesn’t see the individual drives as it usually would as they are being hidden behind the RAID controller. The “virtual drive” created by the RAID controller generally doesn’t implement SMART reporting features for getting information on the drives such as errors and temperatures that you should be watching to minimize the possibility of an impending drive failure taking you by surprise.

I like to use LSI based RAID controllers, even if I’m just using using them for a bunch of SATA or SAS ports and letting the OS do software RAID. That’s because LSI has done a good job of ensuring that drivers are available for most operating systems. Ubuntu’s kernel has included LSI adapter drivers for some time. They have drivers available for several Linux variants, FreeBSD, Windows, and even Solaris and OpenIndiana.

You also see LSI RAID chips included on a number of motherboard products or add-in cards for server and workstation motherboards from Supermicro, ASUS, and others. Intel, IBM, and many other companies are also using LSI chips for their RAID adapters. As a result of so many companies picking LSI RAID chips and ensuring drivers are available for their systems, LSI is one of the safer bets when it comes to getting driver support for a RAID card.

The following example uses a computer on which I’ve got Ubuntu 11.10 installed with a non-RAID hard drive, a USB flash drive, some USB attached memory card reader ports, an optical drive, and a couple of RAID 10 arrays on an LSI SAS2008 based controller using the mpt2sas driver included in the kernel. I suspect most the following will also work with other RAID controllers, too, but it’s always possible that some of them do not implement the features that smartmontools needs to access the SMART data for each drive.

First, install the “smartmontools” package. You can install it like this:

sudo apt-get install smartmontools

Next, you need to identify what your hard drives are. On Ubuntu Linux, generally you can find a series of /dev/sg* names that correspond to the device names for the hard drives. Simply “ls /dev/sg*” and use that as a list of attached storage devices. Note that these sg* (SCSI Generic) names are not the same as the more commonly used /dev/sd* naming pattern you see in a lot of other articles. That’s important because I’ve found that the sg* names let you get to the drives behind the RAID controller to check their status even though there is no sd* name for the individual drives.

For each sg* device, run:

sudo smartctl -x 

where you substitute actual device name for like this:

sudo smartctl -x /dev/sg4

You’ll get some output that will give you an idea what the device is, usually including at least a manufacturer name and model number.

Some of these names correspond to devices such as optical drives (CD, DVD, Blu-ray), memory cards, RAID arrays, or other devices that may not implement the SMART command set. When that happens, you may get output that looks like this output I got from “smartctl -x /dev/sg5″:

smartctl 5.41 2011-06-09 r3365 [x86_64-linux-3.0.0-15-generic] (local build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net

Vendor:               LSI
Product:              Logical Volume
Revision:             3000
User Capacity:        5,999,998,009,344 bytes [5.99 TB]
Logical block size:   512 bytes
Logical Unit id:      0x600508e000000000ae355076e8101a07
Device type:          disk
Local Time is:        Mon Jan 30 02:29:09 2012 PST
Device does not support SMART

Error Counter logging not supported
Device does not support Self Test logging
Device does not support Background scan results logging
scsiPrintSasPhy Log Sense Failed [unsupported scsi opcode]

When this happens, it’s just a note that this device doesn’t support SMART. In the case of the particular RAID array denoted by /dev/sg5, there are four more entries /dev/sg7, /dev/sg8, /dev/sg9, and /dev/sg10 that match up with the drives in the array. The array denoted by /dev/sg6 also has entries /dev/sg11, /dev/sg12, /dev/sg13, and /dev/sg14.

When I execute “smartctl -x /dev/sg7″ the output is much more useful and dumps all the data for the hard drive. (It is hundreds of lines of output, so I’m not going to include it here.)

When a drive supports SMART, you should see a few lines in the output about temperature that look like this:

Current Temperature:                    24 Celsius
Power Cycle Min/Max Temperature:     24/26 Celsius
Lifetime    Min/Max Temperature:     24/33 Celsius
Under/Over Temperature Limit Count:   0/0
SCT Temperature History Version:     2
Temperature Sampling Period:         1 minute
Temperature Logging Interval:        1 minute
Min/Max recommended Temperature:      0/60 Celsius
Min/Max Temperature Limit:           -41/85 Celsius
Temperature History Size (Index):    478 (339)

There are also sections for serial number, firmware revision, power on hours, error logs, error counters, temperature histories, and more. Figure that most drives which implement SMART are going to have 200+ lines of output each, so interpreting the information will take some time. It’s best to save these to text files and inspect them carefully until you understand what the information means and what is important to you.

Here’s a simple script to dump available SMART data for all the devices:

#!/bin/bash
for drive in `ls /dev/sg*`
do
    echo "-----"
    echo "Command: smartctl -x $drive"
    smartctl -x $drive
    echo "====="
    echo
done

Once you know what devices actually do report SMART information, you could easily change the “for drive in” line of that script to list only those particular devices for which you want output instead of using every one of the available /dev/sg* names. For example, something like this would list output for the four drive in one of the RAID 10 arrays on this computer:

for drive in /dev/sg7 /dev/sg8 /dev/sg9 /dev/sg10

The key problem I ran into when trying to apply the smartmontools documentation to systems with LSI RAID controllers is that the drive names in the /dev/sd* or /dev/disk/* patterns simply do not work to get behind the RAID array controller at individual drives.

It may help you a bit to understand this on your computer by installing the “sg3-utils” package and then running “sg_map” to dump out a mapping between the sd* names and the sg* names. As you can see from the example below, the computer I’m using to write this article has eight disks sg7 to sg14 that don’t map to the usual names. Those are the hard drives behind the RAID controller.

# sudo apt-get install sg3-utils

# sg_map
/dev/sg0  /dev/sda
/dev/sg1  /dev/scd0
/dev/sg2  /dev/sdb
/dev/sg3  /dev/sdc
/dev/sg4  /dev/sdd
/dev/sg5  /dev/sde
/dev/sg6  /dev/sdf
/dev/sg7
/dev/sg8
/dev/sg9
/dev/sg10
/dev/sg11
/dev/sg12
/dev/sg13
/dev/sg14

Once you’ve got a handle on the names of the drives behind the RAID controller, then the following documentation in the “Further Reading” section below should be useful for figuring out what else you can do.

Smartmontools is more powerful that just a tool to report current hard drive status. You can also use it to run self-tests on a hard drive. Precisely what it can do depends on what SMART features a particular drive supports.

Note that if you try anything other than simply dumping SMART status information, you’re running a risk of something going wrong. For instance, if you tell a drive in a RAID array to run a self-test, it could cause the RAID controller to drop the drive from the array and switch to degraded mode.

As for me, I’d avoid trying this out on a production system during a time when it needs to be available within a couple of days as some RAID rebuilds can take a very long time when you cause a drive to drop out.

Further Reading

Smartmontools introdution

Ubuntu man page for smartctl

Smartmontools FAQ

I don’t know how long they are offering this combo, so you might want to check it out by clicking to Combo Deal ASUS KGPE-D16 and AMD Opteron 6272 Interlagos 2.1GHz Socket G34 115W 16-Core Server Processor OS6272WKTGGGUWOF and see if they are still offering for $819.98 which is $150 off their regular prices.

If you think spending another $110 for a 5% speed boost is worthwhile, check out the Combo Deal ASUS KGPE-D16 and AMD Opteron 6274 Interlagos 2.2GHz Socket G34 115W 16-Core Server Processor OS6274WKTGGGUWOF combo offer and see if they are still offering it for $929.98 which is also $150 off their regular prices.

They also have a number of other combo deals with the ASUS KGPE-D16 motherboard including a $35 off offer when bundled with an AMD Opteron 6128. The Opteron 6128 is an 8 core 2.0 GHz processor that is probably the lowest priced 8 core server processor out there and still performs well compared to the newer 6200 series chips.

Both this motherboard and these processors are a great choice for a workstation or small to large server builds with a lot of growth potential. You can start with a single processor and a single inexpensive 8GB DDR3 DIMM and upgrade over time to dual processors and 128GB RAM or up to 256GB RAM if you buy 16GB DIMMs.

If you are looking to build a system using new SSDs or a four to eight drive RAID 10 array of hard drives, the ASUS PIKE 2008 RAID card using the LSI SAS2008 chip offers eight 6 Gbps SATA3 or SAS2 ports for around $140-150. It plugs into a dedicated slot on the motherboard that gives it a PCI Express x4 interface good for total disk I/O throughput of about 1.4 gigabytes per second. That’s enough bandwidth for making good use of a pair of fast SSDs for OS and program storage along with a RAID 10 disk array using four drives for your data files.

If you need something more that eight fast disk ports, you can step up to a faster LSI-based SATA/SAS board such as the LSI 9211-8i or IBM M1015 that are similar to the PIKE 2008 but offer a PCI Express x8 interface that is twice as fast. Cards like these and even more powerful siblings such as the LSI 9265-8i (which has 1GB DDR3 cache RAM, writeback caching, a battery-backup interface, RAID 0, 1, 5, 6, 10, 50, and 60 arrays, optional SSD caches for RAID arrays, etc.) can be used with SAS port expanders to control dozens of SSDs and hard drives (specs often say up to 128, but you’d have to add a lot of SAS port expanders to do that. That makes them powerful enough for even some very substantial server configurations with dozens of terabytes of disk storage.

Further Reading

Choosing ECC DIMMs for AMD Opteron Workstation or Server

AMD’s New Bulldozer FX and Opteron Processors Offer Affordable High Memory Capacity and ECC

Unbuffered vs. Registered Memory

First of all, you need to determine whether your motherboard takes unbuffered or registered memory. Many server and workstations boards can take both, but some only work with one or the other. Many (perhaps most) desktop PC motherboards only work with unbuffered memory.

What’s the tradeoff between unbuffered and registered memory? Usually unbuffered memory is a little less expensive and a little faster but will not allow as many DIMMs to be used. This is why server and workstation boards favor registered memory as they often have more than the usual two to four DIMM sockets common on desktop motherboards.

Unbuffered memory is generally a little less expensive only for 4GB and smaller DIMMs. For 8GB and 16GB DIMMs, as of late 2011 and early 2012 often registered DIMMs are far less expensive and much more readily available. This may change as 8GB unbuffered DIMMs become more common due to Intel’s low-end E3 server chip line that maxes out at four modules of 8GB each.

DIMM Rank

DIMM rank is the next thing to check. Quad rank DIMMs are often the least expensive, but if you install more than one per memory channel you will see the memory bus speed dropped one or more speed levels. For instance, if you install quad rank Kingston 8GB 240-Pin DDR3 SDRAM DDR3 1333 ECC Registered model KVR1333D3Q8R9S/8G DIMMs on an ASUS KGPE-D16 motherboard, they will run at 1333MHz with one per memory channel. That can get you up to 32GB RAM per processor. But if you want to add a second DIMM per channel, perhaps to expand to 64GB RAM on a processor, then these DIMMs will only run at 1066MHz instead of 1333MHz. Memory bandwidth will probably still be comparable because the memory controller will interleave between DIMMs on the same channel.

But if you used single or dual rank DIMMs instead of the quad rank ones in the example above, you would probably see the memory bus still running at 1333MHz.

It is possible to get dual rank server grade registered DDR3 ECC DIMMs all the way up to 16GB size today. Samsung SDRAM, which is often one of the more expensive brands, is often used on such DIMMs. The 1333MHz speed grade is most common right now, but 1600MHz has recently become available.

8GB dual rank DIMMs with chips from Hynix, Elpida, Micron, and Samsung are very easy to find these days. These are probably the best option for most people building a small to medium sized server.

Voltage

Most DIMMs suitable for use with the Opteron 4100, 4200, 6100, and 6200 series processors are 1.5V. But these processors also support 1.2V and 1.35V memory and so do some of the motherboards that runs these processors.

Usually the lower voltage memory runs a little cooler and uses less power but is initially more expensive. For computers powered on 7/24, lower voltage memory may be advantageous.

There are a few DIMMs that are rated to multiple voltages. Samsung makes DIMMs from its D-series DDR3 chips that can run at 1.35V and 1.5V so you can use them with a wide variety of motherboards.

Profile or Height

Low profile DIMMs that don’t have large vertical heights from circuit boards or heat spreaders may be needed in some applications because of the overhang from CPU heatsink/fan assemblies.

Recommendations

I’d suggest you buy 8GB DDR3 ECC DIMMs running at 1600MHz for most new builds using AMD Opterons in the 4200 and 6200 series chips that support 1600MHz memory. 8GB DIMMs are probably the most cost effective memory versus both 4GB and 16GB DIMMs. Plus they give you a big system RAM capacity range from a single processor dual channel system with 16GB RAM all the way up to quad processor 16 channel systems with 256GB RAM. That covers most of the market pretty well.

If you know you are going to eventually need huge amounts of RAM, then consider the 16GB DIMMs. But be aware as of early 2012 you’ll probably be paying two or more times the price per gigabyte for them.

Where to Buy

Newegg is often a great place to buy hardware if you don’t have a resale license with relationships with distributors. Unfortunately, they are behind the curve on 1600MHz server DIMMs and still are not carrying any of them months after other vendors have started.

Superbiiz often has a few 8GB and sometimes also 4GB and 16GB DDR3 1600MHz DIMMs in stock. Initial orders are a bit of a hassle due to their anti-fraud policies. Also, sometimes they are sell out of products before they update their web site and then you’re stuck with a backorder. So if you’re buying memory for a new build or customer site installation during scheduled downtime, try to order a few days to a week earlier to help cover for these possibilities.

There’s also Ebay as I’ve found the faster DIMMs there, too, but personally am not as comfortable about these being brand-new high quality DIMMs given how a lot of supposedly “new” hardware on Ebay is coming out of server pulls and other sources that might leave you hanging with no return recourse if it doesn’t work at all or for long.

Further Reading

AMD’s New Bulldozer FX and Opteron Processors Offer Affordable High Memory Capacity and ECC

Why You Should Use ECC Memory In Your Next Computer

Gigabit Ethernet Usually A Nice Performance Boost

Gigabit Ethernet offers 10 times the performance of Fast Ethernet. But you can often get an additional 10% or more performance boost by turning on jumbo frames when your network hardware supports its.

So I turned on jumbo frame support for the NICs on each computer as the switches on the network all support jumbo frames. On most of the computers I got a nice performance boost for big file transfers. Fast Ethernet tops out around 10 megabytes per second transfer for Samba file copies. File copying via Gigabit Ethernet, at least on this network and server, runs from about seven to twelve times faster depending upon the particular client computer, driver settings, etc.

But one of the computers running Windows Vista 32-bit with a Realtek Gigabit Ethernet PCI network interface card no longer worked well on the network. Trying to access network shares produced strange pauses and very slow performance.

Since there’s another Windows Vista 32-bit computer on the same network with the same NIC that did not have these problems, my first inclination was that it must not be a hardware incompatibility or driver problem.

Weirdly, the Samba file sharing seemed to be messed up much worse for some shares than others even though all the shares were on the same Linux server. Certain shares could not be accessed at all. The PC’s file explorer would virtually hang for more than a minute. Other shares would work so that browsing directories was fine, fooling me into thinking this was a security issue such as wrong permissions or passwords somewhere.

After examining the server settings, I could not find any explanation for why the permissions or passwords would be a problem. Further experimenting, I noticed shares which were a problem on this PC worked OK from other PCs with the same user and password for the network access.

Next, I noticed when accessing the Linux server for Linux desktop sessions via Free NX Server and NX Client on this PC, the NX logins timed out and failed. However, it worked just the same as usual for Internet access via a web browser.

When I tried to verify the shares on which directory browsing was working on this PC were in fact functional, I found that even file copies of small files were very slow, taking half a minute or more to copy files around a megabyte in size.

Clearly something was really wrong as even Fast Ethernet should be far quicker than this.

Jumbo Frame Network Driver Problems

As it turns out, the security settings such as ownership of files, permissions, and passwords were not the problem. The trouble came from how this particular combination of network card and driver apparently do not work well when jumbo frames are enabled. The network card is a TrendNET model TEG-PCITXR hardware version 3.0 using a Realtek PCI 32-bit Gigabit Ethernet Controller model RTL8169SC. The driver is the Realtek PCI GBE Family Controller version 6.241.623.2010 dated June 23, 2010.

After some searching, I found there’s a newer version of the Vista driver version 6.25 dated December 1, 2011. I downloaded it from the Realtek website as Microsoft’s automatic driver update didn’t find it. After installation, in the network adapter properties it identifies itself as version 6.250.908.2011 dated September 8, 2011. I turned jumbo frames 7K back on (this NIC only supports up to 7K, the rest of the network hardware support 9K frames) and it seemed to be working OK without the weird pauses, hangs, and slow performance. To be sure, I rebooted the computer and checked the settings again. Still fine and still using 7K jumbo frames.

Why Did The Other Computer’s Networking Not Hang and Slow Down?

I went back to the other Vista 32-bit computer to check its driver. It had the old version and its jumbo frames were set to 7K. So why was it working? Trying some file copying from the server, I noticed that it was not running as fast as it should be copying large test files. It was acting more like it was running Fast Ethernet rather than Gigabit Ethernet. Sure enough, for some reason the link negotiation for this NIC was setting it to run at 100 Mbps rather than 1 Gbps. That would turn off jumbo frames, too, as jumbo frames were not supported on Fast Ethernet.

So I updated the driver and proceeded to try to force the link negotation to 1 Gbps. Still it runs only at Fast Ethernet speed. Very strange as the switch port is gigabit and the cable is category 6 wire, the same as another cable for a nearby computer running Windows 7 that is running gigabit speed just fine. This computer isn’t heavily used with large files on the network so I’ll get around to troubleshooting this network problem later as nobody is likely to notice the slower speed for now.

The lesson to this experience is that turning on jumbo frames on your network may require a bit of testing and troubleshooting to verify that everything is indeed operating correctly. Don’t expect it to be a slam-dunk change even if you have all jumbo frames capable network hardware. If you see weird behavior, try backing out the jumbo frames on the computers with the problems to see if that helps. If it does, try a newer network interface driver and don’t depend upon Windows Update to find it for you. If that doesn’t help, you’re probably best just sticking with regular size frames on those particular computers.

Further Reading

The Promise and Peril of Jumbo Frames

Jumbo Frame Clean Networking Gear

Previous versions of the Weaver theme displayed a simple site layout such as a fixed-width right sidebar with wider left content area decently on a mobile device such as an Android phone. Mobile phone browsers do a pretty decent job dealing with this kind of format, probably because it is really common.

Weaver II version 1.0.12 includes some new mobile support features presumably intended to make sites look better on mobile devices. Unfortunately, my experience so far has been that the typical site using a layout that is designed to look good on a PC screen around 1024 pixels in width ends up horribly mangled on a tiny mobile device screen. Typically you see severely clipped graphics and ads on the sidebar(s). The main content text sometimes gets turned into a vertical line of thousands of characters or a column of single words. Such a mess makes a site virtually unusable at worst and at best highly unpleasant to view.

It looks like this problem was introduced by using CSS max-width for the page wrapper rather than a fixed width. The intent is probably to allow content to flow nicely on smaller screens, but for sites with sidebars it can be a big problem. That’s particularly true for sites with images such as ads or widgets such as those from Facebook in the sidebars.

Fortunately there’s a simple workaround. I’m not bold enough to claim this is the best way to solve the problem, but it is a vast improvement over the default appearance from max-width.

First, open up the WordPress Appearance -> Weaver II Admin page to find the site layout width you configured. The default is 940 pixels, but a lot of people increase that to a bit more somewhere around 1000 to 1020 pixels to be able to fit 250 pixel to 336 pixel wide ad blocks from ad networks. You can find the width setting on the “Main Options” page under the “Layout” tab where it says “Theme Width”.

You’ll notice there’s also a “Theme Fixed Width” option right underneath that field. I tried checking that first, but found it doesn’t work well for some sites on which the page theme width has been changed upwards. For starters, it doesn’t seem to respect the width in the “Theme Width” field and drops back to the default. Maybe this is a bug? If so, hopefully it will be fixed in a future version.

Now switch to the Weaver II “Advanced options” page. Select the “ Section” tab and click on the multi-line entry field under “Custom CSS Rules”. Now simply add a line to force the page wrapper to be a fixed width:

#wrapper { width: 940px; }

Make the width match whatever you used for your theme width. Save the settings. Then refresh your web site page on both your PC and phone to check it out.

On some sites I’m working on updating to use Weaver II, this made them usable again on an Android mobile phone.

If your format can take some resizing of width, there are min-width and max-width options that may help. For example, if you want to both prevent the sidebar from crowding out the main content section but also need to keep it from getting so small it clips out the right edge of ad graphics, you might try playing around with the min-width and max-width settings something like this:

#wrapper { min-width: 800px; max-width: 1020px; }
#sidebar_primary { min-width: 342px; }

That allows some resizing but keeps sections of the page from getting so narrow when displayed on a phone that they are useless. However, if you’ve got banner adds that take up most of the width of your site’s page, you may find they run off the right edge of the page and the formatting starts looking wonky. In that case, it might be best to stick with a fixed width or to increase the min-width value until the appearance is tolerable.

If you’re in the midst of writing a document or program, the interruption of your computer shutting off can take you even half an hour or more to recover your place. If your business depends upon a computer for completing sales orders or you’re running a web server used by your customers and clients, the inconvenience and aggravation it can cause for many people can result in a loss of sales.

Knowing of these problems, a lot of people buy a UPS for their computers as they see inexpensive sub-$100 units in stores and buy them on impulse. Unfortunately, having such a cheap UPS hooked up to your computer isn’t enough any more to reasonably ensure freedom from hassles due to minor power glitches.

Square Wave Power Outputs Don’t Work Well With Newer Power Supplies

It used to be that pretty much any cheap UPS was enough to carry a computer through a few second outage, so long as you sized the UPS to be enough to meet the power requirements. Today, that cheap UPS may give you very little protection from power outages thanks to new EnergyStar high efficiency power supplies being used widely in today’s computers.

With the introduction of a power supply efficiency feature known as Active Power Factor Correction (PFC), the cheap UPS may not be reliable any longer. That’s because when a cheap UPS loses line power, it switches to battery and starts generating a square wave power signal that leaves a brief but significant period many times per second where there is no power being output to the computer power supply. Computer power supplies have capacitors that can cover for this, but with the PFC feature they start to try to draw more power from the battery backup unit when they see there is no power on the line. The result is that the battery backup may trip its circuit breaker sending the whole load crashing down with a power failure. That’s exactly what you wanted to avoid.

EnergyStar labelling requirements have encouraged more computers makers to switch to using PFC power supplies. It’s great for energy efficiency, but not so great when your UPS may fail to keep the computer running when there’s even a few second long blackout or brownout.

Alternative Waves to the Rescue

For decades there has been a solution for this problem. The solution is the sine wave power output UPS. It works great with PFC power supplies. The big problem with it is the price. They can be a couple hundred dollars or more than similar sized inexpensive square wave output UPS units.

In 2010, CyberPower introduced a line of products called their “Adaptive Sinewave UPS” that aims to solve the PFC compatibility problem at lower expense. When on battery, these units output power waves that are essentially “stepped sine waves” that better approximate a pure sine wave but at a much lower cost than pure sine wave hardware. The cost is around $20 to $30 more than similar sized units with square wave power output. This solves the incompability with the PFC power supplies cropping up in many newer computers and manages to keep the cost down, too.

For the last several months, I’ve been using two different models from CyberPower and have been satisfied with their price and performance. For the mix of desktop computers I’ve been supporting, I have found that their 510 watt model CP850PFCLCD and 900 watt model CP1500PFCLCD are good choices. The 510 watt tower units cover most smaller computers with their 300 to 400 watt power supplies plus a power-efficient LCD monitor. The 900 watt tower units can cover larger units, even some servers that don’t need long runtimes on battery.

Both units use the same size batteries which is handy since you will need to periodically replace the lead acid batteries every few years to keep the units working. How can they use the same batteries? Simple, the smaller unit takes one battery and the larger unit takes two. Batteries compatible with the HR1234W model batteries used by these two units cost around $20 each online. You can estimate that you’ll need to replace them every two or three years depending upon how often they are called upon to power your system.

CyberPower is also selling models with 600, 810, and 1320 watt capacities. Unfortunately they do not use the same size batteries as the 510 and 900 watt models. The 600 and 1320 watt models use the same size, quantity 1 and 4 respectively. The 810 watt model uses yet another size. So if you need these other sizes then you may be stuck having to order multiple different replacement batteries which can complicate your maintenance work.

All the sizes except the 900 and 1320 watt are available only as tower units. The 900 is available as both tower and rackmount. The 1320 is rackmount only and is available in both 2000 VA and 2200 VA variants. The 900 watt tower unit is about half the price of the 900 watt rackmount, so try to avoid the rackmount units if space is available under or behind a desk and you don’t need the approximately twice as long runtime provided by the larger number of batteries.

To give you a quick idea of current competitive pricing for these units at their various sizes, I listed one unit at each power capacity as priced by Amazon on the right.

The tower units come with two USB connectors on the front that you can use for charging up your phone or other portable device. It’s handy as it means you can fully turn off your computer and USB hubs when you’re not using them and still conveniently get a charge.

Surge protection for coax TV and phone or Ethernet (RJ45) jacks is also included. Given how it’s possible to fry some of the electronics in a computer through its network port via a power surge or failing network gear, that’s a nice bonus. The network jack surge filters work just fine with typical 100 Mbps and gigabit Ethernet networking gear.

On their front panel display, they can show battery charge, current power usage, and estimated time remaining on battery. The power usage gauge is not as sensitive or accurate as dedicated power measurement devices such as the P3 International P4400 Kill A Watt electricity usage monitor, but it does give a rough estimate that is good enough to understand how much power your system is drawing so you can be sure to not plug in too many power-hungry devices into the UPS battery-backed and surge protected outlet banks.

Automatic shutdown of Windows, Mac, and Linux systems is supported for those who want their computers to automatically power off before battery exhaustion. There are some mixed reviews about compatibility with the latest Mac OS X versions, so if you’re a Mac user you should investigate further if this feature matters to you. Personally I think the common case is power failures that are under a minute in length, so even if the automatic shutdown doesn’t work there’s still a big gain to be had in overall reliability and uptime on computers protected by a UPS.

All these units can do “automatic voltage regulation” that helps to correct for variations from normal 115V to 120V line power without having to switch to battery operation. Many cheap UPS units do not have this feature and so immediately switch to battery if the line voltage drops a small amount (even 10 volts) as may happen on a hot summer day when nearby air conditioning equipment is switching on or the overall power grid is under high load.

Another nice point about these CyberPower units is that they are more energy efficient than most of the competition. A UPS draws some power of its own, both to run little indicator lights and fans and also a side effect of charging inefficiencies. CyberPower runs the fans in the units only when needed and shuts off displays to save power when nobody is playing with the front panel controls.

The big downside to the CyberPower tower units is that they have relatively short on-battery run times. That’s due to their use of smaller batteries than the more expensive competition or their rackmount bigger siblings.

For more information, tests, and comparisons of the CyberPower 900 watt tower unit, see Tom’s Hardware Comparison of Four APC, CyberPower, Opti-UPS, and TrippLite UPS Units

Why You Need ECC Memory

Many desktop motherboards are now supporting 32GB RAM. Many server motherboards are supporting 128GB RAM or more per processor. So for a dual to quad processor mainboard you could be looking at 256GB to 1TB of RAM. That’s hard disk drive territory in terms of capacity. As I discussed previously in Why You Should Use ECC Memory In Your Next Computer, with that much memory it’s likely on many computers a week won’t go by without a bit flipping randomly somewhere in memory. So you really must have ECC when you’re talking about that level of RAM if you intend to avoid weird problems such as random crashes from memory errors.

It used to be that 8GB or more RAM would be fantastically expensive. But today you can pick up 16GB of desktop grade non-ECC memory for less than $100. 16GB of server grade ECC memory costs somewhere around $150.

32GB of server registered SDRAM with ECC protection runs around $300 to $400. That is less than the price of many mid-range SSDs which are all the rage these days for speeding up a slow computer. But without ECC protection against stray bit errors, it’s simply quite risky to be using such large amounts of SDRAM.

What’s particularly nice about the 8GB and larger registered ECC DIMMs is that they are much less expensive than their similar capacity unbuffered SDRAM DIMM counterparts, especially the few unbuffered 8GB and larger DIMMs available with ECC. Getting 32GB of unbuffered SDRAM with ECC as four sticks of 8GB each will probably cost you upwards of $600, maybe even over $1000. The $300 or more in savings from using the registered DIMMs is often enough to pay the added cost of a better server motherboard, processor, case, and power supply versus the commodity equivalents that only use unbuffered memory.

Intel’s Poor Design Choices

Intel, the leading maker of desktop and laptop PC processors, has chosen to cripple its Core i3/i5/i7 desktop and mobile processors and E3 low-end single processor server chips with inferior memory controllers. It appears their motivation is to drive anybody who cares about computers being reliable and needs lots of RAM to pop for their E5 and higher server chips. For Intel that’s a lucrative market as such processors typically cost around $500 to $2000 each. In the case of the really high end Intel E7 server processors, it could be even more than that.

Today’s Intel desktop and mobile processor chips lack support for ECC memory entirely, meaning that any stray bit of radiation or noise in the memory can bring your whole computer crashing down without warning. If you’re just playing games on your PC or watching YouTube videos, maybe that is tolerable. But if you are using your PC for your work, it’s probably not.

Fortunately for Intel, they have a convenient scapegoat — Microsoft. They can just blame weird behaviors and crashes on Windows because so many easily believe that Windows is unstable crap. The problem with this is that Windows 7 actually is quite stable at this point. Sometimes there’s bad drivers or firmware that causes such crashes. But the odds are high that a significant number of the BSODs (Blue Screen of Deaths) that Windows users are seeing are coming from broken or unreliable hardware, particularly SDRAM memory. SDRAM DIMMs that experience random bit flips or electrical noise that corrupts a bit being read or written can trigger crashes or even corrupt data on your hard drive or SSD.

The low-end E3 server chips are very much like the Core i7 desktop processors as they use the same Sandy Bridge processor cores. In the E3 entry server chips, Intel did include support for basic ECC that can correct for single-bit memory errors. Unfortunately, they only support unbuffered memory modules. That means at the moment anything more than 16GB of RAM demands a ridiculous surcharge if you want to use ECC RAM in a server or workstation using an Intel E3 processor.

Unbuffered ECC DIMMS cost typically 50% to 300% more than their registered server grade ECC counterparts at the 8GB size plus they are very hard to find. A few months back, the best prices for such unbuffered 8GB ECC DIMMs were around $400 a crack while the registered 8GB ECC DIMMs, which won’t work with Intel E3 processors, were less than $100 from many vendors. Today the situation isn’t quite as bad, the prices are down to around $150 to $300 versus around $80 to $100 for many registered ECC DIMMs of the same 8GB capacity. Still, that is a hefty price penalty. It’s one of the major reasons why anybody expecting to need or want more than 16GB RAM should be looking at some other processor and motherboard options.

AMD’s FX and Opteron Processors Offer Good “Bang for the Buck”

So you might think you’d simply pay more for a better Intel processor such as an E5 and get a corresponding motherboard, too. You could use the savings from the registered RAM to pay for part of the cost difference.

That’s a great idea, but you might be better off buying AMD processor chips instead of Intel. As has been the case for years, AMD’s “performance for the dollar” ratio is often better than Intel’s.

AMD’s new Bulldozer processor cores introduced in the second half of 2011 have ECC support as incorporated in AMD’s desktop line of Zambezi CPU chips (FX-8120 and FX-8150). If you pick a good motherboard, you can use them with unbuffered ECC memory. There are several reports that the ASUS Crosshair V AM3+ motherboard supported unbuffered ECC just fine. But again, that’s unbuffered RAM and you are going to pay a premium for that. Figure for 32GB RAM, you will pay at least $250 to $500 more than if you could buy registered RAM of the same size.

Oddly, at the moment if you go that route then you’ll be stuck using slower 1333MHz (PC10600) memory versus the 1600MHz (PC12800) that is starting to become widely available for registered ECC DIMMs. Motherboards for AMD FX processors often support overclocked memory at 1866MHz or higher, but unbuffered ECC DIMMs right now are almost impossible to find at any speeds faster than 1333MHz.

AMD’s Opteron 4100 and 6100 series processors have supported ECC memory for the past couple of years. The cheapest processors from that group are around $100. The cheapest mainboards are around $200 to $250. These will work fine with DDR3 ECC registered DIMMs running at up to 1333MHz. For 8GB DIMMs, the price each with such specs is around $70 to $100 depending upon where you buy.

If you are buying new hardware today, I’d suggest you consider AMD’s Opteron 4200 and 6200 series processors. They support memory speeds up to 1600MHz. They often cost less than a comparable performance 4100 or 6100 series processor, often much less. Right now they only provide a small performance boost at the moment versus similar numbered older processors (e.g., Opteron 6272 versus 6172), but that will probably turn into a moderate boost after more programs are recompiled with new compilers to support the advanced instructions the chips include.

Bad Press Undeserved

AMD has recently taken a beating in the press for lower than expected performance of its new Bulldozer cores in its “Zambezi” FX desktop processors and “Valencia” Opteron 4200 and “Interlagos” Opteron 6200 server chips. Yes, they are slower than the very fastest Intel chips. That’s in part because to get top speed out of these new chips, software needs to be recompiled which will happen over time.

Even if you only get the current performance out of these chips, frankly does a 10% to 20% processor speed difference really matter that much for most uses these days? Do you want to pay $1000 for a somewhat faster Intel server CPU or $500-600 for an AMD CPU that is almost as fast? AMD seems to be winning the current “bang for the buck” contest with Intel.

Intel Has Edge on Idle Power Consumption

However, I do think it is only fair to Intel to note they have gotten really good at lowering CPU power usage so over the course of several years it is possible that Intel’s higher prices may be offset substantially by savings in electricity bills. However, that probably depends in part upon how the computer is used. If it is running at a high load all day long, the power savings may simply not be major. Intel’s power savings seem to be highest when the processor is the least loaded.

AMD’s ECC Often Better Than Intel’s

Even on chips where Intel has implemented ECC, such as the E3 line, their ECC protection often doesn’t measure up to AMD’s. Intel usually puts in just simple single bit error correction circuits. While that helps a lot, it doesn’t come close to the benefits of AMD’s “chipkill” ECC implementation found in their Opteron processors. This ECC algorithm can correct for even a flakey memory chip that has cross-bit leakage because it spreads the ECC bits around the DIMM so that even a single totally failed chip cannot crash the system.

AMD didn’t invent the “chipkill” algorithm — credit goes to IBM decades back. But AMD is due credit for putting it into relatively inexpensive server processors.

Consider Overall System Balance

Often the CPU is no longer the bottleneck in a workstation or server. Inadequate RAM or a slow disk system is far more often an obstacle to performance. It’s easy to completely fill the RAM memory even on a desktop single-user PC with 8GB RAM. Simply open up a few dozen web pages and a few other typical programs (word processor, spreadsheet, etc.) and your PC will be busily paging code and data to and from your hard disk drive. For such a PC, often the cheapest and most significant performance upgrade is to add more RAM. But that’s precisely what it is difficult and expensive to do with Intel’s current desktop CPU offerings once you pass about 16GB RAM.

If you’re lucky, you might have a SSD (Solid State Drive) which is way faster at random access virtual memory paging than a mechanical hard drive. But still no mass storage device today beats the performance boost of simply adding more RAM to a PC that doesn’t have enough. And if you make the mistake of having too little RAM and putting your virtual memory paging or swap file on your SSD, you will wear that expensive SSD out a lot faster.

So I’d argue that a lot of folks building workstations or servers today should be giving the AMD Opteron processors serious consideration for the reasons discussed above. What’s particularly neat is that virtually every one of the Opteron 4100 series mainboards will support Opteron 4200 chips with just a BIOS update. The sames goes for most mainboards that took 6100 chips — upgrade the BIOS and they work with the 6200 series.

Single-Threaded Software Favors High Clock Speeds, Newer Multithreaded Software Favors More CPU Cores

If you’re running a lot of single-threaded desktop PC software, this is where the Opteron 4200 series may have the advantage over the 6200 series. The 4200 series chips are basically half a 6200 series chip. Because of power and thermal considerations, the 4200 series with half the cores of the 6200 can be operated at higher clock speeds. So an Opteron 4238 with its 6 cores can be run at 3.3GHz whereas the fastest Opteron 6200, the 6282 SE, runs at 2.6GHz with its 16 cores.

The future is not single-threaded code, however. As quad core CPUs have become common, many software developers are now updating their programs with higher core and thread counts in mind in order to gain performance. A well multithreaded program running on a 16 core CPU at 2.1GHz will almost certainly outperform a single-threaded program running on just one core of a 3.4 GHz processor. The performance difference can be very substantial for users who are running a lot of programs at once.

The typical Internet web surfer can benefit from multicore processors, too. So many web sites have Flash or Javascript code on their pages that consumes CPU time just by having the page open, even at times you are actually looking at some other web page. This is part of the reason why a single or dual core CPU these days gets bogged down pretty fast by a heavily web surfer who opens up a few browser instances at once.

I think what we’ll see happen in the next couple years is that virtually any program for which performance is a serious consideration is going to be modified to support many more processors cores. With Intel and AMD both shipping boatloads of quad core desktop processors, multicore is here to stay. And now that AMD is shipping desktop processors with 8 cores and server processors with up to 16 cores, it is clear that in the future the performance champ is going to be at least a quad core chip and that no single-threaded program is going to be able to take full advantage of it.

AMD Opteron 6100/6200 Offers Good Performance/Price Ratio and Upgrade Path

You can pick up an Opteron 6100/6200 series mainboard such as the excellent ASUS KGPE-D16 dual processor board for around $400 to $450. If you are aiming at saving money and scaling up the system over time, pop in an Opteron 6128 for about $250 and you’ll get solid performance good enough for most uses as it’s around as fast as AMD’s Phenom II six-core chip or Intel’s recent two core i5 chips. Then when you’re ready for a faster CPU in a couple years, you can yank out the cheap 6128 and replace it with a twice as fast 6272 — or two — without having to change any other hardware. It’s great to have an easy upgrade path so you don’t have to waste your time extensively overhauling a system and reinstalling everything from scratch.

If you need something faster right away, look at the Opteron 6200 “Interlagos” series. An Opteron 6272 performs about twice as fast as the 6128 for about $300 more. Plus that leaves you with a second processor socket to upgrade later if you need more CPU or RAM than what the first processor socket and its associated 8 DIMMs sockets can provide.

A helpful resource for getting a rough idea of how fast a processor can be is the PassMark high-end CPU benchmark list. For instance, it shows the Opteron 6272 (sold for around $550) with a score about the same as the cheaper (around $320) Intel i7 2600K desktop CPU and about 50% faster than the Intel E5645 server processor which also sells for around $550. Please don’t use this list as the only resource you consult. Many other sites may offer more applicable benchmarks such as Cinebench, AES encryption, server workload, and others that are more applicable to your use. But this list does do a good job of at least showing a rough ordering of performance along with an approximate price for each CPU. It’s very helpful when you’re in your initial stages of figuring out what CPU to pick for a computer build.

AnandTech’s overview of the Opteron 6200 series has some helpful performance and pricing comparisons, too. They agree that the Opteron 6200 series offers more performance for the dollar than Intel E5 line does right now:

The Opteron 6276 offers a better performance per dollar ratio. It delivers the performance of $1000 Xeon (X5650) at $800. Add to this that the G34 based servers are typically less expensive than their Intel LGA 1366 counterparts and the price bonus for the new Opteron grows. If performance/dollar is your first priority, we think the Opteron 6276 is an attractive alternative.

Opteron 4200 Options

Another nice option that is somewhat less expensive is the ASUS KCMA-D8 dual processor board for around $280 to $300, about $150 cheaper than the KGPE-D16. It supports two C32 socket chips in the AMD Opteron 4100 and 4200 “Valencia” series. You can get performance similar to one 16-core 6200 series processor on the KGPE-D16 by using two C32 socket processors. Not only is the board cheaper, but the case and power supply can be a little less expensive, too, as the KCMA-D8 works with smaller power supplies and is an ATX size board that fits in many cheap computer cases.

The much larger SSI EEB sized KGPE-D16 requires a case like the mid-priced CoolerMaster HAF 932 (around $140-160) or the inexpensive (often around $70-80 when on sale at Newegg) Rosewill RSV-L4000 server chassis. And it needs a power supply with TWO 8-pin 12 volt connectors, generally meaning a 700 watt or higher power supply, whereas the KCMA-D8 only needs one 8-pin 12 volt connector.

Final Notes

I’ve used the ASUS KGPE-D16 in a Rosewill RSV-L400 chassis with a PC Power & Cooling Silencer Mk II 950 watt power supply for some recent builds myself. I’ve been overall happy with this combination. (If you need a tower chassis rather than rackmount, I’d suggest the CoolerMaster HAF 932.)

This setup has a lot of room for growth for processors, RAM, and disks, enough that I expect to be easily able to scale starter systems up to relative monsters with dual 16-core CPUs, 128GB RAM, and 8TB or more hard disk storage without having to replace the SDRAM DIMMs or any parts other than maybe the cheap Opteron 6128 processor that is often good enough for starters. I suspect other people may be doing the same, so there may be a glut of used Opteron 6128 chips on the market in the next year or so as people decide to upgrade to faster CPUs as their needs grow. Already you can find some used 6128 chips on Ebay for $100 to $150.

When you’re picking memory for Opteron systems, try to get 1600MHz DIMMs that are single or dual rank. Quad rank DIMMs are OK if you plan to only populate one DIMM per memory channel, but if you add a second quad rank DIMM to a channel then the speed drops considerably. For instance, putting in two Kingston quad rank 8GB ECC DDR3 1333MHz DIMMs in one Opteron memory channel will get you 1066MHz performance. Dual rank used to be a lot more expensive, but right now it is often only $5 to $10 per DIMM more at the 8GB size. At 16GB, however, there are fewer dual rank offerings and often people are using 1066MHz DIMMs, too, as the 1333MHz ones are still quite expensive.

I’m particularly happy to see the supply of 1600MHz DDR3 registered ECC DIMMs growing quickly. A couple of months ago they were impossible to find. Now at the vendors that have them in stock, they are often priced around the same as the older 1333MHz DIMMs. For instance, Superbiiz is selling 8GB 1600MHz DDR3 ECC registered DIMMS for about $80. Hopefully Newegg and other vendors will catch up and start stocking such DIMMs soon. With such parts it is relatively cheap and reliable to pop 32GB RAM or more into a computer built around the AMD Opteron 4200 and 6200 series processors.

You can get brackets to adapt 2.5″ drives to 3.5″ mounting bays for around $5 to $15 each. Some support just one drive, a few support two. But a better solution is to get a hot swap 2.5″ drive cage. It’s particularly valuable if you’re installing a RAID SSD configuration so that you can easily replace failed drives.

It’s also a handy way to be able to plug in 2.5″ hard drives for uses such as re-imaging or backing a laptop hard drive, repairing a broken boot partition, or making a backup of the SSDs. Of course a USB 3.0 or eSATA docking adapter can work well for those uses, too.

Three similar 2.5″ four drive cages that are worth your consideration are made by Icy Dock, Thermaltake, and iStarUSA. Icy Dock’s cage is model MB994SP-4S. The Thermaltake model is the MAX-1452. The competition from iStarUSA is the model BPU-124V2-SS.

All four are solidly built primarily of metal components with four removable hot swap trays supporting drives up to 15mm thick. They support SAS and SATA interfaces up to 6Gbps and include two 40mm fans to help cool installed drives.

Most SSDs don’t need the added cooling, but these cages also support 2.5″ hard drives, too. A little bit of extra air flow for cooling can make even a barely warm SSD a bit cooler. As heat is the enemy of reliability for most computer electronics, the fans are an OK feature unless you’re really sensitive to audible noise.

The airflow on these cages is enough that a few owners have reported successfully using 2.5″ hard drives such as Western Digital Velociraptor 10,000 rpm disks while maintaining operating temperatures between 30 and 40 degrees Celsius.

The power hookup for the cages varies. The IcyDock has two 4-pin Molex power connectors, the same used on many case fans and IDE disk drives. The Thermaltake and iStarUSA use a single SATA power connector.

The drive trays includes power and access LEDs. The IcyDock LEDs shine green for power and amber for drive activity. The Thermaltake and iStarUSA LEDs are blue and blink to indicate drive activity.

The Thermaltake and iStarUSA models include keylocks on the drive trays making them more secure from accidental removal. They also include four SATA cables.

There is one big difference in favor of the iStarUSA cage for enterprise applications. It includes both primary and secondary SAS connectors for drives that support dual-port SAS. Such connectors are typically used so that each drive can be controlled by two redundant RAID adapters. High availability servers that are using high-end 15,000 rpm SAS hard drives or one of the few dual-port SAS SSDs on the market today could benefit from the dual SAS connectors.

For most home business and small business applications, these 2.5″ drive cages are probably going to be used with SATA2 or SATA3 drives so dual port SAS connectors aren’t necessary.

Visually the Thermaltake cage is the flashiest of the three with its red levers. The Icy Dock and iStarUSA cages are much more conservative in their appearance.

Six 2.5″ Drives in One 5.25″ Bay

There are a couple of drive cages hosting six 2.5″ drives in one 5.25″ drive bay from Thermaltake and iStarUSA. These are limited to 9.5mm height drives. That’s often enough for SSDs, but a few SSDs and many hard drives are taller than that. Heights of 12mm and 15mm are both common.

If the higher density six in one cages interest you because your drives are all 9.5mm and run at cool temperatures, here are a couple which appear reasonable options:

• Thermaltake MAX-1562 Backplane HDD Cage Hot Swap Removable Hard Drive Cage Six 2.5″ drives in one 5.25″ bay -Inches SAS/SATA RAID Ready RC1600101A (click for Thermaltake MAX-1562 specifications)
• iStarUSA BPU-126-SA six 2.5″ in one 5.25″ drive bay hot swap cage, SATA only (click for iStar BPU-126-SA Specifications)

Thermaltake says their MAX-1562 cage supports SAS and SATA up to 6Gbps, but iStarUSA does not list SAS as being compatible with its BPU-126-SA. Both include single a SATA connector per bay and use a single four pin Molex power connector. Both do not have the locking keys featured on the four-in-one cages by the same makers.

From past experience with high density 3.5″ drive cages, I’d be concerned that the ventilation spacing may be too tight for hard drives in these six in one cages to stay at safe operating temperatures unless the ambient temperature in the room is well-controlled at cooler than typical room temperatures. This is less of a concern with most SSDs, but a few of the enterprise SSDs draw much more power than slower consumer grade SSDs so airflow might be a concern for them, too.

Where to Buy

Amazon sells all of these cages discussed above, click through the display links above for more information.

Newegg carries the IcyDock and iStarUSA cages typically priced around $65-70 for IcyDock. They uusually sell the iStarUSA model for around $85 to $90, but recently dropped the price to about $60 for a short time so if you price-watch for several weeks you might catch a bargain price. Check out these links for pictures and customer reviews.

Newegg: Icy Dock MB994SP-4S Hot Swap Cage holding 4 2.5″ drives in one 5.25″ full height drive slot – SATA3 or SAS2 to 6Gbps

Newegg: iStarUSA BPU-124V2-SS Hot Swap Cage holding 4 2.5″ drives in one 5.25″ full height drive slot – SATA3 or SAS2 (single or dual connector per drive) to 6Gbps

Update:
I previously found a seemingly inexpensive source for the iStarUSA cage called AxionTech in Texas. They are selling the iStarUSA BPU-124V2-SS for $64.99. With its better features, the iStarUSA drive cage is my personal pick. But when I ordered from AxionTech, they took the order despite having none in stock. Many weeks later it was still out of stock with no anticipated delivery date.

Why ECC Didn’t Use To Be As Important

Back in the days when “640KB RAM was all that anybody would ever need”, sporadic memory errors weren’t a big problem for two major reasons. The feature size of the circuits on the chip were large enough that a stray gamma particle flying through a memory cell wasn’t all that likely to flip its state. Second, there just weren’t that many memory cells in a typical computer. If the odds of one of them getting flipped was something like one in a billion per year of operation, you could expect to run that 640KB RAM for many decades without an error. For a desktop PC, that’s more than good enough.

But that was then. Today, the story is much different. The RAM memory found in the typical PC today is around 10,000 times that of the MS-DOS era PC. The memory cells sizes have shrunk dramatically, making them more susceptible to stray radiation randomly flipping a bit. The memory runs at much higher speeds, meaning that a poor connection or transient electrical noise is that much more likely to generate a bit read or write error.

SDRAM Errors Far More Common Than Previously Thought

When developing mission critical computer systems that must run 7/24, engineers know that ECC memory is favored for such applications. But exactly how often do memory cells flip and return the wrong value? The answer to this question is not straightforward. The statistics on how often memory errors occur are not well known and at times seem contradictory.

A 1998 article published in EE Times suggested that each 256MB RAM generates one bit error once per week. Given that most desktop PCs of that time had less memory than that, you might have expected your PC with 128MB RAM to run a couple weeks between memory errors.

The educated guess of the early 2000s was the SDRAM quality and computer design had improved and that you didn’t have to worry about memory errors much with less than 1GB RAM. But as the amount of RAM went up, the need for ECC did, too. Any serious server platform had support for ECC memory by around that time.

In 2009, Google published a study that looked at ECC memory controller reports of errors in their server farms. The Google memory error study found that the then-current industry estimates of memory error rates were about 15 to 1000 times too low. ECC correctable memory errors occur on average in one third of servers each year. In the servers in which they are occurring, the error rate averages around 4000 errors per year. Yikes!

The gist of the results is that many memory modules work with very low error rates, but that any module which has an error is likely to have errors much more often. Properly implemented ECC helps detect the bad modules without crashing the servers so they can be replaced at an opportune time. Google uses ECC memory in all its servers and they have so many servers that their study bears a lot of weight compared to previous attempts to quantify memory error rates.

Memory capacities have soared as the industry has transitioned from DDR2 to DDR3 memory. The feature size of memory cells on DDR3 chips has continued to shrink. This refers to the size of the transistors used to build circuits on silicon dies. Most often this is discussed in the press relating to new CPU generations such as the recent shrinks of CPU dies from 45nm to 32nm or the coming shrinks to 28nm or 22nm. As memory chip capacities increase, their feature sizes are likewise shrinking, too. It is unclear if the improving production processes for SDRAM can totally overcome the possibility of increasing errors rates as feature sizes shrink to pack so much more memory on the same silicon.

EE Times recently reported there is no large scale study of error rates in DDR3 that can compare with the Google study of older DDR and DDR2 memories. So it is possible that today’s high capacity DDR3 multigigabit memory chips are more susceptible to bits wrongly flipping their states than was the case for older DDR2 memories a few years ago.

What This Means For You

Today multiple gigabytes of RAM is pretty much a given for any serious workstation or server and even most desktop PCs. 4GB RAM is almost a mininum that you can find in a new laptop or desktop PC. Really 8GB or more is much more reasonable for anything to be used with modern software and operating systems for purposes beyond light web browsing and word processing. If you’re using Photoshop, GIMP, or other serious graphics programs as an artist, doing desktop video editing, or are a software developer, you probably are well aware that 8GB RAM is often far less than adequate for such uses. You should really be looking at 16GB RAM or more. Some of these folks could even benefit from 32GB RAM. Servers routinely have that much RAM or more.

With so much memory, the odds are memory errors happening in your PC in any given week are much higher than they used to be. Imagine what this might mean for you. Instead of your computer running for weeks without memory errors, today’s high capacity RAM PC might have errors every day.

Not every error is going to crash your PC. When memory errors occur, if you’re lucky the flipped bits will be in unused memory where they will cause no harm. In fact some would argue that this is a typical case because of the way DRAM functions as “leaky bucket” memory that has to be recharged to maintain its values. Memory hasn’t been accessed recently is likely to have a weaker charge in the memory cells making it that much more likely a stray gamma particle could flip the cell value. But Google’s study suggests that this may be only a small effect given that even ECC systems with “memory scrubbers” that visit all the memory and read and rewrite any erroneous bits still see similar numbers of memory errors as systems without memory scrubbers.

If the memory with damaged contents is used, you might see a program crash. Or maybe your computer will reboot. Frankly those annoying results are preferable to what could happen. That’s because if you’re very unlucky about where the memory error occurs, that flipped bit might be in an important piece of code or data for the file system on your hard drive or SSD. Oops, sorry, that wrong bit just meant the file system driver overwrote your root directory or trashed your OS installation — time to restore your backups!

But at least you’d soon know something went wrong in that event. Perhaps worse could be the case where the memory error silently trashed some important data file, overwriting good data with bad. You won’t notice it is damaged until you go to read it months later only to find it is damaged beyond the ability to open it. And because it happened silently months ago, you are all the more likely to not have any backups going back before the file was trashed by the memory error.

How To Detect Memory Errors

Many people blame such problems on a buggy program. Microsoft Windows gets the blame for a lot of problems like these. But could they be due to flakey memory? Seriously, how would you know the difference between a buggy program and a flakey memory system without ECC memory?

One approach is to spend several hours or days running an intensive memory test program on your PC. It’s a good thing to do when you are noticing a lot of crashes or other odd behavior. One good choice available as freeware is Memtest86+. However there are lots of other options, too.

Such memory testing programs blast the memory subsystem with read and writes with varying bit patterns looking for errors. While running such a program, your PC is useless for any other purpose. The odds are fairly good that after running several passes over all the memory that any marginal DIMM, bad socket, or flakey CPU memory controller will be detected.

But that still doesn’t deal with the random bit flipping due to environmental radiation or electrical noise problems.

Apart from what memory testing programs can do, it is very difficult to determine if bad memory is causing marginal operation of your PC or server unless you have ECC memory.

ECC cannot make memory totally reliable, especially since some ECC designs today correct single bit errors per memory word but may not be able to correct more than a single bit error. However, even this small improvement can cut the rate of uncorrected errors by a thousand fold or more.

Additional improvements, such as “chipkill” or “chipspare” algorithms implemented in many server processors, can cut the uncorrected error rate by around another magnitude.

Many ECC implementations can report on the number of memory errors being detected, too, so at least you have a clue of whether your SDRAM DIMMs are good or not if you take the time to check your computer’s operational logs.

Drawbacks of ECC Memory

ECC memory has a reputation for being slower than non-ECC memory. That’s true to a degree. The difference is probably a few percent in performance at the same speed grade due to the overhead of the ECC algorithms. But most software gets a lot of L1, L2, and L3 cache hits so the actual difference may be difficult to notice unless you spend some time benchmarking a system carefully. I personally do not think this small performance difference matters much, especially when you consider how much more wasted time you’ll be facing if your memory is having errors that are not being detected and corrected.

ECC memory itself sometimes is more expensive that non-ECC, particular at DIMM capacities that are mainstream for desktop PCs such as 4GB DIMMs are right now. Other times, especially at high capacities such as 8GB per DIMM, ECC DIMMs are cheaper than non-ECC memory. This seems to be a result of economies of scale when it comes to producing high capacity DIMMs. Most of the 8GB and 16GB DIMMs are being made for servers that use ECC registered memory.

The main drawback I see to ECC is that the processors and motherboards that support it tend to be more expensive. But sometimes when you look at the life-cycle cost of maintaining a solid performing PC, a server or workstation motherboard with two CPU sockets that costs an extra $100 or $200 may be worth it over time as you can instantly almost double performance by popping in a second CPU without having to reinstall everything on the computer or pay for new operating system and software licenses.

I look at the extra cost as a form of insurance for my data and time. One badly placed uncorrected flipped bit could do a lot more financial damage than the added cost of better hardware that can correct for such errors.

ECC Becoming More Common, But Buyer Beware

With 4GB RAM and larger amounts being the norm these days, computer manufacturers should be keen on fighting RAM memory errors causing problems in anything but cheap noncritical systems such as video game consoles. They’ve known how to do this for years by adding an additional group of bits to the memory and building logic into the memory controllers to detect and correct single bit errors as it is being read back and to scrub the entire memory for errors by reading through all the memory gradually and rewriting any bits that are found to be wrong. This is not anything new or advanced. ECC memory has been around for multiple decades in mainframe computers. Server-grade microprocessors have included support for it for well more than a decade.

ECC implementations have been either unavailable or very questionable for desktop PCs for years. When a processor or chipset supports ECC, the choice of motherboard matters a lot. Some older Intel desktop chipsets (such as X38) supported ECC, but even motherboards made by generally well regarded grands (Gigabyte, for instance) failed to fully support ECC. In the case of a Gigabyte X38 chipset mainboard I used, the company never fully implemented ECC support in the BIOS. ECC DIMMs would run fine but as far as I could determine their ECC protection bits were not even being used. Yet they printed in their spec sheet that ECC memory was “supported.”

With AMD’s recent introduction of its Bulldozer processor cores in its “Zambezi” FX family of desktop CPUs for enthusiasts, you can get inexpensive ECC memory support in your desktop PC as long as you don’t need more than 16GB RAM. That’s because the unbuffered ECC 8GB DIMMs are still much more expensive than server-grade registered DIMMs of the same capacity. And again that is if you buy the right motherboard. A few ASUS AMD AM3+ socket motherboards such as the Crosshair V Formula are reported to support unbuffered ECC memory up to 32GB and to actually use the ECC bits.

If you’re convinced that ECC is important, then I’d suggest you look at server and workstation mainboards rather than desktop boards the next time you build a PC or server.

Pretty much any server processor today supports ECC memory and will use the ECC bits to do something useful as intended. You’re a lot more likely to see ECC implemented properly in server motherboards. Many of them are not much more expensive than desktop motherboards. And when they are much more expensive, it is usually for a reason such as supporting two processors rather than just one.

The cost of ECC memory itself isn’t that much more these days so long as you pick your processor and motherboard carefully. With good selection, you can get two 8GB DDR3 ECC registered DIMMs for around $150 to $175. That’s about $50 to $75 more than what you’d pay for the same capacity of non-ECC memory using four 4GB unbuffered DIMMs. Ironically, when you look at populating a workstation or server with 32GB RAM or more, ECC registered memory may actually be much cheaper that the commodity-style non-ECC unbuffered memory.

One snafu with buying ECC memory is that it is sold in fewer places than non-ECC. Newegg is one of my favorite vendors. They carry a lot of well-priced ECC memory, but even they have been slow to stock up on the latest and greatest ECC memory such as the new 1600MHz DDR3 ECC DIMMs. I’ve found that Superbiiz may be one of the better sources for such memory at this time. For instance, they are selling 8GB DDR3 1600MHz ECC registered DIMMs for about the same price as Newegg is selling the 1333MHz version.

Next time, I’ll be writing about why I think you should consider buying AMD Opteron chips for workstations and servers versus Intel’s current E3 and E5 server processors. Intel’s Sandy Bridge EP launch coming early this year probably won’t sufficiently shift the playing field to drastically change this assessment, either. I’ll also discuss how registered ECC DIMMs are a better price deal for large RAM capacity workstations and servers, particularly when using 32GB RAM or more.