I have only had a short time to try it out so far. On a Linux workstation running Ubuntu 11.10, it seems to work semi-reliably at sleeping and resuming over a few tries. So far I had one failed resume, similar to the failure rate on a previous BIOS version. I suspect this may be a flaw in the Linux kernel resume. When it does work, it takes about half a minute to resume.

When you install this BIOS update, you will have to manually restore all your BIOS settings which is a hassle but shouldn’t take you more than maybe 15 to 20 minutes to update and restore your settings manually if you didn’t change many of them from defaults.

I selected the “auto” sleep state in the BIOS. There are also “S1 only” and “S3 only” options.

When suspended (sleeping), the power consumption drops to around 11 to 12 watts. That’s about 7 watts more than the same workstation powered off but with the ASMB4-iKVM remote management processor still running.

To get the BIOS update, visit the ASUS website page for KGPE-D16, click on the Downloads tab, then expand the BIOS versions available and pick 2202. You can unzip the file and put the ROM image on a USB flash drive, then use the ASUS EZ Flash 2 feature in the BIOS to upgrade to this new version. But remember, when you do this all your BIOS settings will be lost so be sure to write down anything you had to change from default.

For more information on other ASUS KGPE-D16 observations and tips, see “Gotchas” With ASUS KGPE-D16 Motherboards.

• Carriers are clamping down on wireless plan limits, often by dropping unlimited bandwidth plans or implementing throttling schemes that disable mobile devices from being able to provide anything but very slow web browsing with no streaming audio or video.
• Carriers have failed to build out infrastructure to adequately support growing user bases with high-speed wireless data services.
• Coverage gaps still exists in even densely populated areas in big cities. Consider how in hilly areas of large population centers (California’s big cities, for instance) you often can’t get a wireless signal in certain areas unless you are willing to walk or drive a thousand feet or more to get to a better coverage zone.
• Spectrum goes to waste as big carriers such as AT&T gobble up available spectrum and then sit on it, leaving it unavailable for carriers and their customers that could use it.
• “White space” spectrum (the gaps between bands such as active TV and radio stations in a particular market) could help provide improved coverage and bandwidth, but it sits unused because of inefficient competition between spectrum owners and unreasonable FCC actions and inactions.

Solving the Wireless Bandwidth Crunch

There’s a solution to this problem that lies in using existing technology with new systems that allow devices to hope between varying sizes of wireless cells within a “supernetwork” of bandwidth that is offered by a variety of traditional carriers, new carriers such as cable TV companies, and private homes and businesses. The solution relies upon using a range of network cell sizes, frequencies, and RF transmitter powers to blanket any area with multiple overlapping cells used for different purposes.

Femtocell technology (referring to very tiny wireless cells) has gained a bit of traction with devices from Sprint, Verizon, AT&T, and others that allow people to fill in wireless phone coverage gaps at their homes or businesses with Internet-connected devices that cover up to around a 50 meter range around the device. Sometimes these devices are provided for free to customers with poor coverage problems, other times they are sold with an up-front cost plus monthly fees to customers who have coverage but want better performance for their wireless devices. Estimates are that about two million such devices were deployed in the US by the end of 2011.

But the potential for femtocells is much, much bigger. Envision a system in which a wireless device can hop on a slow but reliable wide coverage macrocell network and find out what femtocells are nearby it. The network consists of layers of slower speeds with high range and building penetration with overlays of medium-range/medium-speed networks for local mobility, plus very small cells that operate at high speeds for devices that are sitting still or moving at no more than the walking or running speed of a typical person. I call this a “layered wireless network”. Each device tries to use the layer of smallest highest-bandwidth cells as much as possible while using the layers of bigger, slower cells as fallbacks or to gain its initial understanding of the overall network. As the device moves through space or femtocells go overloaded or live or dead for various reasons (high usage, power failures, reboots, etc.), the device can retune its radios to use other cells providing coverage in its area.

A system like this would benefit from every home, business, and public venue having one or more femtocells devices. These devices, since there would be so many of them, do not have to be built to high-reliability standards (the “five 9′s” standards referring to 99.999% uptime) typically used by telco operators. Even “three 9′s” at 99.9% uptime would be more than good enough as each physical location should have overlapping coverage from multiple cells of varying sizes.

This system depends upon having good wired connectivity as the backbone. In most areas with any significant population, that’s already a given as you can get cable modem, high-speed DSL, or even better fiber optic network services that offer multi-megabit data rates to the premises at low costs. Widely deployed public access femtocells can provide a way for carriers, including wire-line carriers such as cable TV companies, to leverage their wired infrastructure to make some money on the wireless market and at the same time improve wireless coverage and bandwidth for everybody’s customers.

Many wireless device chipsets today have support for multiple wireless air interfaces such as CDMA, HSPA, 802.11, and Bluetooth networks. Most or all of those can also operate at least two or more of those air interfaces concurrently. What they need to make this system work is public access to femtocells, interoperability standards, and a means to cost-share or revenue-share based upon who is using the femtocells. The obstacle to a system like this is really not a technological one but instead involves business and legal hurdles.

Layered Wireless Network Example

To understand how this would work, consider the following typical use case for a person during the work days when wireless networks often charge peak rates. Joe User is sitting at his desk, using his computer to write a report. His mobile phone rings. That entire phone call is a tiny bandwidth user, consuming just several kilobits per second of bandwidth. The phone and network know that they can route the call through some low-bandwidth small-range air interface that uses the computer sitting on his desk and its Internet connection. So they could choose a Bluetooth connection or some similar personal area network connection to start the call. As Joe User is talking, he realizes that he is going to have to check on something down the hallway. That’s outside the range of the PAN supported by his Internet-connected computer.

As he gets up and starts to walk out of the room, the signal quality feedback helps the phone recognize that it will need to shift wireless connections. It lets the network know that it will be leaving the old PAN connection and joining the femtocell access for this side of the office building that is still, like his computer, owned by his company. The phone and network spread the data over both air interfaces until the handoff is complete. Now Joe User is walking down the hallway, and realizes that he actually left the book he was going to find in his car as he forgot to return it to his coworker down the hall. As he starts to head down the stairs to go to his car, the phone and network both handoff to yet another femtocell for this section of the building that owned by the building owner, not his company. As he exits the building to walk into the parking lot, the phone and network handoff to a parking-lot femtocell owned by the parking lot operator or to a macrocell provided by a wireless carrier in the area. To share the costs and revenues of this system, each time a cell provides some coverage it earns some credits for its owner. These credits can offset usage on other cells first, or there can be revenue-sharing agreements to flip some (such as the parking-lot operator) a bit of money each month to cover the operating cost of the cell based upon its usage.

This use case shows that mobility and network coverage can be preserved while moving around, but that much of the time the air interface really only needs to have a very short range (in the person’s office or office building, or in a restaurant as the person is eating and watching a funny video emailed by a friend) to satisfy many common needs. The macrocell network is there to cover the times when the femtocells cannot cover the needs of the moment.

Most of the time, the wireless data bandwidth used is on a femtocell. Since femtocells have only a small range and they don’t much interfere with other femtocells at a distance, the total amount of wireless bandwidth available to carriers and customers can skyrocket. That’s because most of the time, you’re using wired networks for long-range data transport and the wireless network is like an extension cord used to bridge that last 20 or 30 meters.

Cases for people driving in vehicles get more complicated and require more use of macrocells, but think about how you actually use your phone for calls or Internet access. Most of the time, you are not moving around much. You are sitting in a chair, standing in a line, or walking from one location to another within a building. All of these cases can be predominantly covered by femtocells with little involvement from macrocells. And this will leave the macrocells a lot more bandwidth to handle the truly mobile users who are rushing about in their vehicles.

Steps Needed

Implementing this system widely is going to require a governmental push on the carriers as they don’t appear to be moving ahead at more than a snail’s pace. Right now they are inefficiently using publicly owned RF spectrum. Governments could clear out high frequency short-range bands for femtocell access and also allow use of the “white spaces” TV and radio bands as medium-range bands by the wireless carriers. Then require that the wireless operators using the “white spaces” bands suitable for medium ranges must offload part of their total traffic onto a shared publicly accessible femtocell network connected to the Internet as part of their licensing arrangement.

In the meantime, there’s Republic Wireless which is offering a $19 per month unlimited smart phone plan that largely uses 802.11 access points. Long-term, 802.11 is probably not a total solution. But what this company is doing is a commercial example of how such a system might work.

Who Pays

The hardware cost dropping in the air interfaces into cable set top boxes and cable modems and other widely deployed high bandwidth wired devices is not high. Many of these devices already having processors of some type plus memory, so it seems unlikely that the incremental cost should be more than you’d pay for a mid-range 802.11 network access point. If the means to revenue share can be worked out then it could be an incremental profit opportunity for the wired carriers while reducing costs for wireless carriers and improving services for their customers. Maybe the wired carriers would even bear some or all of the incremental cost of hardware in order to be able to earn a little bit on the use of each femtocell access point over time.

Many customers would also gladly pay a little extra for wired networking hardware if it improved their wireless services, too. For instance, I as a home user might pay $20 more for my cable modem if I were also provided guaranteed floor of 1 megabit per second wireless service for my own devices by registering them on the cable modem/femtocell access point device. This would also ensure that visitors (my parents, siblings, friends, etc.) to my home can get at least some wireless service, even if their carrier doesn’t have great coverage in the area on its own network.

You could also see how this could be scaled up to allow medium and large businesses to act much like small wireless operators of their own. Say you operate a business such as a shopping center that has thousands of visitors per day. You could sign up as a public carrier to help recoup the cost of your femtocall access equipment and related bandwidth costs and at the same time make your shopping center more attractive to retail customers who want to be able to use their smartphones when they are shopping.

Ultimately, a system like this might someday scale to the point that everybody can get at least some basic wireless data services for free. This would be a great move for free speech as all citizens should have access to the Internet in a truly free society.

Such a system would also help ensure that the majority, i.e. paying customers, will virtually always get good quality service without having to worry about being reamed with several hundred dollars of overage fees on their phone bills as wireless unlimited plans are ending with most carriers. Consuming lots of wireless bandwidth, such as viewing streaming media while waiting around in lines at public locations or in their homes or offices, shouldn’t have to cost you an arm and a leg if there are literally many megabits per second of wireless bandwidth per user most places you go thanks to femtocell technology.

More Reading

Unlimited data is dead, so let’s fight a smarter fight

North Carolina launches FCC-approved TV White Space network in Wilmington

Bad Reception? Sprint May Give You a Free Femtocell to Fix It

Two cases I’ve used with good results for AMD Opteron 6100 and 6200 series builds using the ASUS KGPE-D16 dual processor motherboard are the Rosewill RSV-L4000 server chassis (Rosewill is a Newegg house brand) and the Cooler Master HAF 932 workstation chassis. Here I’ll give you some comparisons between the cases to help you pick for your application.

Rosewill RSV-L4000 Server Chassis

The Rosewill RSV-L4000 is a very nice and inexpensive rack mount server chassis. It measures 4RU in height. That’s a nice height for a rackmount case in my opinion because it allows mounting quite a few hard drives plus the use of moderately large CPU heatsink assemblies.

The chassis comes with mounting ears and front pull-out handles. Slide-out rails are an optional accessory.

Four rubber feet are mounted on the bottom, so even if you don’t rack mount you could stack two or three of these successfully so long as they are in a location where they won’t be likely to be knocked over.

Also, you have to consider weight if you stack. They could crush a flimsy table if they are fully loaded with several hard drives (up to 15 hard drives if you use 5 in 3 hotswap cages), dual processors, and large power supplies that are likely to be used with a case like this.

Motherboard Mounting

The motherboard area can hold either ATX or SSI-EEB motherboards. For the ASUS KGPE-D16, 7 out of 9 screw holes are in the correct positions. You’ll need to use plastic board spacers such as the StarTech Plastic Snap-In M3 motherboard standoffs for the other two screw holes. Or if you have the appropriate machine tools and skills, you could use a tap-and-die set to make additional screw holes for the included metal motherboard spacers.

Internally, there is a large metal crossbrace along the top of the chassis in the middle. It solidifies the chassis and is up high enough to keep it out of the way of full-size PCI Express cards. It also clears most CPU heatsinks, for instance the Noctua NH-U9DO that is one of my favorites for Opteron processors. One nice thing to do with this crossbrace is to use it to tie down cables securely using velcro straps or plastic zip ties. For instance, it’s handy to use it to route PCI Express power cables from the power supply over to a video cards.

Cooling and Fans

The overall cooling design for this case is front to back active (fan driven) air. That’s what you typically find in rackmount cases.

The fans included with the RSV-L4000 all use 4-pin Molex connectors. This makes it easy to connect to a power supply, but if you want fan speed monitoring or control then you are going to need to investigate 4-pin Molex to 3-pin motherboard fan adapter cables. I found a few that are around $2 to $4 each. With seven fans in the chassis, you could spend more than $20 on adapter cables. The two rear fans in the case are 80 mm and are positioned over the motherboard I/O panel area. There are three 120 mm fans in the mid-plane between the motherboard and drive areas. This is great because it helps blow air across the entire motherboard and also helps suck air through the drive area to keep hard drives cool. Two more fans are located in the 3.5″ hard drive bays, directly on the outside of the bays.

The mid-plane fan tray can be removed and repositioned slightly closer to the motherboard if you are using a smaller board and want a little more space behind the drive area. It’s also handy to remove it for initial assembly when you are routing and connecting lots of drive cables and installing drives and want some more clearance for your fingers.

The included fans are generally moderately noisy but you may at times get a fan or two that is noisier than the others. For servers in a server room or closet, these fans are not overly noisy. If you are shooting for a quiet workstation, then you may want to replace them with something a bit quieter.

The front cover locks shut and includes a plastic mesh dust filter. You have to remove some screws to get the dust filter out to clean it.

I have not tried the very largest CPU heatsink assemblies in this case, but suspect the Noctua NH-U12DO may not fit because of its very large size.

The case is not designed for liquid cooling. If you want liquid cooling, the Cooler Master HAF 932 is a better choice.

Drive Mounting

The 3.5″ hard drive bays measure 5.25″ wide from top to bottom and do not have any guide of guide rails, so you can easily replace them with either “4 in 3″ or “5 in 3″ hot swap cages to hold 4 or 5 3.5″ hard disk drives. But if you do this, the front fans on these bays go away. Many drive cages include cooling fans, but they are often around 80 mm or 90 mm and spin much faster than the 120 mm fans and are probably going to be considerably louder.

If you are installing or removing drives in the included 3.5″ cages, you will need to fully remove the cages as the trays slide to the rear and there isn’t enough space to get the drives out that way. It’s possible to remove the mid-plan fan tray assembly to increase the clearance and that might allow removing and installing drive trays without removing the entire cages, but I have not tried that.

The remaining drive space on this case is a set of three full-height 5.25″ drive bays on the right front of the case. These also do not have guide rails so you could put another 4 in 3 or 5 in 3 hot swap 3.5″ drive cage in this space, too, giving you a total capacity for up to 15 hot swap 3.5″ drives. Or you could put in a CD, DVD, or Blu-ray optical drive in one and have two more bays for 2.5″ to 5.25″ drive cages for SSDs, laptop hard drives, etc.

If you’re not using all of the sliding rail mounting screw holes, some of the screws are positioned perfectly for securing a number of the battery backup units for LSI RAID controllers using two screws which is probably adequate for most purposes (even though they have four screw holes). I’ve seen the LSI BBU09 for the LSI 9265-8i mounted on the left side of the case using these screw holes.

Front I/O Connectors and Controls

There are two USB 2.0 ports on the front of the chassis, positioned near the power and reset switches. There are power, disk, and two network activity LEDs.

Power Supply Mounting

The standard power supply mount is on right rear. I was able to easily route the motherboard 24 pin and two 8 pin power connectors from a PC Power & Cooling Silencer Mk II 950 watt power supply with plenty of slack. There’s space for stowing the extra lengths or unused power cables on the right side of the case in front of the power supply.

Extras

The RSV-L4000 comes with some extras such as a variety of screws and drive mounting rails.

Cooler Master HAF 932 Workstation Chassis

The Cooler Master HAF 932 is a full size tower case designed to take large motherboards up to E-ATX or SSI-EEB size. It is not designed for rack mounting, so if you’re aiming for high density server installations then this is not the case for you.

Although you could use the Cooler Master HAF 932 to build a server, it’s got workstation features such as audio I/O ports on the front panel that are not of any use on the typical server. It certainly could be used for a small business or home server with a few hard drives installed, but I think of it as more of a workstation case.

Motherboard Mounting

Like the RSV-L4000, the motherboard area can hold either ATX or SSI-EEB motherboards. For the ASUS KGPE-D16, 7 out of 9 screw holes are in the right position. You’ll need to use plastic board spacers for the other two screw holes, or if you are adventurous with machine tools you could use a tap-and-die set to make additional screw holes.

Cooling and Fans

The case is designed for either air or liquid cooling and it is possible to mount a liquid cooler setup on the top rear of the chassis. It has both side and front intakes and rear and top exhausts, so you will probably need to leave some space next to it for the cooling to work well.

The fans included with the HAF 932 are all 3 pin fans with adapters to connect to 4-pin Molex power supply connectors. This is nice because if you’ve got a motherboard with 3-pin fan connectors, most of them can monitor fan RPM and also some can control the fan speed based upon thermal sensors. Since they are large fans, they can move a lot of air while spinning slowly and quietly. It’s not a silent case, but it is quiet enough for a workstation even in a quiet room. Tuck it under a desk and you won’t be likely to hear it a few feet away.

There are plentiful options for changing over to a higher quantity of small fans by replacing the large fans with two or more standard size 120 mm fans.

I have not tried the very largest CPU heatsink assemblies in this case, but have read reports that the Noctua NH-U12DO does fit. Personally I think it is probably expensive overkill, however, even in a dual CPU system.

Drive Mounting

Drive mounting in this case is a bit different than from the RSV-L4000. The 3.5″ internal bays also use rubber grommet plastic trays, but they position the drives sideways to the large fan on the front of the case. You get 5 internal 3.5″ drive mounting positions like this. Two of them include 2.5″ to 3.5″ plastic adapters suitable for mounting a single SSD or laptop hard drive. The 5.25″ bays use a “click to release” mounting system. Five out of the six bays have this. One bay has a dual USB 3.0 connector panel that is screwed into place.

The 5.25″ drive bays have metal guide rails for 5.25″ devices, so these may interfere with many “4 in 3″ and “5 in 3″ drive bays.

Front I/O Connectors and Controls

The top front panel on the HAF 932 includes four USB 2.0 ports plus one eSATA port and one 1394 port. It also includes audio input and output jacks routed to an AC97 style audio connector inside. The large power and small reset buttons are on the top of the case, above the front I/O connectors.

There are power and disk activity LEDs on the front panel. Depending upon the exact model of HAF 932 you get, you may get a case with blue or red LEDs including for power and disk activity as well as lighting on the fans themselves.

Power Supply Mounting

The standard power supply mount is on the bottom. I was able to route the motherboard 24 pin and two 8 pin power connectors from a PC Power & Cooling Silencer Mk II 950 watt power supply successfully, but the 24 pin cable is a bit tight so you probably will not be able to run it behind the motherboard tray like you can with the 8 pin cables.

Extras

The HAF 932 comes with some extras such as a variety of screws, plastic zip ties, and a helpful screwdriver to motherboard mounting nut adapter.

Power Supplies

Many dual processor motherboards need both the usual 20+4 power supply connector plus two 8 pin 12 volt connectors. Power supplies under about 700 watts are unlikely to have two of the 8 pin connectors.

I’ve used the PC Power & Cooling Silencer Mk II 950 watt power supplies in a number of server builds. These are relatively quiet units that come with both a 8 pin 12 volt and a 4+4 pin 12 volt which can easily connect to the second 8 pin power supply connector without any adapters required. It also has six 6+2 PCIe power connectors such as you’re likely to need for high performance video cards. (Many mid-range to high-end video cards need two of these per card.) This power supply unit also includes some nice extras such as four thumbscrews for mounting the power supply and five velcro wrap ties for securing power and other cables inside the chassis.

Final Thoughts

If you’re looking to maximize the number of hard drives in the case and can accept a bit more fan noise and plan to use air cooling (not liquid cooling), the Rosewill RSV-L4000 is probably the case for you. If you don’t need a lot of drives, don’t plan to rack mount the case, plan to use liquid cooling, or prefer quieter fans then the Cooler Master HAF 932 is probably a better choice.

At typically $70 (sale price) to $100 (common price), the RSV-L4000 is usually about 1/3 to 1/2 the price range of around $120 (sale plus rebate) to $150 for which the HAF 932 is being sold. But if you have to start replacing fans and adding 4 pin Molex to 3 pin motherboard fan converters then the cost savings may disappear entirely.

More Reading

“Gotchas” With ASUS KGPE-D16 Motherboards

AMD Opteron Roadmap for G34 and C32 Sockets Include Planned Opteron 6300 and 4300 Series Chips

Notes on Performance and Power When Upgrading ASUS KGPE-D16 Motherboard from Opteron 6100 to 6200 Processors

Choosing ECC DIMMs for AMD Opteron Workstation or Server

AMD’s New Bulldozer FX and Opteron Processors Offer Affordable High Memory Capacity and ECC

None of these problems I’ve found so far are killers, but if you know about them ahead of time it might help you to avoid certain hardware configurations that don’t appear to work properly or get less than optimal performance.

Given that I have access to only a limited number of systems and components to swap around, some of these observations are based upon one or two systems that I’ve observed working as described. So it’s possible that certain problems could be something particular to a single motherboard or add-in card. However, given the information I have at the moment, it’s my belief that these observations are generally applicable.

Use Dual or Single Rank DIMMs If Possible

When you select RAM, if you want top RAM speed and plan to populate both DIMM sockets per channel than you absolutely need single or dual rank DIMMs. You can often save around $5 per DIMM by buying quad rank DIMMs, but then you lose at least one speed grade (i.e., 1333 MHz drops to 1066 MHz) when populating two DIMMs per channel.

For more information on selecting DIMMs for this motherboard, see Choosing ECC DIMMs for AMD Opteron Workstation or Server.

BIOS 2005 and Later Remove S3 Sleep Mode

In BIOS version 0902 and earlier with Opteron 6100 processors, the sleep mode options include S3 sleep which powers down most of the hardware in the computer. Reportedly there is a bug in AMD BIOS code that caused ASUS engineering to disable the S3 sleep mode in the newer 2005 and 2006 BIOS versions that are required for the Opteron 6200 series processors. I do not yet know what the bug is. S1 sleep still works, but saves much less power than S3 sleep.

The ASUS KGPE-D16 S1 sleep mode works reliably with Windows 7. My personal experience is that S1 sleep mode with Ubuntu 11.04 and 11.10 sometimes (perhaps around 10% of the time) fails to come out of sleep and you have to reset the computer to get it running again.

UPDATE April 21, 2012: ASUS has released a BIOS update version 2202 as of April 20, 2012 that enables the S3 sleep mode. I have only tried it for a short time using it so far on a single Linux workstation, but it seems to work reliably at sleeping and resuming after a few attempts using Ubuntu 11.10.

ASUS MIO-888 Audio Card

The ASUS MIO-888 audio board appears to only work in the PCI Express slot that is closest to the processors. I tried it in other slots and never got it to work.

For a time, I noticed a significant amount of audible noise such as hissing and clicking on the speakers connected to the MIO-888. This later disappeared, but I’m not sure why. It’s possible it was a poorly seated speaker cable, but the noise varied with computer activity so I think it was more likely some kind of electrical noise inside the computer affecting the audio output.

I’ve noticed that when trying to install OpenIndiana 151a on a ASUS KGPE-D16 motherboard with no audio card installed, the OS thinks there is a card present but cannot configure it properly. There are also some audio card settings in the motherboard BIOS that hint that the motherboard may have some portions of audio interface circuitry on it even if you don’t install the MIO-888.

My guess is that there is some audio interface functionality on the motherboard that is hooked up to that innermost PCI Express slot and that may explain these observations about the OpenIndiana installation and the BIOS settings.

As for OS support for the MIO-888, it worked fine with both Ubuntu (11.04 and 11.10) and Windows 7. I haven’t tried it with anything else yet.

Given my experience, for workstations on which you need audio you may want to consider using a USB 2.0 connected audio adapter rather than the MIO-888. That should be an easy option for Windows users, but finding drivers may be harder for those running other operating systems.

Highpoint RocketU 1144A Quad Port USB 3.0 PCI Express x4 Card

I found that a Highpoint RocketU 1144A USB 3.0 card (four USB 3.0 ports on a PCI Express x4 interface) appears to cause Ubuntu 11.04 to fail to recognize the USB ports when it is installed in the PCI Express x4 slot on the motherboard. But when it is installed in one of the wider slots (x8 or x16 capable), the USB 3.0 ports work fine. This was with KGPE-D16 BIOS version 0902, I haven’t retried it in the x4 slot with the 2005 or 2006 BIOS.

The system in which I have this card installed is now running Ubuntu 11.10. The throughput via the card to individual hard drives placed in a USB 3.0 dock is in excess of 100 MB/sec for highly sequential reading or writing such as backup and restore operations. I’ve seen up to around 145 MB/sec with newer hard drives, so it looks like anything short of perhaps the very latest SSD can run nearly as fast as what you’d get when attached directly to a SATA 2 (3 Gbps) or SATA 3 (6 Gpbs) interface.

Ubuntu 11.04 and 11.10 support the Highpoint RocketU 1144A with their built-in drivers.

Onboard SATA ports

The six built-in SATA ports on this motherboard only run at 3 Gbps speed. You probably will want to set the first four ports to AHCI mode for better performance and the other two ports to IDE mode so you can attach a CD, DVD, or Blu-ray optical drive and boot from it.

I’ve successfully used LG WH12LS30 and WH12LS38 Blu-ray rewriters for booting CD and DVD OS installation media for Windows 7, Ubuntu 11.04 and 11.10, and OpenIndiana 151a. I’ve also had success writing CDs and DVDs in Windows 7 and Ubuntu 11.04 and 11.10 using these drives attached to the onboard SATA ports.

PIKE 2008 RAID Controller

If you need 6 Gbps SATA ports, the ASUS PIKE 2008 board is a good option. The ASUS PIKE 2008 uses the LSI SAS2008 chipset to provide eight ports of SAS 2 or SATA 3 (6Gbps) for attaching to disks. LSI SAS2008 can support PCI Express x8, but the way that ASUS has built this board with it’s special PIKE slot provides only a PCI Express x4 connection.

PCI Express x4 is enough for one or two SSDs and RAID 10 array of four to six moderately fast hard disk drives, but you can run out of PCI Express bandwidth if you go past this. Theoretically this PCI Express x4 interface limitation will limit your disk I/O throughput to around 1.4 GB/sec.

If you’re using some of the very fastest SAS 2 hard drives such as the 15,000 rpm models from Seagate and Hitachi, they can run at around 200 MB/sec sustained transfer rates per drive so it’s possible to max out the disk I/O bandwidth eight such drives without using any SSDs.

The LSI 9211-8i RAID controller board is basically the same as the PIKE 2008 except it uses the full PCI Express x8 interface and costs about $100 more than the PIKE 2008. So if you’re possibly going to be putting multiple SSDS or other high performance disks in the computer, you might benefit from using the LSI 9211-8i over the PIKE 2008. It’s my understanding that these should max out somewhere around 2.5 GB/sec, but I have not tried one personally.

Another plus point for the 9211-8i is that it provides two SFF-8087 connectors and in theory (as in I have not tried it personally) you can connect these to SAS expanders such as the Intel RAID Expander Card Model RES2SV240 to connect to more than the eight disk drives that the PIKE 2008 can support via the SATA connectors on the KGPE-D16 motherboard.

As for OS support, I’ve gotten the PIKE 2008 to work with Ubuntu 11.04 and 11.0 and Windows 7 in JBOD, RAID 1, and RAID 10 modes. It also worked fine with a quick try with OpenIndiana 151a, but I won’t vouch for long-term stability on that because I didn’t have enough runtime on it to verify anything beyond basic functionality for JBOD and RAID 1 modes. Ubuntu, Windows 7, and OpenIndiana all included drivers that worked with the PIKE 2008 without any installation hassles.

PIKE 2008 so far has worked fine with firmware updates from LSI intended for the LSI 9211-8i card for the 9211′s IR (RAID) mode. I have not tried the IT mode firmware, but think it would probably work OK given the success with upgrading using the IR mode firmware.

So far I’ve used the PIKE 2008 MPT2BIOS version 7.19.00 and firmware 10.00.02 and also have successfully upgraded the board with LSI’s MPT2BIOS version 07.23.01.00 along with corresponding firmware version 12.00.00.00.

Workstation Video Cards

If you’re putting in a mid-range to high-end video card in one of the PCI Express x16 slots, consider that most of these boards are double-wide and take two slot spaces. If you do not have legacy PCI hardware that you need, it might be best to use the outermost PCI Express 16 slot for your video card as that overhangs the PCI slot that you’re unlikely to be using for any high-performance network interface or RAID controller cards.

More Information

Cases for Dual Processor AMD Opteron and Intel Xeon Servers and Workstations, A Comparison of Rosewill RSV-L4000 Server Chassis and Cooler Master HAF 932 Workstation Chassis

Notes on Performance and Power When Upgrading ASUS KGPE-D16 Motherboard from Opteron 6100 to 6200 Processors

The more “obscure” bits of information which are needed to log in to an account, the less likely that an intruder will accidentally stumble upon them. That leads to common sense advice such as picking hard-to-guess passwords with mixed case, numbers, and punctuation. Your also often reminded to not write your passwords down anywhere, or at least not where somebody else can discover them. Both are certainly “security by obscurity” concepts that work pretty well.

Like the usual secure password tips, there are other similar suggestions that are equally helpful at securing a WordPress site. With that thought in mind, here are a few quick tips for hardening your WordPress site so you don’t end up with the unpleasant surprise of your site being trashed by an intruder.

Don’t Use the Default “admin” Username for Administration

Every WordPress installation defaults to the administrative user name being “admin”. Since that’s so common, intruders who are attempting to crack the security on a WordPress site will often try login attempts with the “admin” username. They also often inspect the names of authors on the site to see if they can use them to break in. So if you’ve got authors named “Betty” and “Barney”, a cracker may try logging in as “betty” and “barney” if their “admin” login attempts aren’t working.

There are a few way to change the admin account username. One way that often pops up is to modify your wp_users table via a database tool. But there’s an easier way that doesn’t require anything special beyond what’s included in WordPress.

Using the WordPress control panel, click on “Users” then “Add New User”. Set up a new username for administrative functions by setting its role to “admin”. But don’t use an easy-to-guess username. For instance, if your name is “Betty Simpson” then perhaps you might pick a username of “ytteb4413″ with your first name backwards plus four arbitrary digits to make it extra hard to guess. You can still set the display name to “Betty Simpson”, but nobody is realistically going to be able to guess the real username from the display name without a brute force attack.

Now make sure the display name for the administrative user is not the same as their login username. Once you have the new user account created, go and edit the account and set the “Nickname” to something different than the username and set the “Display name publicly as” field to the nickname. If you don’t do this, the default name displayed publicly is your username that you use for logins. That’s exactly what you want to avoid.

Disable Default “admin” Account

Once you’ve verified that your new admin account works by logging out and logging back in using it, then it is time to disable the old admin account. One common way to do this is to delete the user.

Another way is to edit the user account and set the user’s role to “No role for this site”. While you’re at it, you might update the password to be something extremely cryptic, too, such as string of 30 random characters. If anybody manages to log in as the old “admin” user such as by a brute force attack on the login, they will have wasted their time. Instead of getting to the control panel, they will get a message informing them that they don’t have permission to view any administrative pages they try to access.

They will also be stuck from trying to log in again using another name unless they clear the cookies from their browser or wait however long it takes for the session to expire. The default is 48 hours or 14 days if you clicked the “remember me” box. You can configure it using a plugin configure-login-timeout.

Require Login via SSL

So now you have an obscure username that will make it harder for an intruder to attack your site. But what if the intruder happens to find a way to eavesdrop on your login session on the network? That’s the risk you are running if you allow HTTP logins.

To help significantly reduce this risk, you should change your wp-config.php file to require login via HTTPS which uses the SSL (Secure Socket Layer) encryption protocol to hide the data. Any encryption can be broken with enough time and effort, but SSL makes it vastly more difficult.

To use SSL for logins, you are going to need your web server set up to use HTTPS. How to do that varies from server to server and also depends upon whether you generate your own SSL keys (free, often not hard to do on virtual private servers) or get signed keys (often required on shared servers) that usually cost a yearly fee but do help assure visitors that your site is legitimate.

Configuring your web server for HTTPS is a topic for another article, probably more than one as it varies between Apache, Nginx, and other servers. You can often get information on how to do this from your web hosting provider’s help library, so I’d recommend starting there.

Once you’ve got HTTPS set up for your web server and can verify you can browse your site by visiting “https://yoursitename.com”, then you should edit your wp-config.php file by adding these lines near the top:

// Force SSL for administration
define('FORCE_SSL_ADMIN', true);

This option uses SSL for both login and all administrative pages and all exchanges of cookies that may contain security information, too. This helps prevent eavesdroppers from being able to view your pages after login or to capture your cookies and use them to impersonate you.

Some servers are slow at SSL, particularly older ones that do not have hardware accelerated encryption. If that is a big problem, you could at least secure your logins but leave the pages unsecured by SSL by using this option instead:

// Force SSL for login, but not viewing admin pages
define('FORCE_SSL_LOGIN', true);

I’d recommend FORCE_SSL_ADMIN over FORCE_SSL_LOGIN.

Once you’ve got those changes in place, when you try to visit “http://yoursitename.com/wp-admin” or another page that requires administrative login, it should redirect you to the HTTPS version of the page. This helps keep your logins more secure against eavesdroppers.

Note that if you are using a proxy service such as CloudFlare’s free service to speed up visitor’s access to your website, you may have to change your /etc/hosts file on Linux or on Windows the \Windows\System32\drivers\etc\hosts file to point to your web site’s actual server.

By the way, paid CloudFlare users now automatically get SSL service at no added charge rather than the $1 per month they were paying before November 2011, but they still require you to use a certificate from their authentication provider. For sites using CloudFlare or similar options that don’t need SSL end-to-end for general users, it’s often cheaper to generate your own SSL keys and for administrative access contact your server directly instead of through the proxy service.

Update Your Authentication Keys

In your wp-config.php file, there’s a section that looks like this:

define('AUTH_KEY',         'K=[]#KmL1sT*u:0L(&03+5(q N;RjI)@Ze^fyEL[ Fe>}ds9+F#D.IF8U%R=y4sN');
define('SECURE_AUTH_KEY',  'bm;,?y_3FDfceK>h^[8/9o6k)lyYMj$an&duO-:fS_`]zm48!yoUCi%w,FCEMi$^r!n1y8,>|CI__:[XD-ZA7|%&fclkV');
define('NONCE_KEY',        'L@0+py1fKet4QI1[/bzJ++iGSrk%y2+JHdqR3i.6$oj])&I5)_N(.Sb[0[J@[8LQd');
define('SECURE_AUTH_SALT', '4E)r!GWn`G+JZ5|SrStT0+CtNeqGo+e+R-_*el0f:N6VZMYg@y~i|#fR@G;By[3/');
define('LOGGED_IN_SALT',   '].S9YK~l2cBX-~R`Y9{b(RLvvU/o?2|(4{?_9o^BQGz`LBQ?6JNMhQnBA<.*+-G:');
define('NONCE_SALT',       'JUWg5G*AC@p>VLJ- z$l 0 N!|zoN3#-$-}MFr:Ae6!M(Y(-Lv1#[.kT/=79!$C+');

You should update these once in a while, particularly after any reports of compromised or lost passwords or somebody associated with your site has a parting of ways with you.

To generate a new set randomly, simply visit WordPress Secret Key Generator and copy-and-paste the result into your wp-config.php file.

Limit Login Attempts

So at this point, you’ve got some obscure usernames, hopefully have followed good password policies to make them strong, and your logins are secure from monitoring by somebody eavesdropping on the network packets.

But that is still not enough. A really determined cracker can hit your system with thousands of login attempts per hour, or even more. Eventually they might happen upon a working username and password.

Fortunately there’s an answer to that, too. There’s a WordPress Limit Login Attempts plugin that can help prevent the success of brute force attacks on a WordPress login. It can be configured to block IP addresses that repeatedly fail login for a short period, if they keep trying then for a longer period, and to notify you via email of the lockouts. You can easily install Limit Login Attempts from the WordPress site or your WordPress control panel’s “Plugins -> Add New” feature.

Don’t Forget to Secure Your SSH Shell Logins

If you’re running your web site on shared hosting environment where you don’t have SSH shell access, there’s nothing you can do about this. But sites that are running on virtual or dedicated servers should have some simple security measures followed to help prevent intruders from getting into the backend Linux or Unix server as a root user as that is even more dangerous than them getting in as a WordPress administrative user. See Five Simple Quick Steps to Improve Your Server’s SSH Security for what you can do in about 15 minutes to make it far harder for an intruder to gain access to your server via SSH login.

More Information

Five Simple Quick Steps to Improve Your Server’s SSH Security

WordPress Administration over SSL

WordPress Security Keys

WP Security Plugin: Limit Login Attempts

Review Of The Limit Login Attempts Plugin

If somebody manages to log in to your server, they are probably going to use your web server for something nasty. Hosting botnets and DoS attacks on other web sites come to mind. Stealing your user database with email addresses and passwords is also a distinct possibility.

So you absolutely do not want unauthorized people to get into your web server via SSH.

Fully securing SSH can involve a lot of steps, enough that you might get discouraged and let it slide for much longer than you should.

What I’ve found is that following five simple steps are often enough to get rid of the vast majority of intrusion attempts and to greatly frustrate the few determined attackers who might still be trying to crack your security. These should take you about 15 minutes or less to implement.

Don’t Allow SSH Login Via Common User Names

The first step is to disallow login by any user names other than a handle of esoteric ones. When you look at failed login attempts on a server which hackers are attacking, you often see that the majority of fraudulent login attempts target common names such as “root” and “samba” and other usernames that are commonly associated with Linux computers and the services running on them.

So the first thing you should do is to restrict the SSH logins to some unusual names. Best practice is to not use usernames that appear anywhere on your site, making it harder for even a determined attacker who has studied your website(s) to have any realistic chance of attacking your SSH login except by brute force.

To do this on a web server running Ubuntu or Debian, go to the SSH server configuration file in /etc/ssh/sshd_config and update or add this line:

# Limit userIDs that can log in
AllowUsers HenryEatsTastyTofu2 CuriousMonkeysAloft3

You’ll have to set up accounts with those names, also. Be sure that at least one of those users is set up and works before you save these changes and restart your SSH server as otherwise you may not be able to log in again.

If you want to keep your existing user names for now, at least make sure you don’t have “root” in that list if at all possible. For most purposes, you can use “sudo” to get root privilege when needed so you don’t ever have to log in as root.

Save the file, then restart your SSH server by executing “sudo /etc/init.d/ssh restart”.

Disable SSH Protocol 1

SSH prototocol version 1 is vulnerable to attack by eavesdroppers. You should disable it as any recent SSH client supports the more secure version 2.

In the sshd_config file, update the Protcol line to read:

Protocol 2

Use A Nonstandard SSH Port

SSH usually uses port 22. Intruders are going to target that port on essentially every site on which fraudulent logins are being attempted.

It’s easy to change this to some unusual port number. Once again, go into your SSH server configuration file and find this line:

# What ports, IPs and protocols we listen for
Port 22

Pick some oddball 16-bit number, for instance 7383, and update the file:

# What ports, IPs and protocols we listen for
Port 7383

Then restart your SSH server again.

Now when you login from another computer, you’ll need to specify that nonstandard SSH port number.

From a computer running Unix-style tools (Linux, Windows with Cygwin, etc.) you can simply do this to reach the SSH server:

ssh -p 7383 mysite.org

An attacker simply trying to get into an arbitrary computer may not waste the time to find the SSH port, so you’ve just cut the number of attackers likely to cause problems for you down to the relatively determined and sophisticated ones. Determined attackers can use a port scanner to find SSH, so this isn’t foolproof. But it should vastly cut down on the number of casual attackers hitting your site’s SSH login prompt. What I found is that it cut fraudulent login attempts from hundreds per day to zero.

Use Strong Passwords

Be sure usernames have difficult to guess passwords at least ten or more characters long with mixed numbers, upper and lower case letters, and punctuation.

With strong passwords, you may need to write them down somewhere to remember them. Whatever you do, don’t keep copies of the passwords in plaintext on the server. If somebody finds a security exploit that allows them to read such files on the server, they will be able to log in using the information they found.

Install fail2ban

The package fail2ban monitors the auth.log file for security violations and can temporarily disable access to a password-protected service. It uses Linux iptables to implement a simple firewall block against IP addresses that appear to be attempting brute force login attempts.

Installing fail2ban is easy and the default configuration already will temporarily turn off access to IP addresses that cause several login failures in a row.

For Ubuntu and Debian systems, you can install it like this:

apt-get install fail2ban

To configure it, go find the commonly edited fail2ban configuration file which is usually located in “/etc/fail2ban/jail.conf”.

There are three lines that specify IP addresses for which to ignore failed login attempts, how long to ban an IP address after repeated failures, and how many failed logins it takes to trigger a temporary ban.

# "ignoreip" can be an IP address, a CIDR mask or a DNS host
ignoreip = 127.0.0.1 1.2.3.4
bantime  = 1800
maxretry = 3

I’d suggest you allow local (127.0.0.1) logins and logins from your main IP address from your home or business to bypass fail2ban. That makes it less likely that you will inadvertently prevent yourself from accessing your server if you can’t remember a password correctly right away or misposition your hands on the keyboard and don’t realize it right away.

Simply replace the 1.2.3.4 with your main home or work IP address. You can specify ranges or as many addresses as you want by separating them by spaces.

If you are using a lot of different IP addresses to access your server and cannot predict what they will be, try to list as many as you can that are ones that are unlikely to be used by anybody not associated with your site.

This ignoreip setting is intended to help prevent you from being wrongly locked out, it’s not the main protection that fail2ban offers.

Set a low maxretry number as you want to stop brute force attackers from trying many passwords.

There are generally two SSH related sections in the configuration file that look like these:

[ssh]
enabled = true
port   = ssh
filter  = sshd
logpath  = /var/log/auth.log
maxretry = 5

[ssh-ddos]
enabled = false
port    = ssh
filter  = sshd-ddos
logpath  = /var/log/auth.log
maxretry = 5

Be sure they point to the correct location for the auth.log file and update the “port = ssh” to “port = 7383″ or whatever number you used for your SSH configuration.

Save the file, then restart fail2ban:

sudo /etc/init.d/fail2ban restart

Now you’ve just made it very hard for brute force attackers to try more than about a dozen combinations of usernames and passwords per hour unless they hit your server from multiple IP addresses. Even if they manage to commandeer a few hundred IP addresses and focus all their attacks on you, you will still be slowing them down greatly.

Update Your FTP Settings

Keep in mind that these steps will also affect anything you use that runs on top of SSH. For instance, if you are running a secure FTP server such as vsftp, your FTP session configuration in FileZilla or other FTP clients will also be affected by these changes. In particular, remember to update the port number for your FTP over SSH sessions.

More Steps

The above five steps are by no means a complete solution. But they take very little time to set up and vastly improve the odds that you will keep attackers from getting into your server via SSH.

Some more steps you should investigate as time permits:

Enable SSH key pairs for authentication

Use iptables or another firewall to implement SSH connection rate limiting and other restrictions on allowed IP addresses

And if you’re running a WordPress website, consider that you also need to take steps to secure your WordPress logins from attack, too. See Hardening Your WordPress Site by Avoiding Insecure Admin Login Practices for some easy suggestions on how to improve WordPress login security.

More Information

Hardening Your WordPress Site by Avoiding Insecure Admin Login Practices

Fail2ban Web Site

Securing your ssh server

Bullet proof your server #2 – SSH

Top 20 OpenSSH Server Best Security Practices

Contrast that with Intel’s server chips. Every time they release a new line, it comes with a new socket requiring new motherboards. Intel’s approach makes starting with a small server and easily growing it to a massive server impractical without wholesale replacement of the server.

Intel’s low-end to mid-range server chips have also been incapable of driving adequate amounts of memory, often topping out at 32GB spread over two DDR3 channels. AMD’s G34 socket support for four DDR3 channels means it can easily handle double to quadruple the memory of comparably priced Intel server processors.

In 2012 and 2013, AMD plans to continue this gradual evolution. Right now they are working on new C32 and G34 socket processors that are codenamed “Abu Dhabi” for the G34 chip and “Seoul” for the C32 chip. Speculation is they may be called the Opteron 4300 and 6300 series when the ship. AMD claims they will feature another 10% to 15% performance boost over comparable 4200 and 6200 series processors.

Some earlier reports on the shelved plans for codename “Terramar” and “Sepang” processors reported they would be using new sockets and would feature another die shrink from 32nm to 28nm. However, recent slides from an AMD presentation to financial analysts in February 2012 indicate that the new chips will continue to use 32nm process technology and the same sockets.

AMD Opteron chips provide excellent performance for the dollar, but they are not the performance champs. There’s no doubt you can build a faster server with an Intel E7 chip or the upcoming Sandy Bridge based E5 chips. However, the upfront cash cost of Intel’s processors is often two or more times that of a comparable performance AMD chip.

Intel may earn back some of that cost through lower power consumption in areas with really high electricity costs. But even in those circumstances, it may take more time for the lower power draw to attain payback than the chips are likely to be in use.

As it stands today, the Opteron 6200 series chips are offering about double the performance of the 6100 series with slightly lower power usage at idle and slightly higher power usage at load. AMD offers an easy path for its past 6100 customers to upgrade to 6200 series chips and enjoy benefits in conserving power and rackspace while boosting performance. Hopefully the upcoming 6300 series will continue this trend.

More Reading

AMD’s New Bulldozer FX and Opteron Processors Offer Affordable High Memory Capacity and ECC

Notes on Performance and Power When Upgrading ASUS KGPE-D16 Motherboard from Opteron 6100 to 6200 Processors

Intel brings bigger guns to AMD server chip war: Xeon and Opteron 2012 battle plan

Understanding AMD’s Roadmap & New Direction

AMD’s 2012 – 2013 Server Roadmap: Abu Dhabi, Seoul & Delhi CPUs

AMD: The Flexibility is in the Fabric

AMD Concedes Die-Shrink Race to Intel, Considers ARM Cores

2012, 2013 AMD Fusion Products Detailed in Leaked Slides

My first attempt to fix this was to install TuxOnIce, a set of packages that is supposed to implement a better hibernate. Unfortunately, this made matters even worse. TuxOnIce hung itself trying to access the hibernation state. It chugged along for 40+ minutes doing intermittent disk I/O, until I lost patience. Then I tried rebooting it again via CTRL+ALT+DEL. Next time it complained about not being able to find a disk (dev/dm-2 as I recall) and stated I could press ENTER to reboot, so I did and got a fresh Ubuntu reboot.

With TuxOnIce acting worse than Ubuntu’s default hibernation, I removed that set of packages and was left with a PC that rebooted but again could not hibernate.

Googling for help got me lots of hits, but they all seemed to be about hardware-specific problems mostly to do with laptops. This wasn’t a laptop and most of the advice didn’t seem relevant. The article Ubuntu’s Hibernate Won’t Wake Up left me with the impression that a lot of other people are having problems with hibernate not working and it has never worked well. So I muddled along with hibernate and resume not working for quite some time and didn’t think it was worth the investment to fix the problem on this PC when so many others were having little luck, too.

Then one day I ran into a helpful article on Ubuntu swap partitions that finally gave me the clue I needed to fix the problem in a section starting with the heading “Making the swap partition work for hibernate (optional)”.

After reading that article, I realized that possibly this problem might have happened because this PC has a multitude of disk drives and is also using Ubuntu LVM so at some point during installation it might not have been obvious where the swap space was going to be. Or maybe there is a bug that prevents it from saving the swap space configuration information everywhere it should be saved.

I’m not sure which of those it may be, but upon inspection it turned out the installation didn’t set the hibernation information related to the swap device. So upon reboot, the kernel did not know how to find the hibernation state because it did not know the location of the swap drive where it was stored even though it did know how to find the swap drive for regular use.

It turns out this was reasonably easy to fix, the full fix taking about five minutes including a reboot. I got the UUID for the swap drive by running “sudo blkid” and noting the UUID for the swap device. If that doesn’t work for you, you can also get this information on many system by inspecting the /etc/fstab file for the swap device information.

When making the following changes, it’s best to use the UUID if you can. However, you can use the /dev/sda1 style device/partition names if you can be sure they aren’t going to change out from under you such as might happen when changing disk configurations between reboots.

Next, I edited the /etc/default/grub file to update the GRUB_CMDLINE_LINUX=”" line to include the UUID like this:

GRUB_CMDLINE_LINUX="resume=UUID=12345678-1234-1234-1234-123456789abc"

Of course you have to use your own UUID for your swap device, simply replace the “1234..9abc” text above with your own.

Then I ran “sudo update-grub”. This should update the /boot/grub/grub.cfg file.

Next, I edited the formerly empty /etc/initramfs-tools/conf.d/resume file to set the resume UUID there, too, by adding a line like this:

RESUME=UUID=12345678-1234-1234-1234-123456789abc

The last two steps were to run “sudo update-initramfs -u” and then reboot.

After this, hibernation and resume have worked just fine several times so I think the problem is finally fixed.

In retrospect, it turns out there is at least one article out there Step By Step How to get Hibernate Working for Linux (Ubuntu 11.04, Mint 11) that covered this problem well. Unfortunately, the combinations of search keywords I tried didn’t turn up that article and instead showed lots of articles about laptop and video card related hibernation problems. It’s too bad because it would have saved a lot of time and frustration.

Finally, if you reconfigure your computer later such as changing hard disk drives, moving to an SSD, etc. then it is possible hibernate and resume will stop working again. If so, follow the directions above again to ensure the resume options match your current computer configuration.

I recently upgraded a PC running Ubuntu 11.10 on a ASUS KGPE-D16 motherboard with an Opteron 6128 processor to use a newer Opteron 6274. Most recent OS versions, including Windows 7, can run on the 6200 series processors without requiring a change in configuration or software. But before you go through the whole process, double-check that the OS you have installed does have support for the new processors.

Even when the OS supports the new processor, it may not be optimized for it yet. For instance, Windows 8 is expected to make better use of the dual-core pairs by using different scheduling algorithms than Windows 7 does. So there’s likely some room for improvement as software vendors get their code tuned up for the architectural changes in AMD’s latest processors.

Picking A New AMD Socket G34 CPU

There is a wide variety of 6200 series CPUs. Basically you need to keep in mind four things:

1. Power consumption
2. Core count
3. Clock speed
4. Price

Selecting a processor is a game of trade-offs that depends in large part upon how you use your computer and what you can afford. Absent specific information on your needs and budget, I’d suggest the AMD Opteron 6272 is a good compromise between all the tradeoffs as you can get them in the mid $500 range from SuperBiiz, they should work on pretty much any G34 socket motherboard as they max out at 115W power draw, and the 2.1 GHz cores are still reasonably fast for single threaded or lightly threaded software plus with 16 cores you will be able to get some great multitasking. Others seem to be agreeing as I’ve noticed the 6272 is often sold out more quickly than other 6200 family chips. If that happens, then the 6274 is a good alternative.

But if you’re running only a couple programs that cannot exploit more than about four threads well (many games, old versions of Adobe Photoshop, etc. come to mind), you might be better off with a processor with lower core count and higher clock speed.

If you’re on a tight budget, the AMD Opteron 6212 in the high $200 range is a good choice and will probably perform a bit better than the older AMD Opteron 6128 for most kinds of applications for just slightly more money than roughly $250 that the older Opteron 6128 costs these days.

If you want the absolute fastest G34 processor, be prepared to spend around $1100 for the 6282 SE with its 16 cores running at 2.6 GHz gobbling up 140W maximum power. Some G34 motherboards may not support this processor because of its higher power draw. Others may require a CPU cooler upgrade to handle the higher power dissipation.

Verify You Have Recent BIOS, Upgrade If Necessary

The first thing you need to do to your computer to get ready for the upgrade is to make sure the BIOS on the motherboard supports the newer processors. Reboot the PC and check that the BIOS version is 2005 or newer. If you have an older BIOS such as version 0902 or earlier, you have to upgrade the BIOS to 2005 or 2006 to use the 6200 series processors. To upgrade, download the latest BIOS from the ASUS KGPE-D16 motherboard download site, then extract the BIOS from the ZIP file and then copy it to a USB flash drive. Put the USB flash drive into one of the USB connectors on the PC and then reboot and enter the BIOS setup. Use the Tools menu to select the flash BIOS option and then upgrade the BIOS by picking the right file on the USB flash drive.

UPDATE April 21, 2012: ASUS has released a BIOS update version 2202 as of April 20, 2012 that enables the S3 sleep mode that was broken in the 2005 and 2006 BIOS versions.

If you’re using another motherboard such as from Tyan or Supermicro, similar directions should apply but the BIOS versions and download locations are obviously different than from ASUS.

When buying a new G34 socket motherboard, if you don’t already have a 6100 series processor then you should ask your vendor to verify that the boards have a recent BIOS that supports the 6200 series processors. Otherwise, you are going to have to get your hands on a 6100 series processor to boot the board to be able to upgrade the BIOS. For people buying the ASUS KGPE-D16 motherboards last year, this was certainly a problem for some people at least through December 2011. But as of January 2012, it appears that new motherboards sold through vendors doing some volume of sales do have at least the required 2005 BIOS version.

Replacing the CPU

Next I did the following steps to replace the CPU:

1. Shut down the computer and turn it off at the power supply
2. Pop the computer case top
3. Unclip the two fans on the Noctua NH-U9DO processor cooler
4. Unscrew the NH-U9DO from the mounting point on top of the processor
5. Set the NH-U9DO aside where it won’t get thermal transfer compound on anything else
6. Scrape any excess thermal transfer compound off the metal CPU hold down bracket using a small screw driver
7. Pop the release lever on the CPU and removed the Opteron 6128, setting it aside in a safe place
8. Open up the Opteron 6274 box and place the chip in its socket, matching the positioning triangle on the chip with the triangle on the socket
9. Place the excess thermal compound on the top of the 6274 (the Noctua compound does not harden unlike others do) and squirt a little more on it
10. Put the Noctua cooler back on top of the processor (if you are using a NEW cooler, be sure to remove the plastic protective shield or else you may subject your CPU to 60+ degree Celsius temperatures even just by booting in the BIOS!)
11. First press down the CPU cooler to spread the thermal transfer compound some, then fully screw it into place
12. Clip the fans to the cooler, double-checking the airflow direction so that both fans are blowing the same direction (this is noted by little directional arrows on the fans)
13. Visually recheck everything to be sure the fans are plugged in and everything looks OK
14. Turn on the computer, check that the CPU fans are running. If not, turn off the computer immediately and correct the fan problem.
15. Boot into the BIOS, usually you have to press DEL to select BIOS setup.
16. In the BIOS, check the CPU temperature. If everything is OK, you should be seeing temperatures no higher than around 30 degree Celsius even in a comfortably warm room. (Generally I’m seeing temperatures in the 20s depending upon load and room temperature.)
17. If you saw temperatures much higher than about 30 degrees C, then shut off the computer and go back to step 3 and recheck the thermal transfer compound is OK by again removing the heat sink and verifying that it spread OK. Then follow the rest of the steps again except obviously don’t change out the CPU again.
18. If everything checked out, you can reinstall the case cover and then run longer with a regular software load while watching the CPU temperatures a bit more to verify that all is OK.

Performance and Power Comparison

For typical software that makes use mostly of integer instructions and only lightly uses floating point, the 6272 and 6274 are around twice as fast as the 6128. This is shown by the PassMark CPU testing chart for high-end CPUs where the 6272 is rated at 10,254 and the 6128 is rated at 5105.

Software that uses a lot of floating point instructions may not get as big of a speed boost, particularly if it has not been recompiled with a compiler that optimizes for Bulldozer’s new FPU architecure. One point of discussion about the new Bulldozer processors cores in the 6200 is that each pair of CPU cores shares a FPU. In the 6100 series, each core had its own FPU. The new 6200 FPUs are supposed to be considerably more capable with new 256-bit floating point instruction support and ability to split into executing two 128-bit floating point instructions streams, however fully utilizing these new abilities requires rebuilding code with a new compiler that optimizes for this architecture. If you’re writing your own software, that may be something you can do easily. If not, you may be stuck with older software that is not optimized for the 6200 series processors and will not run as fast as it could. It’s my opinion that this is what causes some of the gap between expected around 100% increase and observed around 67% increase in performance seen in some of the testing done below.

The following is a quick comparison on the performance and power usage between the computer with the 6128 processor and the upgraded version with the 6274. In general, the 6274 draws slightly less power than the 6128 at idle and slightly more at load. Performance goes up mainly due to being able to use more cores. If your typical software mix cannot make use of a lot of cores, you might be better off picking a chip like the 6212 that runs fewer cores at higher speeds.

The particular computer I did this upgrade on has a total of 9 hard disk drives running inside of it along with a AMD/ATI Radeon 6870 graphics card, 32GB of DDR3 ECC 1333MHz memory (four DIMMs), a PC Power and Cooling Silencer 950W Mk2 power supply, a HighPoint USB 3.0 four port RocketU 1144A card, an ASUS PIKE 2008 RAID controller, an ASUS MIO-888 audio card, and a LG WH12LS30 Blu-ray optical drive.

Power monitoring was done by visually watching a Kill-A-Watt power meter into which the computer power supply was the only device plugged in. My guess is there’s about a 1% to 2% margin of error on these measurements between the limitations of the meter and me watching it manually. Of note, I also tried monitoring the power usage with the power meter on a UPS backup but that didn’t seem as consistent.

With the main power off, this computer draws about 5 watts with the IPMI 2.0 hardware (ASUS ASMB4-iKVM) running with either processor. Given that the processors are turned off, that makes sense.

After the BIOS spins up the disks, with the 6128 it draws about 212W and with the 6274 about 210W.

After booting Ubuntu 11.10 and logging in with nothing else running but the OS, the 6128 version draws about 247W and the 6274 about 235W to 238W. What I noticed is that the power draw on the 6274 varies more than I saw when monitoring the 6128.

For software to put load on the CPUs and get some understanding of power consumption and performance differences, I picked the MPrime stress tester and a second program called systester. I monitored these with the Ubuntu System Monitor utility to verify that the expected number of processor cores were at running at 100% load.

For MPrime, I used version 2.66 built as a 64-bit version. I used the torture test, blend model, with 8 threads and for the 6128 saw power usage vary from about 353W to 356W. On the 6274, it varied from about 363W to 365W. But of course this wasn’t fully loaded on the 6274. When I ran it again with 16 threads then the power usage was about 371W to 379W. This program does a good job of burn-in exercising of processors, motherboard, and RAM to verify that everything is working right, so you might want to use it to stress-test a new or upgraded computer for a few hours to a day before putting it into production use.

For systester, I used version 1.4.0 built for x86_64 set to compute Pi to 16 million digits. It’s my understanding that this program using a lot of floating point math which is reputed to be one of the weaker points of the 6200 series processors because a single FPU is shared between pairs of cores whereas on the 6100 series each core had its own FPU.

Here’s a table of the results:

The performance boost shown in the systester output is about a 67% speed bump from the 6128 when all cores are heavily loaded. As shown by the one thread test, each individual core is slightly faster. Each pair of cores on the two thread test seems to be about the same speed. In general on this test, the more threads you have running, the bigger the performance boost.

Overall, upgrading from a 6128 to a 6274 gets you a significant jump in CPU performance with a small bump in power usage at load. Even if you’re just using your computer for typical desktop applications, I think you’ll notice that the additional cores do improve the multitasking performance.

One downside to the jump to the 6200 series processors at the moment is that ASUS disabled the S3 sleep mode in its BIOS versions supporting the 6200 processors reportedly due to related a bug in BIOS code provided by AMD. If you depend upon power-on-suspend to enable quick resume of your work at the start of a workday, realize that the S1 sleep mode that remains is not as power efficient because possibly the only thing it shuts down is the processor cores and maybe hard disk drives depending upon how you have those configured for power management. Hopefully this will be fixed in a future BIOS release.

On a 6128, what I’ve noticed for typical (i.e., not benchmark or stress test) use is that you seldom see more than a couple of cores running at 100% but you often do see every one of the cores being used for extended periods of time. With similar workload on the 6274, you very seldom see 100% load on any core and you do see usually one or two cores running at 0% plus a few more at less than 10%. The computer running with the 6274 should be able to take on some more work without much impact on other tasks already running.

Adding A Second CPU

Adding a second CPU is much like the steps above, but you don’t have to remove an existing CPU and watch out for the thermal transfer compound mess on it. If you need another CPU cooler, I’ve found that usually the least expensive source for the Noctua NH-U9DO A3 is AlwaysLowest. Usually they are around $46 per CPU cooler, but they do charge for shipping. Amazon is sometimes competitive with this, but Newegg is usually considerably more expensive.

More Information

“Gotchas” With ASUS KGPE-D16 Motherboards

References

CPU Based Simple System Stability and Benchmark Tester for Ubuntu Linux – systester

Linux Stress Testing and Benchmarking

This creates a new problem, however, as the usual means (e.g., “Disk Utility”) for checking for drive temperatures and errors won’t work. That’s because the OS doesn’t see the individual drives as it usually would as they are being hidden behind the RAID controller. The “virtual drive” created by the RAID controller generally doesn’t implement SMART reporting features for getting information on the drives such as errors and temperatures that you should be watching to minimize the possibility of an impending drive failure taking you by surprise.

I like to use LSI based RAID controllers, even if I’m just using using them for a bunch of SATA or SAS ports and letting the OS do software RAID. That’s because LSI has done a good job of ensuring that drivers are available for most operating systems. Ubuntu’s kernel has included LSI adapter drivers for some time. They have drivers available for several Linux variants, FreeBSD, Windows, and even Solaris and OpenIndiana.

You also see LSI RAID chips included on a number of motherboard products or add-in cards for server and workstation motherboards from Supermicro, ASUS, and others. Intel, IBM, and many other companies are also using LSI chips for their RAID adapters. As a result of so many companies picking LSI RAID chips and ensuring drivers are available for their systems, LSI is one of the safer bets when it comes to getting driver support for a RAID card.

The following example uses a computer on which I’ve got Ubuntu 11.10 installed with a non-RAID hard drive, a USB flash drive, some USB attached memory card reader ports, an optical drive, and a couple of RAID 10 arrays on an LSI SAS2008 based controller using the mpt2sas driver included in the kernel. I suspect most the following will also work with other RAID controllers, too, but it’s always possible that some of them do not implement the features that smartmontools needs to access the SMART data for each drive.

First, install the “smartmontools” package. You can install it like this:

sudo apt-get install smartmontools

Next, you need to identify what your hard drives are. On Ubuntu Linux, generally you can find a series of /dev/sg* names that correspond to the device names for the hard drives. Simply “ls /dev/sg*” and use that as a list of attached storage devices. Note that these sg* (SCSI Generic) names are not the same as the more commonly used /dev/sd* naming pattern you see in a lot of other articles. That’s important because I’ve found that the sg* names let you get to the drives behind the RAID controller to check their status even though there is no sd* name for the individual drives.

For each sg* device, run:

sudo smartctl -x 

where you substitute actual device name for like this:

sudo smartctl -x /dev/sg4

You’ll get some output that will give you an idea what the device is, usually including at least a manufacturer name and model number.

Some of these names correspond to devices such as optical drives (CD, DVD, Blu-ray), memory cards, RAID arrays, or other devices that may not implement the SMART command set. When that happens, you may get output that looks like this output I got from “smartctl -x /dev/sg5″:

smartctl 5.41 2011-06-09 r3365 [x86_64-linux-3.0.0-15-generic] (local build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net

Vendor:               LSI
Product:              Logical Volume
Revision:             3000
User Capacity:        5,999,998,009,344 bytes [5.99 TB]
Logical block size:   512 bytes
Logical Unit id:      0x600508e000000000ae355076e8101a07
Device type:          disk
Local Time is:        Mon Jan 30 02:29:09 2012 PST
Device does not support SMART

Error Counter logging not supported
Device does not support Self Test logging
Device does not support Background scan results logging
scsiPrintSasPhy Log Sense Failed [unsupported scsi opcode]

When this happens, it’s just a note that this device doesn’t support SMART. In the case of the particular RAID array denoted by /dev/sg5, there are four more entries /dev/sg7, /dev/sg8, /dev/sg9, and /dev/sg10 that match up with the drives in the array. The array denoted by /dev/sg6 also has entries /dev/sg11, /dev/sg12, /dev/sg13, and /dev/sg14.

When I execute “smartctl -x /dev/sg7″ the output is much more useful and dumps all the data for the hard drive. (It is hundreds of lines of output, so I’m not going to include it here.)

When a drive supports SMART, you should see a few lines in the output about temperature that look like this:

Current Temperature:                    24 Celsius
Power Cycle Min/Max Temperature:     24/26 Celsius
Lifetime    Min/Max Temperature:     24/33 Celsius
Under/Over Temperature Limit Count:   0/0
SCT Temperature History Version:     2
Temperature Sampling Period:         1 minute
Temperature Logging Interval:        1 minute
Min/Max recommended Temperature:      0/60 Celsius
Min/Max Temperature Limit:           -41/85 Celsius
Temperature History Size (Index):    478 (339)

There are also sections for serial number, firmware revision, power on hours, error logs, error counters, temperature histories, and more. Figure that most drives which implement SMART are going to have 200+ lines of output each, so interpreting the information will take some time. It’s best to save these to text files and inspect them carefully until you understand what the information means and what is important to you.

Here’s a simple script to dump available SMART data for all the devices:

#!/bin/bash
for drive in `ls /dev/sg*`
do
    echo "-----"
    echo "Command: smartctl -x $drive"
    smartctl -x $drive
    echo "====="
    echo
done

Once you know what devices actually do report SMART information, you could easily change the “for drive in” line of that script to list only those particular devices for which you want output instead of using every one of the available /dev/sg* names. For example, something like this would list output for the four drive in one of the RAID 10 arrays on this computer:

for drive in /dev/sg7 /dev/sg8 /dev/sg9 /dev/sg10

The key problem I ran into when trying to apply the smartmontools documentation to systems with LSI RAID controllers is that the drive names in the /dev/sd* or /dev/disk/* patterns simply do not work to get behind the RAID array controller at individual drives.

It may help you a bit to understand this on your computer by installing the “sg3-utils” package and then running “sg_map” to dump out a mapping between the sd* names and the sg* names. As you can see from the example below, the computer I’m using to write this article has eight disks sg7 to sg14 that don’t map to the usual names. Those are the hard drives behind the RAID controller.

# sudo apt-get install sg3-utils

# sg_map
/dev/sg0  /dev/sda
/dev/sg1  /dev/scd0
/dev/sg2  /dev/sdb
/dev/sg3  /dev/sdc
/dev/sg4  /dev/sdd
/dev/sg5  /dev/sde
/dev/sg6  /dev/sdf
/dev/sg7
/dev/sg8
/dev/sg9
/dev/sg10
/dev/sg11
/dev/sg12
/dev/sg13
/dev/sg14

Once you’ve got a handle on the names of the drives behind the RAID controller, then the following documentation in the “Further Reading” section below should be useful for figuring out what else you can do.

Smartmontools is more powerful that just a tool to report current hard drive status. You can also use it to run self-tests on a hard drive. Precisely what it can do depends on what SMART features a particular drive supports.

Note that if you try anything other than simply dumping SMART status information, you’re running a risk of something going wrong. For instance, if you tell a drive in a RAID array to run a self-test, it could cause the RAID controller to drop the drive from the array and switch to degraded mode.

As for me, I’d avoid trying this out on a production system during a time when it needs to be available within a couple of days as some RAID rebuilds can take a very long time when you cause a drive to drop out.

Further Reading

Smartmontools introdution

Ubuntu man page for smartctl

Smartmontools FAQ