My 71 TiB ZFS NAS has lasted 10 years without a single disk failure
To my server 4U 71 TiB ZFSmade up of twenty-four 4-terabyte drives, is over ten years old and still going strong. Even though it's now on its second motherboard and power supply, the system has yet to experience a single drive failure (knock on wood).
How did I manage to achieve zero disk failures for ten years?
Let's talk about the discs themselves first.
The 4TB HGST drives have clocked about 6,000 hours of service over ten years. You might immediately think that something is wrong, and you would be right. That's only about 250 days of continuous operation. And herein (I think) lies the secret to the drives' longevity.
Turn off the server when not in use.
My NAS is off by default. I only turn it on remotely when I need to use it. I use a script that powers up the server via a smart IoT plug, and after the BMC (Baseboard Management Controller) has finished booting, I use IPMI to power up the disk array itself. But I could also use Wake-on-Lan as an alternative.
Once I'm done using the server, I run a small script that shuts down the server, waits a few seconds, and then turns off the power.
It wasn't enough for me to just power down the drives but leave the motherboard on, because it's the one that draws 7W (about the same as two Raspberry Pis) in standby mode. And with my schedule, that was wasted power.
For me, this mode of operation is comfortable because I run other services on low-power devices such as Raspberry Pi4 or servers that consume much less power in standby mode than my “big” NAS.
The motivation here was a significant reduction in electricity bills, but the “side” effect was the longevity of the hard drives.
You may also object that my case is not indicative and not representative and that I was just lucky and that a large number of disks helped here. But with a successor This NAS with 20 Samsung Spinpoint F1s 1TB drives was the same story: I didn't have a single drive failure in it during its entire 5-year service life.
Motherboard (failed once)
While the drives in the NAS are still fine, I had to replace the motherboard in it a few years ago. It lost access to the BIOS and sometimes wouldn't boot. I tried obvious things like resetting the BIOS, flashing the firmware, and replacing the CMOS battery, but to no avail.
Luckily, this motherboard was still available on Ebay for a good price, so I ended up replacing it with a new one rather than repairing the old one. I needed the same board because the server uses four PCIe slots: 3 x HBA and 1 x 10Gb NIC.
ZFS
The ZFS file system has worked great for all these years. I have changed operating systems over the years and have never had any problems importing the pool into a new OS installation. If I were to build a new data warehouse, I would definitely use ZFS again.
I am running an integrity check zpool scrub on disks several times a year. The check has never found a single checksum error. Over the entire period of checks, more than a petabyte of data was read from the disks. Since the check takes about 20 hours to complete and consumes a lot of electricity during execution, I have to run it on “cheap” days, when the cost of electricity was minimal.
And I am not at all surprised by this result. Disk failure is most often associated with the risk of the following situations:
Complete failure, when the disk is not even detected
Bad sectors (problems reading or writing)
There is a third type of failure, but it is extremely rare: silent data corruption. It is silent because either the drive itself does not realize that it is handling corrupted data, or the SATA connection does not detect any checksum errors when transmitting data.
However, because of all the low-level checksums, this risk is extremely small. Yes, it is a real risk, and should not be minimized, but it is small. In my opinion, this is something to worry about at the data center scale, and for home use, the probability of its occurrence is negligible.
Every time you listen to ZFS enthusiasts, you may get the impression that if you don’t use ZFS, you risk losing all your data one day. I disagree, it all depends on the context and circumstances. Although ZFS is not that difficult to learn, and if you are familiar with Linux or FreeBSD, it is definitely worth trying.
Noise level (very quiet)
My NAS very quiet for NAS. But to achieve this, I had to work hard.
The case contains three powerful 12V fans that cool the 24 drive bays. These fans are very noisy when running at their default speed, so I decided that I would be fine with them running at their lowest speed (idle) where they are almost silent. So I had to add a fan to cool the four PCIe cards (HBA and network), otherwise they would get very hot. This setup provided enough air circulation most of the time, but it was not enough as the drives would eventually get hot, especially when reading/writing data.
Luckily, the Supermicro motherboard I bought for the NAS allowed me to control the fans from Linux, so I decided to create scriptwhich would set the fan speed depending on the temperature of the hottest disk in the case.
I even visited the math subforum and asked its residents for an algorithm that would work best to balance disk cooling and silence. Someone suggested using a “PID controller”, which I knew nothing about.
In the end, I had to learn Python, “borrow” a sample PID controller code, and use trial and error to adjust the parameters to find a balance between fan noise and cooling system performance.
This script has worked great for years and keeps the disks at or below 40 degrees Celsius. PID controllers are a great solution and I think they should be used in most hardware that controls fans, temperatures, etc., instead of the “stupid” threshold-based on/off switch or the less “stupid” Temperature-Speed table.
Net
I started with four-port gigabit network controllers and used network interface aggregation (bonding) to achieve network speeds of around 450 Mbps between different systems. This setup required a huge amount of UTP cables, so I eventually got fed up with it and bought some cheap but working cards. Infiniband. With them I was able to achieve a data transfer rate between systems of about 700 Mbps. When I decided to abandon Ubuntu and go back to Debian, I had a problem: the Infiniband cards in the latter did not work, and I could not find a way to fix it. So I decided to buy some used 10 Gigabit Ethernet cards, which have been working fine ever since.
Problems with power supply (died)
When the NAS is turned on, all the drives in it start up simultaneously (without sequential startup), which consumes about 600 watts for a few seconds. My power supply is rated for 750 watts, and the 12 volt line should have provided enough power, but sometimes the unit would turn off during boot-up, and eventually it couldn't handle it and was replaced.
UPS (threw away)
For years I used a powerful UPS to protect the NAS from power outages so that I could gracefully shut down the NAS during a disaster. It worked fine, but I noticed that the UPS was adding another 10 watts to the power consumption and I decided it was time to remove it.
I just accepted the fact that I might lose some data due to power outages.
Backup (there is none)
My most important data is backed up three times. Much of the data stored on this NAS is not important enough to be backed up. I rely on hardware replacement and ZFS to protect against data loss due to disk failure. If that's not enough, I'm out of luck. And I've lived with that risk for 10 years. My luck may run out someday, but for now I enjoy not having to worry about backups.
Future plans for the storage facility (or lack thereof)
Do I have any plans for what to do next? Honestly, no. I initially built this NAS at this size because I didn't want to move data if I ran out of storage. In the end, I still have plenty of space left.
I have a spare motherboard, CPU, memory, and spare HBA cards, so I can most likely recover the system if anything breaks.
Since hard drive sizes have increased significantly, I may move from a 24-bay enclosure to a smaller form factor. It is possible to create the same storage capacity with just 6-8 hard drives with RAIDZ2 (RAID 6) redundancy. But this would be an equally expensive project.
Another likely scenario is that in the coming years my NAS will eventually break and I will decide not to replace it at all and my “data storage hobby” will come to an end.
Order VPS lines Storage VPS line designed for building network data storage with a 25% discount!
How do you extend the life of your disks?
