Bit-Rot, or silent corruption, is undetected data loss, no alarms, no warnings, just bad data. It can be caused during the writing of the file, by the slow degradation of a physical disk, hard drive controller errors, or user introduced errors. In my case, it was the driver for the eSATA controller on the archive of TV shows in the basement that feed out to
XBMC via shared folders. Data that was written to the disk (or Raid) was intact a month ago, but has since become damaged. The traditional windows file system NTFS has no ability to watch for this kind of corruption, and no ability to fix it once the damage is done. SATA drives have some data checking integrity techniques and correction algorithms
to help handle errors. But many of these schemes were inherited from decades old versions of SATA
when channel speeds were less than a tenth of what they are today. The result is
that the coding isn't robust enough to handle modern-day speeds.
I know from past research that Sun Micro-systems (bought by Oracle) created a new file system for there OS (solaris) called
ZFS.
ZFS was built from the ground up to handle bit-rot properly, using among other things, complete end-to-end data integrity checking. When a files are written to the volume, hash values are generated. The file system can store a backup copy (or 2 or 3...) of every file and will check the hash value again on reading the file, offering you the copy if there is any corruption. Additional overnight "scrubbing" could be scheduled to search the volume for corruption and replace the damaged files from copies. ZFS is the answer to bit-rot if you ask on any forum I've looked at, and I can't blame folks for favoring it, on paper it is the perfect solution... unless you don't already know Unix/Linux/Solaris, in which case you will spend months setting this up properly. Even storage appliance projects like
NexentaStor will take a few weeks to learn well enough to trust with your data.
As I'm not interested in learning Solaris, or rebuilding my data storage server with hardware that Solaris will natively support, I started looking for windows based solutions. What I found, after many days on Google was this blog post.
http://blogs.msdn.com/b/b8/archive/2012/01/16/building-the-next-generation-file-system-for-windows-refs.aspx
If you don't feel like reading it, the important bits are that if you setup a Windows Server 2012 storage space with ReFS as the underling file system windows can do the two most important parts ZFS. End to end file integrity checking, via check-sums for the file data, and the file's metadata and file replacement from backup copy when corruption is found. Although not as robust as ZFS both ReFS and storage spaces are version 1 products and will likely see future improvements from MS.
As a windows admin, I'm happy with the redundancy offered by ReFS & Storage Spaces, though it is less than what ZFS could offer, it does let me stay with the environment I'm experienced in. Data storage is not the place to experiment.