dd utility: the rescue versions

Apart from the traditional implementations of a Unix dd utility (e. g., one found in the GNU Core Utilities), there are some special versions of it, dedicated for rescuing the data from damaged (badly readable) disks:

1. GNU ddrescue (by Antonio Diaz Diaz);
   seems to be the best choice from these two ones [1, 2];
   there's a GUI log viewer for it (ddrescueview), and also some other helper utilities (ddrutility);
2. dd_rescue (by Kurt Garloff);
   there's also a wrapper script for it called dd_rhelp.

The Debian distribution software archives, starting from version 7.0 ("Wheezy"), have only the GNU ddrescue; you can install it the following way:

# apt-get install gddrescue

The convenient way to use this utility is to make an image (a binary dump) of the whole defective disk (FDD, HDD, CD, DVD etc.) with its help, and then apply data recovery / content extraction programs (e. g., TestDisk) directly to the disk image.

Anyway, regular backups & redundant data storage solutions should save you from data recovery problems :-)

References:

1. Disk drive recovery: ddrescue, dd_rescue, dd_rhelp - System Administration Bits of Knowledge (by John Gilmore)
2. Damaged Hard Disk - CGSecurity

---
Last updated: 2013-12-08

TestDisk & PhotoRec: nice tools to recover lost partitions & files

TestDisk + PhotoRec is a nice toolset to recover lost disk partitions (or at least rescue some files from corrupted filesystems) in case of storage media failures, re-partitioning errors etc.
Works for disk images too.
More to say, it's a free & open source (GPL v2+) cross-platform (Linux, *BSD, Apple Mac OS X, Oracle / Sun Solaris, MS-DOS, MS Windows) software.

Presently, in Debian-based Linux distros, TestDisk & PhotoRec have been packaged into a single testdisk package, so you can install both tools at once:

# apt-get install testdisk

Free & open source data encryption software: quick list

Here is a quick list of free & open source on-the-fly filesystem (FS) encryption software.

• Entire disk / volume / partition encryption & "virtual disk" (container file)-based software:
  • LUKS / dm-crypt [Linux (FS driver is implemented at kernel level); encrypted volumes are supported by FreeOTFE in MS Windows];
  • FreeOTFE [MS Windows];
  • DiskCryptor [MS Windows];
  • TrueCrypt [MS Windows, Mac OS X, Linux] (discontinued in 2014-05).
• File-level encryption tools:
  
  • eCryptfs [Linux (FS driver is implemented at kernel level)];
  • EncFS [Linux (FUSE-based FS); experimental MS Windows port exists: encfs4win];
  • antigift (a ccrypt wrapper) [Linux, FreeBSD, Mac OS X, MS Windows].

---
Last updated: 2014-06-19

HOWTO: stress-test a USB flash drive for bad blocks in Linux

Once upon a time you may like to test your USB flash drive [or maybe a SATA HDD, who knows] for bad blocks.

First of all, backup all the data stored on the target drive;
then you should unmount any mounted partitions located on that drive:

# umount /dev/sdz1

Then you can begin the tests (CAUTION: these tests will destroy all the stored data on the tested drive).

You can use dd utility to fill the drive with zeros:

# dd if=/dev/zero of=/dev/sdz

or with pseudorandom data:

# dd if=/dev/urandom of=/dev/sdz

and it will stop unexpectedly on a bad block.

And there's also a little more handy utility called badblocks:

# badblocks -wso /tmp/badblocks.log /dev/sdz

It will sequentially fill the entire /dev/sdz device with some patterns (0xAA, 0x55, 0xFF, 0x00), then read and compare the read data with an actual pattern. A list of bad blocks will be written to /tmp/badblocks.log file.

References:

• http://superuser.com/questions/376274/
• badblocks - Wikipedia
• dd (Unix) - Wikipedia

HOWTO: fix HDD partition order with fdisk in Linux

Sometimes (usually after some repartitioning operations) disk partitions can get into the wrong order (e. g., "sdb1 sdb7 sdb2 sdb5 sdb4 sdb3", as you can see it in lsblk / fdisk / Parted / GParted etc.).

Usually it's harmless, but anyway can be easily fixed in Linux with a fdisk utility (from the util-linux package): just start fdisk with a HDD device as an argument (e. g.: fdisk /dev/sdb), then sequentially press 'x', 'f' and 'w' keys, and finally reboot your system (for the changes to take effect).

Be careful:

1. you can do this from a some sort of Live CD / Live USB environment;
   please don't try this on a running production system;
2. partition numbers get changed, and some OS's (and especially OS bootloaders) sometimes don't like this;
3. the partition table can get corrupted — make backups prior to any experiments.

Here is a sample output log:

# fdisk /dev/sdb
Welcome to fdisk (util-linux 2.21.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.


Command (m for help): m
Command action
   a   toggle a bootable flag
   b   edit bsd disklabel
   c   toggle the dos compatibility flag
   d   delete a partition
   l   list known partition types
   m   print this menu
   n   add a new partition
   o   create a new empty DOS partition table
   p   print the partition table
   q   quit without saving changes
   s   create a new empty Sun disklabel
   t   change a partition's system id
   u   change display/entry units
   v   verify the partition table
   w   write table to disk and exit
   x   extra functionality (experts only)

Command (m for help): x

Expert command (m for help): m
Command action
   b   move beginning of data in a partition
   c   change number of cylinders
   d   print the raw data in the partition table
   e   list extended partitions
   f   fix partition order
   g   create an IRIX (SGI) partition table
   h   change number of heads
   i   change the disk identifier
   m   print this menu
   p   print the partition table
   q   quit without saving changes
   r   return to main menu
   s   change number of sectors/track
   v   verify the partition table
   w   write table to disk and exit

Expert command (m for help): f
Done.

Expert command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

---
Last updated: 2014-06-17

HOWTO: recovering Linux hibernate functionality after modification of the swap partition

If you change the swap partition and forget about adjusting some settings, your GNU/Linux system can lose its hibernate functionality (it will hibernate, but not resume correctly — you'll lose your hibernated session if you boot with your default system settings).
Here follows a quick solution (tested in Debian & Ubuntu).

Assume you've created a new swap partition and listed it in /etc/fstab:

user@machine:~$ cat /etc/fstab | grep swap
UUID=12345678-12ab-cd34-5678-445566778800    none    swap    sw    0    0

You can also get the swap partition UUID with blkid command:

user@machine:~$ sudo blkid | grep swap
[sudo] password for user:
/dev/sda8: UUID="12345678-12ab-cd34-5678-445566778800" TYPE="swap"

Then you need to check and modify resume file to match the UUID of swap partition.

user@machine:~$ cat /etc/initramfs-tools/conf.d/resume
RESUME=UUID=12345678-0000-0000-0000-123456789012

As you can see, the UUID of resume partition is not correct. You can edit that file with any text editor you like (gedit, nano, vi etc.), e. g.:

$ sudo nano /etc/initramfs-tools/conf.d/resume

or

$ sudoedit /etc/initramfs-tools/conf.d/resume

or

$ gksu gedit /etc/initramfs-tools/conf.d/resume

After all these manipulations, it's needed to update the initramfs image (it's usually stored in a file named like 'initrd.img'):

$ sudo update-initramfs -u

Folks say there is an alternative temporary solution based on applying a kernel option like resume=/dev/sda8 (where /dev/sda8 is a hibernate / swap partition), but usually there's no reason to do that (except the urgent manual restoring of the badly hibernated session, at the boot time).

References:

1. https://help.ubuntu.com/community/UsingUUID