Automatic Backup of Running KVM Virtual Machines

I am using KVM to run multiple servers as virtual machines. The host machine is running Ubuntu Server 12.04 LTS, as are the guests. I've described setting up the host and guests here.

Raw images are used for the guest file system. That is, on the host machine, the entire guest file system is simply a (sparse) file:

sgordon@host:~$ ls -lsh /var/vm
total 76G
62G -rwxr-xr-x 1 libvirt-qemu kvm 90G Jun  5 17:51 guest1.img
15G -rwxr-xr-x 1 libvirt-qemu kvm 45G Jun  5 17:51 guest2.img

Above the two guests have file systems of 90GB and 45GB. But note that as the ext4 filesystem on the host supports sparse files, only 62GB and 15GB are occupied for the respective guests. That is, even though 90GB is allocated for guest1, only what is used - 62GB - is stored on the host disk.

One of my desired backup solutions is to simply copy the guest image files (to another disk and eventually another computer). Something like:

sgordon@host:~$ sudo cp /var/vm/guest1.img /mnt/bkp/guest1-2012-06-03.img

Copying the files is ok if the guest virtual machines are stopped. But if they are running its possible that as the copy is taking place changes are made within the guest, leading to an inconsistent copy. So to copy a running KVM virtual machine I make use of LVM snapshots.

The host system is using the Logical Volume Manager. There are two hard disks in the volume group:

sgordon@host:~$ sudo pvs
  PV         VG     Fmt  Attr PSize PFree   
  /dev/sda1  ictweb lvm2 a-   1.36t 1022.07g
  /dev/sdb5  ictweb lvm2 a-   1.36t  769.04g

and several logical volumes:

sgordon@host:~$ sudo lvs --segments -o +pe_ranges
  LV   VG     Attr   #Str Type   SSize   PE Ranges              
  bkp  ictweb -wi-ao    1 linear 600.00g /dev/sdb5:176429-330028
  home ictweb -wi-ao    1 linear  10.00g /dev/sda1:16688-19247  
  root ictweb -wi-ao    1 linear  18.62g /dev/sda1:0-4767       
  swap ictweb -wi-ao    1 linear   9.31g /dev/sdb5:7152-9535    
  tmp  ictweb -wi-ao    1 linear  18.62g /dev/sdb5:2384-7151    
  var  ictweb -wi-ao    1 linear  46.56g /dev/sda1:4768-16687   
  var  ictweb -wi-ao    1 linear 300.00g /dev/sda1:19248-96047  

My KVM guest images are stored on the var logical volume. I want to take a snapshot of that logical volume and then copy the images to a directory on the bkp logical volume (which is on a different physical hard disk to var). In simple terms the snapshot can be considered another logical volume that is identical to the original var volume. Once the snapshot is taken, any subsequent changes on the var volume, specifically in the guest images, are not reflected in the snapshot. Therefore I will copy the guest images from the snapshot instead of the original var volume. Once the copy is complete I will remove the snapshot.

First, create the snapshot of the original volume var. I'll call my snapshot varsnap. Make sure the size is large enough to fit the entire original volume:

sgordon@host:~$ sudo lvcreate -L 350G -s -n varsnap /dev/ictweb/var

Now create a directory for the snapshot and then mount the snapshot to that directory:

sgordon@host:~$ sudo mkdir /mnt/snapshot
sgordon@host:~$ sudo mount /dev/ictweb/varsnap /mnt/snapshot 

You can visit the /mnt/snapshot directory and now list the files - you will see everything from the original volume var. So lets now copy the guest images:

sgordon@host:~$ tar --create --sparse --file /mnt/bkp/guest1-2012-06-03.img.tar /mnt/snapshot/vm/guest1.img

I am using tar in this case because I don't want to worry about spasre files. Recall that the KVM image is a sparse file: although 90GB of hard disk is allocated (and visible within the guest OS) currently only 62GB of the host hard disk is used to store it. I am using tar to "convert" from a sparse file to a ordinary file so I can move it between different computers/OSes in the future. (See the comments below for further details about sparse files). The backup of the KVM guest image is now created:

sgordon@host:~$ ls -lh /mnt/bkp/
total 62G
-rw-r--r-- 1 root root 62G Jun  3 05:07 guest1-2012-06-03.img.tar

Finally, the snapshop logical volume can (and should, to save space) be removed:

sgordon@host:~$ umount /mnt/snapshot
sgordon@host:~$ lvremove -f /dev/ictweb/varsnap

Next time you want to backup the image than repeat the above steps (except no need to make the /mnt/snapshot directory if you didn't remove it). Of course its much easier to automatic this process using a cron job. Assuming the directory /mnt/snapshot exists, put all the commands in a Bash script, say called backup-kvm-images as follows:

# Creates a snapshot of the volume storing the KVM images, and then
# using that snapshot copies the images. The benefit of using a snapshot
# (as opposed to copying the images) is that the VM does not need to be
# shutdown.

# Include appropriate paths (since cron doesn't use your normal environment)

# The date of the backup
bkpdate=`date +%Y-%m-%d`

# Create a LVM snapshot of the var LV
lvcreate -L 350G -s -n varsnap /dev/ictweb/var

# Mount the snapshot
mount /dev/ictweb/varsnap /mnt/snapshot

# Copy the KVM images
tar --create --sparse --file /mnt/bkp/guest1-${bkpdate}.img.tar /mnt/snapshot/vm/guest1.img
tar --create --sparse --file /mnt/bkp/guest2-${bkpdate}.img.tar /mnt/snapshot/vm/guest2.img

# Remove the snapshot
umount /mnt/snapshot
lvremove -f /dev/ictweb/varsnap 

Save the Bash script in the appropriate cron directory for daily, weekly or monthly backups. For example, in /etc/cron.weekly for weekly backups. Make sure the script is executable:

sgordon@host:~$ sudo chmod a+x /etc/cron.weekly/backup-kvm-images

You now have automatic weekly backups of running KVM virtual machines. For those backups to be more useful, you should move them to another computer or even offsite.