Custom Cloud Images for OpenStack pt1

Hi folks,

We previously finished our multi-part series on deploying RHEL-OSP with the RHEL-OSP-Installer. In a few weeks, if all goes according to plan I’ll fire up a new series on the next gen installer… In the mean time, I’d like to show you some useful things to do once you’ve got everything up and running. So what’s up first? Well, as the title suggests, we’re going to create some custom images.

Specifically for RHEL, Fedora, and CentOS. Our new and current multi-part series…

As you go through the steps for each, you’ll find that some steps will work for all 3, but then again, some steps will only work for that distro. What I’m getting at is that there isn’t a unified toolset for creating custom images. That being said, nothing is stopping you from seeing what happens when you use the Fedora tools on RHEL or the CentOS tools on Fedora.. It likely just comes down to ensuring that you have access to the proper package repos..

Regardless, we’ll step through each of the different major “RPM”-based distros and see how things compare. As always, I’d love to hear how you do it differently in the comments section.

Sandbox Environment

For creating your custom images, you should likely have a sandbox environment that is completely separate from your OpenStack environment. The sandbox should be a non-OpenStack KVM host. I won’t go so far as to say it has to be the same distro as the images you will create, but it may help and it may help avoid issues. For example, if you’re creating mostly RHEL-based cloud images, you probably want a RHEL-based KVM host. If you create the odd Fedora or CentOS image, it’s up to you if you want to spend the 15 minutes to setup a second hypervisor. 🙂

Most of the tools that are used to create and/or edit images depend on virtualization packages and/or a running hypervisor. Therefore, your sandbox really needs to be a fully functioning hypervisor, plus extra packages as mentioned below. If you’re creating the host from scratch, you’ll want the following package groups:


If you’re host is already installed, you’ll want to add the following packages:

qemu-kvm qemu-img virt-manager libvirt libvirt-python python-virtinst libvirt-client libguestfs-tools libguestfs-tools-c qemu-kvm-tools

RHEL Custom Cloud Images

This method will actually work for any distro, but it’s not exactly the most eloquent. It requires that you have the ISO image for the version of RHEL that you want, plus 2x the storage space for the image itself. Figure on creating 8GB disk images that will get “sparsified” when we’re all done, leaving us with images that are a fraction of that size. But for our sandbox, figure on at least 16GB of working space.

  1. Create a working disk image:
    # qemu-img create -f qcow2 /tmp/rhel7-web-working.qcow2 8G
  2. Fire up “virt-manager”
  3. Create a new VM as you would normally do from ISO, except..
  4. Don’t create a new disk, select the “working image” created above
  5. When you get to the disk partitioning, blow it all away and only create “/” or “/” and “/boot”. It’s a cloud image. You don’t need LVM. You likely don’t even need “/boot”.
  6. Choose the bare minimum packages you can get away with, for example web server.
  7. Choose bare networking, but turn on at least one interface with DHCP, and no fixed MAC address
  8. Let it install and reboot.
  9. Did you get your packages? Does httpd start? If not enable it. Check other services to enable/disable. What other packages got installed? Maybe remove NetworkManager and other unwanted packages.
  10. Be sure to install the package “cloud-init”
  11. Be sure to add the line

    to “/etc/sysconfig/network in order to allow access to the OpenStack metadata service.

  12. Add the options
    console=tty0 console=ttyS0,115200n8

    To the end of kernel line of /boot/grub2/grub.cfg

  13. Shut it down.
  14. Upload it to OpenStack, and create a test instance with it. Can you connect to it? Did networking start up on it properly? Expect that networking will be different between your KVM sandbox and OpenStack. You may just need to switch your “ensX” device over to “eth0” for the cloud image, then restart networking and “boom”, you’re running. Then double check your access again. Does everything else work as expected? If yes, then shut it down, then delete the instance.
  15. Then delete the image from OpenStack. Yes, you heard (read) me right. We have one more step to get things right. We just ran our “maximum smoke test” against the custom build, now we need to make it properly generic and properly small.
  16. Go back to the image in the sandbox. Again, if everything is as expected we’re almost done. If not, go back and fix packages, services, access, or whatever else is wrong first. Otherwise, do the following:
    # virt-sysprep -a /tmp/rhel7-web-working.qcow2
    # virt-sparsify /tmp/rhel7-web-working.qcow2 /tmp/rhel7-web.qcow2 
    # ls -lsh /tmp/rhel7-web.qcow2

    See the difference between the advertised size and the actual size? The “virt-sysprep” command takes out all of the things like SSH keys, UUIDs, network UDEV rules, and everything else that would make the image not generic. The “virt-sparsify” command then ensures that we’re dealing with the smallest image possible.

  17. Upload the final image to OpenStack, this is your final custom image for RHEL 7

Other Options

Ok, so how do we make this more eloquent? A little more automated? Well, we can certainly script it. Once you know exactly how the networking is going to show up in your OpenStack environment, you can make certain changes to the working image that will allow you to completely avoid the “test upload” process. You’ll get a feel for what’s going to fly and what’s not going to fly. But let’s face it, taking 5-10 minutes to test something “just in case” is still a good idea.

So what commands do we have at our disposal that we can script this? Easy:

  • qemu-img creates all sorts of image files from qcow2 to raw to vmdk.
  • virt-install is the command line version of the “virt-manager” install function, with many many options available, including kickstart! In otherwords, we can get ~very~ specific with the builds.
  • guestfs is a tool that allows us to affect change on disk images while they are powered down. Change passwords, copy files, etc.
  • virt-sysprep is a tool that makes a VM more generic for cloning purposes. Way back when I still worked at NetApp and I was working with the early (2.x !!!) versions of RHEV, I opened many RFE’s (Request for Enhancement) with Red Hat.. I wrote a blog article called Don’t Kickstart.. Clone! back in 2011 and talked about “static artifacts” and “dynamic artifacts”. I wrote some scripts to help make boot LUNs and VMs more generic. Then I opened and RFE for Red Hat to take over.. They responded in spades..
  • virt-sparsify is a tool that effectively rips out the empty space. When fully booted, you might have an 8GB VM, but in storage, the image might be 400MB or 1.5GB.

So how about an example script before we go?



qemu-img create -f qcow2 /tmp/$IMAGE-working.qcow2 8G

virt-install -r 2048 --vcpus 1 \
 --name="$IMAGE-working" \
 --disk="/tmp/$IMAGE-working.qcow2" \
 --os-type=linux --os-variant=rhel7 \
 --location=$KSTARTTREE \
 --graphics none \
 --console pty,target_type=serial \
 --extra-args "console=ttyS0 ks=$KSIMAGE"

virt-sysprep -a /tmp/$IMAGE-working.qcow2
virt-sparsify /tmp/$IMAGE-working.qcow2 /tmp/$IMAGE.qcow2
ls -lsh /tmp/$IMAGE.qcow2

Of course this assumes that everything is just as we want it as it includes the ‘sysprep’ and ‘sparsify’ commands in it, but you get the general idea. And I didn’t include “guestfs” or “guestfish”, as I think I’ll save that for a later post.. it’s pretty cool. Besides, this should be enough to get you going in the mean time. Again, you can likely make this as complex as you need. I just threw this together as an example.

So why didn’t I just start with the commands and the script? Because I want you know, A) how it all fits together and, B) how to break it all down into smaller steps for troubleshooting.

We’ll pick up on Fedora or CentOS next time.

Hope this helps,

Captain KVM

4 thoughts on “Custom Cloud Images for OpenStack pt1”

    1. Hi Naga,

      I’ll be completely honest – I don’t know. I’ve not created any custom. Windows images. I don’t have access to any MS licenses and I won’t do it illegally. If any of my other readers has done it, I’m happy to do a guest post, provided they can prove their method works (like a youtube vid).

      Captain KVM

    1. Hi Kevin,

      So sorry for the really slow response/reply. While there are always multiple ways to shave a yak, this information is still valid. I would certainly look at all of the options though, as the whole point of OpenStack is to automate everything possible.

      Captain KVM

Agree? Disagree? Something to add to the conversation?