OpenStack Installer (for RHEL-OSP) pt2

Hi folks,

We’re back after just a few days.. In part 1, I talked about the RHEL-OSP Installer and it’s current position as the “stop gap” for Red Hat’s OpenStack deployment strategy until everything converges. Supposedly, that convergence will take place with RHEL-OSP 7, summer 2015. In the mean time, I’m here to help provide a relatively painless way to deploy RHEL-OSP.

This weeks focus is on Deploying the Installer.

Update & Goals

Even if you have already gone over part 1, give the last section a re-read. I had to provide an update to the overall architecture. Essentially, even virtualizing the installer and the repo was enough to hose up the networking and it didn’t show up until the tail end of the OpenStack deployments that I was testing.. So take that however you want – a challenge to solve all of the networking issues as presented by mixing physical and nested virtualization, or a suggestion to stick with all physical.

Will I figure out some of the issues related to networking in nested virtualization environment? Most likely. Will it be in the next few weeks as part of this series? Not as likely.

So anyway, this also means I’m down one physical machine. That’s ok, my friend and fellow “hatter” Mike Watkins reminded me how to set up a virtual web server in Apache that will allow us to combine the RHEL-OSP Installer and RHEL 7 Repo on the same physical server. The net result means that I’m not short any servers. Ultimately I want to be able to show folks how to deploy:

  • “Basic 2 Node” – That’s a Cloud Controller running all services (except for compute) with any number of Compute Nodes
  • “Basic 3 Node” – That’s a Cloud Controller running all services (except for compute and some network), a Network Node, and any number of Compute Nodes
  • “Full HA” – That’s 3 Cloud Controllers with all services in High Availability, and any number of Compute Nodes

Now, realistically,  I will only have 1 or 2 compute nodes per any given deployment, but that doesn’t actually change the configuration. 1 Tower server for the Installer/Repo, plus 4 Intel NUCs, plus 3 small switches. For your own environments, you’ll quickly see that you can easily add compute nodes to any given “deployment”. Or not; for our purposes of learning to deploy OpenStack with this tool, 1 node is no different than 10 or 100, except for cost and space.

Goal wise on the network side, I’d like to hopefully show some basic network configurations along the way as well:

  • “Fully Combined” – That’s a basic configuration for kicking the tires.. not really good for production. All traffic traverses the same interface.
  • “Separate Private & Public” – This is probably fine for some fully private clouds as the “Public” doesn’t necessarily mean “outside world”, it just means “not management or storage traffic”. PXE/Management & Tenant (VXLAN, GRE, etc) on one interface, Public on another interface.
  • “Fully Separate” – This is the Full Monty. PXE/Management, Tunnel (GRE, VXLAN, etc), Public (outside world or not) are all on separate interfaces.

This is why I have 3 separate switches. This is also why I have 3 simple switches. I know some networking basics, but I always concentrated on the OS, or the apps, or storage, or anything but the network. I’m in my 40’s now, it’s not changing. Besides, in my home network, I just need to keep the different DHCP servers from competing with each other..

What Specifically are We Deploying Today?

Today we’re working on the RHEL-OSP Installer and RHEL 7 Repo. For me, that happens to be an HP Envy Tower with an Intel I7 CPU, 8GB RAM, and a 1TB of disk space. The disk space is overkill, but everything else is spot on. I’ve done a number of things to try and not only speed things up but better illustrate things.

For example, I’ve got a few BASH scripts that I’ve hacked together that do the host configuration for us (although you will need to edit for your environment). I’ve also got another BASH script to run after the Installer is deployed that configures the virtual Apache server. (Mike Watkins gave me the config, I just converted it to script form because I’m lazy..) Finally, I recorded the deployment so that you could actually watch the process.

How does the RHEL-OSP Installer Work?

Without trying to get too into the weeds, it’s built around Foreman, Puppet, PXE, and Yum, using the same framework used for our Satellite product.. Same look and feel..

The target OpenStack nodes boot using PXE to the Installer, where they are “discovered” and you assign them their particular node roles in a given deployment. At that point, they are rebooted again over PXE to grab their configurations and go. They pull their base OS from the YUM Repo, then pull the OpenStack packages from the Red Hat Mothership.

That’s why they have 2 separate networks and the need for the private to NAT across the public..

Once completed, they let the installer know and then you’re informed that you can hack away on your new OpenStack environment. Or you can edit the deployment and kick it off again. Or you can create a new deployment and re-assign the nodes.. pretty darn cool.

Basic Workflow

  1. Deploy RHEL 6.5 (yes, 6.x, even though we’ll deploy RHEL-OSP 5 on RHEL 7)
  2. Configure host (run 1st script)*
  3. Start RHEL 7 Repo
  4. Deploy Installer (Interactive deployment)
  5. Finish RHEL 7 Repo (run 2nd script)
  6. Log into RHEL-OSP Installer

I don’t have the strongest/fastest connection. It used to be faster, but I’m pretty sure I’m being throttled at the house… Anyway, it took about an hour to get through all 6 steps. I’m willing to bet that on a faster connection you could cut that in half or better – that includes all updates, all scripts, etc.

I strongly suggest you go full screen on the video in order to actually see what’s happening. I recorded it in good resolution. It might blur to start, but give it a sec..

I really hope this helps you out. The next session will include “Installing the Installer” and running the 2nd script (to finish the configuration of the Repo as a virtual web server).

Captain KVM

P.S. – Here’s the 1st script. You’ll need to edit some of the VARS up top to match your environment.

UPDATE – JAN 1, 2015 – Hi folks, in running some tests I realized that I missed a line in the script below that affects the firewall… very important… I inserted the line “-A POSTROUTING -s 192.168.200.0/255.255.0.0 -j MASQUERADE” in the firewall section – specifically at the end of the “*nat” section. The line is what allows the private traffic to traverse the public interface. Without it, “none shall pass!”.

#!/bin/sh
## This is a semi-interactive script meant to speed up the process
## around setting up a RHEL 6.5 server in preparation for configuring
## the RHEL-OSP Installer application
##
## "Semi-interactive" simply means that while most things happen
## behind the scenes, like subscriptions and file copies, things like
## SSH keys still want input.. If you want to take the time to make 
## this fully automated, go for it.
##
## If you have any well thought out, well intended suggestions, find 
## one of the RHEL-OSP Installer related blog posts on my site 
## CaptainKVM dot com and I'd love to see them. You're welcome to 
## disagree respectfully and I will still post your comments. 

# GPL 
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>

# A little reference material, although the first link is for an older version...
# https://access.redhat.com/articles/1150463
# https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux_OpenStack_Platform/5/html/Installer_and_Foreman_Guide/chap-Introduction.html

## VARS for the subscription stuff directly below...
## and the IP info further down...
# The Pool ID comes with your paid subscription or registered 
# evaluation license or partner license.
# The REPO's listed were current at the time of the writing of
# this P.O.S. script - Dec 2014, for RHEL-OSP 5, RHEL-OSP Installer A2 (maybe A3)
POOL_ID=<your_pool_id>
REPO_OS=rhel-6-server-rpms
REPO_OSP=rhel-6-server-openstack-foreman-rpms
REPO_OPT1=rhel-server-rhscl-6-rpms
REPO_OPT2=
LONG=<your_fqdn>
SHORT=<your_short_hostname>
PUB_IP=<pub_ip_for_installer>
PUB_NM=<pub_netmask>
PUB_GW=<pub_gateway>
PUB_DNS1=<pub_dns>
PUB_DNS2=
PRIV_IP=<priv_ip>
PRIV_NM=<priv_netmask>

# setup the proper repos, remove all the unnecessary..
echo -e "\n ## Subscribing to the MotherShip..\n"
subscription-manager register
subscription-manager attach --pool=$POOL_ID
subscription-manager repos --disable=rh*
subscription-manager repos --enable=$REPO_OS
subscription-manager repos --enable=$REPO_OSP
subscription-manager repos --enable=$REPO_OPT1

# If you have the embedded GPU from Intel, you'll need this with RHEL 6.5
# otherwise ignore and wait until updating everything at the end...
# You can comment in/out these particular lines as you need for your particular 
# servers...
echo -e "\n ## Grabbing extra packages and or removing other packages..\n"
#yum -y install xorg-x11-drv-intel
#yum -y update kernel
yum -y remove iwl*
#init 6

# Get rid of the pesky Network Manager...
# Seriously, disable it, or bad things will happen.
echo -e "\n ## Disabling NetworkManager\n"
chkconfig NetworkManager off
service NetworkManager stop

# Add our management hostname to /etc/hosts
echo -e "\n ## Configuring /etc/hosts\n"
cat << EOF > /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
$PRIV_IP $LONG $SHORT
EOF

# and then again to network...
echo -e "\n ## Configuring /etc/sysconfig/network\n"
cat << EOF > /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=$LONG
NETWORKING_IPV6=no
GATEWAY=$PUB_GW
EOF

# Configure the eth0 to be the "public" interface
# I like to put notes in my config files for troubleshooting purposes
# and for sanity checking.. if your interfaces come up in a different
# order, obviously you'll want to switch things around..
# HOWEVER - KEEP IN MIND that this script and the instructs that I use 
# and the instructions I reference expect that eth0 is "public" and eth1
# is "private/PXE"...
echo -e "\n ## Configuring ETHO and ETH1 for public and private networks..\n"
cat << EOF > /etc/sysconfig/network-scripts/ifcfg-eth0
## This is the PCI card
## This should go to the "public" network
## In home lab, the is the 10.0 network
DEVICE=eth0
TYPE=Ethernet
ONBOOT=yes
NM_CONTROLLED=no
BOOTPROTO=static
IPADDR=$PUB_IP
NETMASK=$PUB_NM
EOF

# Configure the eth1 to be the "PXE/mgmt/Provisioning" interface
cat << EOF > /etc/sysconfig/network-scripts/ifcfg-eth1
## This is the onboard card
## This should be the PXE/Mgmt network
## In the home lab this should be the 192.168 network
DEVICE=eth1
TYPE=Ethernet
ONBOOT=yes
NM_CONTROLLED=no
BOOTPROTO=static
IPADDR=$PRIV_IP
NETMASK=$PRIV_NM
EOF

# Configure Name Resolution
cat << EOF > /etc/resolv.conf
nameserver $PUB_DNS1
EOF

# Add the proper MAC addresses to the interface files...
ETHZ=`ifconfig eth0 | grep HWaddr | awk {'print $5'}`
ETHW=`ifconfig eth1 | grep HWaddr | awk {'print $5'}`
echo "HWADDR=$ETHZ">>/etc/sysconfig/network-scripts/ifcfg-eth0
echo "HWADDR=$ETHW">>/etc/sysconfig/network-scripts/ifcfg-eth1

# Get rid of the original UDEV networking rule, just in case
rm -f /etc/udev/rules.d/70-persistent-net.rules

# Make sure NTP is running and off to a good start...
# alternative NTP servers...
# 0.north-america.pool.ntp.org
# 1.north-america.pool.ntp.org
echo -e "\n ## Configuring NTP\n"
service ntpd stop
ntpdate clock.redhat.com
service ntpd start
chkconfig ntpd on

# Backup the original IPtables, then replace with one that handles the 2 interfaces,
# NAT'ing, masqerade, the extra ports for provisioning, etc...
echo -e "\n ## Configuring IPtables\n"
cp /etc/sysconfig/iptables /etc/sysconfig/iptables.orig
echo "" > /etc/sysconfig/iptables
cat << EOF > /etc/sysconfig/iptables
*mangle
:PREROUTING ACCEPT [350:89670]
:INPUT ACCEPT [348:89606]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [282:82514]
:POSTROUTING ACCEPT [282:82514]
COMMIT
*nat
:PREROUTING ACCEPT [12:1231]
:POSTROUTING ACCEPT [26:1797]
:OUTPUT ACCEPT [26:1797]
-A POSTROUTING -s 192.168.200.0/255.255.0.0 -j MASQUERADE
COMMIT
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT 
-A INPUT -p icmp -j ACCEPT 
-A INPUT -i lo -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 80 -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 443 -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 53 -j ACCEPT 
-A INPUT -p udp -m state --state NEW -m udp --dport 53 -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 111 -j ACCEPT 
-A INPUT -p udp -m state --state NEW -m udp --dport 111 -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 32803 -j ACCEPT 
-A INPUT -p udp -m state --state NEW -m udp --dport 32769 -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 2020 -j ACCEPT 
-A INPUT -p udp -m state --state NEW -m udp --dport 2020 -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 662 -j ACCEPT 
-A INPUT -p udp -m state --state NEW -m udp --dport 662 -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 892 -j ACCEPT 
-A INPUT -p udp -m state --state NEW -m udp --dport 892 -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 875 -j ACCEPT 
-A INPUT -p udp -m state --state NEW -m udp --dport 875 -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 2049 -j ACCEPT 
-A INPUT -p udp -m state --state NEW -m udp --dport 2049 -j ACCEPT 
-A INPUT -p udp -m state --state NEW -m udp --dport 67 -j ACCEPT 
-A INPUT -p udp -m state --state NEW -m udp --dport 68 -j ACCEPT 
-A INPUT -p udp -m state --state NEW -m udp --dport 69 -j ACCEPT 
-A INPUT -p udp -m state --state NEW -m udp --dport 8080 -j ACCEPT 
-A INPUT -p udp -m state --state NEW -m udp --dport 8081 -j ACCEPT 
-A INPUT -p udp -m state --state NEW -m udp --dport 8140 -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 8140 -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 8443 -j ACCEPT 
-A INPUT -p tcp -m multiport --dports 3306,9200,4567,4568,4444 -j ACCEPT 
-A FORWARD -o eth+ -j ACCEPT 
-A FORWARD -s 192.168.200.0/24 -j ACCEPT 
-A FORWARD -d 192.168.200.0/24 -j ACCEPT 
-A FORWARD -i eth1 -j ACCEPT 
COMMIT
EOF

# Restart IPtables (yes, the system is about to be rebooted, but this will 
# perform some error checking...
service iptables restart

# Ensure routing can occur between the interfaces:
echo -e "\n ## Enable routing between interfaces\n"
sed -i '/net.ipv4.ip_forward = / s/0/1/' /etc/sysctl.conf
sysctl -p

# Setup SSH Keys
echo -e "\n ## Setup SSH keys\n"
ssh-keygen -t rsa

# Update everything installed
echo -e "\n ## Update the system\n"
yum -y update

# Install the Staypuft/OpenStack Installer
echo -e "\n ## Install RHEL-OSP Installer\n"
yum -y install rhel-osp-installer

# Reboot it!!
echo -e "\n ## OK, if there were no errors in the prep (IPtables restarted fine, etc) then go ahead and reboot now..\n"

11 thoughts on “OpenStack Installer (for RHEL-OSP) pt2”

  1. Hey Jon, I’m glad to hear you talk about the RHEL Openstack Installer in a manner that is commensurate with its maturity. We’re using it at…. well, I can’t say where… but we’re using it and it’s a real bear. Once we’ve got it deploying something(sometimes) we try not to change anything and then try and figure out the missing bits it didn’t deploy or modify correctly. I will say, over the last 6 months I’ve noticed it getting more and more stable as fixes are applied, but, for the love of StayPuft, this thing should really be labeled “not for prime time IMHO(and not necessarily the opinion of my employer or it’s affiliates.)

    1. Hi Tersian,

      Thanks for stopping by and leaving a note. I hear you on the maturity level; I just ask that you open bugs where appropriate. It’s one of those cases where the squeaky wheel really does get the grease. If a product is getting used, and people are opening bugs, guess where the resources go? To that product! If bugs aren’t getting opened, then stuff isn’t going to get fixed. I’ve opened a few bugs already, and I expect to open more in the next several weeks.

      Also, as we work through this multi-part series, I absolutely welcome your lessons learned. “Don’t do this – do this instead”.. “Don’t ever do this. Ever”, “This is a a great hidden feature”, etc.

      thanks again for posting!

      Captain KVM

  2. Jon,

    We’ve opened plenty of tickets and the response has been good. One really problematic area we still struggle with is the need for the installer to own the IPs and the DNS for the default network. We’ve made it work by using non-existent subdomains of our main domain IE: fakesub.example.com when specifying DNS for our private networks for the deployment(default) network but now we’re seeing some issues because they are registering with that subdomain and giving out that bogus info across the public API for metering and things like ceilometer etc. where we would rather see the public interface domain being passed since the monitoring software we are testing correlates the FQDN of it’s base health and welfare with results it is getting from an AMQP feed from ceilometer.

    1. Tersian,

      That’s a cool solution, but now you have to come up w/ a new cool solution to the new cool problem…. ugh. I feel your pain, brother. I just opened an RFE so that users can define software repo’s that they DON’T want to subscribe to. Typical customers will only have a few repos available to them, but employees (like myself) or customers involved with many products and/or beta programs need to be able to say “do not subscribe yourself to the beta channel for RHEL 7”. It’s an easy code addition for and easy solution to a problem that causes lots of issues.

      Thanks for stopping by the blog and thanks for opening BZ’s against the products!!!

      Captain KVM

  3. Here’s another one: Configuring Ceph through the installer after the Ceph infrastructure is already in place. The installer wants to create keys and place them on the ceph nodes but what if we already have ceph running? Well, there’s a few places for keys in the advanced parameters(looks like that just came out in an 11/04 RFE) however, the mon_host array seems to be parsing weird(should be 10.10.10.1, 10,10,10,2, 10,10,10,3 for example) but it doesn’t like “10.10.10.1, 10,10,10,2, 10,10,10,3” nor does it like [“10.10.10.1, 10,10,10,2, 10,10,10,3”] or any other combination I’ve tried. Every puppet run it deploys it as nil, nil, nil. Reminds me of Hogan’s Hero’s, “I see Noooooothing!”

    1. P.S. I’ll reply with the solution when I figure it out tomorrow 🙂 Also, the osp-installer should put in all of the IP4 forwarding and masquerading rules for provisioning when it is installed, including the rp_filter loose settings. If your provisioning network, like you describe, is private, it has to use the build server as a router for traffic to get out to the yum repos unless they are on the build server.

      Coincidentally, we are going to run the installer in a VM on a virtualization host along with all of the repo and install tree servers, that way we don’t need to annoy security with all the IPV4 forwarding, we can simply add an virtual interface into the yum repos from the provisioning network we already have configured on the virtualization server.

      I have yet to see a really good best practices architecture doc/diagram around RedHat’s implementation of the installer and the associated utilities needed to install, especially in an isolated environment where there isn’t open access directly to RHN. Do you know of anything?

      -T

      1. Tersian,

        As for all of the forwarding and masquerading rules, I was trying to condense those along with the puppet, foreman, and NFS rules in one fell swoop with my iptables file as part of my installer script for prepping the Installer host… I’d love to see your Ceph solution and share it with my Ceph buddies. As for running the installer as a VM, I originally intended to do exactly that… but I ran into what I will refer to as ghost issues. In actuality, they were related to networking in a nested virtualization environment.. I know I can solve them, but I wanted to spend my cycles on the actual work and the blog. However, if you share notes and/or “gotchas” from virtualizing it, I’ll give you full credit. If it warrants a full post, I’ll even give you a guest post. Seriously. 🙂

        Lastly, the lack of best practices is really a symptom of a few things. It’s changing enough and will continue to change enough to warrant holding off on any official BP’s. As a matter of fact, as I alluded to in the first post, it’s actually going to change drastically in the next few months for the better. Not as any kind of “bait and switch” or “lets get you addicted to this then pull the plug”… Honestly a “we have to give you something that works in the meantime until we can finish the tool that you deserve”. We’ll be collapsing several tools into one.. like Voltron!!!!! Ok, maybe that’s a little far-fetched, but we’re hard at work..

        Captain KVM

  4. Hi Captain,

    Can I try installing RHEL-OSP HA in VMWare Environment, like running 5 VMs, 3 as Controller node, 1 as compute and 1 as Installer node..? when i gone through some of the docs, for HA, we need to have IPMI Fencing, will this be achievable in the above said environment..? Kindly suggest. Or I need to do this POC with only Physical servers..Please help me.

    1. Vijay,

      You can do this in a VMware environment as you described above. However, you will need to understand that with any nested virtualization environment, your performance will suffer greatly.

      Captain KVM

      1. Thank you Captain, for your reply. This is just to test how HA works on RHEL-OSP. Can i test HA without IPMI Fencing..?

        Regards,
        Vijay

        1. Hi Vijay,

          That’s a different question actually. You need to understand that OpenStack in general (including the RHEL-OSP distribution) is NOT a replacement for vmware or rhev. Anyone that tells you differently is wrong. Trying to put a monolithic application on OpenStack like you would on vmware or rhev or even bare metal is just the completely wrong use case. Follow the longer explanation, then the HA explanation at the end will make more sense.

          The use cases for OpenStack (including RHEL-OSP) dictate that High Availability be built into the application, NOT the infrastructure. For example, in vmware and rhev if a hypervisor fails, the VM’s are restarted elsewhere. If the applications are of low importance, the VM’s are restarted manually; if the applications are of high importance, the VM’s are restarted automatically. In this sense, the hypervisor is still the application in the same sense that a bare metal server is the application.. there is still a high level of dependence on the underlying hardware.

          In OpenStack, if a “compute node” (hypervisor) falls over, no one cares. Really. I mean, someone will figure out why it failed and get it running again, but in terms of the application running, no one cares. The reason is that the application is written and deployed differently. Instead of a monolithic application running once on a hypervisor, the application is broken up into its discrete components and/or microservices. Then those discrete components and/or microservices are deployed many times across many compute nodes across the data center. So if 3 compute nodes go down, the the application utilization goes from maybe 67% to 71%. And if the Cloud Management Platform (like Red Hat CloudForms for example) is told to monitor that application for utilization (among other things) maybe it sees that a threshold has been crossed (70% for more than 20 minutes) and therefor it will spin up a new compute node and more discrete components until the utilization is below 70%.

          The other big difference is that while vmware has an API, it was really meant to be a “point and click” tool. (I’m not saying that as if it were a bad thing.. vsphere is a great platform.) OpenStack (including RHEL-OSP) was designed from the beginning to be driven by API; it just also happens to have a GUI.. but it was really meant to be automated from the get go.

          So, in short, when you’re talking HA with OpenStack, you’re really not talking about fencing a server like in vmware or rhev (although technically you can). When we setup HA in RHEL-OSP, we actually setup the different services (rabbitmq, neutron, etc) in HA so that if a controller goes down, none of the services go unavailable. Unlike the compute nodes, we DO care about controller nodes. 🙂

          Hope this helps,

          Captain KVM

Agree? Disagree? Something to add to the conversation?