The OpenStack project is a libre software cloud computing platform for private and public clouds, which provides an Infrastructure as a Service (IaaS) solution and it’s agnostic in respect kind of applications that can run on it. OpenStack can be used by a software like OpenShift, a libre software PaaS solution in an extremely powerful combination providing a full IaaS+PaaS open source based stack.
OpenShift 3, the last OpenShift version, is not really a smooth evolution of OpenShift 2, it’s a new project based on docker and kubernetes, but Red Hat -the company behind OpenShift- decided to keep the same project name. OpenShift 3 comes in two flavors (AFAIK OpenShift Online has not been updated to version 3 yet): OpenShift Enterprise and OpenShift Origin, available on github. This post describes the installation and configuration of OpenShift Origin version 3 on OpenStack. The procedure followed here is only slightly different from the official documentation, which is strongly recommended to read, but several tips and comments are included in some significant steps.
Specific releases used
It’s always important to describe which software and specific releases are used in a particular implementation, but in this case it’s more important, mainly due to the OpenShift high releasing rate.
OpenStack:
- OpenStack release: Icehouse (2014.1)
- OpenStack nodes OS: Debian 8.2 (Jessie)
- Virtualization: KVM with virtio drivers
- OpenStack neutron: GRE tunnels with OpenvSwitch 2.3.0
OpenShift:
- Origin 1.0.6
- OpenShift nodes OS: CentOS 7.1
Previous steps
Launching instances
A specific flavor is created in OpenStack with 2 vCPU, 8 GiB RAM and 30 GiB of extra ephemeral disk and it’s called m1.xlarge-20. Three CentOS 7.1 instances are launched using this flavor and called master, node1 and node2:
Security group rules applied
Instances are launched with a security group allowing instances to connect to each other on specific ports. Specific ports needed for OpenShift is not yet documented (or maybe I didn’t find it out) and the only reference found with this information is Setting up OpenShift 3 on OpenStack. The security group rules proposed on this post are very permissive because they don’t distinguish between internal and external rules, opening several ports to external connections too. We started using it, but this step is in the TODO list.
Hostnames
It’s necessary to verify that FQDNs are properly defined in all hosts (in this case we’re using novalocal as domain name):
[centos@master ~]$ hostname -f master.novalocal
Furthermore, all FQDNs are defined in the local DNS server using its floating IPs, for example node2:
[centos@master ~]$ dig node2.novalocal ; <> DiG 9.9.4-RedHat-9.9.4-18.el7_1.5 <> node2.novalocal ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 38716 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 2 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;node2.novalocal. IN A ;; ANSWER SECTION: node2.novalocal. 86400 IN A 172.22.202.194 ;; AUTHORITY SECTION: novalocal. 86400 IN NS papion.gonzalonazareno.org. ;; ADDITIONAL SECTION: papion.gonzalonazareno.org. 86400 IN A 192.168.102.2 ;; Query time: 1 msec ;; SERVER: 192.168.102.2#53(192.168.102.2) ;; WHEN: dom oct 11 10:56:35 UTC 2015 ;; MSG SIZE rcvd: 116
Tip: When the instance’s domain name is not properly defined in the OpenStack setup, it can be defined explicitly in the dhcp client configuration file (/etc/dhcp/dhclient.conf in this case), using a line like this:
supersede domain-name "novalocal";
Prerequisites
We’re going to use what is called advanced installation in the OpenShift documentation, this installation is performed with ansible through a complete set of playbooks, but nodes must be prepared before (next steps must be performed in all nodes):
- All nodes have an extra disk associated to /dev/vdb automatically mounted on /mnt. We need to umount it and unset the mount reference in /etc/fstab file:
umount /mnt sed -i '/mnt/d' /etc/fstab
- Some additional packages must be installed:
yum install wget git net-tools bind-utils iptables-services bridge-utils bash-completion
- The system must be updated to the latest packages:
yum update
In our case, this update includes a kernel package, so a reboot is recommended.
- Install LVM2 and activate lvmetad (CentOS 7.1 OpenStack image doesn’t include this basic package):
yum install lvm2 systemctl enable lvm2-lvmetad.service systemctl enable lvm2-lvmetad.socket systemctl start lvm2-lvmetad.service systemctl start lvm2-lvmetad.socket
- Install docker (version 1.7.1 is available from CentOS 7.1 repositories, it’s an important point because version 1.6.2 or later is needed):
yum install docker
- Allow insecure registry from kubernetes cluster IPs (by default 172.30.0.0/16 subnet):
sed -i 's/--selinux-enabled/--selinux-enabled --insecure-registry 172.30.0.0\/16/g' /etc/sysconfig/docker
- Stop docker daemon and remove any containers or images previously created:
systemctl stop docker.service rm -fr /var/lib/docker/*
- Configure docker storage:
cat <<EOF > /etc/sysconfig/docker-storage-setup DEVS=/dev/vdb VG=docker-vg SETUP_LVM_THIN_POOL=yes EOF
- Execute the command «docker-storage-setup» and start docker service. After this two actions the result must be similar to:
[root@master ~]# docker info Containers: 0 Images: 0 Storage Driver: devicemapper Pool Name: docker--vg-docker--pool Pool Blocksize: 524.3 kB Backing Filesystem: xfs Data file: Metadata file: Data Space Used: 20.45 MB Data Space Total: 8.577 GB Data Space Available: 8.557 GB Metadata Space Used: 45.06 kB Metadata Space Total: 25.17 MB Metadata Space Available: 25.12 MB Udev Sync Supported: true Deferred Removal Enabled: true Library Version: 1.02.93-RHEL7 (2015-01-28) Execution Driver: native-0.2 Logging Driver: json-file Kernel Version: 3.10.0-229.el7.x86_64 Operating System: CentOS Linux 7 (Core) CPUs: 2 Total Memory: 7.798 GiB Name: master.novalocal ID: Y3BV:5YMH:XW5C:REHO:7KDS:KWGM:KRI4:5COB:MBD7:DVBH:XICL:TIIJ [root@master ~]# lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert docker-pool docker-vg twi-a-t--- 7.99g 0.24 0.18
- Install ansible 1.8 or later in the external computer (installation methods are described in ansible documentation).
Tip: I do usually install ansible through pip inside a python virtual environment, because it a simple method, easy to update and it doesn’t affect your local python setup. Please, never use «sudo pip install», it can break your system.
- Verify ssh passwordless connections can be performed from the computer where ansible is installed.
- Clone the ansible repository:
git clone https://github.com/openshift/openshift-ansible
- Create file for ansible configuration (ansible.cfg) in current directory with this content:
[defaults] hostfile = ansible_hosts remote_user = centos host_key_checking = False
- Create an ansible inventory file with the name defined before (ansible_hosts):
[OSEv3:children"] masters nodes [OSEv3:vars] ansible_ssh_user=centos osm_default_subdomain=oo.gonzalonazareno.org ansible_sudo=true product_type=openshift deployment_type=origin [masters] 172.22.202.192 openshift_public_hostname=master.novalocal openshift_ip=10.0.0.36 openshift_public_ip=172.22.202.192 [nodes] 172.22.202.192 openshift_public_hostname=master.novalocal openshift_ip=10.0.0.36 openshift_public_ip=172.22.202.192 openshift_node_labels="{'region': 'infra', 'zone': 'default'}" 172.22.202.193 openshift_public_hostname=node1.novalocal openshift_ip=10.0.0.37 openshift_public_ip=172.22.202.193 openshift_node_labels="{'region': 'primary', 'zone': 'east'}" 172.22.202.194 openshift_public_hostname=node2.novalocal openshift_ip=10.0.0.38 openshift_public_ip=172.22.202.194 openshift_node_labels="{'region': 'primary', 'zone': 'west'}"Tip: I think it’s better to use these ansible local definitions instead of using generic ones in /etc/ansible. Different ansible playbooks can use their own ansible configuration files instead a generic ones.
Tip: Two regions are usually defined: infra (infrastructure) and primary for both common containers and apps. Infra region is defined in the master node, so containers like docker-registry or routers will be deployed in this node, whereas normal user applications will be deployed in the region primary in the nodes 1 or 2.
- Verify ansible configuration:
ansible all -m ping 172.22.202.193 | success >> { "changed": false, "ping": "pong" } 172.22.202.194 | success >> { "changed": false, "ping": "pong" } 172.22.202.192 | success >> { "changed": false, "ping": "pong" } - Proceed with the installation launching ansible playbooks:
ansible-playbook openshift-ansible/playbooks/byo/config.yml ... (several minutes later) ... PLAY RECAP ******************************************************************** 172.22.202.192 : ok=154 changed=49 unreachable=0 failed=0 172.22.202.193 : ok=46 changed=18 unreachable=0 failed=0 172.22.202.194 : ok=46 changed=19 unreachable=0 failed=0 localhost : ok=10 changed=0 unreachable=0 failed=0
- In order to verify the correct installation, log in to the master node and type:
oc get nodes
Something similar to this must be shown:
NAME LABELS STATUS AGE master.novalocal kubernetes.io/hostname=master.novalocal,region=infra,zone=default Ready,SchedulingDisabled 3m node1.novalocal kubernetes.io/hostname=node1.novalocal,region=primary,zone=east Ready 3m node2.novalocal kubernetes.io/hostname=node2.novalocal,region=primary,zone=west Ready 3m
- By default master node is marked as unschedulable, but as mentioned before we want to use it in the region infra, so edit master node definition (oc edit node master.novalocal) and delete or comment out the line «unschedulable: true»
- Define region primary as default region editing /etc/origin/master/master-config.yaml and modifying the line:
... projectConfig: defaultNodeSelector: "region=primary"
- Restart the origin master service:
systemctl restart origin-master.service
- Verify master node is in status ready (and is no longer marked as «SchedulingDisabled»:
oc get nodes NAME LABELS STATUS AGE master.novalocal kubernetes.io/hostname=master.novalocal,region=infra,zone=default Ready 10m node1.novalocal kubernetes.io/hostname=node1.novalocal,region=primary,zone=east Ready 10m node2.novalocal kubernetes.io/hostname=node2.novalocal,region=primary,zone=west Ready 10m
- Edit default project (namespace) and define region infra as node selector for this project: oc edit namespace default (add the line ‘openshift.io/node-selector: region=infra’):
apiVersion: v1 kind: Namespace metadata: annotations: openshift.io/node-selector: region=infra openshift.io/sa.initialized-roles: "true" ... - Verify example templates and image streams (is) are installed. Ansible playbooks include the installation of some templates and image streams into the project openshift, readeable by all users:
[centos@master ~]$ oc get is -n openshift NAME DOCKER REPO TAGS UPDATED jenkins openshift/jenkins-1-centos7 1,latest 11 hours ago mongodb openshift/mongodb-24-centos7 2.4,latest 11 hours ago mysql openshift/mysql-55-centos7 5.5,latest 11 hours ago nodejs openshift/nodejs-010-centos7 0.10,latest 11 hours ago perl openshift/perl-516-centos7 5.16,latest 11 hours ago php openshift/php-55-centos7 5.5,latest 11 hours ago postgresql openshift/postgresql-92-centos7 9.2,latest 11 hours ago python openshift/python-33-centos7 3.3,latest 11 hours ago ruby openshift/ruby-20-centos7 2.0,latest 11 hours ago wildfly openshift/wildfly-81-centos7 latest,8.1 11 hours ago [centos@master ~]$ oc get templates -n openshift NAME DESCRIPTION PARAMETERS OBJECTS cakephp-example An example CakePHP application with no database 13 (8 blank) 5 cakephp-mysql-example An example CakePHP application with a MySQL database 14 (3 blank) 7 dancer-example An example Dancer application with no database 6 (3 blank) 5 dancer-mysql-example An example Dancer application with a MySQL database 13 (3 blank) 7 django-example An example Django application with no database 12 (9 blank) 5 django-psql-example An example Django application with a PostgreSQL database 13 (4 blank) 7 jenkins-ephemeral Jenkins service, without persistent storage. WARNING: Any data stored will be... 3 (all set) 3 jenkins-persistent Jenkins service, with persistent storage. 4 (all set) 4 mongodb-ephemeral MongoDB database service, without persistent storage. WARNING: Any data store... 5 (3 generated) 2 mongodb-persistent MongoDB database service, with persistent storage. Scaling to more than one r... 6 (3 generated) 3 mysql-ephemeral MySQL database service, without persistent storage. WARNING: Any data stored... 4 (2 generated) 2 mysql-persistent MySQL database service, with persistent storage. Scaling to more than one rep... 5 (2 generated) 3 nodejs-example An example Node.js application with no database 10 (8 blank) 5 nodejs-mongodb-example An example Node.js application with a MongoDB database 11 (3 blank) 7 postgresql-ephemeral PostgreSQL database service, without persistent storage. WARNING: Any data st... 4 (2 generated) 2 postgresql-persistent PostgreSQL database service, with persistent storage. Scaling to more than on... 5 (2 generated) 3 rails-postgresql-example An example Rails application with a PostgreSQL database 16 (3 blank) 7
Advanced installation
This installation method is performed by ansible, and excellent configuration management tool (if you didn’t use it before this is a great opportunity to do it). Ansible doesn’t need any kind of agent installed on remote machines, it just uses ssh passwordless connections. In this case we use an external computer with ansible able to connect to the nodes through ssh. Let’s comment a detailed step by step procedure:
Configuration
Installation is finished and now it’s time to configure OpenShift, deploying an integrated docker-registry and a router (used to communicate deployed apps with external networks).
Deploying a Docker Registry
Ansible playbooks creates the service account «register» and add it to the privileged security context constraint (SCC), so this step explained in official doc is not needed. If you’re not going to use persistent storage for docker registry (not recommended for production use), just deploy the registry with this two commands as root (or with sudo) in the master node:
[root@master ~]# cd /etc/origin/master/ [root@master master]# oadm registry --config=admin.kubeconfig --credentials=openshift-registry.kubeconfig deploymentconfigs/docker-registry services/docker-registry
Docker registry image is pulled from docker.io and deploy starts:
oc get pods NAME READY STATUS RESTARTS AGE docker-registry-1-deploy 0/1 Pending 0 50s
origin-master and origin-node at master show these logs:
[root@master ~]# journalctl -u origin-master -o cat ... 31216 controller.go:72] Ignoring change for DeploymentConfig default/docker-registry:1; no existing Deployment found 31216 factory.go:216] About to try and schedule pod docker-registry-1-deploy 31216 factory.go:319] Attempting to bind docker-registry-1-deploy to master.novalocal 31216 controller.go:85] Ignoring DeploymentConfig change for default/docker-registry:1 (latestVersion=1); same as Deployment default/docker-registry-1 [root@master ~]# journalctl -u origin-node -o cat ... proxier.go:310] Adding new service "default/docker-registry:5000-tcp" at 172.30.42.251:5000/TCP proxier.go:251] Proxying for service "default/docker-registry:5000-tcp" on TCP port 54737 roundrobin.go:281] LoadBalancerRR: Removing endpoints for default/docker-registry:5000-tcp manager.go:1430] Need to restart pod infra container for "docker-registry-1-deploy_default" because it is not found provider.go:91] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider docker.go:156] Pulling image openshift/origin-pod:v1.0.6 without credentials ovs|00001|vsctl|INFO|Called as ovs-vsctl add-port br0 vethbeb3daa docker.go:156] Pulling image openshift/origin-deployer:v1.0.6 without credentials manager.go:1430] Need to restart pod infra container for "docker-registry-1-z72ff_default" because it is not found ovs|00001|vsctl|INFO|Called as ovs-vsctl add-port br0 veth81e5052 provider.go:91] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider docker.go:156] Pulling image openshift/origin-docker-registry:v1.0.6 without credentials roundrobin.go:263] LoadBalancerRR: Setting endpoints for default/docker-registry:5000-tcp to [10.1.0.3:5000] master.novalocal ovs-vsctl[32733]: ovs|00001|vsctl|INFO|Called as ovs-vsctl del-port vethbeb3daa master.novalocal origin-node[30535]: I1007 10:15:14.663275 30535 manager.go:1180] Killing container with id "b4d6155a25558221b025e670a397ed37674ddc3b2e9250a68a882e70b1d5b783"
After a few moment docker registry deploy finishes, the docker registry pod remains running and docker registry service becomes available:
oc get pods NAME READY STATUS RESTARTS AGE docker-registry-1-z72ff 1/1 Running 0 5m oc get svc NAME CLUSTER_IP EXTERNAL_IP PORT(S) SELECTOR AGE docker-registry 172.30.42.251 <none> 5000/TCP docker-registry=default 23m kubernetes 172.30.0.1 <none> 443/TCP <none>
Three docker images are now into the node master docker (they’ve been used to deploy docker-registry):
[root@master ~]# docker images REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE docker.io/openshift/origin-deployer v1.0.6 3738f952ffbe 3 weeks ago 421.1 MB docker.io/openshift/origin-docker-registry v1.0.6 293aa25bf219 3 weeks ago 285.2 MB docker.io/openshift/origin-pod v1.0.6 47bcd97081ef 3 weeks ago 1.105 MB
Two containers are running into the node master docker (both belonging to docker-registry pod):
[root@master ~]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 0b1d121b66ac openshift/origin-docker-registry:v1.0.6 "/bin/sh -c 'REGISTR 16 minutes ago Up 16 minutes k8s_registry.e5761bf2_docker-registry-1-z72ff_default_2f0c0f4b-6cdc-11e5-839e-fa163e8c6e5b_06315515 bf5a1b9461bc openshift/origin-pod:v1.0.6 "/pod" 17 minutes ago Up 17 minutes k8s_POD.75f3f5d2_docker-registry-1-z72ff_default_2f0c0f4b-6cdc-11e5-839e-fa163e8c6e5b_ef862b76
Verifying docker registry
It’s recommended to verify the docker-registry correct operation. Following the steps indicated in the official doc, log into OpenShift as a normal user from node1 or node2 (this will also test network communications and OpenShift SDN):
[centos@node1 ~]$ oc login master.novalocal --certificate-authority /etc/origin/node/ca.crt Authentication required for https://master.novalocal:8443 (openshift) Username: test Password: (any passwaord can be used)
Get the access token:
[centos@node1 ~]$ oc whoami -t gfr9IR1pgkJBDtc8ElJXMjWsQtkQ7F2jcUTof49FaFo
Log in to the Docker registry:
[centos@node1 ~]$ sudo docker login -u test -e test@localhost -p gfr9IR1pgkJBDtc8ElJXMjWsQtkQ7F2jcUTof49FaFo 172.30.42.251:5000 WARNING: login credentials saved in /root/.docker/config.json Login Succeeded
You can now perform docker pull and docker push operations against your registry. For example:
[centos@node1 ~]$ sudo docker pull docker.io/busybox [centos@node1 ~]$ sudo docker tag docker.io/busybox 172.30.124.220:5000/test/busybox [centos@node1 ~]$ sudo docker push 172.30.124.220:5000/test/busybox The push refers to a repository [172.30.214.2208:5000/test/busybox] (len: 1) d7057cb02084: Image already exists cfa753dfea5e: Image successfully pushed
Tip: This has been the major issue found during OpenShift installation on OpenStack, because the push never ended and after a while a timeout was shown. Finally, we discovered that the issue was caused by a specific network configuration of our OpenStack setup, because it didn’t have network virtio drivers enabled (due to a previous OpenStack bug currently solved) and this comment pointed me in the right direction. Turning on network virtio driver for OpenStack solves the issue.
Deploying a router
In a similar manner as was done before with the docker registry, a router can be deployed:
[root@master ~]# cd /etc/origin/master/ [root@master master]# oadm router router --credentials=openshift-router.kubeconfig --service-account=router deploymentconfigs/router services/router
After a few moments both router and docker-registry are correctly deployed into the master node (infra region):
oc get pods -o wide NAME READY STATUS RESTARTS AGE NODE docker-registry-1-z72ff 1/1 Running 0 22m master.novalocal router-1-5khf4 1/1 Running 0 1m master.novalocal
An OpenShift router is really a ha-proxy listening on master external interface (note: this is just the default plugin). This ha-proxy allows web applications deployed inside the kubernetes cluster be accessible from external networks. In order to see ha-proxy statistics we can see the generated username and password using the extended description of a pod (using JSON or YAML output):
oc get pod router-1-5khf4 -o json|jq '.spec.containers[0].env[]
...
"name": "ROUTER_SERVICE_NAME",
"value": "router"
}
{
"name": "ROUTER_SERVICE_NAMESPACE",
"value": "default"
}
{
"name": "STATS_PASSWORD",
"value": "AjEbR2Fr7q"
}
{
"name": "STATS_PORT",
"value": "1936"
}
{
"name": "STATS_USERNAME",
"value": "admin"
Tip: The last command was filtered using jq, a command line JSON processor.
At this moment is not possible to accesss to ha-proxy statistics on port 1936/tcp, but there is a bug opened reporting this issue and it’s expected than can be solved in the next OpenShift release.
As a final conclusion, we can say that installing OpenShift on OpenStack is not a difficult procedure, especially when you have some previous experience with kubernetes and docker. The major issue found was not related to OpenShift but a specific OpenStack setup. Official documentation is an excellent starting point and this post just add several tips and comment that can be useful to others.
Desde lo alto del Cerro 