OpenShift 3 on OpenStack


The OpenStack project is a libre software cloud computing platform for private and public clouds, which provides an Infrastructure as a Service (IaaS) solution and it’s agnostic in respect kind of applications that can run on it. OpenStack can be used by a software like OpenShift, a libre software PaaS solution in an extremely powerful combination providing a full IaaS+PaaS open source based stack.

OpenShift 3, the last OpenShift version, is not really a smooth evolution of OpenShift 2, it’s a new project based on docker and kubernetes, but Red Hat -the company behind OpenShift- decided to keep the same project name. OpenShift 3 comes in two flavors (AFAIK OpenShift Online has not been updated to version 3 yet): OpenShift Enterprise and OpenShift Origin, available on github. This post describes the installation and configuration of OpenShift Origin version 3 on OpenStack. The procedure followed here is only slightly different from the official documentation, which is strongly recommended to read, but several tips and comments are included in some significant steps.

Specific releases used

It’s always important to describe which software and specific releases are used in a particular implementation, but in this case it’s more important, mainly due to the OpenShift high releasing rate.

OpenStack:

  • OpenStack release: Icehouse (2014.1)
  • OpenStack nodes OS: Debian 8.2 (Jessie)
  • Virtualization: KVM with virtio drivers
  • OpenStack neutron: GRE tunnels with OpenvSwitch 2.3.0

OpenShift:

  • Origin 1.0.6
  • OpenShift nodes OS: CentOS 7.1

Previous steps

Launching instances

A specific flavor is created in OpenStack with 2 vCPU, 8 GiB RAM and 30 GiB of extra ephemeral disk and it’s called m1.xlarge-20. Three CentOS 7.1 instances are launched using this flavor and called master, node1 and node2:
Captura de pantalla de 2015-10-11 11:40:54

Security group rules applied

Instances are launched with a security group allowing instances to connect to each other on specific ports. Specific ports needed for OpenShift is not yet documented (or maybe I didn’t find it out) and the only reference found with this information is Setting up OpenShift 3 on OpenStack. The security group rules proposed on this post are very permissive because they don’t distinguish between internal and external rules, opening several ports to external connections too. We started using it, but this step is in the TODO list.

Captura de pantalla de 2015-10-11 12:37:05

Hostnames

It’s necessary to verify that FQDNs are properly defined in all hosts (in this case we’re using novalocal as domain name):

[centos@master ~]$ hostname -f
master.novalocal

Furthermore, all FQDNs are defined in the local DNS server using its floating IPs, for example node2:

[centos@master ~]$ dig node2.novalocal

; <> DiG 9.9.4-RedHat-9.9.4-18.el7_1.5 <> node2.novalocal
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 38716
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 2

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;node2.novalocal.		IN	A

;; ANSWER SECTION:
node2.novalocal.	86400	IN	A	172.22.202.194

;; AUTHORITY SECTION:
novalocal.		86400	IN	NS	papion.gonzalonazareno.org.

;; ADDITIONAL SECTION:
papion.gonzalonazareno.org. 86400 IN	A	192.168.102.2

;; Query time: 1 msec
;; SERVER: 192.168.102.2#53(192.168.102.2)
;; WHEN: dom oct 11 10:56:35 UTC 2015
;; MSG SIZE  rcvd: 116

Tip: When the instance’s domain name is not properly defined in the OpenStack setup, it can be defined explicitly in the dhcp client configuration file (/etc/dhcp/dhclient.conf in this case), using a line like this:

supersede domain-name "novalocal";

Prerequisites

We’re going to use what is called advanced installation in the OpenShift documentation, this installation is performed with ansible through a complete set of playbooks, but nodes must be prepared before (next steps must be performed in all nodes):

  1. All nodes have an extra disk associated to /dev/vdb automatically mounted on /mnt. We need to umount it and unset the mount reference in /etc/fstab file:

    umount /mnt
    sed -i '/mnt/d' /etc/fstab
    

  2. Some additional packages must be installed:

    yum install wget git net-tools bind-utils iptables-services bridge-utils bash-completion
    

  3. The system must be updated to the latest packages:

    yum update
    

    In our case, this update includes a kernel package, so a reboot is recommended.

  4. Install LVM2 and activate lvmetad (CentOS 7.1 OpenStack image doesn’t include this basic package):

    yum install lvm2
    systemctl enable lvm2-lvmetad.service
    systemctl enable lvm2-lvmetad.socket
    systemctl start lvm2-lvmetad.service
    systemctl start lvm2-lvmetad.socket
    

  5. Install docker (version 1.7.1 is available from CentOS 7.1 repositories, it’s an important point because version 1.6.2 or later is needed):

    yum install docker
    

  6. Allow insecure registry from kubernetes cluster IPs (by default 172.30.0.0/16 subnet):

    sed -i 's/--selinux-enabled/--selinux-enabled --insecure-registry 172.30.0.0\/16/g' /etc/sysconfig/docker
    

  7. Stop docker daemon and remove any containers or images previously created:

    systemctl stop docker.service
    rm -fr /var/lib/docker/*
    

  8. Configure docker storage:

    cat <<EOF > /etc/sysconfig/docker-storage-setup
    DEVS=/dev/vdb
    VG=docker-vg
    SETUP_LVM_THIN_POOL=yes
    EOF
    

  9. Execute the command “docker-storage-setup” and start docker service. After this two actions the result must be similar to:

    [root@master ~]# docker info
    Containers: 0
    Images: 0
    Storage Driver: devicemapper
     Pool Name: docker--vg-docker--pool
     Pool Blocksize: 524.3 kB
     Backing Filesystem: xfs
     Data file: 
     Metadata file: 
     Data Space Used: 20.45 MB
     Data Space Total: 8.577 GB
     Data Space Available: 8.557 GB
     Metadata Space Used: 45.06 kB
     Metadata Space Total: 25.17 MB
     Metadata Space Available: 25.12 MB
     Udev Sync Supported: true
     Deferred Removal Enabled: true
     Library Version: 1.02.93-RHEL7 (2015-01-28)
    Execution Driver: native-0.2
    Logging Driver: json-file
    Kernel Version: 3.10.0-229.el7.x86_64
    Operating System: CentOS Linux 7 (Core)
    CPUs: 2
    Total Memory: 7.798 GiB
    Name: master.novalocal
    ID: Y3BV:5YMH:XW5C:REHO:7KDS:KWGM:KRI4:5COB:MBD7:DVBH:XICL:TIIJ
    [root@master ~]# lvs
      LV          VG        Attr       LSize Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
      docker-pool docker-vg twi-a-t--- 7.99g             0.24   0.18  
    

  10. Advanced installation

    This installation method is performed by ansible, and excellent configuration management tool (if you didn’t use it before this is a great opportunity to do it). Ansible doesn’t need any kind of agent installed on remote machines, it just uses ssh passwordless connections. In this case we use an external computer with ansible able to connect to the nodes through ssh. Let’s comment a detailed step by step procedure:

    1. Install ansible 1.8 or later in the external computer (installation methods are described in ansible documentation).

      Tip: I do usually install ansible through pip inside a python virtual environment, because it a simple method, easy to update and it doesn’t affect your local python setup. Please, never use “sudo pip install”, it can break your system.

    2. Verify ssh passwordless connections can be performed from the computer where ansible is installed.
    3. Clone the ansible repository:

      git clone https://github.com/openshift/openshift-ansible
      

    4. Create file for ansible configuration (ansible.cfg) in current directory with this content:
      [defaults]
      hostfile = ansible_hosts
      remote_user = centos
      host_key_checking = False
      
    5. Create an ansible inventory file with the name defined before (ansible_hosts):
      [OSEv3:children"]
      masters
      nodes
      
      [OSEv3:vars]
      ansible_ssh_user=centos
      osm_default_subdomain=oo.gonzalonazareno.org
      ansible_sudo=true
      
      product_type=openshift
      deployment_type=origin
      
      [masters]
      172.22.202.192 openshift_public_hostname=master.novalocal openshift_ip=10.0.0.36 openshift_public_ip=172.22.202.192
      
      [nodes]
      172.22.202.192 openshift_public_hostname=master.novalocal openshift_ip=10.0.0.36 openshift_public_ip=172.22.202.192 openshift_node_labels="{'region': 'infra', 'zone': 'default'}" 
      172.22.202.193 openshift_public_hostname=node1.novalocal openshift_ip=10.0.0.37 openshift_public_ip=172.22.202.193 openshift_node_labels="{'region': 'primary', 'zone': 'east'}" 
      172.22.202.194 openshift_public_hostname=node2.novalocal openshift_ip=10.0.0.38 openshift_public_ip=172.22.202.194 openshift_node_labels="{'region': 'primary', 'zone': 'west'}" 
      

      Tip: I think it’s better to use these ansible local definitions instead of using generic ones in /etc/ansible. Different ansible playbooks can use their own ansible configuration files instead a generic ones.

      Tip: Two regions are usually defined: infra (infrastructure) and primary for both common containers and apps. Infra region is defined in the master node, so containers like docker-registry or routers will be deployed in this node, whereas normal user applications will be deployed in the region primary in the nodes 1 or 2.

    6. Verify ansible configuration:

      ansible all -m ping
      172.22.202.193 | success >> {
          "changed": false, 
          "ping": "pong"
      }
      
      172.22.202.194 | success >> {
          "changed": false, 
          "ping": "pong"
      }
      
      172.22.202.192 | success >> {
          "changed": false, 
          "ping": "pong"
      }
      

    7. Proceed with the installation launching ansible playbooks:

      ansible-playbook openshift-ansible/playbooks/byo/config.yml
      ...
      (several minutes later)
      ...
      PLAY RECAP ******************************************************************** 
      172.22.202.192             : ok=154  changed=49   unreachable=0    failed=0   
      172.22.202.193             : ok=46   changed=18   unreachable=0    failed=0   
      172.22.202.194             : ok=46   changed=19   unreachable=0    failed=0   
      localhost                  : ok=10   changed=0    unreachable=0    failed=0 
      

    8. In order to verify the correct installation, log in to the master node and type:

      oc get nodes
      

      Something similar to this must be shown:

      NAME               LABELS                                                              STATUS                     AGE
      master.novalocal   kubernetes.io/hostname=master.novalocal,region=infra,zone=default   Ready,SchedulingDisabled   3m
      node1.novalocal    kubernetes.io/hostname=node1.novalocal,region=primary,zone=east     Ready                      3m
      node2.novalocal    kubernetes.io/hostname=node2.novalocal,region=primary,zone=west     Ready                      3m
      

    9. By default master node is marked as unschedulable, but as mentioned before we want to use it in the region infra, so edit master node definition (oc edit node master.novalocal) and delete or comment out the line “unschedulable: true”
    10. Define region primary as default region editing /etc/origin/master/master-config.yaml and modifying the line:
      ...
      projectConfig:
        defaultNodeSelector: "region=primary" 
      

    11. Restart the origin master service:
      systemctl restart origin-master.service
      

    12. Verify master node is in status ready (and is no longer marked as “SchedulingDisabled”:

      oc get nodes
      NAME               LABELS                                                              STATUS    AGE
      master.novalocal   kubernetes.io/hostname=master.novalocal,region=infra,zone=default   Ready     10m
      node1.novalocal    kubernetes.io/hostname=node1.novalocal,region=primary,zone=east     Ready     10m
      node2.novalocal    kubernetes.io/hostname=node2.novalocal,region=primary,zone=west     Ready     10m
      

    13. Edit default project (namespace) and define region infra as node selector for this project: oc edit namespace default (add the line ‘openshift.io/node-selector: region=infra’):
      apiVersion: v1
      kind: Namespace
      metadata:
        annotations:
          openshift.io/node-selector: region=infra
          openshift.io/sa.initialized-roles: "true"
      ...
      
    14. Verify example templates and image streams (is) are installed. Ansible playbooks include the installation of some templates and image streams into the project openshift, readeable by all users:

      [centos@master ~]$ oc get is -n openshift
      NAME         DOCKER REPO                       TAGS          UPDATED
      jenkins      openshift/jenkins-1-centos7       1,latest      11 hours ago
      mongodb      openshift/mongodb-24-centos7      2.4,latest    11 hours ago
      mysql        openshift/mysql-55-centos7        5.5,latest    11 hours ago
      nodejs       openshift/nodejs-010-centos7      0.10,latest   11 hours ago
      perl         openshift/perl-516-centos7        5.16,latest   11 hours ago
      php          openshift/php-55-centos7          5.5,latest    11 hours ago
      postgresql   openshift/postgresql-92-centos7   9.2,latest    11 hours ago
      python       openshift/python-33-centos7       3.3,latest    11 hours ago
      ruby         openshift/ruby-20-centos7         2.0,latest    11 hours ago
      wildfly      openshift/wildfly-81-centos7      latest,8.1    11 hours ago
      
      [centos@master ~]$ oc get templates -n openshift
      NAME                       DESCRIPTION                                                                        PARAMETERS        OBJECTS
      cakephp-example            An example CakePHP application with no database                                    13 (8 blank)      5
      cakephp-mysql-example      An example CakePHP application with a MySQL database                               14 (3 blank)      7
      dancer-example             An example Dancer application with no database                                     6 (3 blank)       5
      dancer-mysql-example       An example Dancer application with a MySQL database                                13 (3 blank)      7
      django-example             An example Django application with no database                                     12 (9 blank)      5
      django-psql-example        An example Django application with a PostgreSQL database                           13 (4 blank)      7
      jenkins-ephemeral          Jenkins service, without persistent storage. WARNING: Any data stored will be...   3 (all set)       3
      jenkins-persistent         Jenkins service, with persistent storage.                                          4 (all set)       4
      mongodb-ephemeral          MongoDB database service, without persistent storage. WARNING: Any data store...   5 (3 generated)   2
      mongodb-persistent         MongoDB database service, with persistent storage. Scaling to more than one r...   6 (3 generated)   3
      mysql-ephemeral            MySQL database service, without persistent storage. WARNING: Any data stored...    4 (2 generated)   2
      mysql-persistent           MySQL database service, with persistent storage. Scaling to more than one rep...   5 (2 generated)   3
      nodejs-example             An example Node.js application with no database                                    10 (8 blank)      5
      nodejs-mongodb-example     An example Node.js application with a MongoDB database                             11 (3 blank)      7
      postgresql-ephemeral       PostgreSQL database service, without persistent storage. WARNING: Any data st...   4 (2 generated)   2
      postgresql-persistent      PostgreSQL database service, with persistent storage. Scaling to more than on...   5 (2 generated)   3
      rails-postgresql-example   An example Rails application with a PostgreSQL database                            16 (3 blank)      7
      

    Configuration

    Installation is finished and now it’s time to configure OpenShift, deploying an integrated docker-registry and a router (used to communicate deployed apps with external networks).

    Deploying a Docker Registry

    Ansible playbooks creates the service account “register” and add it to the privileged security context constraint (SCC), so this step explained in official doc is not needed. If you’re not going to use persistent storage for docker registry (not recommended for production use), just deploy the registry with this two commands as root (or with sudo) in the master node:

    [root@master ~]# cd /etc/origin/master/
    [root@master master]# oadm registry --config=admin.kubeconfig --credentials=openshift-registry.kubeconfig 
    deploymentconfigs/docker-registry
    services/docker-registry
    

    Docker registry image is pulled from docker.io and deploy starts:

    oc get pods
    NAME                       READY     STATUS    RESTARTS   AGE
    docker-registry-1-deploy   0/1       Pending   0          50s
    

    origin-master and origin-node at master show these logs:

    [root@master ~]# journalctl -u origin-master -o cat
    ...
    31216 controller.go:72] Ignoring change for DeploymentConfig default/docker-registry:1; no existing Deployment found
    31216 factory.go:216] About to try and schedule pod docker-registry-1-deploy
    31216 factory.go:319] Attempting to bind docker-registry-1-deploy to master.novalocal
    31216 controller.go:85] Ignoring DeploymentConfig change for default/docker-registry:1 (latestVersion=1); same as Deployment default/docker-registry-1
    
    [root@master ~]# journalctl -u origin-node -o cat
    ...
    proxier.go:310] Adding new service "default/docker-registry:5000-tcp" at 172.30.42.251:5000/TCP
    proxier.go:251] Proxying for service "default/docker-registry:5000-tcp" on TCP port 54737
    roundrobin.go:281] LoadBalancerRR: Removing endpoints for default/docker-registry:5000-tcp
    manager.go:1430] Need to restart pod infra container for "docker-registry-1-deploy_default" because it is not found
    provider.go:91] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
    docker.go:156] Pulling image openshift/origin-pod:v1.0.6 without credentials
    ovs|00001|vsctl|INFO|Called as ovs-vsctl add-port br0 vethbeb3daa
    docker.go:156] Pulling image openshift/origin-deployer:v1.0.6 without credentials
    manager.go:1430] Need to restart pod infra container for "docker-registry-1-z72ff_default" because it is not found
    ovs|00001|vsctl|INFO|Called as ovs-vsctl add-port br0 veth81e5052
    provider.go:91] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
    docker.go:156] Pulling image openshift/origin-docker-registry:v1.0.6 without credentials
    roundrobin.go:263] LoadBalancerRR: Setting endpoints for default/docker-registry:5000-tcp to [10.1.0.3:5000]
    master.novalocal ovs-vsctl[32733]: ovs|00001|vsctl|INFO|Called as ovs-vsctl del-port vethbeb3daa
    master.novalocal origin-node[30535]: I1007 10:15:14.663275   30535 manager.go:1180] Killing container with id "b4d6155a25558221b025e670a397ed37674ddc3b2e9250a68a882e70b1d5b783" 
    

    After a few moment docker registry deploy finishes, the docker registry pod remains running and docker registry service becomes available:

    oc get pods
    NAME                      READY     STATUS    RESTARTS   AGE
    docker-registry-1-z72ff   1/1       Running   0          5m
    
    oc get svc
    NAME              CLUSTER_IP      EXTERNAL_IP   PORT(S)    SELECTOR                  AGE
    docker-registry   172.30.42.251   <none>        5000/TCP   docker-registry=default   23m
    kubernetes        172.30.0.1      <none>        443/TCP    <none> 
    

    Three docker images are now into the node master docker (they’ve been used to deploy docker-registry):

    [root@master ~]# docker images
    REPOSITORY                                   TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
    docker.io/openshift/origin-deployer          v1.0.6              3738f952ffbe        3 weeks ago         421.1 MB
    docker.io/openshift/origin-docker-registry   v1.0.6              293aa25bf219        3 weeks ago         285.2 MB
    docker.io/openshift/origin-pod               v1.0.6              47bcd97081ef        3 weeks ago         1.105 MB
    

    Two containers are running into the node master docker (both belonging to docker-registry pod):

    [root@master ~]# docker ps
    CONTAINER ID        IMAGE                                     COMMAND                CREATED             STATUS              PORTS               NAMES
    0b1d121b66ac        openshift/origin-docker-registry:v1.0.6   "/bin/sh -c 'REGISTR   16 minutes ago      Up 16 minutes                           k8s_registry.e5761bf2_docker-registry-1-z72ff_default_2f0c0f4b-6cdc-11e5-839e-fa163e8c6e5b_06315515   
    bf5a1b9461bc        openshift/origin-pod:v1.0.6               "/pod"                 17 minutes ago      Up 17 minutes                           k8s_POD.75f3f5d2_docker-registry-1-z72ff_default_2f0c0f4b-6cdc-11e5-839e-fa163e8c6e5b_ef862b76
    

    Verifying docker registry

    It’s recommended to verify the docker-registry correct operation. Following the steps indicated in the official doc, log into OpenShift as a normal user from node1 or node2 (this will also test network communications and OpenShift SDN):

    [centos@node1 ~]$ oc login master.novalocal --certificate-authority /etc/origin/node/ca.crt
    Authentication required for https://master.novalocal:8443 (openshift)
    Username: test
    Password: (any passwaord can be used)
    

    Get the access token:

    [centos@node1 ~]$ oc whoami -t
    gfr9IR1pgkJBDtc8ElJXMjWsQtkQ7F2jcUTof49FaFo
    

    Log in to the Docker registry:

    [centos@node1 ~]$ sudo docker login -u test -e test@localhost -p gfr9IR1pgkJBDtc8ElJXMjWsQtkQ7F2jcUTof49FaFo 172.30.42.251:5000
    WARNING: login credentials saved in /root/.docker/config.json
    Login Succeeded
    

    You can now perform docker pull and docker push operations against your registry. For example:

    [centos@node1 ~]$ sudo docker pull docker.io/busybox
    [centos@node1 ~]$ sudo docker tag docker.io/busybox 172.30.124.220:5000/test/busybox
    [centos@node1 ~]$ sudo docker push 172.30.124.220:5000/test/busybox
    The push refers to a repository [172.30.214.2208:5000/test/busybox] (len: 1)
    d7057cb02084: Image already exists 
    cfa753dfea5e: Image successfully pushed
    

    Tip: This has been the major issue found during OpenShift installation on OpenStack, because the push never ended and after a while a timeout was shown. Finally, we discovered that the issue was caused by a specific network configuration of our OpenStack setup, because it didn’t have network virtio drivers enabled (due to a previous OpenStack bug currently solved) and this comment pointed me in the right direction. Turning on network virtio driver for OpenStack solves the issue.

    Deploying a router

    In a similar manner as was done before with the docker registry, a router can be deployed:

    [root@master ~]# cd /etc/origin/master/
    [root@master master]# oadm router router --credentials=openshift-router.kubeconfig --service-account=router
    deploymentconfigs/router
    services/router
    

    After a few moments both router and docker-registry are correctly deployed into the master node (infra region):

    oc get pods -o wide
    NAME                      READY     STATUS    RESTARTS   AGE       NODE
    docker-registry-1-z72ff   1/1       Running   0          22m       master.novalocal
    router-1-5khf4            1/1       Running   0          1m        master.novalocal
    

    An OpenShift router is really a ha-proxy listening on master external interface (note: this is just the default plugin). This ha-proxy allows web applications deployed inside the kubernetes cluster be accessible from external networks. In order to see ha-proxy statistics we can see the generated username and password using the extended description of a pod (using JSON or YAML output):

    oc get pod router-1-5khf4 -o json|jq '.spec.containers[0].env[]
    ...
      "name": "ROUTER_SERVICE_NAME",
      "value": "router" 
    }
    {
      "name": "ROUTER_SERVICE_NAMESPACE",
      "value": "default" 
    }
    {
      "name": "STATS_PASSWORD",
      "value": "AjEbR2Fr7q" 
    }
    {
      "name": "STATS_PORT",
      "value": "1936" 
    }
    {
      "name": "STATS_USERNAME",
      "value": "admin" 
    

    Tip: The last command was filtered using jq, a command line JSON processor.

    At this moment is not possible to accesss to ha-proxy statistics on port 1936/tcp, but there is a bug opened reporting this issue and it’s expected than can be solved in the next OpenShift release.

    As a final conclusion, we can say that installing OpenShift on OpenStack is not a difficult procedure, especially when you have some previous experience with kubernetes and docker. The major issue found was not related to OpenShift but a specific OpenStack setup. Official documentation is an excellent starting point and this post just add several tips and comment that can be useful to others.

    References

, , ,

  1. Deja un comentario

Responder

Introduce tus datos o haz clic en un icono para iniciar sesión:

Logo de WordPress.com

Estás comentando usando tu cuenta de WordPress.com. Cerrar sesión / Cambiar )

Imagen de Twitter

Estás comentando usando tu cuenta de Twitter. Cerrar sesión / Cambiar )

Foto de Facebook

Estás comentando usando tu cuenta de Facebook. Cerrar sesión / Cambiar )

Google+ photo

Estás comentando usando tu cuenta de Google+. Cerrar sesión / Cambiar )

Conectando a %s

A %d blogueros les gusta esto: