How to install Cumin

Cumin is open Source program written in Python that allows you to execute multiple commands on multiple hosts in parallel. For more information on Cumin please see the references section.

Prerequisites

For this tutorial, you need:

- pupetmaster node

- puppetdb node

- cumin node

- some nodes for testing

Environment

Nodes names

All nodes in this tutorial are running Debian Stretch with Puppet 6

- Puppetmaster1002 : 10.192.32.4 Puppet master node

- Puppetdb1002 : IP address 10.192.32.10 PuppetDb node

- cumin1001 : IP address 10.192.48.13 Running cumin

- lab1001: Test node

Installation

One of the required package for Cumin 3.0.2 is python3-clustershell. Debian Stretch doesn't have that package so you will need to download it manually on your puppetmaster and the cumin class will install it. ( See cumin class below)

ppaul@cumin1001:~$ sudo apt-get install python3-clustershell
Reading package lists... Done
Building dependency tree       
Reading state information... Done
E: Unable to locate package python3-clustershell

You can also install Debian Buster if you want on the cumin node since Debian Buster has the package.

- Test from a node running Debian Buster

ppaul@lab1002:~$ sudo apt-get install python3-clustershell
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Suggested packages:
 vim-addon-manager
The following NEW packages will be installed:
 python3-clustershell
0 upgraded, 1 newly installed, 0 to remove and 1 not upgraded.
Need to get 126 kB of archives.
After this operation, 625 kB of additional disk space will be used.
Get:1 http://apt1001.dfw.labnet/debian buster/main amd64 python3-clustershell all 1.8.1-1 [126 kB]
Fetched 126 kB in 1s (249 kB/s)          
Selecting previously unselected package python3-clustershell.
(Reading database ... 57033 files and directories currently installed.)
Preparing to unpack .../python3-clustershell_1.8.1-1_all.deb ...
Unpacking python3-clustershell (1.8.1-1) ...
Setting up python3-clustershell (1.8.1-1) ...

cumin Class

On your puppetmaster create the cumin class

cd /etc/puppetlabs/code/environments/production/modules
sudo mkdir cumin
cd cumin
sudo mkdir manifests
sudo mkdir templates
cd manifests

Create the master.pp file, copy and paste the contain below

class profile::cumin::master (
   $puppetdb_host  = hiera('puppetdb_host'),
) {
   $cumin_log_path = '/var/log/cumin'
     require_package([
       'python3-dnspython',
       'python3-colorama',
       'python3-pyparsing',
       'python3-tqdm',
       'python3-requests',
       'python3-yaml',
   ])
   file { '/srv/cumin_3.0.2-1_amd64.deb':
     ensure    => present,
     owner      => 'root',
     group      => 'root',
     source     => 'puppet:///modules/cumin/cumin_3.0.2-1_amd64.deb',
   }
   file { '/srv/python3-clustershell_1.8.1-2_all.deb':
     ensure     => present,
     owner      => 'root',
     group      => 'root',
     source     => 'puppet:///modules/cumin/python3-clustershell_1.8.1-2_all.deb',
   }
  package { 'python3-clusterhsell':
     ensure     => installed,
     provider   => dpkg,
     source     => '/srv/python3-clustershell_1.8.1-2_all.deb',
     require    => File['/srv/python3-clustershell_1.8.1-2_all.deb'],
  }
   package { 'cumin':
     ensure     => installed,
     provider   => dpkg,
     source     => '/srv/cumin_3.0.2-1_amd64.deb',
     require    => File['/srv/cumin_3.0.2-1_amd64.deb'],
   } 
   file { $cumin_log_path:
       ensure => directory,
       owner  => 'root',
       group  => 'root',
       mode   => '0750',
   } 
file { '/etc/cumin':
       ensure => directory,
       owner  => 'root',
       group  => 'root',
       mode   => '0755',
   }
  file { '/etc/cumin/config.yaml':
       ensure  => present,
       owner   => 'root',
       group   => 'root',
       mode    => '0640',
       content => template('cumin/config.yaml.erb'),
       require => File['/etc/cumin'],
   }
   file { '/etc/cumin/config-installer.yaml':
       ensure  => present,
       owner   => 'root',
       group   => 'root',
       mode    => '0640',
       content => template('cumin/config-installer.yaml.erb'),
       require => File['/etc/cumin'],
   }
   file { '/etc/cumin/aliases.yaml':
       ensure  => present,
       owner   => 'root',
       group   => 'root',
       mode    => '0644',
       content => template('cumin/aliases.yaml.erb'),
       require => File['/etc/cumin'],
   }
if os_version('debian == stretch') {
       $python_version = '3.5'
   } else {
       $python_version = '3.6'
   }
}

File needed

Download cumin_3.0.2-1_amd64.deb and python3-clustershell_1.8.1-2_all.deb and save those files in /etc/puppetlabs/code/environments/production/modules/cumin/files

http://ftp.us.debian.org/debian/pool/main/c/clustershell/python3-clustershell_1.8.1-2_all.deb
https://github.com/wikimedia/cumin/releases/download/v3.0.2/cumin_3.0.2-1_amd64.deb

Create aliases.yaml.erb, config-installer.yaml.erb and config.yaml.erb in /etc/puppetlabs/code/environments/production/modules/cumin/templates For now we are going just to focus on the config.yaml.erb file. Open the config.yaml.erb and copy and paste the contain below. Leave the other files empty for now.

transport: clustershell
 log_file: <%= @cumin_log_path %>/cumin.log
default_backend: puppetdb
environment:
    SSH_AUTH_SOCK: /run/keyholder/proxy.sock
puppetdb:
    host: <%= @puppetdb_host %>
    port: 8081
    api_version: 4
<% if scope.function_os_version(['debian >= stretch']) -%>
    urllib3_disable_warnings:
      - SubjectAltNameWarning
<% end -%>
knownhosts:
    files:
        - /etc/ssh/ssh_known_hosts
clustershell:
    ssh_options:
        - '-o BatchMode=yes'
        - '-o ConnectTimeout=2'
        - '-o StrictHostKeyChecking=yes'

Configure hiera

-Add puppetdb_host to cumin1001.yaml in hiera

puppetdb_host: puppetdb1002.dfw.labnet

-Add cumin IP address to common.yaml in hiera

 cumin_masters:
- 10.192.48.13

Apply the cumin class

Open your site.pp file and apply the cumin class to your cumin node

node 'cumin1001.dfw.labnet' {
       include cumin::master
}

Install Cumin

On the cumin node run

sudo puppet agent -t

verification

navigate to /usr/lib/python3/dist-packages/ and make sure that the cumin folder is there

ppaul@cumin1001:/usr/lib/python3/dist-packages/cumin$ ls -l
total 60
drwxr-xr-x 3 root root  4096 Oct  5 12:56 backends
-rw-r--r-- 1 root root 19656 Jul 30  2018 cli.py
-rw-r--r-- 1 root root  6051 Jul 30  2018 grammar.py
-rw-r--r-- 1 root root  4852 Jul 30  2018 __init__.py
drwxr-xr-x 2 root root  4096 Oct  5 12:56 __pycache__
-rw-r--r-- 1 root root  6877 Jul 30  2018 query.py
-rw-r--r-- 1 root root  2068 Jul 30  2018 transport.py
drwxr-xr-x 3 root root  4096 Oct  5 12:56 transports

We have now Cumin installed on our node. The next setup is to configure our cumin node to talk to the puppetdb node

Configuration

Firewall

On the puppetdb node puppetdb1002 only the puppetmaster node and the icinga node are allowed to talk to the puppetdb node.

8081/tcp                   ALLOW       10.192.16.12      #allow icinga1002.dfw.labnet        
8081/tcp                   ALLOW       10.192.32.4       #allow puppetmaster1002.dfw.labnet

We are going to allow cumin1001.dfw.labnet to talk to the puppetdb node

8081/tcp                   ALLOW       10.192.48.13 

-Testing

ppaul@cumin1001:/srv$ telnet puppetdb1002.dfw.labnet 8081
Trying 10.192.32.10...
Connected to puppetdb1002.dfw.labnet.
Escape character is '^]'.

We see that our cumin is allow to connect on port 8081 on the puppetdb node

===Change cumin port===

By default cumin is set to connect to puppetdb node on port 443 or our puppetdb node is running ssl on port 8081. We are going to change the cumin puppetdb backup end to use port 8081.

Navigate to /usr/lib/python3/dist-packages/cumin/backends and open the puppetdb.py file and change line 168 from

168             port=puppetdb_config.get('port', 443))

to

168             port=puppetdb_config.get('port', 8081))

Puppetdb 6 and Cumin ssl issue

Right now we have cumin node is allow to connect to the puppetdb node. Let try to see if we can run a cumin command

sudo cumin -b 1 -s 5 'lab1001.dfw.labnet' 'run-puppet-agent'

Result

Caught SSLError exception: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:720)

Let us try to curl from our cumin node to the puppetdb node

 ppaul@cumin1001:~$ sudo curl -vG 'https://puppetdb1002.dfw.labnet:8081/pdb/query/v4/facts'
*   Trying 10.192.32.10...
* TCP_NODELAY set
* Connected to puppetdb1002.dfw.labnet (10.192.32.10) port 8081 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS alert, Server hello (2):
* SSL certificate problem: unable to get local issuer certificate
* Curl_http_done: called premature == 1
* stopped the pause stream!
* Closing connection 0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

We can see that the puppetdb node is not allowing any connection without a certificate.

Let us try to run the same command now by bypassing the certificate. just add -k to the previous command

ppaul@cumin1001:~$ sudo curl -vkG 'https://puppetdb1002.dfw.labnet:8081/pdb/query/v4/facts'
*   Trying 10.192.32.10...
* TCP_NODELAY set
* Connected to puppetdb1002.dfw.labnet (10.192.32.10) port 8081 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS alert, Server hello (2):
* error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate
* Curl_http_done: called premature == 1
* stopped the pause stream!
* Closing connection 0
curl: (35) error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate

Same problem we can not bypass the certificate verification

According to the PuppetDB documentation in the link below, to make secured requests from other hosts, you will need to supply

--cacert
--cert
--key

https://puppet.com/docs/puppetdb/5.2/api/query/curl.html

Let us try to curl again but this time we we follow the documentation. create the bash script below

#!/bin/bash
curl -vG 'https://puppetdb1002.dfw.labnet:8081/pdb/query/v4/facts' \
 --tlsv1 \
 --cacert /etc/puppetlabs/puppet/ssl/certs/ca.pem \
 --cert /etc/puppetlabs/puppet/ssl/certs/cumin1001.dfw.labnet.pem \
 --key /etc/puppetlabs/puppet/ssl/private_keys/cumin1001.dfw.labnet.pem

make it excutable and run it

 ppaul@cumin1001:~$ sudo ./puppetdbquery 
*   Trying 10.192.32.10...
* TCP_NODELAY set
* Connected to puppetdb1002.dfw.labnet (10.192.32.10) port 8081 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/puppetlabs/puppet/ssl/certs/ca.pem
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS handshake, CERT verify (15):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / DHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=puppetdb1002.dfw.labnet
*  start date: Sep 30 00:52:53 2019 GMT
*  expire date: Sep 29 00:52:53 2024 GMT
*  common name: puppetdb1002.dfw.labnet (matched)
*  issuer: CN=Puppet CA: puppetmaster1002.dfw.labnet
*  SSL certificate verify ok.
> GET /pdb/query/v4/facts HTTP/1.1
> Host: puppetdb1002.dfw.labnet:8081
> User-Agent: curl/7.52.1
> Accept: */*
> 
< HTTP/1.1 200 OK
< Date: Fri, 04 Oct 2019 02:33:28 GMT
< Content-Type: application/json;charset=utf-8
< Vary: Accept-Encoding, User-Agent
< Transfer-Encoding: chunked

Now we are able to connect to our puppetdb node with out a problem

What is the fix for now/

The fix for now is to install a SSL proxy using Apache2 or Nginx on the puppetdb node. I decided to use Nginx in my case.

I am not going to discuss about how to install and configure Nginx here but only provide the Nginx config file if that will help

- Nginx config file

server {
       listen 8081 ssl http2;
       listen [::]:8081 ssl http2;
       server_name <%= @server_name %>;
       root /usr/share/nginx/www;
       ssl_certificate /etc/ssl/certs/wildcard.tx.labnet.crt;
       ssl_certificate_key /etc/ssl/private/wildcard.tx.labnet.key;
       add_header Strict-Transport-Security "max-age=15768000; includeSubDomains; preload";
       location / {
               proxy_pass http://localhost:8080/;
       }

}