DevOps With Ansible

Owned by Colin Samuel Rosenthal
Last updated: May 05, 2017

The current situation in Netarkivet

  • About 100 hosts
  • About 600 jvms
  • Home-made deploy framework to turn huge config file into a mass of install scripts
  • A huge support/operations job to maintain/monitor/deploy system

What Is DevOps Anyway?

DevOps (a clipped compound of "software DEVelopment" and "information technology OPerationS") is a term used to refer to a set of practices that emphasize the collaboration and communication of both software developers and information technology (IT) professionals while automating the process of software delivery and infrastructure changes. It aims at establishing a culture and environment where building, testing, and releasing software can happen rapidly, frequently, and more reliably.

What Is Ansible?

Ansible is an IT automation tool. It can configure systems, deploy software, and orchestrate more advanced IT tasks such as continuous deployments or zero downtime rolling updates.

And What's So Good About It?

  • Agentless. Just ssh to the target machine. No client/server.
  • Only requires python on the master machine (installed some non-root user with pip on Netarkivet devel platform)
  • Works on state, not procedures.
  • Parallel
  • Idempotent
  • Easy to get started ...

So Let's Put That Last To The Test

(See https://github.com/netarchivesuite/nas_ansible )

Minimal ansible needs two things

  1. An inventory 

    [all:children]
kb
sb

[kb:children]
adm

[adm]
kb-test-adm-001.kb.dk

[harvesters:children]
sb_harvesters

[sb:children]
sb_harvesters

[sb_harvesters]
sb-test-har-001.statsbiblioteket.dk

  2. A playbook 

    ---

- name: Deploy of basic NAS system
  hosts: all
  roles:
     - nas

- name: Deploy of harvester
  hosts: harvesters
  roles:
     - harvester

    which refers to some roles, which contain (among other things) tasks, templates, and variables. So, for example, the host sb-test-har-001.statsbiblioteket.dk has the following variable defined

    ---
apps:
  [
  {"app_name":"HarvestControllerApplication", "app_id": "sbhigh_h3", "channel":"HIGHPRIORITY"},
  {"app_name":"HarvestControllerApplication", "app_id": "sblow_h3", "channel":"LOWPRIORITY"}
  ]

    which is intended to specify which NetarchiveSuite applications we want on that host. (You can also define variables scoped particular host groups.)  Tasks are defined for each role, for example one task would be to generate start scripts for NetarchoveSuite applications 

    - name: Install start and stop scripts
  template:
     src: "start_Application.sh"
     dest: "/home/{{ansible_ssh_user}}/{{nas_env}}/conf/start_{{ item.app_name }}_{{ item.app_id }}.sh"
  with_items: "{{ apps }}"

    This says to apply a template to each item in the list "apps". The template file (start_Application.sh) uses jinja2 templating 

    {% set app = item.app_name+"_"+item.app_id %}
{% set dir = "/home/"+ansible_ssh_user+"/"+nas_env %}
echo Starting linux application: {{app}}
cd {{dir}}
PIDS=$(ps -wwfe | grep dk.netarkivet.harvester.heritrix3.HarvestControllerApplication | grep -v grep | grep /home/netarkdv/SystemTest/conf/settings_{{app}}.xml | awk "{print \$2}")
if [ -n "$PIDS" ] ; then
  echo Application already running.
else
  export CLASSPATH={{dir}}/lib/netarchivesuite-monitor-core.jar:{{dir}}/lib/netarchivesuite-heritrix3-controller.jar:$CLASSPATH;
  java -Xmx1024m  -Ddk.netarkivet.settings.file={{dir}}/conf/settings_{{app}}.xml -Dlogback.configurationFile={{dir}}/conf/logback_{{app}}.xml dk.netarkivet.harvester.heritrix3.{{item.app_name}} < /dev/null > start_{{app}}.log 2>&1 &
fi

Testing It All

Log into devel@kb-prod-udv-001.kb.dk

## Login
ssh devel@kb-prod-udv-001.kb.dk
## Activate virtualenv
. csr_ansible/bin/activate
## Delete the existing conf dir to show what happens
ssh netarkdv@sb-test-har-001.statsbiblioteket.dk rm AnsibleTest/conf/*
## Run ansible
ansible-playbook nas_ansible_pb/site.yaml -i nas_ansible_pb/hosts
##
ssh netarkdv@sb-test-har-001.statsbiblioteket.dk ls AnsibleTest/conf/* 
ssh netarkdv@sb-test-har-001.statsbiblioteket.dk cat AnsibleTest/conf/start_HarvestControllerApplication_sbhigh_h3.sh

Other Things To Think About

  • Server config (java, jms-broker, postgres)
  • Windows
  • Cloud deploy (for testing a distributed instance)
  • Ansible Tower - fancy GUI
  • Structuring an ansible project best practice - when and how to use groups and roles, for example
  • Triggers/handlers (e.g. which parts of the system to restart and when)