The current situation in Netarkivet

About 100 hosts
About 600 jvms
Home-made deploy framework to turn huge config file into a mass of install scripts
A huge support/operations job to maintain/monitor/deploy system

What Is DevOps Anyway?

DevOps (a clipped compound of "software DEVelopment" and "information technology OPerationS") is a term used to refer to a set of practices that emphasize the collaboration and communication of both software developers and information technology (IT) professionals while automating the process of software delivery and infrastructure changes. It aims at establishing a culture and environment where building, testing, and releasing software can happen rapidly, frequently, and more reliably.

What Is Ansible?

Ansible is an IT automation tool. It can configure systems, deploy software, and orchestrate more advanced IT tasks such as continuous deployments or zero downtime rolling updates.

And What's So Good About It?

Agentless. Just ssh to the target machine. No client/server.
Only requires python on the master machine (installed some non-root user with pip on Netarkivet devel platform)
Works on state, not procedures.
Parallel
Idempotent
Easy to get started ...

So Let's Put That Last To The Test

(See https://github.com/netarchivesuite/nas_ansible )

Minimal ansible needs two things

An inventory

[all:children]
kb
sb

[kb:children]
adm

[adm]
kb-test-adm-001.kb.dk

[harvesters:children]
sb_harvesters

[sb:children]
sb_harvesters

[sb_harvesters]
sb-test-har-001.statsbiblioteket.dk

A playbook

---

- name: Deploy of basic NAS system
  hosts: all
  roles:
     - nas

- name: Deploy of harvester
  hosts: harvesters
  roles:
     - harvester

which refers to some roles, which contain (among other things) tasks, templates, and variables. So, for example, the host sb-test-har-001.statsbiblioteket.dk has the following variable defined

---
apps:
  [
  {"app_name":"HarvestControllerApplication", "app_id": "sbhigh_h3", "channel":"HIGHPRIORITY"},
  {"app_name":"HarvestControllerApplication", "app_id": "sblow_h3", "channel":"LOWPRIORITY"}
  ]

which is intended to specify which NetarchiveSuite applications we want on that host. (You can also define variables scoped particular host groups.) Tasks are defined for each role, for example one task would be to generate start scripts for NetarchoveSuite applications

- name: Install start and stop scripts
  template:
     src: "start_Application.sh"
     dest: "/home/{{ansible_ssh_user}}/{{nas_env}}/conf/start_{{ item.app_name }}_{{ item.app_id }}.sh"
  with_items: "{{ apps }}"

This says to apply a template to each item in the list "apps". The template file (start_Application.sh) uses jinja2 templating

{% set app = item.app_name+"_"+item.app_id %}
{% set dir = "/home/"+ansible_ssh_user+"/"+nas_env %}
echo Starting linux application: {{app}}
cd {{dir}}
PIDS=$(ps -wwfe | grep dk.netarkivet.harvester.heritrix3.HarvestControllerApplication | grep -v grep | grep /home/netarkdv/SystemTest/conf/settings_{{app}}.xml | awk "{print \$2}")
if [ -n "$PIDS" ] ; then
  echo Application already running.
else
  export CLASSPATH={{dir}}/lib/netarchivesuite-monitor-core.jar:{{dir}}/lib/netarchivesuite-heritrix3-controller.jar:$CLASSPATH;
  java -Xmx1024m  -Ddk.netarkivet.settings.file={{dir}}/conf/settings_{{app}}.xml -Dlogback.configurationFile={{dir}}/conf/logback_{{app}}.xml dk.netarkivet.harvester.heritrix3.{{item.app_name}} < /dev/null > start_{{app}}.log 2>&1 &
fi

Testing It All

Log into devel@kb-prod-udv-001.kb.dk

## Login
ssh devel@kb-prod-udv-001.kb.dk
## Activate virtualenv
. csr_ansible/bin/activate
## Delete the existing conf dir to show what happens
ssh netarkdv@sb-test-har-001.statsbiblioteket.dk rm AnsibleTest/conf/*
## Run ansible
ansible-playbook nas_ansible_pb/site.yaml -i nas_ansible_pb/hosts
##
ssh netarkdv@sb-test-har-001.statsbiblioteket.dk ls AnsibleTest/conf/* 
ssh netarkdv@sb-test-har-001.statsbiblioteket.dk cat AnsibleTest/conf/start_HarvestControllerApplication_sbhigh_h3.sh

Other Things To Think About

Server config (java, jms-broker, postgres)
Windows
Cloud deploy (for testing a distributed instance)
Ansible Tower - fancy GUI
Structuring an ansible project best practice - when and how to use groups and roles, for example
Triggers/handlers (e.g. which parts of the system to restart and when)

Browser not supported