Note that this documentation is for the old 6.0 release.
For the newest documentation, please see the current release documentation.

Installation of the Quickstart system

Owned by Colin Samuel Rosenthal
Jun 12, 2020
3 min read

We have prepared a bash shell script that starts all the necessary components on one machine. We will use this script throughout this quickstart manual to allow you to get a feel for what the system can do and how it works without having to deal with issues of distributing to other servers.

Base system required

For the quick startup, NetarchiveSuite requires:

  • A Linux system with a minimum of 2GB free diskspace.  The QuickStart guide was tested with Ubuntu.
  • Java 8 SE runtime or higher. In production at the Danish Netarkivet we run with OpenJDK 1.8.

  • An additional user named "test". The commands to install NetarchiveSuite are run from your own login. The commands install and run the NetarchiveSuite software under user "test". This simulates the more realistic productions situation where the software runs under various logins on one or more machines in a distributed network. For convenience, it is a good idea to configure the test-user to have password-free ssh access - i.e. you should be able to execute "ssh test@localhost" in a shell without entering the test-user's password.

Setup JMS

NetarchiveSuite uses Java Messaging Service (JMS) for communication between the different components. (The implementation used is OpenMQ 5.1)

Download the openmq installation script:

wget https://raw.githubusercontent.com/netarchivesuite/netarchivesuite/master/deploy/deploy-core/scripts/openmq/mq.sh
chmod +x mq.sh

Install the openmq broker:

./mq.sh install

This will download openmq, install and start it. 


Note: OpenMQ will be installed to ~/MessageQueue5.1.

Download NetarchiveSuite

Binary releases are available from https://sbforge.org/nexus/content/groups/public/org/netarchivesuite/distribution/ and  https://sbforge.org/nexus/content/groups/public/org/netarchivesuite/heritrix3-bundler/.

The Netarchivesuite-bundle is in the 'distribution' folder, and the heritrix3-bundle in the 'heritrix3-bundler' folder.


Create a working directory and navigate to it

mkdir netarchive; cd netarchive


Download the latest release

NAS_VERSION=6.0
wget -O NetarchiveSuite.zip  https://sbforge.org/nexus/content/groups/public/org/netarchivesuite/distribution/$NAS_VERSION/distribution-$NAS_VERSION.zip
together with the latest bundled harvester 
NAS_VERSION=6.0
wget -O NetarchiveSuite-heritrix3-bundler.zip https://sbforge.org/nexus/content/groups/public/org/netarchivesuite/heritrix3-bundler/$NAS_VERSION/heritrix3-bundler-$NAS_VERSION.zip

Download RunNetarchiveSuite.sh and deploy_standalone_example.xml to the netarchive directory:

wget -N https://raw.githubusercontent.com/netarchivesuite/netarchivesuite/master/deploy/distribution/src/main/resources/examples/deploy_standalone_example.xml
wget -N https://raw.githubusercontent.com/netarchivesuite/netarchivesuite/master/deploy/deploy-core/scripts/RunNetarchiveSuite.sh
chmod +x RunNetarchiveSuite.sh

The first script is a simple script for doing all the steps during deployment. It takes a NetarchiveSuite package ('.zip'), a configuration file (the second file), a temporary installation directory, and the heritrix3 bundler zipfile as arguments (in the given order). The different ports used by the application for communication are included in the deploy_standalone_example.xml file.

In the configuration file all the applications are placed on one machine, the current machine (localhost).

When the installation script is run it will unpack the installation files into the netarchive/deploy directory as the current user and then - as the user "test" - install NetarchiveSuite into the /home/test/QUICKSTART directory (using ssh). 

If you already have a Quickstart installation, the existing bitarchive, database and admin.data files will be untouched. You must explicitly remove any previous installation, if you want a clean empty installation.

To do the deployment:

./RunNetarchiveSuite.sh NetarchiveSuite.zip deploy_standalone_example.xml deploy NetarchiveSuite-heritrix3-bundler.zip

Note that if you have not setup your automatic ssh test user login (using key based login), you need to login multiple times before the installation finishes successfully. 

The script creates a deployment folder named "QUICKSTART" under the test users home directory, which contains methods for starting and stopping NetarchiveSuite, and starts the whole NetarchiveSuite. The files to run the installation will be placed in the directory deploy under the directory where you ran the RunNetarchiveSuite.sh command,

Now configure your browser:

  • Start a web browser. 
  • Setup the browser to proxy on port 8070 on the host executing the netarchive, and exclude localhost and the hostname (used by the Heritrix GUI).  In Firefox it is done as follows:
Choose in the Firefox toolbar:
Edit->Preferences->Advanced->Network->Settings
Checkmark:
Manual Proxy Configuration
and add:
Http proxy: name-of-host
Port: 8070

No Proxy for: localhost, name-of-host
  • Write the following url in the browser http://localhost:8074/HarvestDefinition (if running on the local machine, otherwise go to port 8074 on the host running NetarchiveSuite).  

    You should see the Netarchive harvest definition page:

    If you see a stacktrace instead of the Netarchive pages, this is most likely because the browser tries to go through the ViewerProxy.    Revisit the browser configuration and try again.
  • You can now see the webinterface in the browser. You can now create jobs, run them, and browse output following the basic instructions in the rest of this manual, or the more complete description in the User Manual.
  • You can stop and start the entire NAS system with:
ssh test@localhost
cd QUICKSTART
./conf/killall.sh
./conf/startall.sh
  • If you want to try other deploy examples, then go to "Examples of deploy configuration files" in the Installation Manual.