A Service Oriented Architecture for the NetarchiveSuite Frontend

.... or "How Hard Can It Be To Find A Webdeveloper?"

.... or "How To Build A Full-stack Webdev in Two Easy Lessons"

.... or "How I Learnt To Stop Worrying And Love Javascript"

(Based on work from Innovation Week September 2017 at KB-Aarhus.)

Where We Are Now

http://kb-test-adm-001.kb.dk:8078/

It looks ok, so why is further development so slow and such hard work?

Well look at the code ....

What do we see?

A big ball of jsp and servlets
No clear separation between business logic and presentation
Multiple back ends
- Database via DAO classes
- External NAS applications via JMX/JMS
- Heritrix via REST

So can we learn to make a web frontend like real web-developers?

Let's do it in two stages

Create a Service Layer For the NAS Backend

All communication between the NAS GUI and the backend system should be via a well-defined REST API

Create new webmodule in the existing NAS GUI

Configure its web.xml to scan for classes that expose REST methods

<servlet>
   <servlet-name>Jersey REST Service</servlet-name>
   <servlet-class>org.glassfish.jersey.servlet.ServletContainer</servlet-class>
   <init-param>
     <param-name>jersey.config.server.provider.packages</param-name>
     <param-value>dk.netarkivet.common.api</param-value>
   </init-param>
  <load-on-startup>1</load-on-startup>
</servlet>

Implement some methods

@Path("/status")
public class Status {

    @GET
    @Path("/all")
    @Produces(MediaType.APPLICATION_JSON)
    public JMXStatusEntry[] getAllStatus() {
        String query = "dk.netarkivet.common.logging:*,index=0";
        ArrayList<JMXStatusEntry> entries = new ArrayList<>();

        List<StatusEntry> entries1 = null;
        try {
            entries1 = JMXStatusEntry.queryJMX(query);
        } catch (MalformedObjectNameException e) {
            throw new RuntimeException(e);
        }
        for (StatusEntry entry: entries1) {
            entry.getLogMessage(Locale.getDefault());
            entries.add((JMXStatusEntry) entry);
        }
        return entries.toArray(new JMXStatusEntry[]{});
    }
}

Profit! http://kb-test-adm-001.kb.dk:8078/rest/rest/status/all

Now Just Write a Modern Dynamic Interactive WebGui ...

On advice I started with jQuery.

As before, created a new web-context for the GUI in the existing NAS tomcat - although any web-server will do

Enabling jQuery just means adding some scripts in your html body and some css in your html head

<head>
    <meta charset="utf-8" />
    <link rel="stylesheet" type="text/css" href="//cdn.datatables.net/1.10.16/css/jquery.dataTables.css">
    <link rel="stylesheet" href="//code.jquery.com/ui/1.12.1/themes/base/jquery-ui.css">
.
.
.
</head>

<body>
.
.
.
<script src="//code.jquery.com/jquery-1.11.2.min.js"></script>
<script src="//code.jquery.com/jquery-migrate-1.2.1.min.js"></script>
<script src="https://code.jquery.com/ui/1.12.1/jquery-ui.js"></script>
<script type="text/javascript" charset="utf8" src="https://cdn.datatables.net/1.10.16/js/jquery.dataTables.min.js"></script>
<script src="site.js"></script>
</body>

The html for the Status page now looks like this

<table id="status_table">
    <thead></thead>
    <tbody></tbody>
</table>

... literally, that's the whole thing, because all the action happens in the site.js:

$(document).ready(function() {
    $('#tabs').tabs({
      load: function( event, ui ) {
          if(event.currentTarget != null && event.currentTarget.firstChild.textContent === "Status") {
              $('#status_table').DataTable( {
                  ajax: {
                      url: '../rest/rest/status/all',
                      dataSrc: ''
                  },
                  columns: [
                      { data: 'physicalLocation' },
                      { data: 'applicationName' },
                      { data: 'logMessage'}
                  ]
              } );
          }
      }
    });

What matters here is that we are leveraging the entire functionality of the jQuery-UI DataTable plugin to format the JSON we get from the REST service.

And here's what it looks like http://kb-test-adm-001.kb.dk:8078/webgui/e

Note that even with absolutely minimal configuration and styling we get

Pagination
Column sorting
Search

None of which we have now.

So Where To Now

... or can I really ask the Steering Committee for permission to start a complete frontend rewrite?

The Good Stuff

This is the result of three days' work by one developer, of which two days were on the backend. Getting started with this web-development stuff isn't as hard as I expected.
We can leverage a universe of existing solutions to instead of having to reinvent the wheel.
A clean API should make testing NAS much easier.
This is already way more fun than hacking through a mess of servlets & jsp and should make it easier to integrate new developers in the project.

The Challenges (not all technical!)

Known unknowns and unknown unknowns - we need a much longer pilot-program to be able to present a really convincing case. We haven't even looked at interactions yet.
Is jQuery the right technology?
How much reskilling do we need to do in the NAS developer group?
What about "higher level" skills - how to manage, structure, test, standardise a web-development project? These are not skills we currently have in NAS.

Code for this project is available in a branch of the main netarchivesuite github. As of presentation-day, see this commit: https://github.com/netarchivesuite/netarchivesuite/commit/b02a8773e6b1a6f20cd57fe0099885406a0ea179