TEST6 Robustness test

Prerequisites

This test requires restart of infrastructure components (database and jms broker). These steps must be coordinated with the other testers.

Install and Start System

On devel@kb-prod-udv-001.kb.dk:

Start a selective harvest

Start a hourly selective harvest for the 'netarkivet.dk' domain.

Create a new template:

• Download the template "default_orderxml" 
• Edit the template so that max-size-bytes is 5000 in the WARCWriterProcessor
• Upload the template with a new name, for example "Small Warcfiles"

Modify domain templates 

• Configure the defaultconfig for  kum.dk to use template 'small_arcfiles_order.xml'.
• Configure the  defaultconfig for  dbc.dk to use template 'frontpages'. 
• Configure the  defaultconfig for  bs.dk to use template 'path_10levels_order.xml'. 

Make a new snapshot harvest definition with a name you can remember

• Create a new snapshot harvest Set 'Max number of bytes per domain’ 1.000.000 bytes (1 mbyte).
• Check that the job is started correctly in the  'Harvest status'->'All Jobs' in the left menu and that no errors or warnings are present in the system overview.

Stop the Test Automatically During Upload

• Using the GUI, find the job number and the name of the harvest machine for the job in which kum.dk is being harvested.
• Download the attached script and modify it to point at the correct harvester and job number
• Copy the script to kb-prod-udv-001.kb.dk and run it. It monitors the "warcs" directory and as soon as the first warcfile is uploaded it detects that uploading has started and shuts down the test instance.

Save the Metadata Warcfile

• Log into the harvester where kum.dk was being harvested
• Find the crawldir in TEST6/harvester_low
• Find the metadata warcfile in the metadata subdirectory and copy it to /tmp

Create a Fake Crawl Dir

• ssh netarkiv@sb-test-har-001.statsbiblioteket.dk
• cd TEST6/harvester_high 
• cp -r ~netarkiv/testdata/TEST6/23-fakejobdir
• mkdir 23-fakejobdir/logs
• touch 23-fakejobdir/logs/crawl.log

Wait 3 Hours then Restart the System

Wait 3 Hours then Restart the System

Verify the restarted system. On kb-test-adm-001

1. Check the log for warnings and errors.

   cd /home/test/$TESTX/log/
   grep SEVERE *.log.0
   grep WARNING *.log.0

   The following entries are normal: 

   arcrepositoryapplication0.log.0:WARNING: AdminDataFile (./admin.data) was not found.
   guiapplication0.log.0:WARNING: Refusing to schedule harvest definition 'netarkivet' in the past. Skipped 18 events. Old nextDate was Mon Dec 18 14:29:30 CET 2006 new nextDate is Tue Dec 19 09:29:30 CET 2006
   GUIApplication0.log.0:WARNING: Job 2 failed: HarvestErrors = dk.netarkivet.common.exceptions.IOFailure: Crawl probably interrupted by shutdown of HarvestController

   The following warning may occur after a while: 

   WARNING: Error processing message '
   Class:                  com.sun.messaging.jmq.jmsclient.ObjectMessageImpl
   getJMSMessageID():      ID:40-130.225.27.140(d2:1:3:b1:10:de)-46478-1197902260630
   getJMSTimestamp():      1197902260630
   getJMSCorrelationID():  null
   JMSReplyTo:             null
   JMSDestination:         TEST6_COMMON_THE_SCHED
   getJMSDeliveryMode():   PERSISTENT
   getJMSRedelivered():    false
   getJMSType():           null
   getJMSExpiration():     0
   getJMSPriority():       4
   Properties:             null'
   dk.netarkivet.common.exceptions.UnknownID: Job id 23 is not known in persistent storage
           at dk.netarkivet.harvester.datamodel.JobDBDAO.read(JobDBDAO.java:294)
           at dk.netarkivet.harvester.scheduler.HarvestSchedulerMonitorServer.processCrawlStatusMessage(HarvestSchedulerMonitorServer.java:103)
           at dk.netarkivet.harvester.scheduler.HarvestSchedulerMonitorServer.visit(HarvestSchedulerMonitorServer.java:285)
           at dk.netarkivet.harvester.harvesting.distribute.CrawlStatusMessage.accept(CrawlStatusMessage.java:133)
           at dk.netarkivet.harvester.distribute.HarvesterMessageHandler.onMessage(HarvesterMessageHandler.java:67)
           at com.sun.messaging.jmq.jmsclient.MessageConsumerImpl.deliverAndAcknowledge(MessageConsumerImpl.java:330)
           at com.sun.messaging.jmq.jmsclient.MessageConsumerImpl.onMessage(MessageConsumerImpl.java:265)
           at com.sun.messaging.jmq.jmsclient.SessionReader.deliver(SessionReader.java:102)
           at com.sun.messaging.jmq.jmsclient.ConsumerReader.run(ConsumerReader.java:174)
           at java.lang.Thread.run(Thread.java:595)

2. Go to the system overview page and check that all the expected applications are listen and are without warnings or errors.
3. Check that the scheduler schedules only one job for the hourly selective harvest.

Check that a job can be resubmitted

1. Check that you can reject a job for resubmission using the "Reject?" button so that it is no longer visible when you list failed jobs.
2. Check that you can see the rejected job when you now list all jobs.
3. Click on one or more "Genstart"/"Resubmit" buttons. Note that you only can resubmit jobs failed due to harvesting errors, not due to upload errors.
4. Check that the job-status changes to "resubmitted" and that a new Job is made from the same harvestdefinition with the same configurations.
5. Check that resubmitted jobs contain information about which job they were resubmitted (FR770)

Check Report Generation

Use a browser set up as a viewerproxy connection for this test. Select any completed job and click on the "Browse reports for jobs" link.

You should see a list like

    metadata://netarkivet.dk/crawl/setup/duplicatereductionjobs?majorversion=1&minorversion=0&harvestid=1&harvestnum=0&jobid=1
    metadata://netarkivet.dk/crawl/setup/crawl-manifest.txt?heritrixVersion=1.14.4&harvestid=1&jobid=1
    metadata://netarkivet.dk/crawl/setup/harvestInfo.xml?heritrixVersion=1.14.4&harvestid=1&jobid=1
    metadata://netarkivet.dk/crawl/setup/order.xml?heritrixVersion=1.14.4&harvestid=1&jobid=1
    metadata://netarkivet.dk/crawl/setup/seeds.txt?heritrixVersion=1.14.4&harvestid=1&jobid=1
    metadata://netarkivet.dk/crawl/reports/arcfiles-report.txt?heritrixVersion=1.14.4&harvestid=1&jobid=1
    metadata://netarkivet.dk/crawl/reports/crawl-report.txt?heritrixVersion=1.14.4&harvestid=1&jobid=1
    metadata://netarkivet.dk/crawl/reports/frontier-report.txt?heritrixVersion=1.14.4&harvestid=1&jobid=1
    metadata://netarkivet.dk/crawl/reports/hosts-report.txt?heritrixVersion=1.14.4&harvestid=1&jobid=1
    metadata://netarkivet.dk/crawl/reports/mimetype-report.txt?heritrixVersion=1.14.4&harvestid=1&jobid=1
    metadata://netarkivet.dk/crawl/reports/processors-report.txt?heritrixVersion=1.14.4&harvestid=1&jobid=1
    metadata://netarkivet.dk/crawl/reports/responsecode-report.txt?heritrixVersion=1.14.4&harvestid=1&jobid=1
    metadata://netarkivet.dk/crawl/reports/seeds-report.txt?heritrixVersion=1.14.4&harvestid=1&jobid=1
    metadata://netarkivet.dk/crawl/logs/crawl.log?heritrixVersion=1.14.4&harvestid=1&jobid=1
    metadata://netarkivet.dk/crawl/logs/heritrix.out?heritrixVersion=1.14.4&harvestid=1&jobid=1
    metadata://netarkivet.dk/crawl/logs/heritrix_dmesg.log?heritrixVersion=1.14.4&harvestid=1&jobid=1
    metadata://netarkivet.dk/crawl/logs/local-errors.log?heritrixVersion=1.14.4&harvestid=1&jobid=1
    metadata://netarkivet.dk/crawl/logs/progress-statistics.log?heritrixVersion=1.14.4&harvestid=1&jobid=1
    metadata://netarkivet.dk/crawl/logs/runtime-errors.log?heritrixVersion=1.14.4&harvestid=1&jobid=1
    metadata://netarkivet.dk/crawl/logs/uri-errors.log?heritrixVersion=1.14.4&harvestid=1&jobid=1
    metadata://netarkivet.dk/crawl/index/cdx?majorversion=1&minorversion=0&harvestid=1&jobid=1×tamp=20130814073013&serialno=00000

Check that all the entries are present and browse each in turn.

Shutdown the system