Legal Batch job test

Owned by M
Last updated: Apr 16, 2018 by Sr
Uses a command-line tool several times and check that the output from the tool is correct as well as that the appropriate bit archives have been queried. Note that the number of files mentioned in the output is not necessarily exact.

Setup

  1. Login to the indexserver (kb-test-acs-001.kb.dk)

    ssh devel@kb-test-acs-001
export TESTX=TEST11A
cd ${TESTX}

  2. Create dir for batchprograms and copy contents of /home/devel/backup_TEST11A/batchprogs to /home/devel/TEST11A/

    cp -av /home/devel/backup_TEST11A/batchprogs .

Run batch programs on archived files

ChecksumJob

Calculating the MD5 checksum on the archive files (only .arc files).

Run

java -cp lib/netarchivesuite-archive-core.jar -Dsettings.common.applicationInstanceId=CHECKSUM -Ddk.netarkivet.settings.file=conf/settings_IndexServerApplication.xml dk.netarkivet.archive.tools.RunBatch -Cbatchprogs/ChecksumJob.class -R.*\.arc -Ooutput.checksum

This should produce a file output.checksum with be something like:

1-metadata-1.arc##9dca301f9ecd3663bbb3cdc91009562c
1-1-20130114144130-00001-kb-test-har-001.kb.dk.arc##1abe93f92569154da3f629255146dcfd
1-1-20130114144130-00002-kb-test-har-001.kb.dk.arc##6b2801ca68d7c96af8b05ba9e9ae47b4
1-1-20130114144130-00003-kb-test-har-001.kb.dk.arc##5c77d44ad9fc9290f2cd6639562ed11a
1-1-20130114144130-00000-kb-test-har-001.kb.dk.arc##c1e533f574a46576fc14393581428129

EvilBatch

Writes the sentence 'evil' for each bitarchive.

Run

java -cp lib/netarchivesuite-archive-core.jar -Dsettings.common.applicationInstanceId=CHECKSUM -Ddk.netarkivet.settings.file=conf/settings_IndexServerApplication.xml dk.netarkivet.archive.tools.RunBatch -Cbatchprogs/EvilBatch.class -R.*\.arc -Ooutput.evil

This should produce a file output.evil with:

Legal
Legal
Legal
Legal

This is run on the KBN replica, which has four bitapps (according to the default configuration file ). Therefore the sentence 'Legal' is written four times.


The SBN replica one has one bitapp, and it should therefore only have the sentence, 'Legal', written once.

Run

java -cp lib/netarchivesuite-archive-core.jar -Dsettings.common.applicationInstanceId=EVILSB -Ddk.netarkivet.settings.file=conf/settings_IndexServerApplication.xml dk.netarkivet.archive.tools.RunBatch -Cbatchprogs/EvilBatch.class -R.*\.arc -Ooutput.evil.sb -BSBN

This should produce a file output.evil with:

Legal

This is run on the SBN replica, which has only one bitapp (according to the default configuration file'). Therefore the sentence 'Legal' is written once.

EvilBatch2

Tries to delete archived files, thus gets failed for all files.

Run

java -cp lib/netarchivesuite-archive-core.jar -Dsettings.common.applicationInstanceId=EVIL2 -Ddk.netarkivet.settings.file=conf/settings_IndexServerApplication.xml dk.netarkivet.archive.tools.RunBatch -Cbatchprogs/EvilBatch2.class -R.*\.arc -Eerror.evil2

Since the batch program is supposed to fail, nothing should be written to std-out, only to std-error. It is therefore the std-err which should be catched, which can be done with the -E argument.

This should produce a file error.evil2 with be something like:

Failed files:
1-1-20130114144130-00000-kb-test-har-001.kb.dk.arc
1-1-20130114144130-00003-kb-test-har-001.kb.dk.arc
1-1-20130114144130-00001-kb-test-har-001.kb.dk.arc
1-1-20130114144130-00002-kb-test-har-001.kb.dk.arc
1-metadata-1.arc

ExceptionBatch

Checks if a file is a metadata file, in which case it fails with an exception. Otherwise it writes out that the given file is not a metadata file.

Run

java -cp lib/netarchivesuite-archive-core.jar -Dsettings.common.applicationInstanceId=EXCEPTION -Ddk.netarkivet.settings.file=conf/settings_IndexServerApplication.xml dk.netarkivet.archive.tools.RunBatch -Cbatchprogs/ExceptionBatch.class -R.*\.arc -Eerror.exception -Ooutput.exception

This should produce a file 'output.exception''  with something like:

non-metadata file: 1-1-20130114144130-00000-kb-test-har-001.kb.dk.arc
non-metadata file: 1-1-20130114144130-00003-kb-test-har-001.kb.dk.arc
non-metadata file: 1-1-20130114144130-00001-kb-test-har-001.kb.dk.arc
non-metadata file: 1-1-20130114144130-00002-kb-test-har-001.kb.dk.arc

And a file 'error.exception'  with something like:

Failed files:
1-metadata-1.arc

ExceptionBatchFinish

Does not work properly (does not give exception, only writes out name).

Run

java -cp lib/netarchivesuite-archive-core.jar -Dsettings.common.applicationInstanceId=FINISH -Ddk.netarkivet.settings.file=conf/settings_IndexServerApplication.xml dk.netarkivet.archive.tools.RunBatch -Cbatchprogs/ExceptionBatchFinish.class -R.*\.arc -Ooutput.exception.finish

This should produce a file ' output.exception.finish''  with something like:

1-1-20130114144130-00000-kb-test-har-001.kb.dk.arc
1-metadata-1.arc
1-1-20130114144130-00001-kb-test-har-001.kb.dk.arc
1-1-20130114144130-00003-kb-test-har-001.kb.dk.arc
1-1-20130114144130-00002-kb-test-har-001.kb.dk.arc

ExtractDeduplicateCDXBatchJob

Run

java -Dsettings.common.applicationInstanceId=DEDUP -Ddk.netarkivet.settings.file=conf/settings_IndexServerApplication.xml -cp lib/netarchivesuite-archive-core.jar dk.netarkivet.archive.tools.RunBatch -Ndk.netarkivet.wayback.batch.DeduplicationCDXExtractionBatchJob -Jlib/netarchivesuite-wayback-indexer.jar -R'.*metadata.*\.arc' -BSBN -Ocdx.output

The output in the cdx.output file should be well formed cdx records with each line starting with a canonicalized url like these:

jigsaw.w3.org/css-validator/images/vcss-blue 20091001090817 http://jigsaw.w3.org/css-validator/images/vcss-blue image/gif 200 YEX3QI4MLIW3EPJCYCA2OZWT4ZEQGA36 - 8221 3-3-20091001084354-00004-kb-test-har-001.kb.dk.arc

123hjemmeside.dk/picture.aspx?id=13195185 20090114041208 http://www.123hjemmeside.dk/picture.aspx?id=13195185 image/jpeg 200 SY7VL2B5HODA3TKBR6DQLCAN74BSUL3G - 33623059 27020-70-20080505183112-00095-kb-prod-har-002.kb.dk.arc
123hjemmeside.dk/picture.aspx?id=13243517 20090114105138 http://www.123hjemmeside.dk/picture.aspx?id=13243517 image/jpeg 200 XOEJDTALKDZNB2KKRCUS3K6X2LGZKEAC - 24666098 27063-70-20080513145144-00069-kb-prod-har-001.kb.dk.arc
123hjemmeside.dk/picture.aspx?id=13244248 20090114105514 http://www.123hjemmeside.dk/picture.aspx?id=13244248 image/jpeg 200 R7AEJVSL6Z3U2CAQGNDKKJ2NL5MSOCJU - 38159302 27063-70-20080513144604-00068-kb-prod-har-001.kb.dk.arc

The fields are:
<canonical url> <original url> <mime-type> <http code> <checksum> <redirect url> <length> <filename>

(In the above cases, <redirect-url> is empty, which it will always be unless there is a 3xx http code.)

Run CDX Job On WARC Files

java -Dsettings.common.applicationInstanceId=DEDUP -Ddk.netarkivet.settings.file=conf/settings_IndexServerApplication.xml -cp lib/netarchivesuite-archive-core.jar dk.netarkivet.archive.tools.RunBatch -Ndk.netarkivet.wayback.batch.DeduplicationCDXExtractionBatchJob -Jlib/netarchivesuite-wayback-indexer.jar -R'.*metadata.*\.warc' -BSBN -Ocdx.warc.output

Check that there is non-empty output. This verifies that CDX deduplicate extraction also works on warcfiles. 

ExceptionBatchInit 

This method should fail before starting to process the files, and thus no files should be processed.

Run

java -cp lib/netarchivesuite-archive-core.jar -Dsettings.common.applicationInstanceId=INIT -Ddk.netarkivet.settings.file=conf/settings_IndexServerApplication.xml dk.netarkivet.archive.tools.RunBatch -Cbatchprogs/ExceptionBatchInit.class -R.*\.arc

The following text message should be written in the console:

Running batch job 'batchprogs/ExceptionBatchInit.class' on files matching '.*.arc' on replica 'KBN', output written to stdout errors written to stderr 
Processed 0 files with 0 failures

Run batch programs on records within archive files

These methods does not check on the '.arc' files, but on the content of these '.arc' files.

SimpleArcBatchJob

Checks that it is posible to access the records within a arc file.

Run

java -cp lib/netarchivesuite-archive-core.jar -Dsettings.common.applicationInstanceId=SIMPLE -Ddk.netarkivet.settings.file=conf/settings_IndexServerApplication.xml dk.netarkivet.archive.tools.RunBatch -Cbatchprogs/SimpleArcBatchJob.class -R.*\.arc -Ooutput.simple

The output file should be quite large compared to the previous output files.

The size of the output files and the error files can be found by running the command: du -h output.* error.*. This gives the following results:

4,0K    output.evil
4,0K    output.exception
4,0K    output.exception.finish
212K    output.simple
4,0K    error.evil2
4,0K    error.exception

The content of the output file contain a line of data for each record. This line will have the following structure:

org.archive.io.arc.ARCRecord@f16070 available: q:\bitarkiv\JOLF\filedir\1-1-20090316092641-00001-kb-test-har-002.kb.dk.arc: {ip-address=0.0.0.0, content-type=text/plain, absolute-offset=0, subject-uri=filedesc://1-1-20090316092641-00001-kb-test-har-002.kb.dk.arc.open, length=1343, creation-date=20090316092641, version=1.1}

Furthermore, a list of failed files should be printed to stdout (all of them WARC files).

Running methods from jar files, including Jhove based methods

Before running any Jhove methods, run the following command:

cp -av /home/devel/backup_TEST11A/lib_jhove/* /home/devel/TEST11A/lib

eu.planets.batch.jar -> eu.planets.JhoveArcJob 

Get some metadata from all records within the arc-files through Jhove.

This method writes the following metadata on the all the records of handable dataformats: parsing errors, format, version,mimetype.

For the dataformats which are not handled, these values are replaced by 'null', except mimetype which is replaced by the mimetype of the record.

Run

java -cp lib/netarchivesuite-archive-core.jar -Dsettings.common.applicationInstanceId=JHOVE -Ddk.netarkivet.settings.file=conf/settings_IndexServerApplication.xml dk.netarkivet.archive.tools.RunBatch -Jbatchprogs/eu.planets.batch.jar,lib/jhove.jar,lib/jhove-module.jar -Neu.planets.JhoveArcJob -R.*\.arc -Ooutput.jhove.arc

The output.jhove.arc file should be in the following format:

0,HTML,null,text/html
null,null,null,image/png
null,null,null,text/css
0,GIF,null,image/gif
0,JPEG,null,image/jpeg
.....

Furthermore, a list of failed files should be printed to stdout (all of them WARC files).

Check the content of metadata files and 'content' 

These three methods takes a copy of the content from different arc files based on a regular expression.

All these tests are basically stress-tests, which by dumping the content of all records within a arc file to the output stream. First the content of all the files are collected, then only the content of the metadata files, and finally the data from the harvesting. These are all put into different output files, and the first file should have the same size as the other two combined.

CopyArcContent

This tests the ability to copy the content of all the files in the bitarchives. It can be seen as a stress test of the batch system.

Run

java -cp lib/netarchivesuite-archive-core.jar -Dsettings.common.applicationInstanceId=COPYARC -Ddk.netarkivet.settings.file=conf/settings_IndexServerApplication.xml dk.netarkivet.archive.tools.RunBatch -Cbatchprogs/CopyArcContent.class -R.*\.arc -Ooutput.copy.class  

This method copies all the content from the records to the output file, and therefore it takes some time to run the command (approx. 25 sek on 60 MB). Takes about 40 min's on 80 files (6GB).

The content of the output file is not easy to confirm. It should start by having the metadata of a record and then the actual content of this record.

Check the size of the output file (run the command du -h output.copy.class): 

59M     output.copy.class

This value should be approximately the same as the combined size of all the harvests.

Furthermore, a list of failed files should be printed to stdout (all of them WARC files).

Test a WARC Batch Job

java -Dsettings.common.applicationInstanceId=DEDUP -Ddk.netarkivet.settings.file=conf/settings_IndexServerApplication.xml -cp lib/netarchivesuite-archive-core.jar dk.netarkivet.archive.tools.RunBatch -Ndk.netarkivet.common.utils.cdx.WARCExtractCDXJob -Jlib/netarchivesuite-wayback-indexer.jar -R'.*dk.*\.warc' -BSBN -Ocdx.warc.all.output

Check that the output file (cdx.warc.all.output) has a significant amount of content.

eu.planets.batch.jar -> eu.planets.CopyArcContent: 'metadata'

Copy the content of the metadata files only.

 

Run

 

java -cp lib/netarchivesuite-archive-core.jar -Dsettings.common.applicationInstanceId=META -Ddk.netarkivet.settings.file=conf/settings_IndexServerApplication.xml dk.netarkivet.archive.tools.RunBatch -Cbatchprogs/CopyArcContent.class -Ooutput.copy.meta -R'.*-metadata-.*\.arc' -BKBN

 

This should give the following output in the console:

Running batch job 'batchprogs/CopyArcContent.class' on files matching '.*-metadata-.*' on replica 'KBN', output written to file 'output.copy.meta', errors written to stderr
Processed 2 files with 0 failures

eu.planets.batch.jar -> eu.planets.CopyArcContent: ’content’

Copy only the content from of the files collected in the harvest. This test assumes that the harvesters in the test system is in the '.dk' domain.

 

Run

 

java -cp lib/netarchivesuite-archive-core.jar -Dsettings.common.applicationInstanceId=CONTENT -Ddk.netarkivet.settings.file=conf/settings_IndexServerApplication.xml dk.netarkivet.archive.tools.RunBatch -Cbatchprogs/CopyArcContent.class -Ooutput.copy.content -R'.*.dk.*\.arc' -BKBN

 

The regular expression should handle any files besides the metadata files, since they don't contain the sequence '.dk' in their name. This means that they handles all the other files, the 'content' files.

This should give the following output in the console:

 

Running batch job 'batchprogs/CopyArcContent.class' on files matching '.*.dk.*' on replica 'KBN', output written to file 'output.copy.content', errors written to stderr
Processed 9 files with 0 failures

Check that the system properties can be read

 

Run

 

java -cp lib/netarchivesuite-archive-core.jar -Dsettings.common.applicationInstanceId=PROPS -Ddk.netarkivet.settings.file=conf/settings_IndexServerApplication.xml dk.netarkivet.archive.tools.RunBatch -Cbatchprogs/SystemReaderJob.class -Ooutput.system -Eerror.system -R'.*.dk.*' -BSBN

 

Be aware of, that all metadata files are excluded.

Check that the error.system file is empty.

The output.system should contain system properties and a list of files. First the java version, then the operating system name, the operating system architecture and the operating system version. Then the list of files should be written, followed by a count of the files and the system property user name. It could look something like this:

 

System properties!
java version: 1.6.0_24
os name: Linux
os architecture: i386
os version: 2.6.32-220.4.1.el6.x86_64
File: 1-1-20130114144130-00001-kb-test-har-001.kb.dk.arc
File: 1-1-20130114144130-00002-kb-test-har-001.kb.dk.arc
File: 1-1-20130114144130-00003-kb-test-har-001.kb.dk.arc
File: 1-1-20130114144130-00000-kb-test-har-001.kb.dk.arc
File count: 4
User: netarkiv