Versions Compared

Version Old Version 2 New Version Current
Changes made by

M

Sr

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Excerpt

Uses a command-line tool several times and check that the output from the tool is correct as well as that the appropriate bit archives have been queried. Note that the number of files mentioned in the output is not necessarily exact.

Table of Contents

Setup

  1. Login to a bitarchive server, eg.the indexserver (kb-test-acs-001.kb.dk)

    Code Block
    $ ssh netarkiv@sbdevel@kb-test-baracs-001.statsbiblioteket.dk
$ export TESTX=TEST11A
$
cd ${TESTX}


  2. Create dir for batchprograms :and copy contents of /home/devel/backup_TEST11A/batchprogs to /home/devel/TEST11A/

    Code Block
    $cp mkdir batchprogs
$ scp test@kb-prod-udv-001.kb.dk:-av /home/test/test-batch/* batchprogs/devel/backup_TEST11A/batchprogs .


Run batch programs on archived files

ChecksumJob

Calculating the MD5 checksum on the archive files (only .arc files).

Run

Code Block
java -cp lib/dk.netarkivet.archivenetarchivesuite-archive-core.jar -Dsettings.common.applicationInstanceId=CHECKSUM -Ddk.netarkivet.settings.file=conf/settings_IndexServerApplication.xml dk.netarkivet.archive.tools.RunBatch -Cbatchprogs/ChecksumJob.class -R.*\.arc -Ooutput.checksum

Consol output should be something like:

...

This should produce a file output.checksum with be something like:

Code Block
1-metadata-1.arc##9dca301f9ecd3663bbb3cdc91009562c
1-1-20130114144130-00001-kb-test-har-001.kb.dk.arc##1abe93f92569154da3f629255146dcfd
1-1-20130114144130-00002-kb-test-har-001.kb.dk.arc##6b2801ca68d7c96af8b05ba9e9ae47b4
1-1-20130114144130-00003-kb-test-har-001.kb.dk.arc##5c77d44ad9fc9290f2cd6639562ed11a
1-1-20130114144130-00000-kb-test-har-001.kb.dk.arc##c1e533f574a46576fc14393581428129

EvilBatch

Writes the sentence 'evil' for each bitarchive.

Run

Code Block
java -cp lib/netarchivesuite-archive-core.jar -Dsettings.common.applicationInstanceId=CHECKSUM -Ddk.netarkivet.settings.file=conf/settings_IndexServerApplication.xml dk.netarkivet.archive.arcrepository.distribute.JMSArcRepositoryClient <init>
INFO: JMSArcRepositoryClient will retry a store 3 times and timeout on each try after 3600000 milliseconds, and timeout on each getrequest after 60000 milliseconds.
Mar 16, 2009 11:23:12 AM dk.netarkivet.common.distribute.JMSConnectionSunMQ <init>
INFO: Creating instance of dk.netarkivet.common.distribute.JMSConnectionSunMQ
Mar 16, 2009 11:23:12 AM dk.netarkivet.common.distribute.JMSConnection initConnection
INFO: Initializing a JMS connection of type 'class dk.netarkivet.common.distribute.JMSConnectionSunMQ' to Broker at kb-test-adm-001.kb.dk:7676.
Mar 16, 2009 11:23:13 AM dk.netarkivet.archive.arcrepository.distribute.JMSArcRepositoryClient <init>
INFO: JMSArcRepository listens for replies on channel '[Queue 'JOLF_COMMON_THIS_REPOS_CLIENT_130_226_228_5_ISA']'
tools.RunBatch -Cbatchprogs/EvilBatch.class -R.*\.arc -Ooutput.evil

This should produce a file output.evil with:

Code Block
Legal
Legal
Legal
Legal

This is run on the KBN replica, which has four bitapps (according to the default configuration file ). Therefore the sentence 'Legal' is written four times.


The SBN replica one has one bitapp, and it should therefore only have the sentence, 'Legal', written once.

Run

Code Block
java -cp lib/netarchivesuite-archive-core.jar -Dsettings.common.applicationInstanceId=EVILSB -Ddk.netarkivet.settings.file=conf/settings_IndexServerApplication.xml dk.netarkivet.archive.tools.RunBatch -Cbatchprogs/EvilBatch.class -R.*\.arc -Ooutput.evil.sb -BSBN

This should produce a file output.evil with:

Code Block
Legal

This is run on the SBN replica, which has only one bitapp (according to the default configuration file'). Therefore the sentence 'Legal' is written once.

EvilBatch2

Tries to delete archived files, thus gets failed for all files.

Run

Code Block
java -cp lib/netarchivesuite-archive-core.jar -Dsettings.common.applicationInstanceId=EVIL2 -Ddk.netarkivet.settings.file=conf/settings_IndexServerApplication.xml dk.netarkivet.archive.tools.RunBatch -Cbatchprogs/EvilBatch2.class -R.*\.arc -Eerror.evil2

Since the batch program is supposed to fail, nothing should be written to std-out, only to std-error. It is therefore the std-err which should be catched, which can be done with the -E argument.

This should produce a file error.evil2 with be something like:

Code Block
Failed files:
1-1-20130114144130-00000-kb-test-har-001.kb.dk.arc
1-1-20130114144130-00003-kb-test-har-001.kb.dk.arc
1-1-20130114144130-00001-kb-test-har-001.kb.dk.arc
1-1-20130114144130-00002-kb-test-har-001.kb.dk.arc
1-metadata-1.arc

ExceptionBatch

Checks if a file is a metadata file, in which case it fails with an exception. Otherwise it writes out that the given file is not a metadata file.

Run

Code Block
java -cp lib/netarchivesuite-archive-core.jar -Dsettings.common.applicationInstanceId=EXCEPTION -Ddk.netarkivet.settings.file=conf/settings_IndexServerApplication.xml dk.netarkivet.archive.tools.RunBatch -Cbatchprogs/ExceptionBatch.class -R.*\.arc -Eerror.exception -Ooutput.exception

This should produce a file 'output.exception''  with something like:

Code Block
non-metadata file: 1-1-20130114144130-00000-kb-test-har-001.kb.dk.arc
non-metadata file: 1-1-20130114144130-00003-kb-test-har-001.kb.dk.arc
non-metadata file: 1-1-20130114144130-00001-kb-test-har-001.kb.dk.arc
non-metadata file: 1-1-20130114144130-00002-kb-test-har-001.kb.dk.arc

And a file 'error.exception'  with something like:

Code Block
Failed files:
1-metadata-1.arc

ExceptionBatchFinish

Does not work properly (does not give exception, only writes out name).

Run

Code Block
java -cp lib/netarchivesuite-archive-core.jar -Dsettings.common.applicationInstanceId=FINISH -Ddk.netarkivet.settings.file=conf/settings_IndexServerApplication.xml dk.netarkivet.archive.tools.RunBatch -Cbatchprogs/ExceptionBatchFinish.class -R.*\.arc -Ooutput.exception.finish

This should produce a file ' output.exception.finish''  with something like:

Code Block
1-1-20130114144130-00000-kb-test-har-001.kb.dk.arc
1-metadata-1.arc
1-1-20130114144130-00001-kb-test-har-001.kb.dk.arc
1-1-20130114144130-00003-kb-test-har-001.kb.dk.arc
1-1-20130114144130-00002-kb-test-har-001.kb.dk.arc

ExtractDeduplicateCDXBatchJob

Run

Code Block
java -Dsettings.common.applicationInstanceId=DEDUP -Ddk.netarkivet.settings.file=conf/settings_IndexServerApplication.xml -cp lib/netarchivesuite-archive-core.jar dk.netarkivet.archive.tools.RunBatch -Ndk.netarkivet.wayback.batch.DeduplicationCDXExtractionBatchJob -Jlib/netarchivesuite-wayback-indexer.jar -R'.*metadata.*\.arc' -BSBN -Ocdx.output

The output in the cdx.output file should be well formed cdx records with each line starting with a canonicalized url like these:

Code Block
jigsaw.w3.org/css-validator/images/vcss-blue 20091001090817 http://jigsaw.w3.org/css-validator/images/vcss-blue image/gif 200 YEX3QI4MLIW3EPJCYCA2OZWT4ZEQGA36 - 8221 3-3-20091001084354-00004-kb-test-har-001.kb.dk.arc

123hjemmeside.dk/picture.aspx?id=13195185 20090114041208 http://www.123hjemmeside.dk/picture.aspx?id=13195185 image/jpeg 200 SY7VL2B5HODA3TKBR6DQLCAN74BSUL3G - 33623059 27020-70-20080505183112-00095-kb-prod-har-002.kb.dk.arc
123hjemmeside.dk/picture.aspx?id=13243517 20090114105138 http://www.123hjemmeside.dk/picture.aspx?id=13243517 image/jpeg 200 XOEJDTALKDZNB2KKRCUS3K6X2LGZKEAC - 24666098 27063-70-20080513145144-00069-kb-prod-har-001.kb.dk.arc
123hjemmeside.dk/picture.aspx?id=13244248 20090114105514 http://www.123hjemmeside.dk/picture.aspx?id=13244248 image/jpeg 200 R7AEJVSL6Z3U2CAQGNDKKJ2NL5MSOCJU - 38159302 27063-70-20080513144604-00068-kb-prod-har-001.kb.dk.arc

The fields are:
<canonical url> <original url> <mime-type> <http code> <checksum> <redirect url> <length> <filename>

(In the above cases, <redirect-url> is empty, which it will always be unless there is a 3xx http code.)

Run CDX Job On WARC Files

Code Block
java -Dsettings.common.applicationInstanceId=DEDUP -Ddk.netarkivet.settings.file=conf/settings_IndexServerApplication.xml -cp lib/netarchivesuite-archive-core.jar dk.netarkivet.archive.tools.RunBatch -Ndk.netarkivet.wayback.batch.DeduplicationCDXExtractionBatchJob -Jlib/netarchivesuite-wayback-indexer.jar -R'.*metadata.*\.warc' -BSBN -Ocdx.warc.output

Check that there is non-empty output. This verifies that CDX deduplicate extraction also works on warcfiles. 

ExceptionBatchInit 

This method should fail before starting to process the files, and thus no files should be processed.

Run

Code Block
java -cp lib/netarchivesuite-archive-core.jar -Dsettings.common.applicationInstanceId=INIT -Ddk.netarkivet.settings.file=conf/settings_IndexServerApplication.xml dk.netarkivet.archive.tools.RunBatch -Cbatchprogs/ExceptionBatchInit.class -R.*\.arc

The following text message should be written in the console:

Code Block
Running batch job 'batchprogs/ChecksumJobExceptionBatchInit.class' on files matching 'arc.*.arc' on replica 'KBN', output written to file 'output.checksum', stdout errors written to stderr 
Processed 110 files with 0 failures
Cleaning up dk.netarkivet.common.distribute.JMSConnectionSunMQ
Cleaned up dk.netarkivet.common.distribute.JMSConnectionSunMQ

Content of file output.checksum should be something like:

Code Block
Run batch programs on records within archive files
These methods does not check on the '.arc' files, but on the content of these '.arc' files.
SimpleArcBatchJob
Checks that it is posible to access the records within a arc file.
Run
 Code Block
java -cp lib/netarchivesuite-archive-core.jar -Dsettings.common.applicationInstanceId=SIMPLE -Ddk.netarkivet.settings.file=conf/settings_IndexServerApplication.xml dk.netarkivet.archive.tools.RunBatch -Cbatchprogs/SimpleArcBatchJob.class -R.*\.arc -Ooutput.simple
The output file should be quite large compared to the previous output files.
The size of the output files and the error files can be found by running the command: du -h output.* error.*. This gives the following results:
 Code Block
4,0K    output.evil
4,0K    output.exception
4,0K    output.exception.finish
212K    output.simple
4,0K    error.evil2
4,0K    error.exception
The content of the output file contain a line of data for each record. This line will have the following structure:
 Code Block
org.archive.io.arc.ARCRecord@f16070 available: q:\bitarkiv\JOLF\filedir\1-1-20090316092641-0000300001-kb-test-har-002.kb.dk.arc##c68b3e18f7b870b76d86de7970a822c2
2-2-20090316092643-00003.arc: {ip-address=0.0.0.0, content-type=text/plain, absolute-offset=0, subject-uri=filedesc://1-1-20090316092641-00001-kb-test-har-001002.kb.dk.arc##7d723dd4d374437c5e29e995521bf014
1-1-20090316092641-00001-kb-test-har-002.kb.dk.arc##038c1a09bed89a6f7bdb7db029c98c9b
2-2-20090316092643-00002-kb-test-har-001.kb.dk.arc##7eef07cc6332c5e97ef80b92d1428e2d
2-metadata-1.arc##1ef96326228f9e705db6d14087be9fe8
1-1-20090316092641-00002-kb-test-har-002.kb.dk.arc##b2ce92e8381b3e4ee4780af0be24b0ee
1-metadata-1.arc##ff2b5bdd901a4045f06e8c3171f7b6e9
2-2-20090316092643-00000.arc.open, length=1343, creation-date=20090316092641, version=1.1}
Furthermore, a list of failed files should be printed to stdout (all of them WARC files).
Running methods from jar files, including Jhove based methods
Before running any Jhove methods, run the following command:
 Code Block
cp -av /home/devel/backup_TEST11A/lib_jhove/* /home/devel/TEST11A/lib
eu.planets.batch.jar -> eu.planets.JhoveArcJob 
Get some metadata from all records within the arc-files through Jhove.
This method writes the following metadata on the all the records of handable dataformats: parsing errors, format, version,mimetype.
For the dataformats which are not handled, these values are replaced by 'null', except mimetype which is replaced by the mimetype of the record.
Run
 Code Block
java -cp lib/netarchivesuite-archive-core.jar -Dsettings.common.applicationInstanceId=JHOVE -Ddk.netarkivet.settings.file=conf/settings_IndexServerApplication.xml dk.netarkivet.archive.tools.RunBatch -Jbatchprogs/eu.planets.batch.jar,lib/jhove.jar,lib/jhove-module.jar -Neu.planets.JhoveArcJob -R.*\.arc -Ooutput.jhove.arc
The output.jhove.arc file should be in the following format:
 Code Block
0,HTML,null,text/html
null,null,null,image/png
null,null,null,text/css
0,GIF,null,image/gif
0,JPEG,null,image/jpeg
.....
Furthermore, a list of failed files should be printed to stdout (all of them WARC files).
Check the content of metadata files and 'content' 
These three methods takes a copy of the content from different arc files based on a regular expression.
All these tests are basically stress-tests, which by dumping the content of all records within a arc file to the output stream. First the content of all the files are collected, then only the content of the metadata files, and finally the data from the harvesting. These are all put into different output files, and the first file should have the same size as the other two combined.
CopyArcContent
This tests the ability to copy the content of all the files in the bitarchives. It can be seen as a stress test of the batch system.
Run
 Code Block
java -cp lib/netarchivesuite-archive-core.jar -Dsettings.common.applicationInstanceId=COPYARC -Ddk.netarkivet.settings.file=conf/settings_IndexServerApplication.xml dk.netarkivet.archive.tools.RunBatch -Cbatchprogs/CopyArcContent.class -R.*\.arc -Ooutput.copy.class  
This method copies all the content from the records to the output file, and therefore it takes some time to run the command (approx. 25 sek on 60 MB). Takes about 40 min's on 80 files (6GB).
The content of the output file is not easy to confirm. It should start by having the metadata of a record and then the actual content of this record.
Check the size of the output file (run the command du -h output.copy.class): 
59M     output.copy.class

This value should be approximately the same as the combined size of all the harvests.
Furthermore, a list of failed files should be printed to stdout (all of them WARC files).
Test a WARC Batch Job
 Code Block
java -Dsettings.common.applicationInstanceId=DEDUP -Ddk.netarkivet.settings.file=conf/settings_IndexServerApplication.xml -cp lib/netarchivesuite-archive-core.jar dk.netarkivet.archive.tools.RunBatch -Ndk.netarkivet.common.utils.cdx.WARCExtractCDXJob -Jlib/netarchivesuite-wayback-indexer.jar -R'.*dk.*\.warc' -BSBN -Ocdx.warc.all.output
Check that the output file (cdx.warc.all.output) has a significant amount of content.
eu.planets.batch.jar -> eu.planets.CopyArcContent: 'metadata'
Copy the content of the metadata files only.
 
Run
 
 Code Block
java -cp lib/netarchivesuite-archive-core.jar -Dsettings.common.applicationInstanceId=META -Ddk.netarkivet.settings.file=conf/settings_IndexServerApplication.xml dk.netarkivet.archive.tools.RunBatch -Cbatchprogs/CopyArcContent.class -Ooutput.copy.meta -R'.*-metadata-.*\.arc' -BKBN
 
This should give the following output in the console:
 Code Block
Running batch job 'batchprogs/CopyArcContent.class' on files matching '.*-metadata-.*' on replica 'KBN', output written to file 'output.copy.meta', errors written to stderr
Processed 2 files with 0 failures
eu.planets.batch.jar -> eu.planets.CopyArcContent: ’content’
Copy only the content from of the files collected in the harvest. This test assumes that the harvesters in the test system is in the '.dk' domain.
 
Run
 
 Code Block
java -cp lib/netarchivesuite-archive-core.jar -Dsettings.common.applicationInstanceId=CONTENT -Ddk.netarkivet.settings.file=conf/settings_IndexServerApplication.xml dk.netarkivet.archive.tools.RunBatch -Cbatchprogs/CopyArcContent.class -Ooutput.copy.content -R'.*.dk.*\.arc' -BKBN
 
The regular expression should handle any files besides the metadata files, since they don't contain the sequence '.dk' in their name. This means that they handles all the other files, the 'content' files.
This should give the following output in the console:
 
 Code Block
Running batch job 'batchprogs/CopyArcContent.class' on files matching '.*.dk.*' on replica 'KBN', output written to file 'output.copy.content', errors written to stderr
Processed 9 files with 0 failures
Check that the system properties can be read
 
Run
 
 Code Block
java -cp lib/netarchivesuite-archive-core.jar -Dsettings.common.applicationInstanceId=PROPS -Ddk.netarkivet.settings.file=conf/settings_IndexServerApplication.xml dk.netarkivet.archive.tools.RunBatch -Cbatchprogs/SystemReaderJob.class -Ooutput.system -Eerror.system -R'.*.dk.*' -BSBN
 
Be aware of, that all metadata files are excluded.
Check that the error.system file is empty.
The output.system should contain system properties and a list of files. First the java version, then the operating system name, the operating system architecture and the operating system version. Then the list of files should be written, followed by a count of the files and the system property user name. It could look something like this:
 
 Code Block
System properties!
java version: 1.6.0_24
os name: Linux
os architecture: i386
os version: 2.6.32-220.4.1.el6.x86_64
File: 1-1-20130114144130-00001-kb-test-har-001.kb.dk.arc##a050db8b02c05270eeacac6de3f328e9arc
File: 1-1-2009031609264120130114144130-0000000002-kb-test-har-002001.kb.dk.arc##e0dedb1584f422d54a36f2684ce8ee56arc
File: 1-1-2009031609340320130114144130-0000400003-kb-test-har-002001.kb.dk.arc##bb3f96f74d9c8d1ae545ec84d914d976
2-2-20090316092643-00001arc
File: 1-1-20130114144130-00000-kb-test-har-001.kb.dk.arc##a321400d7a168230e5ed7d8427be05bearc
File count: 4
User: netarkiv