Setup
Login to the indexserver (kb-test-acs-001.kb.dk)
$ ssh devel@kb-test-acs-001 $ export TESTX=TEST11A $ cd ${TESTX}Create dir for batchprograms and copy contents of /home/devel/backup_TEST11A/batchprogs to /home/devel/TEST11A/
$ mkdir batchprogs $ cp -av /home/devel/backup_TEST11A/batchprogs batchprogs/
Run batch programs on archived files
ChecksumJob
Calculating the MD5 checksum on the archive files (only .arc files).
Run
java -cp lib/netarchivesuite-archive-core.jar -Dsettings.common.applicationInstanceId=CHECKSUM -Ddk.netarkivet.settings.file=conf/settings_IndexServerApplication.xml dk.netarkivet.archive.tools.RunBatch -Cbatchprogs/ChecksumJob.class -R.*\.arc -Ooutput.checksum
This should produce a file output.checksum with be something like:
1-metadata-1.arc##9dca301f9ecd3663bbb3cdc91009562c 1-1-20130114144130-00001-kb-test-har-001.kb.dk.arc##1abe93f92569154da3f629255146dcfd 1-1-20130114144130-00002-kb-test-har-001.kb.dk.arc##6b2801ca68d7c96af8b05ba9e9ae47b4 1-1-20130114144130-00003-kb-test-har-001.kb.dk.arc##5c77d44ad9fc9290f2cd6639562ed11a 1-1-20130114144130-00000-kb-test-har-001.kb.dk.arc##c1e533f574a46576fc14393581428129
EvilBatch
Writes the sentence 'evil' for each bitarchive.
Run
java -cp lib/netarchivesuite-archive-core.jar -Dsettings.common.applicationInstanceId=CHECKSUM -Ddk.netarkivet.settings.file=conf/settings_IndexServerApplication.xml dk.netarkivet.archive.tools.RunBatch -Cbatchprogs/EvilBatch.class -R.*\.arc -Ooutput.evil
This should produce a file output.evil with:
Legal Legal Legal Legal
This is run on the KBN replica, which has four bitapps (according to the default configuration file ). Therefore the sentence 'Legal' is written four times.
The SBN replica one has one bitapp, and it should therefore only have the sentence, 'Legal', written once.
Run
java -cp lib/netarchivesuite-archive-core.jar -Dsettings.common.applicationInstanceId=EVILSB -Ddk.netarkivet.settings.file=conf/settings_IndexServerApplication.xml dk.netarkivet.archive.tools.RunBatch -Cbatchprogs/EvilBatch.class -R.*\.arc -Ooutput.evil.sb -BSBN
This should produce a file output.evil with:
Legal
This is run on the SBN replica, which has only one bitapp (according to the default configuration file'). Therefore the sentence 'Legal' is written once.
EvilBatch2
Tries to delete archived files, thus gets failed for all files.
Run
java -cp lib/netarchivesuite-archive-core.jar -Dsettings.common.applicationInstanceId=EVIL2 -Ddk.netarkivet.settings.file=conf/settings_IndexServerApplication.xml dk.netarkivet.archive.tools.RunBatch -Cbatchprogs/EvilBatch2.class -R.*\.arc -Eerror.evil2
Since the batch program is supposed to fail, nothing should be written to std-out, only to std-error. It is therefore the std-err which should be catched, which can be done with the -E argument.
This should produce a file error.evil2 with be something like:
Failed files: 1-1-20130114144130-00000-kb-test-har-001.kb.dk.arc 1-1-20130114144130-00003-kb-test-har-001.kb.dk.arc 1-1-20130114144130-00001-kb-test-har-001.kb.dk.arc 1-1-20130114144130-00002-kb-test-har-001.kb.dk.arc 1-metadata-1.arc
ExceptionBatch
Checks if a file is a metadata file, in which case it fails with an exception. Otherwise it writes out that the given file is not a metadata file.
Run
java -cp lib/netarchivesuite-archive-core.jar -Dsettings.common.applicationInstanceId=EXCEPTION -Ddk.netarkivet.settings.file=conf/settings_IndexServerApplication.xml dk.netarkivet.archive.tools.RunBatch -Cbatchprogs/ExceptionBatch.class -R.*\.arc -Eerror.exception -Ooutput.exception
This should produce a file 'output.exception'' with something like:
non-metadata file: 1-1-20130114144130-00000-kb-test-har-001.kb.dk.arc non-metadata file: 1-1-20130114144130-00003-kb-test-har-001.kb.dk.arc non-metadata file: 1-1-20130114144130-00001-kb-test-har-001.kb.dk.arc non-metadata file: 1-1-20130114144130-00002-kb-test-har-001.kb.dk.arc
And a file 'error.exception' with something like:
Failed files: 1-metadata-1.arc
ExceptionBatchFinish
Does not work properly (does not give exception, only writes out name).
Run
java -cp lib/netarchivesuite-archive-core.jar -Dsettings.common.applicationInstanceId=FINISH -Ddk.netarkivet.settings.file=conf/settings_IndexServerApplication.xml dk.netarkivet.archive.tools.RunBatch -Cbatchprogs/ExceptionBatchFinish.class -R.*\.arc -Ooutput.exception.finish
This should produce a file ' output.exception.finish'' with something like:
1-1-20130114144130-00000-kb-test-har-001.kb.dk.arc 1-metadata-1.arc 1-1-20130114144130-00001-kb-test-har-001.kb.dk.arc 1-1-20130114144130-00003-kb-test-har-001.kb.dk.arc 1-1-20130114144130-00002-kb-test-har-001.kb.dk.arc
ExtractDeduplicateCDXBatchJob
Run
java -Dsettings.common.applicationInstanceId=DEDUP -Ddk.netarkivet.settings.file=conf/settings_IndexServerApplication.xml -cp lib/netarchivesuite-archive-core.jar dk.netarkivet.archive.tools.RunBatch -Ndk.netarkivet.wayback.batch.DeduplicationCDXExtractionBatchJob -Jlib/netarchivesuite-wayback-indexer.jar -R'.*metadata.*\.arc' -BSBN -Ocdx.output
The output in the cdx.output file should be well formed cdx records with each line starting with a canonicalized url like these:
jigsaw.w3.org/css-validator/images/vcss-blue 20091001090817 http://jigsaw.w3.org/css-validator/images/vcss-blue image/gif 200 YEX3QI4MLIW3EPJCYCA2OZWT4ZEQGA36 - 8221 3-3-20091001084354-00004-kb-test-har-001.kb.dk.arc 123hjemmeside.dk/picture.aspx?id=13195185 20090114041208 http://www.123hjemmeside.dk/picture.aspx?id=13195185 image/jpeg 200 SY7VL2B5HODA3TKBR6DQLCAN74BSUL3G - 33623059 27020-70-20080505183112-00095-kb-prod-har-002.kb.dk.arc 123hjemmeside.dk/picture.aspx?id=13243517 20090114105138 http://www.123hjemmeside.dk/picture.aspx?id=13243517 image/jpeg 200 XOEJDTALKDZNB2KKRCUS3K6X2LGZKEAC - 24666098 27063-70-20080513145144-00069-kb-prod-har-001.kb.dk.arc 123hjemmeside.dk/picture.aspx?id=13244248 20090114105514 http://www.123hjemmeside.dk/picture.aspx?id=13244248 image/jpeg 200 R7AEJVSL6Z3U2CAQGNDKKJ2NL5MSOCJU - 38159302 27063-70-20080513144604-00068-kb-prod-har-001.kb.dk.arc
The fields are:
<canonical url> <original url> <mime-type> <http code> <checksum> <redirect url> <length> <filename>
(In the above cases, <redirect-url> is empty, which it will always be unless there is a 3xx http code.)
Run CDX Job On WARC Files
java -Dsettings.common.applicationInstanceId=DEDUP -Ddk.netarkivet.settings.file=conf/settings_IndexServerApplication.xml -cp lib/netarchivesuite-archive-core.jar dk.netarkivet.archive.tools.RunBatch -Ndk.netarkivet.wayback.batch.DeduplicationCDXExtractionBatchJob -Jlib/netarchivesuite-wayback-indexer.jar -R'.*metadata.*\.warc' -BSBN -Ocdx.warc.output
Check that there is non-empty output. This verifies that CDX deduplicate extraction also works on warcfiles.
ExceptionBatchInit
This method should fail before starting to process the files, and thus no files should be processed.
Run
java -cp lib/netarchivesuite-archive-core.jar -Dsettings.common.applicationInstanceId=INIT -Ddk.netarkivet.settings.file=conf/settings_IndexServerApplication.xml dk.netarkivet.archive.tools.RunBatch -Cbatchprogs/ExceptionBatchInit.class -R.*\.arc
The following text message should be written in the console:
Running batch job 'batchprogs/ExceptionBatchInit.class' on files matching '.*.arc' on replica 'KBN', output written to stdout errors written to stderr Processed 0 files with 0 failures
Run batch programs on records within archive files
These methods does not check on the '.arc' files, but on the content of these '.arc' files.
SimpleArcBatchJob
Checks that it is posible to access the records within a arc file.
Run
java -cp lib/netarchivesuite-archive-core.jar -Dsettings.common.applicationInstanceId=SIMPLE -Ddk.netarkivet.settings.file=conf/settings_IndexServerApplication.xml dk.netarkivet.archive.tools.RunBatch -Cbatchprogs/SimpleArcBatchJob.class -R.*\.arc -Ooutput.simple
The output file should be quite large compared to the previous output files.
The size of the output files and the error files can be found by running the command: du -h output.* error.*. This gives the following results:
4,0K output.evil 4,0K output.exception 4,0K output.exception.finish 212K output.simple 4,0K error.evil2 4,0K error.exception
The content of the output file contain a line of data for each record. This line will have the following structure:
org.archive.io.arc.ARCRecord@f16070 available: q:\bitarkiv\JOLF\filedir\1-1-20090316092641-00001-kb-test-har-002.kb.dk.arc: {ip-address=0.0.0.0, content-type=text/plain, absolute-offset=0, subject-uri=filedesc://1-1-20090316092641-00001-kb-test-har-002.kb.dk.arc.open, length=1343, creation-date=20090316092641, version=1.1}
Running methods from jar files, including Jhove based methods
Before running any Jhove methods, run the following command:
cp -av /home/devel/backup_TEST11A/lib_jhove/* /home/devel/TEST11A/lib
eu.planets.batch.jar -> eu.planets.JhoveArcJob
Get some metadata from all records within the arc-files through Jhove.
This method writes the following metadata on the all the records of handable dataformats: parsing errors, format, version,mimetype.
For the dataformats which are not handled, these values are replaced by 'null', except mimetype which is replaced by the mimetype of the record.
Run
java -cp lib/netarchivesuite-archive-core.jar -Dsettings.common.applicationInstanceId=JHOVE -Ddk.netarkivet.settings.file=conf/settings_IndexServerApplication.xml dk.netarkivet.archive.tools.RunBatch -Jbatchprogs/eu.planets.batch.jar,lib/jhove.jar,lib/jhove-module.jar -Neu.planets.JhoveArcJob -R.*\.arc -Ooutput.jhove.arc
The output.jhove.arc file should be in the following format:
0,HTML,null,text/html null,null,null,image/png null,null,null,text/css 0,GIF,null,image/gif 0,JPEG,null,image/jpeg .....
Check the content of metadata files and 'content'
These three methods takes a copy of the content from different arc files based on a regular expression.
All these tests are basically stress-tests, which by dumping the content of all records within a arc file to the output stream. First the content of all the files are collected, then only the content of the metadata files, and finally the data from the harvesting. These are all put into different output files, and the first file should have the same size as the other two combined.
CopyArcContent
This tests the ability to copy the content of all the files in the bitarchives. It can be seen as a stress test of the batch system.
Run
java -cp lib/netarchivesuite-archive-core.jar -Dsettings.common.applicationInstanceId=COPYARC -Ddk.netarkivet.settings.file=conf/settings_IndexServerApplication.xml dk.netarkivet.archive.tools.RunBatch -Cbatchprogs/CopyArcContent.class -R.*\.arc -Ooutput.copy.class
This method copies all the content from the records to the output file, and therefore it takes some time to run the command (approx. 25 sek on 60 MB). Takes about 40 min's on 80 files (6GB).
The content of the output file is not easy to confirm. It should start by having the metadata of a record and then the actual content of this record.
Check the size of the output file (run the command du -h output.copy.class):
59M output.copy.class
This value should be approximately the same as the combined size of all the harvests.
Test a WARC Batch Job
java -Dsettings.common.applicationInstanceId=DEDUP -Ddk.netarkivet.settings.file=conf/settings_IndexServerApplication.xml -cp lib/netarchivesuite-archive-core.jar dk.netarkivet.archive.tools.RunBatch -Ndk.netarkivet.common.utils.cdx.WARCExtractCDXJob -Jlib/netarchivesuite-wayback-indexer.jar -R'.*dk.*\.warc' -BSBN -Ocdx.warc.all.output
Check that the output file has a significant amount of content.
eu.planets.batch.jar -> eu.planets.CopyArcContent: 'metadata'
Copy the content of the metadata files only.
Run
java -cp lib/netarchivesuite-archive-core.jar -Dsettings.common.applicationInstanceId=META -Ddk.netarkivet.settings.file=conf/settings_IndexServerApplication.xml dk.netarkivet.archive.tools.RunBatch -Cbatchprogs/CopyArcContent.class -Ooutput.copy.meta -R'.*-metadata-.*\.arc' -BKBN
This should give the following output in the console:
Running batch job 'batchprogs/CopyArcContent.class' on files matching '.*-metadata-.*' on replica 'KBN', output written to file 'output.copy.meta', errors written to stderr Processed 2 files with 0 failures
eu.planets.batch.jar -> eu.planets.CopyArcContent: ’content’
Copy only the content from of the files collected in the harvest. This test assumes that the harvesters in the test system is in the '.dk' domain.
Run
java -cp lib/netarchivesuite-archive-core.jar -Dsettings.common.applicationInstanceId=CONTENT -Ddk.netarkivet.settings.file=conf/settings_IndexServerApplication.xml dk.netarkivet.archive.tools.RunBatch -Cbatchprogs/CopyArcContent.class -Ooutput.copy.content -R'.*.dk.*\.arc' -BKBN
The regular expression should handle any files besides the metadata files, since they don't contain the sequence '.dk' in their name. This means that they handles all the other files, the 'content' files.
This should give the following output in the console:
Running batch job 'batchprogs/CopyArcContent.class' on files matching '.*.dk.*' on replica 'KBN', output written to file 'output.copy.content', errors written to stderr Processed 9 files with 0 failures
Check that the system properties can be read
Run
java -cp lib/netarchivesuite-archive-core.jar -Dsettings.common.applicationInstanceId=PROPS -Ddk.netarkivet.settings.file=conf/settings_IndexServerApplication.xml dk.netarkivet.archive.tools.RunBatch -Cbatchprogs/SystemReaderJob.class -Ooutput.system -Eerror.system -R'.*.dk.*' -BSBN
Be aware of, that all metadata files are excluded.
Check that the error.system file is empty.
The output.system should contain system properties and a list of files. First the java version, then the operating system name, the operating system architecture and the operating system version. Then the list of files should be written, followed by a count of the files and the system property user name. It could look something like this:
System properties! java version: 1.6.0_24 os name: Linux os architecture: i386 os version: 2.6.32-220.4.1.el6.x86_64 File: 1-1-20130114144130-00001-kb-test-har-001.kb.dk.arc File: 1-1-20130114144130-00002-kb-test-har-001.kb.dk.arc File: 1-1-20130114144130-00003-kb-test-har-001.kb.dk.arc File: 1-1-20130114144130-00000-kb-test-har-001.kb.dk.arc File count: 4 User: netarkiv