Table of Contents | ||||
---|---|---|---|---|
|
Excerpt |
---|
Stress test and crashtests of batch jobs and third-party batch jobs on 14 22 TB. |
Table of Contents | ||
---|---|---|
|
Start The Test
Code Block |
---|
export TESTX=TEST11B export PORT=807? export MAILRECEIVERS=foo@bar.dk stop_test.sh cleanup_all_test.sh prepare_test.sh deploy_config_TEST11B.xml install_test.sh start_test.sh export VERSION=5.4-RC1 export H3ZIP=/home/devel/nas_versions/bundler/NetarchiveSuite-heritrix3-bundler-$VERSION.zip export CONF=deploy_config_TEST11B.xml prepare_test.sh -d $CONF -v $VERSION -3 $H3ZIP #### modify the prepared scripts in folder prepared_software/TEST11B to use <drive>:\bitarchive_X\ instead of <drive>:\bitarchive_X\TEST11B ### The following files need to be modified: dir_kb-test-bar-01.bitarkiv.kb.dk.bat, kb-test-bar-01.bitarkiv.kb.dk/security.policy, kb-test-bar-01.bitarkiv.kb.dk/settings_BitarchiveApplication_BitApp_[E|F|G|H|I|J].xml cd prepared_software/TEST11B sed -i -- 's/\\TEST11B\\filedir/\\filedir/g' dir_kb-test-bar-01.bitarkiv.kb.dk.bat sed -i -- 's/\\TEST11B\\tempdir/\\tempdir/g' dir_kb-test-bar-01.bitarkiv.kb.dk.bat sed -i -- 's/\\TEST11B\\atticdir/\\atticdir/g' dir_kb-test-bar-01.bitarkiv.kb.dk.bat sed -i -- 's/bitarchive_1\\TEST11B/bitarchive_1/g' dir_kb-test-bar-01.bitarkiv.kb.dk.bat sed -i -- 's/bitarchive_2\\TEST11B/bitarchive_2/g' dir_kb-test-bar-01.bitarkiv.kb.dk.bat sed -i -- 's/bitarchive_3\\TEST11B/bitarchive_3/g' dir_kb-test-bar-01.bitarkiv.kb.dk.bat sed -i -- 's/bitarchive_4\\TEST11B/bitarchive_4/g' dir_kb-test-bar-01.bitarkiv.kb.dk.bat sed -i -- 's/bitarchive_5\\TEST11B/bitarchive_5/g' dir_kb-test-bar-01.bitarkiv.kb.dk.bat sed -i -- 's/bitarchive_6\\TEST11B/bitarchive_6/g' dir_kb-test-bar-01.bitarkiv.kb.dk.bat sed -i -- 's/\\TEST11B/\\/g' kb-test-bar-01.bitarkiv.kb.dk/settings_BitarchiveApplication_BitApp_?.xml sed -i -- 's/TEST11B${\/}filedir${\/}/filedir${\/}/g' kb-test-bar-01.bitarkiv.kb.dk/security.policy cd install_test.sh -d $CONF -v $VERSION -3 $H3ZIP start_test.sh -d $CONF -v $VERSION -3 $H3ZP |
Update The Filelist
In the GUI, update the Filestatus for KB. This will take several minutes to run. In the GUIApplication log you should see lines like "WARNING: There have been found multiple files with the name '9999-MB100.arc'" and "INFO: The file 'MB100' was not found in the database. Thus creating entry for the file. " and it Most of the time is spent on updating the empty admin database, so you will get a lot of logs in GUIApplication saying stuff like
Code Block |
---|
2018-04-17 17:16:07.803 [Thread-41] INFO dk.netarkivet.archive.arcrepositoryadmin.ReplicaCacheHelpe
rs.addFileInformation
- The file '44854-91-20090509044357-00076-sb-prod-har-005.arc' was not
found in the database. Thus creating entry for the file. |
The last logentry should end with "INFO: Completed findMissingFiles for replica 'BITARCHIVEReplica (KB) KBN'."
...
Code Block |
---|
Filestatus for: KBN Number of files: 25180,890569 Missing files: 0 |
Update Checksum and FileStatus
...
Wait until the job has finished (about seven hours) with a message like
Code Block |
---|
INFO: Finished batch job dk.netarkivet.archive.arcrepository.bitpreservation.ChecksumJob with result: 0 failures in processing 27620 files at 172.17.0.176_BitApp_E |
on the BitApp and
Code Block |
---|
Aug 15, 2013 10:11:34 PM dk.netarkivet.archive.arcrepositoryadmin.ReplicaCacheDatabase updateChecksumStatus
INFO: UpdateChecksumStatus operation completed!
Aug 15, 2013 10:11:34 PM dk.netarkivet.archive.arcrepository.bitpreservation.DatabaseBasedActiveBitPreservation findChangedFiles
INFO: Completed findChangedFiles for replica 'CHECKSUMReplica (CS2) CS2N'. |
in the GUIApplication log.
Start Some Batch Jobs
On devel@kb-test-adm-001:
Code Block |
---|
export TESTX=TEST11B
cd /home/devel/$TESTX/
mkdir batchprogs
### IN DEVELOPMENT ¤##################################
## The path below does not exist
##scp test@kb-prod-udv-001.kb.dk:/home/test/test
cp -pv test-batch/* batchprogs/. |
Then start the following four jobs:
Code Block |
---|
nohup java -cp lib/netarchivesuite-archive-core.jar -Ddk.netarkivet.settings.file=conf/settings_ArcRepositoryApplication.xml -Dsettings.common.applicationInstanceId=BATCH dk.netarkivet.archive.tools.RunBatch -Cbatchprogs/ChecksumJob.class -Ooutput.checksum &
nohup java -cp lib/netarchivesuite-archive-core.jar -Ddk.netarkivet.settings.file=conf/settings_ArcRepositoryApplication.xml -Dsettings.common.applicationInstanceId=BATCH2 dk.netarkivet.archive.tools.RunBatch -Cbatchprogs/GoodPostProcessingJob.class -Ogood.out &
nohup java -cp lib/netarchivesuite-archive-core.jar -Ddk.netarkivet.settings.file=conf/settings_ArcRepositoryApplication.xml -Dsettings.common.applicationInstanceId=BATCH3 dk.netarkivet.archive.tools.RunBatch -Cbatchprogs/EvilPostProcessingJob.class -Oevil.out &
nohup java -cp lib/netarchivesuite-archive-core.jar -Ddk.netarkivet.settings.file=conf/settings_ArcRepositoryApplication.xml -Dsettings.common.applicationInstanceId=BATCH4 dk.netarkivet.archive.tools.RunBatch -Jbatchprogs/mime.jar -Nbatchprogs.MimeSize -Omimesize.out & |
The "good" and "evil" jobs complete quickly. You can follow the progress of all the jobs simply with
Code Block |
---|
ps -fe | grep BATCH |
Check The Output of "Good"
The output is put into the file 'good.out'. This file should contain the following text (sorted):
Code Block |
---|
0G5.arc
0G5.arc
... |
Go to the status page and check the log for the BitarchiveMonitor . There should be the following messages:
Code Block |
---|
Jun 11, 2010 9:24:31 AM dk.netarkivet.archive.bitarchive.distribute.BitarchiveMonitorServer doBatchReply
INFO: BatchReplyMessage: 'BatchReplyMessage for batch job ID:10-130.226.228.6(d3:b5:49:8b:d6:94)-37536-1276241061823
FilesProcessed = 156775
FilesFailed = 0
ID:1906780-130.226.228.6(d4:77:b5:34:b8:ae)-52334-1276241071022: To TEST11B_COMMON_THIS_REPOS_CLIENT_130_226_228_6_GUIA_BATCH ReplyTo TEST11B_KB_THE_BAMON OK' sent from BA monitor to queue: '[Queue 'TEST11B_COMMON_THIS_REPOS_CLIENT_130_226_228_6_GUIA_BATCH']'
Jun 11, 2010 9:24:27 AM dk.netarkivet.common.utils.batch.GoodPostProcessingJob postProcess
INFO: Sorting the filenames
Jun 11, 2010 9:24:27 AM dk.netarkivet.common.utils.batch.GoodPostProcessingJob postProcess
INFO: Reading all the filenames.
Jun 11, 2010 9:24:27 AM dk.netarkivet.archive.bitarchive.distribute.BitarchiveMonitorServer doBatchReply
INFO: Post processing batchjob results for 'dk.netarkivet.common.utils.batch.LoadableFileBatchJob' with id 'ID:10-130.226.228.6(d3:b5:49:8b:d6:94)-37536-1276241061823' |
Check The Output of "Evil"
The output is put into the file 'evil.out'. This file should contain the following text (unsorted):
Code Block |
---|
0G5.arc
1G5.arc
... |
Check The Output Of The Checksum Job
This may take about eight hours to run. The output is put into the file 'output.checksum'. This file should contain the following text:
Code Block |
---|
1-1-20090316092641-00003-kb-test-har-002.kb.dk.arc##c68b3e18f7b870b76d86de7970a822c2
2-2-20090316092643-00003-kb-test-har-001.kb.dk.arc##7d723dd4d374437c5e29e995521bf014
....... |
Check The Output Of MimeSize
This may take up to 25 hours to run. The output is put into the file 'mimesize.out'. This file should contain the following text:
Code Block |
---|
..
text/html##567890
image/jpeg##1234567
... |
Further tests
Child pages (Children Display) | ||||
---|---|---|---|---|
|