Table of Contents | ||||
---|---|---|---|---|
|
Excerpt |
---|
Stress test and crashtests of batch jobs and third-party batch jobs on 14 22 TB. |
Table of Contents | ||
---|---|---|
|
Start The Test
Code Block |
---|
export TESTX=TEST11B export PORT=807? export MAILRECEIVERS=foo@bar.dk stop_test.sh cleanup_all_test.sh prepare_test.sh deploy_config_TEST11B.xml install_test.sh start_test.sh export VERSION=5.4-RC1 export H3ZIP=/home/devel/nas_versions/bundler/NetarchiveSuite-heritrix3-bundler-$VERSION.zip export CONF=deploy_config_TEST11B.xml prepare_test.sh -d $CONF -v $VERSION -3 $H3ZIP #### modify the prepared scripts in folder prepared_software/TEST11B to use <drive>:\bitarchive_X\ instead of <drive>:\bitarchive_X\TEST11B ### The following files need to be modified: dir_kb-test-bar-01.bitarkiv.kb.dk.bat, kb-test-bar-01.bitarkiv.kb.dk/security.policy, kb-test-bar-01.bitarkiv.kb.dk/settings_BitarchiveApplication_BitApp_[E|F|G|H|I|J].xml cd prepared_software/TEST11B sed -i -- 's/\\TEST11B\\filedir/\\filedir/g' dir_kb-test-bar-01.bitarkiv.kb.dk.bat sed -i -- 's/\\TEST11B\\tempdir/\\tempdir/g' dir_kb-test-bar-01.bitarkiv.kb.dk.bat sed -i -- 's/\\TEST11B\\atticdir/\\atticdir/g' dir_kb-test-bar-01.bitarkiv.kb.dk.bat sed -i -- 's/bitarchive_1\\TEST11B/bitarchive_1/g' dir_kb-test-bar-01.bitarkiv.kb.dk.bat sed -i -- 's/bitarchive_2\\TEST11B/bitarchive_2/g' dir_kb-test-bar-01.bitarkiv.kb.dk.bat sed -i -- 's/bitarchive_3\\TEST11B/bitarchive_3/g' dir_kb-test-bar-01.bitarkiv.kb.dk.bat sed -i -- 's/bitarchive_4\\TEST11B/bitarchive_4/g' dir_kb-test-bar-01.bitarkiv.kb.dk.bat sed -i -- 's/bitarchive_5\\TEST11B/bitarchive_5/g' dir_kb-test-bar-01.bitarkiv.kb.dk.bat sed -i -- 's/bitarchive_6\\TEST11B/bitarchive_6/g' dir_kb-test-bar-01.bitarkiv.kb.dk.bat sed -i -- 's/\\TEST11B/\\/g' kb-test-bar-01.bitarkiv.kb.dk/settings_BitarchiveApplication_BitApp_?.xml sed -i -- 's/TEST11B${\/}filedir${\/}/filedir${\/}/g' kb-test-bar-01.bitarkiv.kb.dk/security.policy cd install_test.sh -d $CONF -v $VERSION -3 $H3ZIP start_test.sh -d $CONF -v $VERSION -3 $H3ZP |
Update The Filelist
In the GUI, update the Filestatus for KB. This will take several minutes to run. In the GUIApplication log you should see lines like "WARNING: There have been found multiple files with the name '9999-MB100.arc'" and "INFO: The file 'MB100' was not found in the database. Thus creating entry for the file. " and it Most of the time is spent on updating the empty admin database, so you will get a lot of logs in GUIApplication saying stuff like
Code Block |
---|
2018-04-17 17:16:07.803 [Thread-41] INFO dk.netarkivet.archive.arcrepositoryadmin.ReplicaCacheHelpe
rs.addFileInformation
- The file '44854-91-20090509044357-00076-sb-prod-har-005.arc' was not
found in the database. Thus creating entry for the file. |
The last logentry should end with "INFO: Completed findMissingFiles for replica 'BITARCHIVEReplica (KB) KBN'."
...
Code Block |
---|
Filestatus for: KBN Number of files: 25180,890569 Missing files: 0 |
Update Checksum and FileStatus
...
in the GUIApplication log.
Start Some Batch Jobs
On test@kbdevel@kb-test-adm-001:
Code Block |
---|
export TESTX=TEST11B cd /home/testdevel/$TESTX/ mkdir batchprogs scp### IN DEVELOPMENT ¤################################## ## The path below does not exist ##scp test@kb-prod-udv-001.kb.dk:/home/test/test cp -pv test-batch/* batchprogs/. |
...
Code Block |
---|
nohup java -cp lib/dk.netarkivet.archivenetarchivesuite-archive-core.jar -Ddk.netarkivet.settings.file=conf/settings_ArcRepositoryApplication.xml -Dsettings.common.applicationInstanceId=BATCH dk.netarkivet.archive.tools.RunBatch -Cbatchprogs/ChecksumJob.class -Ooutput.checksum & nohup java -cp lib/dk.netarkivet.archivenetarchivesuite-archive-core.jar -Ddk.netarkivet.settings.file=conf/settings_ArcRepositoryApplication.xml -Dsettings.common.applicationInstanceId=BATCH2 dk.netarkivet.archive.tools.RunBatch -Cbatchprogs/GoodPostProcessingJob.class -Ogood.out & nohup java -cp lib/dk.netarkivet.archivenetarchivesuite-archive-core.jar -Ddk.netarkivet.settings.file=conf/settings_ArcRepositoryApplication.xml -Dsettings.common.applicationInstanceId=BATCH3 dk.netarkivet.archive.tools.RunBatch -Cbatchprogs/EvilPostProcessingJob.class -Oevil.out & nohup java -cp lib/dk.netarkivet.archivenetarchivesuite-archive-core.jar -Ddk.netarkivet.settings.file=conf/settings_ArcRepositoryApplication.xml -Dsettings.common.applicationInstanceId=BATCH4 dk.netarkivet.archive.tools.RunBatch -Jbatchprogs/mime.jar -Nbatchprogs.MimeSize -Omimesize.out & |
...
Code Block |
---|
0G5.arc 1G5.arc ... |
...
Check The Output Of The Checksum Job
This may take about eight hours to run.
The output is put into the file 'output.checksum'. This file should contain the following text:
Code Block |
---|
1-1-20090316092641-00003-kb-test-har-002.kb.dk.arc##c68b3e18f7b870b76d86de7970a822c2 2-2-20090316092643-00003-kb-test-har-001.kb.dk.arc##7d723dd4d374437c5e29e995521bf014 ....... |
Check The Output Of MimeSize
This may take up to 25 hours to run. The output is put into the file 'mimesize.out'. This file should contain the following text:
Code Block |
---|
..
text/html##567890
image/jpeg##1234567
... |
Further tests
Child pages (Children Display) | ||||
---|---|---|---|---|
|