...
| Excerpt |
|---|
| Stress test and crashtests of batch jobs and third-party batch jobs on 14 22 TB. |
| Table of Contents | ||
|---|---|---|
|
...
| Code Block |
|---|
export TESTX=TEST11B export PORT=807? export MAILRECEIVERS=foo@bar.dk stop_test.sh cleanup_all_test.sh prepare_test.sh deploy_config_TEST11B.xml install_test.sh start_test.sh export VERSION=5.4-RC1 export H3ZIP=/home/devel/nas_versions/bundler/NetarchiveSuite-heritrix3-bundler-$VERSION.zip export CONF=deploy_config_TEST11B.xml prepare_test.sh -d $CONF -v $VERSION -3 $H3ZIP #### modify the prepared scripts in folder prepared_software/TEST11B to use <drive>:\bitarchive_X\ instead of <drive>:\bitarchive_X\TEST11B ### The following files need to be modified: dir_kb-test-bar-01.bitarkiv.kb.dk.bat, kb-test-bar-01.bitarkiv.kb.dk/security.policy, kb-test-bar-01.bitarkiv.kb.dk/settings_BitarchiveApplication_BitApp_[E|F|G|H|I|J].xml cd prepared_software/TEST11B sed -i -- 's/\\TEST11B\\filedir/\\filedir/g' dir_kb-test-bar-01.bitarkiv.kb.dk.bat sed -i -- 's/\\TEST11B\\tempdir/\\tempdir/g' dir_kb-test-bar-01.bitarkiv.kb.dk.bat sed -i -- 's/\\TEST11B\\atticdir/\\atticdir/g' dir_kb-test-bar-01.bitarkiv.kb.dk.bat sed -i -- 's/bitarchive_1\\TEST11B/bitarchive_1/g' dir_kb-test-bar-01.bitarkiv.kb.dk.bat sed -i -- 's/bitarchive_2\\TEST11B/bitarchive_2/g' dir_kb-test-bar-01.bitarkiv.kb.dk.bat sed -i -- 's/bitarchive_3\\TEST11B/bitarchive_3/g' dir_kb-test-bar-01.bitarkiv.kb.dk.bat sed -i -- 's/bitarchive_4\\TEST11B/bitarchive_4/g' dir_kb-test-bar-01.bitarkiv.kb.dk.bat sed -i -- 's/bitarchive_5\\TEST11B/bitarchive_5/g' dir_kb-test-bar-01.bitarkiv.kb.dk.bat sed -i -- 's/bitarchive_6\\TEST11B/bitarchive_6/g' dir_kb-test-bar-01.bitarkiv.kb.dk.bat sed -i -- 's/\\TEST11B/\\/g' kb-test-bar-01.bitarkiv.kb.dk/settings_BitarchiveApplication_BitApp_?.xml sed -i -- 's/TEST11B${\/}filedir${\/}/filedir${\/}/g' kb-test-bar-01.bitarkiv.kb.dk/security.policy cd install_test.sh -d $CONF -v $VERSION -3 $H3ZIP start_test.sh -d $CONF -v $VERSION -3 $H3ZP |
Update The Filelist
In the GUI, update the Filestatus for KB. This will take several minutes to run. In the GUIApplication log you should see lines like "WARNING: There have been found multiple files with the name '9999-MB100.arc'" and "INFO: The file 'MB100' was not found in the database. Thus creating entry for the file. " and it Most of the time is spent on updating the empty admin database, so you will get a lot of logs in GUIApplication saying stuff like
| Code Block |
|---|
2018-04-17 17:16:07.803 [Thread-41] INFO dk.netarkivet.archive.arcrepositoryadmin.ReplicaCacheHelpe
rs.addFileInformation
- The file '44854-91-20090509044357-00076-sb-prod-har-005.arc' was not
found in the database. Thus creating entry for the file. |
The last logentry should end with "INFO: Completed findMissingFiles for replica 'BITARCHIVEReplica (KB) KBN'."
...
| Code Block |
|---|
Filestatus for: KBN Number of files: 25180,890569 Missing files: 0 |
Update Checksum and FileStatus
...
in the GUIApplication log.
Start Some Batch Jobs
On test@kbdevel@kb-test-adm-001:
| Code Block |
|---|
export TESTX=TEST11B cd /home/testdevel/$TESTX/ mkdir batchprogs scp ### IN DEVELOPMENT ¤################################## ## The path below does not exist ##scp test@kb-prod-udv-001.kb.dk:/home/test/test cp -pv test-batch/* batchprogs/. |
Then start the following four jobs:
| Code Block |
|---|
nohup java -cp lib/dk.netarkivet.archivenetarchivesuite-archive-core.jar -Ddk.netarkivet.settings.file=conf/settings_ArcRepositoryApplication.xml -Dsettings.common.applicationInstanceId=BATCH dk.netarkivet.archive.tools.RunBatch -Cbatchprogs/ChecksumJob.class -Ooutput.checksum & nohup java -cp lib/dk.netarkivet.archivenetarchivesuite-archive-core.jar -Ddk.netarkivet.settings.file=conf/settings_ArcRepositoryApplication.xml -Dsettings.common.applicationInstanceId=BATCH2 dk.netarkivet.archive.tools.RunBatch -Cbatchprogs/GoodPostProcessingJob.class -Ogood.out & nohup java -cp lib/dk.netarkivet.archivenetarchivesuite-archive-core.jar -Ddk.netarkivet.settings.file=conf/settings_ArcRepositoryApplication.xml -Dsettings.common.applicationInstanceId=BATCH3 dk.netarkivet.archive.tools.RunBatch -Cbatchprogs/EvilPostProcessingJob.class -Oevil.out & nohup java -cp lib/dk.netarkivet.archivenetarchivesuite-archive-core.jar -Ddk.netarkivet.settings.file=conf/settings_ArcRepositoryApplication.xml -Dsettings.common.applicationInstanceId=BATCH4 dk.netarkivet.archive.tools.RunBatch -Jbatchprogs/mime.jar -Nbatchprogs.MimeSize -Omimesize.out & |
...
| Code Block |
|---|
.. text/html##567890 image/jpeg##1234567 ... |
Further tests
| Child pages (Children Display) | ||||
|---|---|---|---|---|
|