TEST 3A

Owned by Colin Samuel Rosenthal
Last updated: Mar 26, 2018 by Sr
4 min read
Test of the Bitpreservation functionality

Recent issues: NAS-2078 - Getting issue details... STATUS

Goals

  • Database based Bitpreservation

Prerequisites

None special.

Procedure

Prepare Installation

See Setup DK test environment.

Upload a Small Bitarchive

ssh devel@kb-prod-udv-001.kb.dk
scp -r ${HOME}/bitarchive_testdata kb-test-adm-001:
ssh kb-test-adm-001 chmod 755  bitarchive_testdata/upload.sh
#ssh kb-test-adm-001 'source .bash_profile;bitarchive_testdata/upload.sh $TESTX arcfiles'
#ssh kb-test-adm-001 'source .bash_profile;bitarchive_testdata/upload.sh $TESTX warcfiles'
ssh devel@kb-test-adm-001
# If TESTX is not set to TEST3A in your test-system, change the value below
TESTX=TEST3A
bash bitarchive_testdata/upload.sh $TESTX arcfiles
bitarchive_testdata/upload.sh $TESTX warcfiles

Check Bitpreservation Status

Go to the Bitpreservation section of the GUI and update information for each replica.

Each replica should have the same number of files (25) and there should be no errors.

Delete a File

ssh devel@kb-prod-udv-001.kb.dk
mkdir -p temp_test3a
scp netarkdv@sb-test-bar-001.statsbiblioteket.dk:/netarkiv/0001/netarkdv/${TESTX}/filedir/2-metadata* temp_test3a/
ssh netarkdv@sb-test-bar-001.statsbiblioteket.dk rm /netarkiv/0001/netarkdv/${TESTX}/filedir/2-metadata*

Check that File is Missing in GUI

In the GUI, update the file status for SB. There should be one file missing.

Restore File Manually

Copy the file back:

## as devel@kb-prod-udv-001.kb.dk
scp temp_test3a/2-metadata* netarkdv@sb-test-bar-001.statsbiblioteket.dk:/netarkiv/0001/netarkdv/${TESTX}/filedir/

Now go to the Bitpreservation GUI and and check the box Get info for the "missing" file. The page should now say "There are no more missing files".

Delete the File Again

Delete the file again as in the above procedure.

Restore the File via the GUI

  • Click on Show missing files.  

  • Click to check the box where it says Get info for <foobar>.arc, and then press execute.  

  • Verify that upload status for the file is UPLOAD_COMPLETED for KBN and CSN, whereas it is UPLOAD_FAILED for SBN.  
  • Verify that the file has the same checksum in KBN and CSN, whereas it should not have any checksum for SBN.  

  • Click to check the box where it says Add to replica SBN, and then press execute.  

  • Verify that you get the message: The file <foobar>.arc has been restored in replica SBN.

Corrupt a File in the Bitarchive

## as devel@kb-prod-udv-001.kb.dk
ssh netarkdv@sb-test-bar-001.statsbiblioteket.dk ls -l /netarkiv/0001/netarkdv/${TESTX}/filedir/2-metadata*
ssh netarkdv@sb-test-bar-001.statsbiblioteket.dk "echo foobar >> /netarkiv/0001/netarkdv/${TESTX}/filedir/2-metadata*"
ssh netarkdv@sb-test-bar-001.statsbiblioteket.dk ls -l /netarkiv/0001/netarkdv/${TESTX}/filedir/2-metadata*

The file size should have increased by a few bytes.

In the GUI, update the checksums for the replica SBN.

There should now be one file with an error. The file should have status UPLOAD_FAILED for SBN. Remain on this page in the GUI.

The number of files in all the replicas should be equal.

Corrupt the File Further

## as devel@kb-prod-udv-001.kb.dk
ssh netarkdv@sb-test-bar-001.statsbiblioteket.dk "echo barfoo >> /netarkiv/0001/netarkdv/${TESTX}/filedir/2-metadata*"

Try to Correct the File Through the GUI

Try to correct the file through the GUI (the password used should be "examplecredentials") using the button "Replace the file in replica BITARCHIVEReplica (SB) SBN".

It should fail with a message like this: Attempt at restoring 2-metadata-1.arc in replica replica BITARCHIVEReplica (SB) SBN failed

CorrectMessage failed: The RemoveAndGetFileMessage has returned the error: 'Attempt to remove /netarkiv/0001/netarkdv/TEST3A/filedir/2-metadata-1.arc failed due to checksum mismatch: eab093f9a5770785348513b30a2d126a != d284044cf0d07cf2ac86eab21689b36b'. Reply to CorrectMessage with the same error.'

Correct the File Through the GUI

  • Now go back to the Bitpreservation.
  • Click on "Show files with errors" and "info".
  • Now try to replace the file but with the wrong password. This should fail.
  • Now replace the file using the correct password (examplecredentials). This should succeed with a suitable information message like "The file 2-metadata-1.arc has been replaced at replica BITARCHIVEReplica (SB) SBN"

Delete a Checksum Entry

## as devel@kb-prod-udv-001.kb.dk
ssh kb-test-acs-001 "sed -i '/2-metadata/d' ${TESTX}/CS/checksum_CS.md5"

Verify the Deleted Entry

  • Click on Update filelist for CS

  • Check that it now contains one missing file.  
  • Check that the number of files are 1 lower than the other replicas.  

  • Check that the Last update for the filelist is now (or just a moment ago).

Correct the Deleted Entry

  • Click on Show missing files.  

  • Click to check the box where it says Get info for 2-metadata-1.arc, and then press execute.  

  • Verify that upload status for the file is UPLOAD_COMPLETED for KBN and SBN, whereas it is UPLOAD_FAILED for CSN.  
  • Verify that the file has the same checksum in KBN and SBN, whereas it should not have any checksum for CSN.  

  • Click to check the box where it says Add to replica, and then press execute.  

  • Verify that you get the message: The file 2-metadata-1.arc has been restored in replica CSN.

Corrupt a Checksum Entry

## as devel@kb-prod-udv-001.kb.dk
ssh kb-test-acs-001 "sed -i '/2-metadata/s/$/foobar/' ${TESTX}/CS/checksum_CS.md5"

  • Click on Update checksum and filestatus for CS

  • Verify that it now contains one file with error.  
  • Verify that no files are missing from the replica.  
  • Verify that the number of files are the same as the other replicas.  

  • Verify that the Last update for the checksum status and filelist is now (or just a moment ago).

Correct the Corrupted Entry

  • Click on Show files with error.  

  • Verify that only one file is shown.  

  • Click on the info link for the file 2-metadata-1.arc.  

  • Verify that upload status for the file is UPLOAD_COMPLETED for KBN and SBN, whereas it is UPLOAD_FAILED for CSN.  
  • Verify that the file has the same checksum in KBN and SBN, whereas it should be a different one for CSN.  

  • Write the credentials (default examplecredentials) and press the button Replace the file in replica CHECKSUMReplica (CS) CSN

    • The resulting message should: The file 2-metadata-1.arc has been replaced at replica CHECKSUMReplica (CS) CSN.

  • Return to the Filestatus page and check that there are no errors or missing files in any replica.

Stop the Test and Clean-Up

## as devel@kb-prod-udv-001.kb.dk
cleanup_all_test.sh