Integrity Service tests

Workflows

Verify existing workflows

1. Verify that the following workflows are listed in the Integrity service configuration dropdown and the Workflow table:
   
   1. CompleteIntegrityCheck
   2. IncrementalIntegrityCheck
   3. RepairMissingFilesWorkflow
   4. SaltedChecksumWorkflow

Manually start workflow

1. Start the CompleteIntegrityCheck by clicking the Start button.
2. Verify that the current state begins to show the different steps of the workflow.
3. Verify that the workflow can finish and go to the state 'idle'
4. Click on the date for the 'last run' and validate that all of the previously found steps are shown with a duration time.

Test missing file

1. Delete a file from one of the pillars (e.g. move it away from the fileDir in the ReferencePillar)
   - preferably a file with fileID starting with an alphabetically early letter as the later  RepairMissingFilesWorkflow will handle files in alphabetical order.
2. Run the CompleteIntegrityCheck by clicking the Start button - and wait until it is finished.
3. Verify that no corrupt files have been found anywhere.
4. Verify that the given pillar is missing a file (or one more file than before the running of the workflow).
5. Look at the list of existing files at the pillar, and verify that the deleted file is not on the list.
6. Look at the list of existing files at another pillar, and verify that the deleted file is on the list for that pillar.
7. Look at the list of missing files at the pillar, and verify that the deleted file is on the list.
8. Verify that the file is not missing at any other pillar
9. Verify that an alarm has been sent to the AlarmService about the missing file. All pillars are said in the alarm message to have "<pillarid> is missing X files".
   1. If files were already missing on the pillar, you may be able to check the newest alarm against the last one and confirm that the count for missing files has been incremented by 1.

Restoring the missing file

1. Use the RepairMissingFilesWorkflow to restore the missing file - and wait until it is finished.
2. Run the CompleteIntegrityCheck by clicking the Start button - and wait until it is finished.
3. Verify that the file is not longer missing at the pillar.

Test corrupt file at single pillar

1. Change in the content of a file at one of the pillars (e.g. run replace-file, or add a character to it in the fileDir of the ReferencePillar) - The file must be present at all pillars initially and not have any checksum errors.
   1. If the file is manually edited the checksum error will not be picked up until the pillar itself recalculates checksums - prefer using replace-file if you're not editing the pillar's recalculation interval.
2. Run the CompleteIntegrityCheck by clicking the Start button - and wait until it is finished.
3. Verify that no pillar is missing the corrupted file.
4. Verify that only the pillar, where it has been corrupted, has the given file on their list of 'files with checksum error'.
5. Verify that an alarm has been sent to the AlarmService about the corrupt file.

Test corrupt file at multiple pillars

1. Choose a collection with more than 3 pillars.
2. Upload a file to all pillars
3. Run replace-file command with another file for two of the pillars (thus at least two has the original and at least two has the new file)
4. Run the CompleteIntegrityCheck by clicking the Start button - and wait until it is finished.
5. Verify that no pillar is missing the corrupted file.
6. Verify that all pillars have the given file on their list of 'Number of inconsistent checksums'.
7. Verify that an alarm has been sent to the AlarmService about the corrupt file.

Restoring the corrupt file

1. Restore the file by running a deleteFile on the pillars with the corrupted files followed by a putFile with the original file
2. Run the CompleteIntegrityCheck by clicking the Start button - and wait until it is finished.
3. Verify that the file is no longer on the list of 'files with checksum error'.

Testing the Integrity Information for collection

1. Check the current integrity information about a given collection (i.e. 'Latest ingest', 'Size', and 'Number of files').
2. Upload a new file in the current measurement size of the collection information (i.e. if the size is currently measured in MB, then upload a new file of at least 1 MB).
3. Run the CompleteIntegrityCheck by clicking the Start button - and wait until it is finished.
4. Verify that the number of files has increased by 1, and the total file size has increased accordingly with the size of the new file.

Verify that workflow dates are not identical for different collections

1. Check whether the workflows have different 'last run' dates for the different collections
2. Run the CompleteIntegrityCheck for one of the collections by clicking the Start button - and wait until it is finished.
3. Verify that a new 'last run' date is updated to the time when the workflow was started.
4. Verify that the other workflows, which has not been run, are not updated by this.

Test the salted checksums integrity check

1. Run the SaltedChecksumWorkflow workflow and wait for its completion
2. Verify that the workflow was a success and that it has not resulted in sending any alarms.
   - NB: the workflow picks a random file in the collection to make a salted checksum for, so if the picked file only exists on a checksum pillar it will fail and throw an alarm - if so, just ignore it and try running the workflow again