...
In "Harvest status > All Running Jobs" wait until one of the jobs has nonzero progressstarted. Click on the Job ID to enter H3 Remote Access.
...
- Progression/Queues
- Crawllog: check cache update, filtering, paging
- Reports: click on at least two of them. Check that the Processors report shows that deduplication has been disabled.
- Show/delete frontier: delete some items from the frontier
- Add RejectRules: add a new rule
- Modify budget: add an object limit to some domain or subdomain
Now restart (unpause) the job. Then immediatiely ...
...
- Stop NAS with the "stop_test.sh" command
- On kb-test-adm-001/TEST2TESTX/conf change the settings for HarvestJobManager so deduplicaiton.enabled is "true"
- restart the NAS system with the "start_test.sh" command. After some time, the job you paused and unpaused should appear in the state "Failed".
- Restart the job by clicking the "Restart?" button. A new job should be created and the old one should have the status "Resubmitted (Job X)".
- Wait for the job to finish.
- Meanwhile click on the JobID for the failed job, then click on "Browse reports for jobs".
- You should see a list of available reports including one called "scripting_events.log". This is the log of alterations you made to the frontier via H3 Remote Access. Click on it.
Assuming you have the correct viewerproxy setup (See Setup DK test environment) you should have a log line describing your action. Something similar to
Code Block 2016-02-02T14:05:59.170Z Action from user CSR: Deleted 563 uris matching regex '.*kb.dk.*'
Check the Processors report. You should see a report on the non-zero number of objects which has been deduplicated.
...
- Check that netarkivet.dk and sulnudu.dk are not listed as being harvested.
- Check that neither domain appears in the order templates harvest template's crawler beans for any of the jobs with the (possible) exception of the following lines:
...
| Code Block |
|---|
[devel@kb-prod-udv-001 ~]$ ssh netarkiv@sbnetarkdv@sb-test-bar-001.statsbiblioteket.dk grep netarkivet.dk /netarkiv/0001/TEST2/filedir/*-metadata-1.warc | grep -v 'metadata:' |
...