Page Comparison

• • 3rd Broadcrawl 2024- step 2 - started
  • Browsertrix
    • Local Production-instance will be ready this week - the existing test-platform will be used as a staging platform 

    • Trying to see if we can support Webrecorder with subscription to ge textra crawling instance + support

    • 500K+ front pages harvested. 3.5 TB 8 days?
  • We participate in a broad national collection/metholodigal effort on climate change/debate and more. It´s called "More water in the system".  https://www.rigsarkivet.dk/nyheder/dokumentation-af-reaktioner-paa-klimaforandringer/
  • Crawl parallel with curl and Brave and comparing links to find holes in the NAS/aheritrix crawls.

Ongoing pass 2 of broad crawl, works well.

Have asked questions on how to handle network outage during crawl. Got some answers. Have some follow up questions.

Found problems with servers not accepting HTTP 1.0, see Slack Heritrix channel.

Have some problem with Pywb 2.8.3 (also in Slack, Pywb).

NAS meeting i Stockholm in March or in Norway 6-7 April, before IIPC meeting? (Olga suggestion.)