2026-04-07 Statusmeeting

• Broad crawl.

  • About to start 3rd crawl - step 1

• Crawlers blocked from big official Dansih sites

  • Allow lisiting not possible due to security

  • good bot vs. bad bot

  • Update

    • user agents?

    • Specific site with bot-info?

• IIPC WAC 2026

  • Making the last preparations

• Browsertrix

  • Event harvest of general election 2026

    • Lot of experience with crawl of Tiktok, Facebook, Instagram, advanced visualizations etc.

    • Heritrix vs. Browsertrix vs. Archiveweb.page

  • Deduplication tested!

We began a new internal harvesting workshop about Browsertrix on March 19th. We tested the cloud version in 2023 and we are currently testing a local installation of the tool. At the same time, we are meeting with different colleagues from the British Library, the Royal Danish Library and the National Library of Luxembourg who use Browsertrix in order to obtain feedback.
At this stage, we observe that the crawl of social media is better with Browsertrix, but other questions arise regarding our workflow. We will continue the tests until April 15th.

Our annual selective harvest is going to be launch next week. There are 11,127 selections divided into 22 jobs. It will last for 6 weeks.

Finally, we will launch a new Videos harvest concerning the Elections and the Olympic Games. 67 channels have been selected for these two subjects. We will see if we can complete the harvest without any issues.

Over the past month, we have been exploring the transition from OpenWayback to PyWB. Our current OpenWayback setup has several limitations in terms of content playback, so we are assessing how to migrate and what adjustments would be required.

We are also continuing to analyse the tests carried out with Browsertrix, particularly those involving content that is difficult to capture using NAS.