2026-05-05 Statusmeeting

• Update to http1.1 → chunking issues in playback

• Q2 broadcrawl on track

• We are looking for collaboration and exchange of regexes - both general malicious patterns created by extractors, widely used frameworks, and large international websites.

• Have suggested improved handling of soft404 to Heritrix: https://github.com/internetarchive/heritrix3/issues/729

• IIPC WAC 2026

  • A great conference -still need to digest all the input and knowledge

• Maybe SOlrWayback workshop at iPRES

• Browsertrix

  • Testing deduplication and scaling even more

  • Testing queue-system

In mid-March, we launched a new internal harvesting workshop about Browsertrix. We finally decided to extend the tests stage in order to focus on angles we were unable to fully explore:

• the harvest settings and behaviors

• the scalability: to test the harvest of several hundreds of accounts simultaneously

• the harvest scheduling: daily, weekly, etc.

• the harvest organization

• to quantify the time required for the harvest monitoring part
  The tests will go on until June 4th.

In early April, we were conducting tests to launch our next Videos harvest. We tried different settings to reduce blocking issues and we finally launched the crawl the week of April 27th. Unfortunately, the harvest had to be paused due to these blockages.

Last week, we launched a special blog crawl with a new system: 4 queues are opened at the same time, one per blogging platform, and the blogs in each queue are archived one by one. This concerns the eklablog, overblog, canalblog, and centerblog platforms. We hope we will shorten the crawl period this way.

Finally, we started the first steps of the preparation of our broad crawl in order to obtain the different domains lists to archive.

We have contacted Cloudflare to avoid CAPTCHAs and IP blocks. They have told us there are two ways to resolve this. The first is for the bot to verify and identify itself to the provider using its IP address and bot name. The second way is to use digital signatures. Using Web Bot Auth, essentially they publish a public key and then the bot signs all requests with the private key, so they can ensure it comes from BNE.

Today we have updated to NAS version 8.0.