2026-07-07 Statusmeeting

• Begun step 1 of 2026 third broadcrawl. Now NAS 8.2

• Experimenting with tool to recommend Heritrix or Browsertrix as crawler

• Experimenting with tool to extract samples from Heritrix crawllogs to be processed with high fidelity (Browsertrix)

• Interested in hearing from anyone that have used/tried Browserprocessor with NAS?

The preparations for the 2026 broad crawl have begun. We are simultaneously working to find a solution concerning the issue of blockages and we have planned several lines of action to establish a state of play concerning these blockages. We also plan to install the new version of NAS and Heritrix and we want to work on several developments.

Our biannual crawl has been launched last week. It includes a little less than 4,500 selections and it will last until July 24th.

We also have launched last week a new OpenStreetMap harvest concerning the France and the world. We plan to crawl 317,912 tiles for a fortnight.

Finally, after the Pharmacovigilance harvest launched last month, we are preparing a new research corpus crawl entitled "History, Technologies, and Practices of the Web in the 2000s". It is a retrospective crawl and the goal is to document the web publishing practices in the 2000s. Selections are based on several criteria, including older websites from the 2000s, content publication methods, the history of the code and the technologies used, and web theories. Several interviews have been conducted with researchers to define the scope of the harvest.

We have completed the broad crawl of the .eus domain, with 23,000 domains saved and approximately 6 TB archived.

We are currently working on the .cat domain harvest, which we expect to complete this week.

As every year, we are archiving the online impact of Madrid Pride while also finalizing several event collections on the recently concluded regional elections.