/
2026-03-03 Statusmeeting

2026-03-03 Statusmeeting

Agenda for the joint NetarchiveSuite teleconference 2026-03-03, 13:00-14:00.

Participants

  • BNF:  Sara, Leslie, Auriane

  • ONB: Andreas, Antares

  • KB/DK - Copenhagen: Thomas, Stephen, Tue, Anders,

  • KB/DK - Aarhus: Colin, Lars, Bolette, Halina

  • BNE: José, Miguel, Eva, Alberto

  • KB/Sweden: Peter, Pär

Update on NAS latest tests and developments

  • 8.1 release on February 9th: NetarchiveSuite 8.x Release Notes

  • Proposal from Peter:

    • https://github.com/netarchivesuite/netarchivesuite/issues/149

    • The proposed changes are in three categories:

      1. Ignore non-200 status from requests to Heritrix (PR, PR)
        This was just partly done before. Current NAS can sometimes stop a harvest when there are Heritrix problems (due to lack of memory or cpu resources), leaving the job with status DONE. This change fixes that issue

      2. Structured handling when there are communication problems (PR)
        If there are problems, don’t just go on but try to give Hertrix and the system a chance to recover. Loop with increasing sleep each turn (“back off”) and give up after N retries or Y minutes. And when it gives up, throw IOFailure, which is caught higher úp, making the harvest get the status FAILED instead of DONE.

      3. Adding timeout on the Heritrix communication (PR)
        This is important given (2) above. Currently there are no timeouts.

       What are you view? Could (1) be kept in next release? It can be seen as a bug fix and would solve some of our problems.

      We will see if we can get help to compile in (2) and (3) locally.

    •  

  • Question: do we want to plan a NAS discussion in Brussels?

Status of the production sites

Netarkivet

BnF

 

ONB

 

BNE

The IT unit is currently working on upgrading NAS to version 8.0, which we expect to complete next week.

We are also testing Browsertrix and assessing how it could be integrated into our workflows. In particular, we would like to focus on content that cannot be harvested with NAS, such as social media and other dynamic content.

KB-Sweden

 

147 When job request to Heritrix fails: Before: No status report; job ended as completed With change: Report status of why it failed; job ended as failed 148 When FullFrontierReport request to Heritrix fails: Before: No status report; (Nullpointerexception?); job continues With change: Report status of why it failed; job continues 150 Handle Heritrix communication problems in the crawl loop Three statuses after HTTP request to Hertrix: OK: continue as before – make a frontierreport etc CRAWL_OVER: break the loop FAILED: problem, perhaps temporary, with the communication In the latter case, don’t make a frontierreport, loop with increasing sleep each turn and give up after N retries or Y minutes. And when it gives up it throws IOFailure, which is caught higher úp, making the harvest get the status FAILED instead of DONE. 11 Explicit HTTP timeouts added to avoid indefinite blocking when Heritrix is overloaded. Required for retry/backoff and failure-budget logic to trigger deterministically and prevent silent stalls. HttpClient may retry NoHttpResponseException up to 3 times internally (setRetryHandler). This means a single poll can take up to ~3× socket timeout in worst case before failing. Our failure-budget (in other PR) therefore operates on “poll failures” rather than raw TCP failures, which reduces flapping and still guarantees bounded time to abort.

 

Next meetings

  • April 7th

  • May 5th

  • June 2nd

  • July 7th

  • September 1st

  • October 6th

  • November 3rd

  • December 1st

  • January 5th 2027

Any other business?

  •