2020-10-06 Statusmeeting

Broad crawl
Step 2 is proceeding in a great fashion.

Event crawl
We decided to continue with the event crawl on Corona in Denmark but with lower frequency and. 0-hop sites reduced greatly, and with minimal curational activity.

Alexandre, trainee
Arrived and is up and running, working remotely from Copenhagen with the rest of the team. We are almost done with the intro-program and are looking into what will give most value to Alexandre, Netarkivet and also BnF.

IT-University in Copenhagen:
The collaboration with the IT-University in Copenhagen is moving forward.

Youtube
We have experimented with getting embedded video-content and so far the results are great (except WARC-validation is not valid with re-visits)

WARC-file-validation
We are working on finalizing a workflow from Webrecorder/Conifer.org to Netarkivet. To be able to validate WARC-files correctly is a big part of getting the right level of preservation (we use JWAT for this). But it´s a bit complicated – see for instance this OPF blog by Remco van Veenendaal from Holland: https://openpreservation.org/blogs/warc-validation-tool-experiences/

(How) are you validating WARC-files? And what is the future on this?

Our annual broad crawl will be lanched this tuesday, 6th of October. This will be our first broad crawl with the new NAS version including the official Heritrix 3 IIPC and we expect a better efficiency of the crawling. We have reduced to 2000 the maximal number of URLs per domain (instead of 2500 last year) and we expect to harvest between 110 and 115 TB.

Next week, we'll put in production a new version of our public GUI "Archives de l'internet" making available for the readers the video channels harvested in July. These channels cover the topics of covid19 outbreak and French local elections. The new version includes also a mechanism allowing to display Instagram account pages in our web archive and to browse the posts using Picuki directly from the Instagram page. Finally, this new version gives access to 3 titles of paid online newspapers, harvested with authentication.

Even Crawl

We are still working in Coronavirus collection. Now we are focused on audio and video content.

Serials Broad Crawl

At the beginning of the year we launched a broad crawl of open-access electronic serials and now we are focused in the quality assurance of this collection

BCWeb

We are having some problems with the old version of BCWeb, so we hope this will speed up the installation of the new version that we have available, 6.1

IT Department

The new head of the IT department has just arrived.