Page Comparison

Highlights in 7.1

Current Snapshot: 

https://sbforge.org/nexus/service/local/repositories/snapshots/content/org/netarchivesuite/distribution/7.1-SNAPSHOT/distribution-7.1-20210525.063538-46.zip 

https://sbforge.org/nexus/service/local/repositories/snapshots/content/org/netarchivesuite/heritrix3-bundler/7.1-SNAPSHOT/heritrix3-bundler-7.1-20210525.063651-46.zip

1. Added 3 new link extractors (from the British Library) to heritrix :

   • org.archive.modules.extractor.ExtractorRobotsTxt

   • org.archive.modules.extractor.ExtractorSitemap
   • org.archive.modules.extractor.ExtractorJson
2. Added caching of crawl logs when hadoop is used for processing
3. Added caching of metadata-file indexes when hadoop is used for processing
4. Added retry functionality to improve the robustness of the WarcRecordClient
5. Fixed a bug whereby files uploaded from a harvester were not being deleted when the Bitmagasin backend is in use

The new caching functionality stores data in a directory specified by the setting settings.common.webinterface.metadata_cache_dir whose default value is "metadata_cache" (relative to the current working directory where the GUIApplication is started). At present there is no automatic cleaning of this directory.

Highlights in 7.0

NetarchiveSuite 7.0 introduces an entirely new backend storage and mass-processing implementation based on software from bitrepository.org and hadoop. The new functionality is enabled by defining the following key in the settings file for all applications: 

<settings>
   <common>
      <arcrepositoryClient>
         <class>dk.netarkivet.archive.arcrepository.distribute.BitmagArcRepositoryClient</class>   

and additionally 

<settings>
   <common>
      <useBitmagHadoopBackend>true</useBitmagHadoopBackend>

The older arcrepositoryClient implementation dk.netarkivet.archive.arcrepository.distribute.JMSArcRepositoryClient will be deprecated in future releases. (The developers are unaware of any other organisations currently using the older client, but please contact us if you still rely on it.)

The new architecture introduces many new keys and external configuration files. There is therefore a separate Guide To Configuring the NetarchiveSuite 7.0 Backend.

Upgrading From Previous NetarchiveSuite Releases

For those using either JMSArcRepositoryClient or LocalArcRepositoryClient there should be no special requirements to upgrade.

Issues Resolved in Release 7.0