...

The current archive functionality can be split into the following 3 areas:

Wayback Access

Current solution

The current solution is to fetch resources through jms messages to the bitarchives.

Pros: 

• Read access is handled the same way as write access, and the same model for logging and autherization can be used.
• Both replicas can be used for retrieving data (can they).

Disadvantage:

Alternate solution

Switch to let Wayback access files directly though mounted RO drives. 

Pros: 

• Wouldn't put stress on the message broker or bitarchive infrastructure.
• Wouldn't depend on the message broker or bitarchive infrastructure scalability.

Disadvantage:

Massprocessing

Current solution

This is currently implemented through the BatchJob functionality, which has been develop

AdvantagePros: 

1. Full control of source code

...

1. All improvements have to be done with NetarchiveSuite ressources.
2. Somewhat unstable.

...

Alternate solution

The de facto standard platform for massproccessing massprocessing is Hadoop, which is used at an increasingly number of webarchiving institutions for analysing the stored web data. Both SB and KB have established Hadoop clusteresclusters, which are already used for processing the Netarkivet.dk archive.

AdvantagePros: 

1. Hadoop is an mature standard processing platform for large datasets.
2. Comes with a huge set of tools, including Webarchiving tools.
3. Very robust and scalable.
4. Enables processing resources and data to be seperated separated (SB setup)

Disadvantage:

1. Migration cost

Bitpreservation

Current solution

Pros: 

Cons:

Alternate solution

Switch to using a Bitrepository system for the Bitpreservation

Pros: 

Disadvantage: