Contents

Database

Description of the individual tables and their columns can be found in the Derby SQL create script createfullhddb.sql.

Harvesting roundtrip

This section describes what goes on during a harvest, from when the templates are uploaded and harvests created till the WARC files are uploaded to the archive and historical information about the harvest is stored.

Initial steps

To create a harvest in the first place, we need to have a template (crawler bean definition) to base it on. Additionally, we need a schedule for a selective harvest, or some domains to harvest for a snapshot harvest.

Uploading templates

Templates (Heritrix .cxml files) are uploaded using the Definitions-upload-harvest-template.jsp page. See Appendix B2: Managing Heritrix 3 Crawler-Beans for more information on requirements for .cxml templates. The template is stored in the templates table in the database.

Creating domains

Domains can either be auto-created through selective harvest definitions, or mass uploaded, or they can be created manually through the interface. Domains are represented by a Domain object, which in turn contains a list of SeedList objects and a list of DomainConfiguration objects. These are stored in the domains, seedlists, and configurations tables respectively. New domains are created with a single configuration, and a single seedlist with only one seed

 http://www.<domain>

and a limit on the number of bytes and/or documents to download.

Creating schedules

NetarchiveSuite comes with four default schedules, repeating respectively hourly, daily, weekly and monthly. More schedules can be created using the web interface. Schedules are represented with the Schedule object and stored in the schedules table in the database.

Creating a selective harvest

Harvests are created using the web-based user interface, as described in the User Manual.

Job creation and dispatching

This functionality is handled by the classes in the harvest scheduler package. This is also a separate maven module harvest-scheduler.

The HarvestJobGenerator selects all harvests that are ready to harvest, i.e. harvests that are activated and where the next harvest date is in the past (or, for snapshot harvest, that haven't run yet). For each harvest, a new thread is started to perform the operations required to send off the jobs – this keeps snapshot harvests from blocking scheduling of selective harvests. Since the threads may run for a while, we maintain a list of harvests currently undergoing scheduling and use it to avoid scheduling the same harvest several times concurrently.

Splitting the harvest into jobs

Splitting a harvest into jobs and sending those jobs off to the harvesters happens, as mentioned, in a separate thread. The job of splitting the harvest into jobs is delegated to the Job generator class defined by setting settings.harvester.scheduler.jobGen.class. All implementations must implement the dk.netarkivet.harvester.scheduler.jobgen.JobGenerator interface, and there are currently two different implementations, the dk.netarkivet.harvester.scheduler.jobgen.DefaultJobGenerator and dk.netarkivet.harvester.scheduler.jobgen.FixedDomainConfigurationCountJobGenerator. Both implementations both extend the same class, AbstractJobGenerator.

The first part of splitting shared by both implementations is to reduce the harvests into chunks that can be handled by the JobGenerator class itself – since the data structures for domain configurations and their associated domains contain a fair amount of information, we cannot keep all of them in memory at the same time. For that reason alone, we split harvests into arbitrary chunks with no more domain configurations per chunk than the domainConfigSubsetSize setting allows. We use an iterator to avoid keeping all the domains and their configurations in memory for this operation, which iterates over all configurations in sorted order. The configurations are sorted by .cxml template, then by maximum number of bytes to harvest, and finally by the expected number of objects harvested. This makes sure that configurations that should be in the same job are sorted next to each other. A FilterIterator weeds out configurations in a snapshot harvest whose domains either are marked as aliases or were completely harvested in the snapshot harvest that the current snapshot harvest is based on.

For each chunk, we iteratively create new jobs by taking one domain configuration at a time and checking if it can be added to the job we're building. If it cannot, we store the job and start making a new one. Note that the jobs created are not submitted to the harvesters yet; that happens asynchronously as part of the dispatching check. For both implementations of the JobGenerator, there is an upperlimit to the number of configurations for each job domainConfigSubsetSize, so values of fixedDomainCountSnapshot or fixedDomainCountFocused higher than domainConfigSubsetSize are effectively set to domainConfigSubsetSize, as only domainConfigSubsetSize is processed in each iteration of the jobgeneration.

The check for whether a configuration can be added to a job is the most complex part of the scheduling system, and the two implementations have their own strategy:

The strategy for the FixedDomainConfigurationCountJobGenerator is only to have a limit for how many configurations you can have in each job, irrespectively of how much was harvested the last time by these configurations.
The strategy for the DefaultJobGenerator, however, attempts to partition the configurations into chunks such that all configurations in a job take approximately the same amount of time to harvest and don't exceed Heritrix memory limits. The estimation of the size of a configuration is complicated by the fact that a configuration for a previously unharvested domain has an unknown size, and that domains over time can easily increase in size by several orders of magnitude by adding forums, image galleries or crawler traps.

The basic check-list for the DefaultJobGenerator for whether or not a domain configuration can be added to a job is as follows. If any of the checks is invalid, the current job is finalized, and a new job initialized with the current configuration as its first configuration:

The configuration must not already have been added to the job.
The job must not end up with more than CONFIG_COUNT_SNAPSHOT (settings.harvester.scheduler.jobGen.config.fixedDomainCountSnapshot) for snapshot jobs. This check is not enforced for selective jobs.
The configuration must use the same crawl template as the other configurations in the job.
Any attributes (MAX_HOPS, HONOR_ROBOTS_DOT_TXT, EXTRACT_JAVASCRIPT) to be inserted in the crawl template must have the same values as for the other configurations in the job.
If the byte limit for this job is determined by the harvest definition, the configuration must not have a smaller byte limit than the definition specifies. If the byte limit for the job is determined by the other configurations in the job, this configuration must have the same byte limit as the other configurations.
The expected number of objects harvested by all configurations in the job, based on previous harvests of the configurations, must not exceed the maxTotalSize setting.
The relative difference between the largest and smallest expected numbers of objects harvested by configurations in the job must be no more than the maxRelativeSizeDifference setting. Note that the default setting for this is 100, so expectations within a job differ by a factor 100, not just 100%. This is to prevents jobs from finishing many small configurations quickly and then taking a long time to finish a few, large configurations.
However, if the absolute difference between the largest and smallest expected numbers of objects harvested by configurations in the job is less than the minAbsoluteSizeDifference setting, the relative difference is ignored. This allows the very smallest configurations to be lumped together in fewer jobs even if they are very different in relative size.

The expected number of objects used by the DefaultJobGenerator class is found based on previous harvests of a given configuration and a few assumptions about the development of web sites. If a configuration hasn't been harvested before, defaults from the settings file are used. Expectations for previously harvested domains are calculated as follows:

The "best" previous harvest to estimate from is found by picking the most recent complete harvest using the configuration, or the harvest that harvested the most objects if the configuration never completed.
The expected size per object is found based on the average size in the "best" previous harvest, if that harvest got enough objects to be considered (at least 50), but at least as many as the expectedAverageBytesPerObject setting.
A maximum number of objects is found based on the current limits of the configuration and the harvest and the expected size per object. If neither configuration nor harvest imposes any limits, an artificial limit for estimation purposes is taken from the maxDomainSize setting.
A minimum number of objects is the number of objects found in the the "best" previous harvest, or is 0 if no previous harvest was found.
If the configuration had previously been completed, the estimated number of objects is the difference of minimum and maximum divided by the errorFactorPrevResult setting plus the minimum.
Otherwise, the estimated number of objects is the difference of minimum and maximum divided by the errorFactorBestGuess setting plus the minimum.
The expected number of objects is capped by the maximum based on the limits.

The errorFactorBestGuess setting should generally be smaller than the the errorFactorPrevResult setting, since there is more uncertainty about the actual number of objects when the harvest has never been completed. These two settings are best understood as the largest possible factor of error between our estimate and reality. If we use an error-factor of 10, we accept that while configurations could end up growing by as much as the hard limits allow, we split as if they only grow by one-tenth that amount. In most cases, growth will be limited, but it is likely that if a new archive, forum or such is added to a site, the site can grow significantly between harvests. These settings determine the trade-off between the likelihood that some sites have grown a lot and the desire to keep similar-sized configurations in the same job.

The basic check-list for the FixedDomainConfigurationCountJobGenerator for whether or not a domain configuration can be added to a job is much simpler:

The configuration must not already have been added to the job.
The job must not end up with more than CONFIG_COUNT_SNAPSHOT(settings.harvester.scheduler.jobGen.config.fixedDomainCountSnapshot) configurations for snapshot jobs or CONFIG_COUNT_FOCUSED (settings.harvester.scheduler.jobGen.config.fixedDomainCountFocused) configurations for selective jobs.
The configuration must use the same crawl template as the other configurations in the job.
Any attributes (MAX_HOPS, HONOR_ROBOTS_DOT_TXT, EXTRACT_JAVASCRIPT) to be inserted in the crawl template must have the same values as for the other configurations in the job.
If the byte limit for this job is determined by the harvest definition, the configuration must not have a smaller byte limit than the definition specifies. If the byte limit for the job is determined by the other configurations in the job, this configuration must have the same byte limit as the other configurations.

Once the job does not get any more domain configurations added to it, it is added to the database with status 'New', and cannot change further except for status updates.

When all domain configurations for a harvest have been placed in jobs, the time for the next execution of the harvest is set. Note that the execution time is updated regardless of whether the jobs are actually successful, or even have been run. Additionally, the counter of number of runs is updated.

If there are any errors in the scheduling process, including the creation of jobs, the harvest is deactivated to prevent the system from being overloaded with broken scheduling attempts.

Talking to the harvesters

New Jobs are sent to any available harvesters as a DoOneCrawlMessage . This message contains not only the Job object, but also some metadata entries that are associated with the job. Currently, the metadata consists of a listing of the aliases that were used in the job creation and of a listing of the job IDs that should be used to get the deduplication index. The harvests report back via JMS when they are ready to process harvest jobs (see JobDispatcher).

The DoOneCrawlMessages are placed on a JMS queue for the relevant Harvest Channels. At the same time, the job is set to status 'Submitted', indicating that it's in queue for being picked up by a harvester. Notice that since the JMS messages are expected to be cleaned from the queues at system restart, we assume that any messages about jobs in state "Submitted" are lost after a restart, and they are therefore automatically resubmitted at system startup.

Each HarvestControllerServer application listens to just one of the two queues. When it receives a message (remember that JMS guarantees exactly-once delivery for queues), it immediately sends a message back that tells the HarvestJobManager that the job has been picked up and can be put in state 'Started'.

At this point, the HarvestControllerServer has accepted that it will attempt to run the job and can start to set up the files necessary for running Heritrix.

Harvest setup

The directory used in a crawl is created by the HarvestControllerServer, using the job id and timestamp in the directory name. Details on what Heritrix reads and writes can be found in the Heritrix "outside the GUI" page.

Running Heritrix3

The HeritrixLauncher class sets the correct file paths in the Heritrix .cxml file and keeps an eye on the progress of the harvest.

Heritrix is run by the harvester system as a standalone process. This allows access to Heritrix' web interface. The interfacing to the Heritrix process is controlled by the heritirix3 HeritrixController, an implementation of the HeritrixController interface. Heritrix 3 is controlled entirely by web (REST) calls, and its API is well documented and exampled.

The HeritrixController starts a new process as part of its constructor, putting the jar files in lib/heritrix/lib and the NetarchiveSuite jar files in the classpath. The process is started in the directory created by the HarvestControllerServer, and all files created as part of the crawl are put into that directory. Stdout and stderr from Heritrix, along with a dump of the startup environment, are put in the heritrix.out file. The full command line used for running Heritrix is put in the log file.

Before the process is started, a shutdown-hook is added to attempt proper cleanup in case the harvest controller is shut down prematurely. Notice that this hook is removed if the process finishes normally.

Access to the Heritrix user interface can be had by connecting to the port specified by the heritrixAdminGui setting, using the admin name and password specified by the heritrixAdminName and heritrixAdminPassword settings, respectively.

The cleanup of the HeritrixController involves issuing the shutdown command to Heritrix, then waiting for a while (duration defined by the processTimeout setting) for Heritrix to end crawls and write its reports. If Heritrix doesn't stop within the timeout period, we forcibly kill it. After that, we collect the exit code and wait for the stdout/stderr collector processes to finish.

Creating metadata

After the heritrix has finished with the harvesting, the harvest is documented, and the result of this documentation is stored in a separate warcfile prefixed with the job id, and ending with "-metadata-<metadata version number>.warc(.gz)". This metadata file contains all heritrix logs and reports associated with this harvest(crawl.log, local-errors.log, progress-statistics.log, runtime-errors.log, uri-errors.log, heritrix.out, crawl-report.txt, frontier-report.txt, hosts-report.txt, mimetype-report.txt, processors-report.txt, responsecode-report.txt, seeds-report.txt), some metadata about the job itself, and CDX'es of the contents of the warcfiles created by Heritrix. A CDX line points to where an object is located in an warcfile, its length and mimetype. This metadata warcfile is uploaded along with the rest of the warcfiles.

Uploading

When Heritrix is finished, and the metadata warcfile created, all warcfiles are uploaded to the archive using a ArcrepositoryClient.

Storing harvest statistics

When uploading is done, a status message is sent back to the scheduler, containing error reports and harvest statistics. Errors are split into harvest errors and upload errors, since upload is attempted even if the harvest fails. For each, a short error description and a longer, detailed description are sent. The statistics sent are the following for each domain harvested:

Number of objects harvested
Number of bytes harvested
Reason the harvest stopped, one of completed (no more objects found), object-limit (hit maximum allowed number of objects), size-limit (hit maximum allowed number of bytes, as specified by the harvest), config-size-limit (hi maximum allowed number of bytes, as specified by the configuration), and unfinished (the harvest was interrupted before any of the other stop reasons applied).

If sourceTagSeeds is enabled on the SeedModule bean in the harvest-template and the setting settings.harvester.harvesting.harvestReport.disregardSeedURLInfo is false (the default value), the harvested data will count as belonging to the domain of the Seed-URI. Otherwise, it will count as belonging to the domain of the url harvested.

When the status message is received from the harvester, the statistics are stored per domain in the database, along with the job number, the harvest number, the domain name, the configuration name, and a timestamp for receipt of the information.

After the harvest statistics have been sent to the database, the HarvestController application checks if there is free space on the machine for a new harvestjob. If this is the case, it starts to listen on the job queue. If not, it goes into a dormant mode.

Old jobs

When a harvester application starts up, it checks whether any jobs are left from previous runs, in case the harvest or the upload was aborted. If there is, the last three steps described above are taken for the old jobs before the harvest application starts listening for new jobs.

Deduplication in NetarchiveSuite

deduplication is performed by using the DeDuplicator module developed by Kristinn Sigurdsson as the first of the Heritrix write-processors:

<bean id="DeDuplicator" class="is.hi.bok.deduplicator.DeDuplicator">
       <property name="index-location" value="%{DEDUPLICATION_INDEX_LOCATION}">
       <property name="matching-method" value="URL"/>
        <property name="try-equivalent" value="TRUE"/>
        <property name="change-content-size" value="false"/>
        <property name="mime-filter" value="^text/.*"/>
        <property name="filter-mode" value="BLACKLIST"/>
        <property name="analysis-mode">value="Timestamp"</>
        <property name="log-level" value="SEVERE"/>
        <property name="origin" value=""/>
        <property name="origin-handling" value="INDEX"/>
        <property name="stats-per-host" value="true"/>
        <property name="revisitInWarcs" value="true"/>       
        <property name="enabled" value="true"/>
  </bean>

Note the presence of the placeholder. NetarchiveSuite replaces this with the correct path for the deduplication index. The deduplication uses a Lucene (v. 4.4.0) index with information about previously harvested objects. This index may be empty. In NetarchiveSuite, the index contains entries for objects fetched in an earlier harvest job, which this harvestjob is likely to revisit, i.e. this new harvestjob revisits some of same domains that the previously harvest jobs did.

Each entry in the index contains the URL (unNormalized), content-digest, mimetype , Etag (if available), and the origin of the entry <arcfile>,<offset>, <timestamp>.

Only non-text fetched URLs are indexed, as only URIs with non-text mimetypes are looked up by the DeDuplicator.

During the actual crawl-time, all non-text URIs are looked up in the index, and if a match (A URI is matched, if the URI is found, and it has the same digest as the current URI) is found in the index, the URI is ignored by the WARCWriterProcessor, but an entry is written to the crawl-log that contains the reference to the original stored URI. This reference is written to the Annotations part of the crawl-line (12th part of the crawl-line): deduplicate:arcfile,offset,original-timestamp . NetarchiveSuite can also write a WARC-revisit record for this deduplication event (for NetarchiveSuite 5.2+). The writing of the WARC-revisit records is only done by the Heritrix WARCWriterProcessor module if the DeDuplicator property revisitInWarcs is set to true.

Generation of deduplication indices is made by merging information in the crawl-log with information in the CDX-files generated for each job at the conclusion of the harvest.

The CDX’es contain information about the contents of the warc-files generated by Heritrix, but they lack information on the deduplicated objects. CDX-files containing references also to deduplicated objects can be generated from the crawl-logs by the tools provided in the wayback module. These tools are described in the Additional Tools manual.

The merging of this information in NetarchiveSuite was necessitated by the way we do Quality Assurance of the harvestJobs, which is done on a job by job basis, so we needed a way to refer to the deduplicated objects.

Browser not supported