Page Comparison

...

What matters is not that there is a surveillance interface but that there is a monitor components component that records a persistent state for each batch. - CSR

Add something like "The state of each batch is stored in DOMS and accessed through an API which queries a caching layer (e.g. a lucene index of batch objects in DOMS). So there will be some latency between updates to DOMS batch objects and the result of API queries."

...

Robots can occupy the same location on the assembly line. Here we have the first example of this. The "Autonomous JPylyzer" is a robot that, like the "Autonomous Doms Ingester" polls for "Bitrepository Ingest" events. The task of this robot is to run jpylyzer on the jpeg2000 files in the batch. The task will be done as a hadoop job. . This assumes that the ABI ensures that the jp2 files are in hdfs, or that the Autonomous Jpylyzer can bring them in if necessary. -CSR

• As the map step, run JPylyzer on each jpeg2000 file
• As the reduce step, sdd add the output of this process to the file object in DOMS

...

We have two robots, that might end up working concurrently. The first is the "Autonomous Batch Structure Checker". This robot might in fact be a set of robots (TO BE DECIDED), but for now we can think of it as a single step. It polls for "Metadata Ingest" events in the SBOI (Batch Status API -CSR). It will perform a series of checks of the metadata in DOMS. When done, it will add the "Batch Structure Checked" event to the Batch object.

...

• Query the bit repository for any files pertaining previous versions of the batch
• If any is found, request the delete key
• Delete the files
• Query DOMS for any previous run-number objects. If any is found, purge them and their subtree.
• Need to clarify what metadata we want to keep from failed batches, since it could be very useful for comparison with the new version. -CSR

...