Please add status summary hereLego had a game zone on the web called Lego Universe, which has been closed recently (2012-01-30). In order to document Lego Universe we did som extra crawls of lego.com and videos from YouTube.com, which were embeddet in Lego Universe.
Quite some webpages are displayed differently by different browsers and browser versions. We studied som examples closely and documented the result in our wiki.
Two of the most important sites with display problems are Facebook and Twitter. The crawllog tells us, that most of the given url’s are harvested, but the viewerproxy does not at all show them.
As you know, heritrix cannot harvest streamed video or sound. As to sound we succeeded in harvesting mp3-files from rss feeds from a page with radio streams. We used a template with xml extractor.
Upgrading to NAS 3.18 caused us some trouble - NAS was down for about a week. |