...
For some years ago we tried with a virtual farm of 35 broadcrawl harvesters with one H3 instance on each with NFS diskstorage. It was not cheap to scale or to run a stabil broadcrawl because of the NFS diskstorage and it was too expensive to have storage locally compared to physical servers. So we are now using physical servers for broad crawl.
| Quantity | Phy/Vir | Disk TB | RAM GB | Swap GB | CPUs | Year/gen | H3 inst | Country/place | Tot H3 inst | ||
|---|---|---|---|---|---|---|---|---|---|---|---|
| broad | sel | broad | sel | ||||||||
| 5 | phy | 4,3 | 32 | 16 | 28 | 2013-2018 | 8 | DK/CPH | 40 | ||
| 1 | phy | 4,4 | 32 | 16 | 70 | 2020 | 10 | DK/CPH | 10 | ||
| 1 | phy | 2,7 | 64 | 64 | 32 | 2022 | 17 | DK/CPH | 17 | ||
| 2 | phy | 4,4 | 32 | 32 | 24 | old | 6 | DK/CPH | 12 | ||
| 1 | vir | 1,5 | 4 | 2 | DK/CPH | ||||||
| 1 | vir | 0,4 | 4 | 2 | DK/CPH | ||||||
| 5 | vir | 0,4 | 4 | 2 | DK/CPH | ||||||
| 5 | vir | 3 | 20 | 1 | 8 | 8 | DK/AAR | 40 | |||
(Temp DK pasted text:)
001-005: 4,3 TB 32 G RAM 16 G Swap 28xCPU about 5-10 years old HW generation 8xH3 instances per server
...
For some years ago we tried to use this virtuel platform also for broadcrawl harvest. It was not possible.
Preliminary Sweden
| Quantity | Phy/Vir | Disk |
|---|
GB | RAM GB | Swap GB | CPUs | Year/gen | H3 inst | Country/place | Tot H3 inst | ||
|---|---|---|---|---|---|---|---|---|---|
| broad | sel | broad | sel | ||||||
| ? | vir | ||||||||
| 300 | 10 |
| 8-18 | 4 | 5 |
| 1 | SE/STH |