Crawler BenchCrawling the Rust Book
Ten crawlers in 25 setups, each pointed at the same 114-page site (176 files) served from a remote mirror. Each run was screen-recorded and is replayed here on a unified clock, so you can watch them in real time.
Crawl Results
00:05.51140114 / 01161 MB114 pages · 0 assets · 1161 MB peak
00:43.71140114 / 01334 MB114 pages · 0 assets · 1334 MB peak
00:04.31140114 / 0935 MB114 pages · 0 assets · 935 MB peak
00:42.91140114 / 0900 MB114 pages · 0 assets · 900 MB peak
00:02.011462114 / 623.5 MB114 pages · 62 assets · 3.5 MB peak
00:48.911562115 / 6283.4 MB115 pages · 62 assets · 83.4 MB peak
00:52.111562115 / 6229.9 MB115 pages · 62 assets · 29.9 MB peak
00:37.611562115 / 6231.8 MB115 pages · 62 assets · 31.8 MB peak
00:03.111449114 / 49491 MB114 pages · 49 assets · 491 MB peak
00:44.611449114 / 491088 MB114 pages · 49 assets · 1088 MB peak
00:33.511350113 / 50993 MB113 pages · 50 assets · 993 MB peak
00:01.611449114 / 49484 MB114 pages · 49 assets · 484 MB peak
00:02.511449114 / 49214 MB114 pages · 49 assets · 214 MB peak
00:43.111449114 / 49742 MB114 pages · 49 assets · 742 MB peak
00:30.911449114 / 49761 MB114 pages · 49 assets · 761 MB peak
00:01.611449114 / 49209 MB114 pages · 49 assets · 209 MB peak
00:12.311350113 / 5058.5 MB113 pages · 50 assets · 58.5 MB peak
00:04.211548115 / 481089 MB115 pages · 48 assets · 1089 MB peak
00:04.911548115 / 48603 MB115 pages · 48 assets · 603 MB peak
00:12.211551115 / 51270 MB115 pages · 51 assets · 270 MB peak
00:12.211551115 / 51254 MB115 pages · 51 assets · 254 MB peak
00:14.811461114 / 61357 MB114 pages · 61 assets · 357 MB peak
00:11.911461114 / 61356 MB114 pages · 61 assets · 356 MB peak
00:05.411428114 / 28157 MB114 pages · 28 assets · 157 MB peak
00:04.311561115 / 615.7 MB115 pages · 61 assets · 5.7 MB peak
Resource Usage
Watch in Realtime
Every crawl starts together at 0:07. Click a screen to enlarge it.
0:00.0 / 1:07.1
Setups
| Beam Us Up | 2.0.0 | GUI | 16 threads (auto) | Default settings | Mac mini (M1) |
| Beam Us Up (JS) | 2.0.0 | GUI | 16 threads (auto) | JavaScript rendering on | Mac mini (M1) |
| Beam Us Up CLI | 2.0.0 | CLI | 10 threads | Default settings | Mac mini (M1) |
| Beam Us Up CLI (JS) | 2.0.0 | CLI | 10 threads | JavaScript rendering on | Mac mini (M1) |
| GNU Wget | 1.25.0 | CLI | Single thread | Mirror preset | Mac mini (M1) |
| HTTrack | 3.50-4 | GUI | 4 threads | Default settings | Mac mini (M1) |
| HTTrack CLI | 3.50-4 | CLI | 4 threads | Default settings | Mac mini (M1) |
| HTTrack CLI (Max) | 3.50-4 | CLI | 8 threads | 8 threads; no rate cap; 20 connections/s | Mac mini (M1) |
| InterroBot | 2.14.1 | GUI | Default threads | Default settings | Mac mini (M1) |
| InterroBot (JS) | 2.14.1 | GUI | Default threads | JavaScript rendering on | Mac mini (M1) |
| InterroBot (Turbo JS) | 2.14.1 | GUI | Turbo threads | Turbo threads; JavaScript rendering on | Mac mini (M1) |
| InterroBot (Turbo) | 2.14.1 | GUI | Turbo threads | Turbo threads | Mac mini (M1) |
| InterroBot CLI | 2.14.1 | CLI | Default threads | Default settings | Mac mini (M1) |
| InterroBot CLI (JS) | 2.14.1 | CLI | Default threads | JavaScript rendering on | Mac mini (M1) |
| InterroBot CLI (Turbo JS) | 2.14.1 | CLI | Turbo threads | Turbo threads; JavaScript rendering on | Mac mini (M1) |
| InterroBot CLI (Turbo) | 2.14.1 | CLI | Turbo threads | Turbo threads | Mac mini (M1) |
| Katana | 1.7.0 | CLI | 10 threads | Full depth; no subdomains; images kept | Mac mini (M1) |
| Screaming Frog | 24.3.0 | GUI | Default threads | Default settings | Mac mini (M1) |
| Screaming Frog CLI | 24.3.0 | CLI | Default threads | Default settings | Mac mini (M1) |
| SEO Macroscope | 1.7.6.1 | GUI | Default threads | Default settings | Windows 10 (VM) |
| SEO Macroscope (High) | 1.7.6.1 | GUI | 10 threads | 10 threads | Windows 10 (VM) |
| SiteOne Crawler | 1.0.8 | GUI | Default threads | Report upload off | Mac mini (M1) |
| SiteOne Crawler (High) | 1.0.8 | GUI | 8 threads | 8 threads; 50 requests/s; report upload off | Mac mini (M1) |
| Visual SEO Studio | 2.5.3.1 | GUI | Default threads | Max crawl depth 9 | Windows 10 (VM) |
| Xenu's Link Sleuth | 1.3.8 | GUI | 30 threads | Default settings | Windows 10 (VM) |
How This Was Measured
- Test Machines
Mac mini (M1/Tahoe)8 GB of RAM · macOS 26.6.2
Windows 10 (Hyper-V)15.8 GB of RAM · Windows 10 · virtual machine
Each crawler ran by itself. Crawl time, counts, memory, and CPU are captured consistently, per OS. See more on how it’s measured.
- Sites
Rust Bookrust-book-remote.crawlerbench.com114 HTML pages176 files
GIMP 3.0 Manualgimp-remote.crawlerbench.com711 HTML pages2,616 files
Served from a remote server, reached over the internet.