Crawling the Rust Book
Nine crawlers in 22 setups, each pointed at the same 114-page site (176 files) served from a web server on the same machine. Each run was screen-recorded and is replayed here on a unified clock, so you can watch them in real time.
Crawl Results
Beam Us Up
00:05.91140114 / 01029 MB114 pages · 0 assets · 1029 MB peak
Beam Us Up (JS)
00:42.91140114 / 01198 MB114 pages · 0 assets · 1198 MB peak
Beam Us Up CLI
00:04.91140114 / 01087 MB114 pages · 0 assets · 1087 MB peak
Beam Us Up CLI (JS)
00:42.61140114 / 0862 MB114 pages · 0 assets · 862 MB peak
GNU Wget
00:00.111462114 / 623.4 MB114 pages · 62 assets · 3.4 MB peak
InterroBot
00:01.811450114 / 50433 MB114 pages · 50 assets · 433 MB peak
InterroBot (JS)
00:39.411450114 / 50934 MB114 pages · 50 assets · 934 MB peak
InterroBot (Turbo JS)
00:21.611450114 / 501027 MB114 pages · 50 assets · 1027 MB peak
InterroBot (Turbo)
00:01.011450114 / 50433 MB114 pages · 50 assets · 433 MB peak
InterroBot CLI
00:00.811450114 / 50146 MB114 pages · 50 assets · 146 MB peak
InterroBot CLI (JS)
00:39.311450114 / 50582 MB114 pages · 50 assets · 582 MB peak
InterroBot CLI (Turbo JS)
00:21.311450114 / 50771 MB114 pages · 50 assets · 771 MB peak
InterroBot CLI (Turbo)
00:00.911450114 / 5043.4 MB114 pages · 50 assets · 43.4 MB peak
Katana
00:12.111350113 / 5067.2 MB113 pages · 50 assets · 67.2 MB peak
Screaming Frog
00:05.111548115 / 481299 MB115 pages · 48 assets · 1299 MB peak
Screaming Frog CLI
00:05.411548115 / 48555 MB115 pages · 48 assets · 555 MB peak
SEO Macroscope
00:12.211551115 / 51241 MB115 pages · 51 assets · 241 MB peak
SEO Macroscope (High)
00:12.311551115 / 51252 MB115 pages · 51 assets · 252 MB peak
SiteOne Crawler
00:15.111461114 / 61371 MB114 pages · 61 assets · 371 MB peak
SiteOne Crawler (High)
00:06.411461114 / 61375 MB114 pages · 61 assets · 375 MB peak
Visual SEO Studio
00:05.911428114 / 28142 MB114 pages · 28 assets · 142 MB peak
Xenu's Link Sleuth
00:02.011561115 / 615.5 MB115 pages · 61 assets · 5.5 MB peak
Resource Usage
Watch in Realtime
Every crawl starts together at 0:07. Click a screen to enlarge it.
0:00.0 / 1:01.9
Setups
| Crawler | Version | Interface | Threads | Settings | Machine |
|---|---|---|---|---|---|
| Beam Us Up | 2.0.0 | GUI | 16 threads (auto) | Default settings | Mac mini (M1) |
| Beam Us Up (JS) | 2.0.0 | GUI | 16 threads (auto) | JavaScript rendering on | Mac mini (M1) |
| Beam Us Up CLI | 2.0.0 | CLI | 10 threads | Default settings | Mac mini (M1) |
| Beam Us Up CLI (JS) | 2.0.0 | CLI | 10 threads | JavaScript rendering on | Mac mini (M1) |
| GNU Wget | 1.25.0 | CLI | Single thread | Mirror preset | Mac mini (M1) |
| InterroBot | 2.15.0 | GUI | Default threads | Default settings | Mac mini (M1) |
| InterroBot (JS) | 2.15.0 | GUI | Default threads | JavaScript rendering on | Mac mini (M1) |
| InterroBot (Turbo JS) | 2.15.0 | GUI | Turbo threads | Turbo threads; JavaScript rendering on | Mac mini (M1) |
| InterroBot (Turbo) | 2.15.0 | GUI | Turbo threads | Turbo threads | Mac mini (M1) |
| InterroBot CLI | 2.15.0 | CLI | Default threads | Default settings | Mac mini (M1) |
| InterroBot CLI (JS) | 2.15.0 | CLI | Default threads | JavaScript rendering on | Mac mini (M1) |
| InterroBot CLI (Turbo JS) | 2.15.0 | CLI | Turbo threads | Turbo threads; JavaScript rendering on | Mac mini (M1) |
| InterroBot CLI (Turbo) | 2.15.0 | CLI | Turbo threads | Turbo threads | Mac mini (M1) |
| Katana | 1.7.0 | CLI | 10 threads | Full depth; no subdomains; images kept | Mac mini (M1) |
| Screaming Frog | 24.3.0 | GUI | Default threads | Default settings | Mac mini (M1) |
| Screaming Frog CLI | 24.3.0 | CLI | Default threads | Default settings | Mac mini (M1) |
| SEO Macroscope | 1.7.6.1 | GUI | Default threads | Default settings | Windows 10 (VM) |
| SEO Macroscope (High) | 1.7.6.1 | GUI | 10 threads | 10 threads | Windows 10 (VM) |
| SiteOne Crawler | 1.0.8 | GUI | Default threads | Report upload off | Mac mini (M1) |
| SiteOne Crawler (High) | 1.0.8 | GUI | 8 threads | 8 threads; 50 requests/s; report upload off | Mac mini (M1) |
| Visual SEO Studio | 2.5.3.1 | GUI | Default threads | Max crawl depth 9 | Windows 10 (VM) |
| Xenu's Link Sleuth | 1.3.8 | GUI | 30 threads | Default settings | Windows 10 (VM) |
How This Was Measured
- Test Machines
Mac mini (M1/Tahoe)8 GB of RAM · macOS 26.6.2
Windows 10 (Hyper-V)15.8 GB of RAM · Windows 10 · virtual machine
Each crawler ran by itself. Crawl time, counts, memory, and CPU are captured consistently, per OS. See more on how it’s measured.
- Sites
Rust Bookrust-book-local.crawlerbench.com114 HTML pages176 files
GIMP 3.0 Manualgimp-local.crawlerbench.com711 HTML pages2,616 files
Served from a local web server on the test machine.