Crawling the GIMP 3.0 Manual
Nine crawlers in 22 setups, each pointed at the same 711-page site (2,616 files) served from a web server on the same machine. Each run was screen-recorded and is replayed here on a unified clock, so you can watch them in real time.
Crawl Results
Beam Us Up
00:08.47120712 / 02172 MB712 pages · 0 assets · 2172 MB peak
Beam Us Up (JS)
03:28.07120712 / 02237 MB712 pages · 0 assets · 2237 MB peak
Beam Us Up CLI
00:08.47120712 / 02234 MB712 pages · 0 assets · 2234 MB peak
Beam Us Up CLI (JS)
03:35.07120712 / 01444 MB712 pages · 0 assets · 1444 MB peak
GNU Wget
00:01.37111,912711 / 1,9125.2 MB711 pages · 1,912 assets · 5.2 MB peak
InterroBot
00:04.77121,867712 / 1,867541 MB712 pages · 1,867 assets · 541 MB peak
InterroBot (JS)
03:35.77121,867712 / 1,867886 MB712 pages · 1,867 assets · 886 MB peak
InterroBot (Turbo JS)
01:47.47121,867712 / 1,867909 MB712 pages · 1,867 assets · 909 MB peak
InterroBot (Turbo)
00:03.97121,867712 / 1,867544 MB712 pages · 1,867 assets · 544 MB peak
InterroBot CLI
00:02.97121,867712 / 1,867133 MB712 pages · 1,867 assets · 133 MB peak
InterroBot CLI (JS)
03:39.27121,867712 / 1,867506 MB712 pages · 1,867 assets · 506 MB peak
InterroBot CLI (Turbo JS)
01:57.77121,867712 / 1,867664 MB712 pages · 1,867 assets · 664 MB peak
InterroBot CLI (Turbo)
00:03.07121,867712 / 1,867130 MB712 pages · 1,867 assets · 130 MB peak
Katana
00:28.17111,863711 / 1,86371.9 MB711 pages · 1,863 assets · 71.9 MB peak
Screaming Frog
Hit the 500-URL free limit at 00:08.01037 MB1037 MB peak
Screaming Frog CLI
Hit the 500-URL free limit at 00:06.7560 MB560 MB peak
SEO Macroscope
05:36.37121,869712 / 1,869463 MB712 pages · 1,869 assets · 463 MB peak
SEO Macroscope (High)
05:25.67121,869712 / 1,8692150 MB712 pages · 1,869 assets · 2150 MB peak
SiteOne Crawler
02:56.07171,905717 / 1,905570 MB717 pages · 1,905 assets · 570 MB peak
SiteOne Crawler (High)
00:56.07171,905717 / 1,905563 MB717 pages · 1,905 assets · 563 MB peak
Visual SEO Studio
Hit the 501-URL free limit at 00:07.296.0 MB96.0 MB peak
Xenu's Link Sleuth
00:08.67121,912712 / 1,9126.9 MB712 pages · 1,912 assets · 6.9 MB peak
Resource Usage
Watch in Realtime
Every crawl starts together at 0:07. Click a screen to enlarge it.
0:00.0 / 5:49.1
Setups
| Crawler | Version | Interface | Threads | Settings | Machine |
|---|---|---|---|---|---|
| Beam Us Up | 2.0.0 | GUI | 16 threads (auto) | Default settings | Mac mini (M1) |
| Beam Us Up (JS) | 2.0.0 | GUI | 16 threads (auto) | JavaScript rendering on | Mac mini (M1) |
| Beam Us Up CLI | 2.0.0 | CLI | 10 threads | Default settings | Mac mini (M1) |
| Beam Us Up CLI (JS) | 2.0.0 | CLI | 10 threads | JavaScript rendering on | Mac mini (M1) |
| GNU Wget | 1.25.0 | CLI | Single thread | Mirror preset | Mac mini (M1) |
| InterroBot | 2.15.0 | GUI | Default threads | Default settings | Mac mini (M1) |
| InterroBot (JS) | 2.15.0 | GUI | Default threads | JavaScript rendering on | Mac mini (M1) |
| InterroBot (Turbo JS) | 2.15.0 | GUI | Turbo threads | Turbo threads; JavaScript rendering on | Mac mini (M1) |
| InterroBot (Turbo) | 2.15.0 | GUI | Turbo threads | Turbo threads | Mac mini (M1) |
| InterroBot CLI | 2.15.0 | CLI | Default threads | Default settings | Mac mini (M1) |
| InterroBot CLI (JS) | 2.15.0 | CLI | Default threads | JavaScript rendering on | Mac mini (M1) |
| InterroBot CLI (Turbo JS) | 2.15.0 | CLI | Turbo threads | Turbo threads; JavaScript rendering on | Mac mini (M1) |
| InterroBot CLI (Turbo) | 2.15.0 | CLI | Turbo threads | Turbo threads | Mac mini (M1) |
| Katana | 1.7.0 | CLI | 10 threads | Full depth; no subdomains; images kept | Mac mini (M1) |
| Screaming Frog | 24.3.0 | GUI | Default threads | Default settings | Mac mini (M1) |
| Screaming Frog CLI | 24.3.0 | CLI | Default threads | Default settings | Mac mini (M1) |
| SEO Macroscope | 1.7.6.1 | GUI | Default threads | Default settings | Windows 10 (VM) |
| SEO Macroscope (High) | 1.7.6.1 | GUI | 10 threads | 10 threads | Windows 10 (VM) |
| SiteOne Crawler | 1.0.8 | GUI | Default threads | Report upload off | Mac mini (M1) |
| SiteOne Crawler (High) | 1.0.8 | GUI | 8 threads | 8 threads; 50 requests/s; report upload off | Mac mini (M1) |
| Visual SEO Studio | 2.5.3.1 | GUI | Default threads | Max crawl depth 9 | Windows 10 (VM) |
| Xenu's Link Sleuth | 1.3.8 | GUI | 30 threads | Default settings | Windows 10 (VM) |
How This Was Measured
- Test Machines
Mac mini (M1/Tahoe)8 GB of RAM · macOS 26.6.2
Windows 10 (Hyper-V)15.8 GB of RAM · Windows 10 · virtual machine
Each crawler ran by itself. Crawl time, counts, memory, and CPU are captured consistently, per OS. See more on how it’s measured.
- Sites
GIMP 3.0 Manualgimp-local.crawlerbench.com711 HTML pages2,616 files
Rust Bookrust-book-local.crawlerbench.com114 HTML pages176 files
Served from a local web server on the test machine.
- Free Limits
- Screaming Frog, Screaming Frog CLI, and Visual SEO Studio stopped at the free edition’s 500-URL limit, so they could not finish this site.