Crawler BenchCrawling the GIMP 3.0 Manual
Ten crawlers in 24 setups, each pointed at the same 711-page site (2,616 files) served from a remote mirror. Each run was screen-recorded and is replayed here on a unified clock, so you can watch them in real time.
Crawl Results
00:08.27120712 / 02250 MB712 pages · 0 assets · 2250 MB peak
03:32.27120712 / 01986 MB712 pages · 0 assets · 1986 MB peak
00:08.57120712 / 02087 MB712 pages · 0 assets · 2087 MB peak
03:40.77120712 / 01279 MB712 pages · 0 assets · 1279 MB peak
00:30.27111,913711 / 1,9135.1 MB711 pages · 1,913 assets · 5.1 MB peak
13:35.87121,913712 / 1,913272 MB712 pages · 1,913 assets · 272 MB peak
13:39.37121,913712 / 1,91330.0 MB712 pages · 1,913 assets · 30.0 MB peak
09:57.07121,913712 / 1,91331.8 MB712 pages · 1,913 assets · 31.8 MB peak
00:08.77121,867712 / 1,867392 MB712 pages · 1,867 assets · 392 MB peak
04:04.87121,867712 / 1,867740 MB712 pages · 1,867 assets · 740 MB peak
03:23.46981,853698 / 1,853858 MB698 pages · 1,853 assets · 858 MB peak
00:05.87121,867712 / 1,867362 MB712 pages · 1,867 assets · 362 MB peak
00:08.17121,867712 / 1,867139 MB712 pages · 1,867 assets · 139 MB peak
04:04.27121,867712 / 1,867497 MB712 pages · 1,867 assets · 497 MB peak
03:12.37011,853701 / 1,853613 MB701 pages · 1,853 assets · 613 MB peak
00:05.77121,867712 / 1,867144 MB712 pages · 1,867 assets · 144 MB peak
00:28.17111,864711 / 1,86455.2 MB711 pages · 1,864 assets · 55.2 MB peak
Hit the 500-URL free limit at 00:07.0573 MB573 MB peak
05:36.17121,869712 / 1,869450 MB712 pages · 1,869 assets · 450 MB peak
05:20.57121,869712 / 1,8691936 MB712 pages · 1,869 assets · 1936 MB peak
02:55.47171,905717 / 1,905564 MB717 pages · 1,905 assets · 564 MB peak
03:07.26801,875680 / 1,875426 MB680 pages · 1,875 assets · 426 MB peak
Hit the 501-URL free limit at 00:07.6109 MB109 MB peak
00:23.17121,912712 / 1,9128.0 MB712 pages · 1,912 assets · 8.0 MB peak
Resource Usage
Watch in Realtime
Every crawl starts together at 0:07. Click a screen to enlarge it.
0:00.0 / 13:50.6
Setups
| Beam Us Up | 2.0.0 | GUI | 16 threads (auto) | Default settings | Mac mini (M1) |
| Beam Us Up (JS) | 2.0.0 | GUI | 16 threads (auto) | JavaScript rendering on | Mac mini (M1) |
| Beam Us Up CLI | 2.0.0 | CLI | 10 threads | Default settings | Mac mini (M1) |
| Beam Us Up CLI (JS) | 2.0.0 | CLI | 10 threads | JavaScript rendering on | Mac mini (M1) |
| GNU Wget | 1.25.0 | CLI | Single thread | Mirror preset | Mac mini (M1) |
| HTTrack | 3.50-4 | GUI | 4 threads | Default settings | Mac mini (M1) |
| HTTrack CLI | 3.50-4 | CLI | 4 threads | Default settings | Mac mini (M1) |
| HTTrack CLI (Max) | 3.50-4 | CLI | 8 threads | 8 threads; no rate cap; 20 connections/s | Mac mini (M1) |
| InterroBot | 2.14.1 | GUI | Default threads | Default settings | Mac mini (M1) |
| InterroBot (JS) | 2.14.1 | GUI | Default threads | JavaScript rendering on | Mac mini (M1) |
| InterroBot (Turbo JS) | 2.14.1 | GUI | Turbo threads | Turbo threads; JavaScript rendering on | Mac mini (M1) |
| InterroBot (Turbo) | 2.14.1 | GUI | Turbo threads | Turbo threads | Mac mini (M1) |
| InterroBot CLI | 2.14.1 | CLI | Default threads | Default settings | Mac mini (M1) |
| InterroBot CLI (JS) | 2.14.1 | CLI | Default threads | JavaScript rendering on | Mac mini (M1) |
| InterroBot CLI (Turbo JS) | 2.14.1 | CLI | Turbo threads | Turbo threads; JavaScript rendering on | Mac mini (M1) |
| InterroBot CLI (Turbo) | 2.14.1 | CLI | Turbo threads | Turbo threads | Mac mini (M1) |
| Katana | 1.7.0 | CLI | 10 threads | Full depth; no subdomains; images kept | Mac mini (M1) |
| Screaming Frog CLI | 24.3.0 | CLI | Default threads | Default settings | Mac mini (M1) |
| SEO Macroscope | 1.7.6.1 | GUI | Default threads | Default settings | Windows 10 (VM) |
| SEO Macroscope (High) | 1.7.6.1 | GUI | 10 threads | 10 threads | Windows 10 (VM) |
| SiteOne Crawler | 1.0.8 | GUI | Default threads | Report upload off | Mac mini (M1) |
| SiteOne Crawler (High) | 1.0.8 | GUI | 8 threads | 8 threads; 50 requests/s; report upload off | Mac mini (M1) |
| Visual SEO Studio | 2.5.3.1 | GUI | Default threads | Max crawl depth 9 | Windows 10 (VM) |
| Xenu's Link Sleuth | 1.3.8 | GUI | 30 threads | Default settings | Windows 10 (VM) |
How This Was Measured
- Test Machines
Mac mini (M1/Tahoe)8 GB of RAM · macOS 26.6.2
Windows 10 (Hyper-V)15.8 GB of RAM · Windows 10 · virtual machine
Each crawler ran by itself. Crawl time, counts, memory, and CPU are captured consistently, per OS. See more on how it’s measured.
- Sites
GIMP 3.0 Manualgimp-remote.crawlerbench.com711 HTML pages2,616 files
Rust Bookrust-book-remote.crawlerbench.com114 HTML pages176 files
Served from a remote server, reached over the internet.
- Free Limits
- Screaming Frog CLI and Visual SEO Studio stopped at the free edition’s 500-URL limit, so they could not finish this site.