Crawler Bench

Filter Crawls

Site
Rust BookGIMP 3.0 Manual
Server
Remote ServerLocal Server
Filters
Interface
Rendering
Type
Price
Crawlers
Crawlers

Crawling the GIMP 3.0 Manual

Ten crawlers in 25 setups, each pointed at the same 711-page site (2,616 files) served from a web server on the same machine. Each run was screen-recorded and is replayed here on a unified clock, so you can watch them in real time.

Crawl Results

00:08.57120712 / 02546 MB712 pages · 0 assets · 2546 MB peak
03:28.47120712 / 01758 MB712 pages · 0 assets · 1758 MB peak
00:08.37120712 / 02199 MB712 pages · 0 assets · 2199 MB peak
03:35.87120712 / 01288 MB712 pages · 0 assets · 1288 MB peak
00:01.47111,912711 / 1,9125.0 MB711 pages · 1,912 assets · 5.0 MB peak
13:35.47121,912712 / 1,912283 MB712 pages · 1,912 assets · 283 MB peak
13:45.17121,912712 / 1,91229.9 MB712 pages · 1,912 assets · 29.9 MB peak
09:47.67121,912712 / 1,91230.2 MB712 pages · 1,912 assets · 30.2 MB peak
00:03.87121,867712 / 1,867409 MB712 pages · 1,867 assets · 409 MB peak
03:39.17121,867712 / 1,867750 MB712 pages · 1,867 assets · 750 MB peak
01:52.27121,867712 / 1,867891 MB712 pages · 1,867 assets · 891 MB peak
00:03.27121,867712 / 1,867364 MB712 pages · 1,867 assets · 364 MB peak
00:04.57121,867712 / 1,867135 MB712 pages · 1,867 assets · 135 MB peak
03:39.57121,867712 / 1,867509 MB712 pages · 1,867 assets · 509 MB peak
01:55.67121,867712 / 1,867638 MB712 pages · 1,867 assets · 638 MB peak
00:03.77121,867712 / 1,867137 MB712 pages · 1,867 assets · 137 MB peak
00:29.17111,863711 / 1,86365.4 MB711 pages · 1,863 assets · 65.4 MB peak
Hit the 500-URL free limit at 00:07.8945 MB945 MB peak
Hit the 500-URL free limit at 00:06.6562 MB562 MB peak
05:36.37121,869712 / 1,869463 MB712 pages · 1,869 assets · 463 MB peak
05:25.67121,869712 / 1,8692150 MB712 pages · 1,869 assets · 2150 MB peak
02:56.67171,905717 / 1,905563 MB717 pages · 1,905 assets · 563 MB peak
00:53.87171,905717 / 1,905559 MB717 pages · 1,905 assets · 559 MB peak
Hit the 501-URL free limit at 00:07.296.0 MB96.0 MB peak
00:08.67121,912712 / 1,9126.9 MB712 pages · 1,912 assets · 6.9 MB peak

Resource Usage

Watch in Realtime

Every crawl starts together at 0:07. Click a screen to enlarge it.

0:00.0 / 13:56.5

Setups

Beam Us Up2.0.0GUI16 threads (auto)Default settingsMac mini (M1)
Beam Us Up (JS)2.0.0GUI16 threads (auto)JavaScript rendering onMac mini (M1)
Beam Us Up CLI2.0.0CLI10 threadsDefault settingsMac mini (M1)
Beam Us Up CLI (JS)2.0.0CLI10 threadsJavaScript rendering onMac mini (M1)
GNU Wget1.25.0CLISingle threadMirror presetMac mini (M1)
HTTrack3.50-4GUI4 threadsDefault settingsMac mini (M1)
HTTrack CLI3.50-4CLI4 threadsDefault settingsMac mini (M1)
HTTrack CLI (Max)3.50-4CLI8 threads8 threads; no rate cap; 20 connections/sMac mini (M1)
InterroBot2.14.1GUIDefault threadsDefault settingsMac mini (M1)
InterroBot (JS)2.14.1GUIDefault threadsJavaScript rendering onMac mini (M1)
InterroBot (Turbo JS)2.14.1GUITurbo threadsTurbo threads; JavaScript rendering onMac mini (M1)
InterroBot (Turbo)2.14.1GUITurbo threadsTurbo threadsMac mini (M1)
InterroBot CLI2.14.1CLIDefault threadsDefault settingsMac mini (M1)
InterroBot CLI (JS)2.14.1CLIDefault threadsJavaScript rendering onMac mini (M1)
InterroBot CLI (Turbo JS)2.14.1CLITurbo threadsTurbo threads; JavaScript rendering onMac mini (M1)
InterroBot CLI (Turbo)2.14.1CLITurbo threadsTurbo threadsMac mini (M1)
Katana1.7.0CLI10 threadsFull depth; no subdomains; images keptMac mini (M1)
Screaming Frog24.3.0GUIDefault threadsDefault settingsMac mini (M1)
Screaming Frog CLI24.3.0CLIDefault threadsDefault settingsMac mini (M1)
SEO Macroscope1.7.6.1GUIDefault threadsDefault settingsWindows 10 (VM)
SEO Macroscope (High)1.7.6.1GUI10 threads10 threadsWindows 10 (VM)
SiteOne Crawler1.0.8GUIDefault threadsReport upload offMac mini (M1)
SiteOne Crawler (High)1.0.8GUI8 threads8 threads; 50 requests/s; report upload offMac mini (M1)
Visual SEO Studio2.5.3.1GUIDefault threadsMax crawl depth 9Windows 10 (VM)
Xenu's Link Sleuth1.3.8GUI30 threadsDefault settingsWindows 10 (VM)

How This Was Measured

Test Machines
  • Mac mini (M1/Tahoe)8 GB of RAM · macOS 26.6.2
  • Windows 10 (Hyper-V)15.8 GB of RAM · Windows 10 · virtual machine

Each crawler ran by itself. Crawl time, counts, memory, and CPU are captured consistently, per OS. See more on how it’s measured.

Sites
  • The GIMP 3.0 Manual homepageGIMP 3.0 Manualgimp-local.crawlerbench.com711 HTML pages2,616 files
  • The Rust Book homepageRust Bookrust-book-local.crawlerbench.com114 HTML pages176 files

Served from a local web server on the test machine.

Free Limits
Screaming Frog, Screaming Frog CLI, and Visual SEO Studio stopped at the free edition’s 500-URL limit, so they could not finish this site.