Crawler Bench

Filter Crawls

Filters
Interface
Rendering
Type
Price
Crawlers
Crawlers

Crawling the GIMP 3.0 Manual

Nine crawlers in 22 setups, each pointed at the same 711-page site (2,616 files) served from a web server on the same machine. Each run was screen-recorded and is replayed here on a unified clock, so you can watch them in real time.

Crawl Results

Beam Us UpFreeGUI · 16 threads (auto)
00:08.47120712 / 02172 MB712 pages · 0 assets · 2172 MB peak
Beam Us Up (JS)FreeGUI · 16 threads (auto)
03:28.07120712 / 02237 MB712 pages · 0 assets · 2237 MB peak
Beam Us Up CLIFreeCLI · 10 threads
00:08.47120712 / 02234 MB712 pages · 0 assets · 2234 MB peak
Beam Us Up CLI (JS)FreeCLI · 10 threads
03:35.07120712 / 01444 MB712 pages · 0 assets · 1444 MB peak
GNU WgetFreeCLI · Single thread
00:01.37111,912711 / 1,9125.2 MB711 pages · 1,912 assets · 5.2 MB peak
InterroBotFreeGUI · Default threads
00:04.77121,867712 / 1,867541 MB712 pages · 1,867 assets · 541 MB peak
InterroBot (JS)GUI · Default threads
03:35.77121,867712 / 1,867886 MB712 pages · 1,867 assets · 886 MB peak
InterroBot (Turbo JS)GUI · Turbo threads
01:47.47121,867712 / 1,867909 MB712 pages · 1,867 assets · 909 MB peak
InterroBot (Turbo)GUI · Turbo threads
00:03.97121,867712 / 1,867544 MB712 pages · 1,867 assets · 544 MB peak
InterroBot CLIFreeCLI · Default threads
00:02.97121,867712 / 1,867133 MB712 pages · 1,867 assets · 133 MB peak
InterroBot CLI (JS)CLI · Default threads
03:39.27121,867712 / 1,867506 MB712 pages · 1,867 assets · 506 MB peak
InterroBot CLI (Turbo JS)CLI · Turbo threads
01:57.77121,867712 / 1,867664 MB712 pages · 1,867 assets · 664 MB peak
InterroBot CLI (Turbo)CLI · Turbo threads
00:03.07121,867712 / 1,867130 MB712 pages · 1,867 assets · 130 MB peak
KatanaFreeCLI · 10 threads
00:28.17111,863711 / 1,86371.9 MB711 pages · 1,863 assets · 71.9 MB peak
Screaming FrogFreeGUI · Default threads
Hit the 500-URL free limit at 00:08.01037 MB1037 MB peak
Screaming Frog CLIFreeCLI · Default threads
Hit the 500-URL free limit at 00:06.7560 MB560 MB peak
SEO MacroscopeFreeGUI · Default threads
05:36.37121,869712 / 1,869463 MB712 pages · 1,869 assets · 463 MB peak
SEO Macroscope (High)FreeGUI · 10 threads
05:25.67121,869712 / 1,8692150 MB712 pages · 1,869 assets · 2150 MB peak
SiteOne CrawlerFreeGUI · Default threads
02:56.07171,905717 / 1,905570 MB717 pages · 1,905 assets · 570 MB peak
SiteOne Crawler (High)FreeGUI · 8 threads
00:56.07171,905717 / 1,905563 MB717 pages · 1,905 assets · 563 MB peak
Visual SEO StudioFreeGUI · Default threads
Hit the 501-URL free limit at 00:07.296.0 MB96.0 MB peak
Xenu's Link SleuthFreeGUI · 30 threads
00:08.67121,912712 / 1,9126.9 MB712 pages · 1,912 assets · 6.9 MB peak

Resource Usage

Watch in Realtime

Every crawl starts together at 0:07. Click a screen to enlarge it.

0:00.0 / 5:49.1
Beam Us UpReady
Beam Us Up (JS)Ready
Beam Us Up CLIReady
Beam Us Up CLI (JS)Ready
GNU WgetReady
InterroBotReady
InterroBot (JS)Ready
InterroBot (Turbo JS)Ready
InterroBot (Turbo)Ready
InterroBot CLIReady
InterroBot CLI (JS)Ready
InterroBot CLI (Turbo JS)Ready
InterroBot CLI (Turbo)Ready
KatanaReady
Screaming FrogReady
Screaming Frog CLIReady
SEO MacroscopeReady
SEO Macroscope (High)Ready
SiteOne CrawlerReady
SiteOne Crawler (High)Ready
Visual SEO StudioReady
Xenu's Link SleuthReady

Setups

CrawlerVersionInterfaceThreadsSettingsMachine
Beam Us Up2.0.0GUI16 threads (auto)Default settingsMac mini (M1)
Beam Us Up (JS)2.0.0GUI16 threads (auto)JavaScript rendering onMac mini (M1)
Beam Us Up CLI2.0.0CLI10 threadsDefault settingsMac mini (M1)
Beam Us Up CLI (JS)2.0.0CLI10 threadsJavaScript rendering onMac mini (M1)
GNU Wget1.25.0CLISingle threadMirror presetMac mini (M1)
InterroBot2.15.0GUIDefault threadsDefault settingsMac mini (M1)
InterroBot (JS)2.15.0GUIDefault threadsJavaScript rendering onMac mini (M1)
InterroBot (Turbo JS)2.15.0GUITurbo threadsTurbo threads; JavaScript rendering onMac mini (M1)
InterroBot (Turbo)2.15.0GUITurbo threadsTurbo threadsMac mini (M1)
InterroBot CLI2.15.0CLIDefault threadsDefault settingsMac mini (M1)
InterroBot CLI (JS)2.15.0CLIDefault threadsJavaScript rendering onMac mini (M1)
InterroBot CLI (Turbo JS)2.15.0CLITurbo threadsTurbo threads; JavaScript rendering onMac mini (M1)
InterroBot CLI (Turbo)2.15.0CLITurbo threadsTurbo threadsMac mini (M1)
Katana1.7.0CLI10 threadsFull depth; no subdomains; images keptMac mini (M1)
Screaming Frog24.3.0GUIDefault threadsDefault settingsMac mini (M1)
Screaming Frog CLI24.3.0CLIDefault threadsDefault settingsMac mini (M1)
SEO Macroscope1.7.6.1GUIDefault threadsDefault settingsWindows 10 (VM)
SEO Macroscope (High)1.7.6.1GUI10 threads10 threadsWindows 10 (VM)
SiteOne Crawler1.0.8GUIDefault threadsReport upload offMac mini (M1)
SiteOne Crawler (High)1.0.8GUI8 threads8 threads; 50 requests/s; report upload offMac mini (M1)
Visual SEO Studio2.5.3.1GUIDefault threadsMax crawl depth 9Windows 10 (VM)
Xenu's Link Sleuth1.3.8GUI30 threadsDefault settingsWindows 10 (VM)

How This Was Measured

Test Machines
  • Mac mini (M1/Tahoe)8 GB of RAM · macOS 26.6.2
  • Windows 10 (Hyper-V)15.8 GB of RAM · Windows 10 · virtual machine

Each crawler ran by itself. Crawl time, counts, memory, and CPU are captured consistently, per OS. See more on how it’s measured.

Sites
  • The GIMP 3.0 Manual homepageGIMP 3.0 Manualgimp-local.crawlerbench.com711 HTML pages2,616 files
  • The Rust Book homepageRust Bookrust-book-local.crawlerbench.com114 HTML pages176 files

Served from a local web server on the test machine.

Free Limits
Screaming Frog, Screaming Frog CLI, and Visual SEO Studio stopped at the free edition’s 500-URL limit, so they could not finish this site.