Crawler Bench

Filter Crawls

Filters
Interface
Rendering
Type
Price
Crawlers
Crawlers

Crawling the GIMP 3.0 Manual

Nine crawlers in 22 setups, each pointed at the same 711-page site (2,616 files) served from a remote mirror. Each run was screen-recorded and is replayed here on a unified clock, so you can watch them in real time.

Crawl Results

Beam Us UpFreeGUI · 16 threads (auto)
00:08.67120712 / 02013 MB712 pages · 0 assets · 2013 MB peak
Beam Us Up (JS)FreeGUI · 16 threads (auto)
03:32.27120712 / 01762 MB712 pages · 0 assets · 1762 MB peak
Beam Us Up CLIFreeCLI · 10 threads
00:08.77120712 / 02144 MB712 pages · 0 assets · 2144 MB peak
Beam Us Up CLI (JS)FreeCLI · 10 threads
03:39.17120712 / 01267 MB712 pages · 0 assets · 1267 MB peak
GNU WgetFreeCLI · Single thread
00:31.07111,913711 / 1,9135.1 MB711 pages · 1,913 assets · 5.1 MB peak
InterroBotFreeGUI · Default threads
00:08.97121,868712 / 1,868479 MB712 pages · 1,868 assets · 479 MB peak
InterroBot (JS)GUI · Default threads
03:44.27121,868712 / 1,868827 MB712 pages · 1,868 assets · 827 MB peak
InterroBot (Turbo JS)GUI · Turbo threads
01:53.27121,868712 / 1,868941 MB712 pages · 1,868 assets · 941 MB peak
InterroBot (Turbo)GUI · Turbo threads
00:05.77121,868712 / 1,868484 MB712 pages · 1,868 assets · 484 MB peak
InterroBot CLIFreeCLI · Default threads
00:07.27121,868712 / 1,868140 MB712 pages · 1,868 assets · 140 MB peak
InterroBot CLI (JS)CLI · Default threads
03:47.37121,868712 / 1,868504 MB712 pages · 1,868 assets · 504 MB peak
InterroBot CLI (Turbo JS)CLI · Turbo threads
02:01.97121,868712 / 1,868673 MB712 pages · 1,868 assets · 673 MB peak
InterroBot CLI (Turbo)CLI · Turbo threads
00:05.47121,868712 / 1,868139 MB712 pages · 1,868 assets · 139 MB peak
KatanaFreeCLI · 10 threads
00:29.17111,863711 / 1,86355.8 MB711 pages · 1,863 assets · 55.8 MB peak
Screaming FrogFreeGUI · Default threads
Hit the 500-URL free limit at 00:07.61114 MB1114 MB peak
Screaming Frog CLIFreeCLI · Default threads
Hit the 500-URL free limit at 00:06.8557 MB557 MB peak
SEO MacroscopeFreeGUI · Default threads
05:36.17121,869712 / 1,869450 MB712 pages · 1,869 assets · 450 MB peak
SEO Macroscope (High)FreeGUI · 10 threads
05:20.57121,869712 / 1,8691936 MB712 pages · 1,869 assets · 1936 MB peak
SiteOne CrawlerFreeGUI · Default threads
02:56.47171,905717 / 1,905567 MB717 pages · 1,905 assets · 567 MB peak
SiteOne Crawler (High)FreeGUI · 8 threads
00:55.67171,905717 / 1,905523 MB717 pages · 1,905 assets · 523 MB peak
Visual SEO StudioFreeGUI · Default threads
Hit the 501-URL free limit at 00:07.6109 MB109 MB peak
Xenu's Link SleuthFreeGUI · 30 threads
00:23.17121,912712 / 1,9128.0 MB712 pages · 1,912 assets · 8.0 MB peak

Resource Usage

Watch in Realtime

Every crawl starts together at 0:07. Click a screen to enlarge it.

0:00.0 / 5:48.9
Beam Us UpReady
Beam Us Up (JS)Ready
Beam Us Up CLIReady
Beam Us Up CLI (JS)Ready
GNU WgetReady
InterroBotReady
InterroBot (JS)Ready
InterroBot (Turbo JS)Ready
InterroBot (Turbo)Ready
InterroBot CLIReady
InterroBot CLI (JS)Ready
InterroBot CLI (Turbo JS)Ready
InterroBot CLI (Turbo)Ready
KatanaReady
Screaming FrogReady
Screaming Frog CLIReady
SEO MacroscopeReady
SEO Macroscope (High)Ready
SiteOne CrawlerReady
SiteOne Crawler (High)Ready
Visual SEO StudioReady
Xenu's Link SleuthReady

Setups

CrawlerVersionInterfaceThreadsSettingsMachine
Beam Us Up2.0.0GUI16 threads (auto)Default settingsMac mini (M1)
Beam Us Up (JS)2.0.0GUI16 threads (auto)JavaScript rendering onMac mini (M1)
Beam Us Up CLI2.0.0CLI10 threadsDefault settingsMac mini (M1)
Beam Us Up CLI (JS)2.0.0CLI10 threadsJavaScript rendering onMac mini (M1)
GNU Wget1.25.0CLISingle threadMirror presetMac mini (M1)
InterroBot2.15.0GUIDefault threadsDefault settingsMac mini (M1)
InterroBot (JS)2.15.0GUIDefault threadsJavaScript rendering onMac mini (M1)
InterroBot (Turbo JS)2.15.0GUITurbo threadsTurbo threads; JavaScript rendering onMac mini (M1)
InterroBot (Turbo)2.15.0GUITurbo threadsTurbo threadsMac mini (M1)
InterroBot CLI2.15.0CLIDefault threadsDefault settingsMac mini (M1)
InterroBot CLI (JS)2.15.0CLIDefault threadsJavaScript rendering onMac mini (M1)
InterroBot CLI (Turbo JS)2.15.0CLITurbo threadsTurbo threads; JavaScript rendering onMac mini (M1)
InterroBot CLI (Turbo)2.15.0CLITurbo threadsTurbo threadsMac mini (M1)
Katana1.7.0CLI10 threadsFull depth; no subdomains; images keptMac mini (M1)
Screaming Frog24.3.0GUIDefault threadsDefault settingsMac mini (M1)
Screaming Frog CLI24.3.0CLIDefault threadsDefault settingsMac mini (M1)
SEO Macroscope1.7.6.1GUIDefault threadsDefault settingsWindows 10 (VM)
SEO Macroscope (High)1.7.6.1GUI10 threads10 threadsWindows 10 (VM)
SiteOne Crawler1.0.8GUIDefault threadsReport upload offMac mini (M1)
SiteOne Crawler (High)1.0.8GUI8 threads8 threads; 50 requests/s; report upload offMac mini (M1)
Visual SEO Studio2.5.3.1GUIDefault threadsMax crawl depth 9Windows 10 (VM)
Xenu's Link Sleuth1.3.8GUI30 threadsDefault settingsWindows 10 (VM)

How This Was Measured

Test Machines
  • Mac mini (M1/Tahoe)8 GB of RAM · macOS 26.6.2
  • Windows 10 (Hyper-V)15.8 GB of RAM · Windows 10 · virtual machine

Each crawler ran by itself. Crawl time, counts, memory, and CPU are captured consistently, per OS. See more on how it’s measured.

Sites

Served from a remote server, reached over the internet.

Free Limits
Screaming Frog, Screaming Frog CLI, and Visual SEO Studio stopped at the free edition’s 500-URL limit, so they could not finish this site.