Crawler Bench

Filter Crawls

Filters
Interface
Rendering
Type
Price
Crawlers
Crawlers

Crawling the Rust Book

Nine crawlers in 22 setups, each pointed at the same 114-page site (176 files) served from a web server on the same machine. Each run was screen-recorded and is replayed here on a unified clock, so you can watch them in real time.

Crawl Results

Beam Us UpFreeGUI · 16 threads (auto)
00:05.91140114 / 01029 MB114 pages · 0 assets · 1029 MB peak
Beam Us Up (JS)FreeGUI · 16 threads (auto)
00:42.91140114 / 01198 MB114 pages · 0 assets · 1198 MB peak
Beam Us Up CLIFreeCLI · 10 threads
00:04.91140114 / 01087 MB114 pages · 0 assets · 1087 MB peak
Beam Us Up CLI (JS)FreeCLI · 10 threads
00:42.61140114 / 0862 MB114 pages · 0 assets · 862 MB peak
GNU WgetFreeCLI · Single thread
00:00.111462114 / 623.4 MB114 pages · 62 assets · 3.4 MB peak
InterroBotFreeGUI · Default threads
00:01.811450114 / 50433 MB114 pages · 50 assets · 433 MB peak
InterroBot (JS)GUI · Default threads
00:39.411450114 / 50934 MB114 pages · 50 assets · 934 MB peak
InterroBot (Turbo JS)GUI · Turbo threads
00:21.611450114 / 501027 MB114 pages · 50 assets · 1027 MB peak
InterroBot (Turbo)GUI · Turbo threads
00:01.011450114 / 50433 MB114 pages · 50 assets · 433 MB peak
InterroBot CLIFreeCLI · Default threads
00:00.811450114 / 50146 MB114 pages · 50 assets · 146 MB peak
InterroBot CLI (JS)CLI · Default threads
00:39.311450114 / 50582 MB114 pages · 50 assets · 582 MB peak
InterroBot CLI (Turbo JS)CLI · Turbo threads
00:21.311450114 / 50771 MB114 pages · 50 assets · 771 MB peak
InterroBot CLI (Turbo)CLI · Turbo threads
00:00.911450114 / 5043.4 MB114 pages · 50 assets · 43.4 MB peak
KatanaFreeCLI · 10 threads
00:12.111350113 / 5067.2 MB113 pages · 50 assets · 67.2 MB peak
Screaming FrogFreeGUI · Default threads
00:05.111548115 / 481299 MB115 pages · 48 assets · 1299 MB peak
Screaming Frog CLIFreeCLI · Default threads
00:05.411548115 / 48555 MB115 pages · 48 assets · 555 MB peak
SEO MacroscopeFreeGUI · Default threads
00:12.211551115 / 51241 MB115 pages · 51 assets · 241 MB peak
SEO Macroscope (High)FreeGUI · 10 threads
00:12.311551115 / 51252 MB115 pages · 51 assets · 252 MB peak
SiteOne CrawlerFreeGUI · Default threads
00:15.111461114 / 61371 MB114 pages · 61 assets · 371 MB peak
SiteOne Crawler (High)FreeGUI · 8 threads
00:06.411461114 / 61375 MB114 pages · 61 assets · 375 MB peak
Visual SEO StudioFreeGUI · Default threads
00:05.911428114 / 28142 MB114 pages · 28 assets · 142 MB peak
Xenu's Link SleuthFreeGUI · 30 threads
00:02.011561115 / 615.5 MB115 pages · 61 assets · 5.5 MB peak

Resource Usage

Watch in Realtime

Every crawl starts together at 0:07. Click a screen to enlarge it.

0:00.0 / 1:01.9
Beam Us UpReady
Beam Us Up (JS)Ready
Beam Us Up CLIReady
Beam Us Up CLI (JS)Ready
GNU WgetReady
InterroBotReady
InterroBot (JS)Ready
InterroBot (Turbo JS)Ready
InterroBot (Turbo)Ready
InterroBot CLIReady
InterroBot CLI (JS)Ready
InterroBot CLI (Turbo JS)Ready
InterroBot CLI (Turbo)Ready
KatanaReady
Screaming FrogReady
Screaming Frog CLIReady
SEO MacroscopeReady
SEO Macroscope (High)Ready
SiteOne CrawlerReady
SiteOne Crawler (High)Ready
Visual SEO StudioReady
Xenu's Link SleuthReady

Setups

CrawlerVersionInterfaceThreadsSettingsMachine
Beam Us Up2.0.0GUI16 threads (auto)Default settingsMac mini (M1)
Beam Us Up (JS)2.0.0GUI16 threads (auto)JavaScript rendering onMac mini (M1)
Beam Us Up CLI2.0.0CLI10 threadsDefault settingsMac mini (M1)
Beam Us Up CLI (JS)2.0.0CLI10 threadsJavaScript rendering onMac mini (M1)
GNU Wget1.25.0CLISingle threadMirror presetMac mini (M1)
InterroBot2.15.0GUIDefault threadsDefault settingsMac mini (M1)
InterroBot (JS)2.15.0GUIDefault threadsJavaScript rendering onMac mini (M1)
InterroBot (Turbo JS)2.15.0GUITurbo threadsTurbo threads; JavaScript rendering onMac mini (M1)
InterroBot (Turbo)2.15.0GUITurbo threadsTurbo threadsMac mini (M1)
InterroBot CLI2.15.0CLIDefault threadsDefault settingsMac mini (M1)
InterroBot CLI (JS)2.15.0CLIDefault threadsJavaScript rendering onMac mini (M1)
InterroBot CLI (Turbo JS)2.15.0CLITurbo threadsTurbo threads; JavaScript rendering onMac mini (M1)
InterroBot CLI (Turbo)2.15.0CLITurbo threadsTurbo threadsMac mini (M1)
Katana1.7.0CLI10 threadsFull depth; no subdomains; images keptMac mini (M1)
Screaming Frog24.3.0GUIDefault threadsDefault settingsMac mini (M1)
Screaming Frog CLI24.3.0CLIDefault threadsDefault settingsMac mini (M1)
SEO Macroscope1.7.6.1GUIDefault threadsDefault settingsWindows 10 (VM)
SEO Macroscope (High)1.7.6.1GUI10 threads10 threadsWindows 10 (VM)
SiteOne Crawler1.0.8GUIDefault threadsReport upload offMac mini (M1)
SiteOne Crawler (High)1.0.8GUI8 threads8 threads; 50 requests/s; report upload offMac mini (M1)
Visual SEO Studio2.5.3.1GUIDefault threadsMax crawl depth 9Windows 10 (VM)
Xenu's Link Sleuth1.3.8GUI30 threadsDefault settingsWindows 10 (VM)

How This Was Measured

Test Machines
  • Mac mini (M1/Tahoe)8 GB of RAM · macOS 26.6.2
  • Windows 10 (Hyper-V)15.8 GB of RAM · Windows 10 · virtual machine

Each crawler ran by itself. Crawl time, counts, memory, and CPU are captured consistently, per OS. See more on how it’s measured.

Sites
  • The Rust Book homepageRust Bookrust-book-local.crawlerbench.com114 HTML pages176 files
  • The GIMP 3.0 Manual homepageGIMP 3.0 Manualgimp-local.crawlerbench.com711 HTML pages2,616 files

Served from a local web server on the test machine.