Crawler Bench

SiteOne Crawler

Crawler/Analyzer · GUI/CLI

The free Swiss-Army-knife for your website.

In their wordscrawler.siteone.io
Version Tested
1.0.8 (released Aug 24, 2024)
Vendor
SiteOne
Platforms
Windows, macOS, and Linux
Interfaces
GUI/CLI (both included)
JavaScript Rendering
No
Type
Crawler/Analyzer
Test Machine
Mac mini (M1/Tahoe), 8 GB of RAM, macOS 26.6.2

Across the Test Sites

Tested in two setups on four sites. Its best finish: SiteOne Crawler (High) on the GIMP 3.0 Manual (remote server), 00:55.6, 10th of 22.

SetupRust Book (local)Rust Book (remote)GIMP 3.0 Manual (local)GIMP 3.0 Manual (remote)
SiteOne CrawlerGUI · Default threads00:15.116th of 2200:14.416th of 2202:56.013th of 2202:56.413th of 22
SiteOne Crawler (High)GUI · 8 threads00:06.412th of 2200:06.412th of 2200:56.010th of 2200:55.610th of 22

Site Crawls

Two SiteOne Crawler setups on the Rust Book: 114 pages (176 files) served from a web server on the same Mac, replayed on one clock.

Filter Runs

Site
Server
Filters
Interface
Rendering
Price

Crawl Results

SiteOne CrawlerFreeGUI · Default threads
00:15.111461114 / 61371 MB114 pages · 61 assets · 371 MB peak
SiteOne Crawler (High)FreeGUI · 8 threads
00:06.411461114 / 61375 MB114 pages · 61 assets · 375 MB peak

Resource Usage

Watch in Realtime

Every crawl starts together at 0:07. Click a screen to enlarge it.

0:00.0 / 0:26.1
SiteOne CrawlerReady
SiteOne Crawler (High)Ready

Setups

CrawlerVersionInterfaceThreadsSettings
SiteOne Crawler1.0.8GUIDefault threadsReport upload off
SiteOne Crawler (High)1.0.8GUI8 threads8 threads; 50 requests/s; report upload off

How This Was Measured

Test Machine
  • Mac mini (M1/Tahoe)8 GB of RAM · macOS 26.6.2

Each crawler ran by itself. Crawl time, counts, memory, and CPU are captured consistently, per OS. See more on how it’s measured.

Sites
  • The Rust Book homepageRust Bookrust-book-local.crawlerbench.com114 HTML pages176 files
  • The GIMP 3.0 Manual homepageGIMP 3.0 Manualgimp-local.crawlerbench.com711 HTML pages2,616 files

Served from a local web server on the test machine.