Crawler BenchSiteOne Crawler
Crawler/Analyzer · GUI/CLI
The free Swiss-Army-knife for your website.
- Version Tested
- 1.0.8 (released Aug 24, 2024)
- Website
- crawler.siteone.io
- Source Code
- github.com/janreges/siteone-crawler
- Vendor
- SiteOne
- Platforms
- Windows, macOS, and Linux
- Interfaces
- GUI/CLI (both included)
- JavaScript Rendering
- No
- Type
- Crawler/Analyzer
- Test Machine
- Mac mini (M1/Tahoe), 8 GB of RAM, macOS 26.6.2
Across the Test Sites
Tested in two setups on four sites. Its best finish: SiteOne Crawler (High) on the GIMP 3.0 Manual (local server), 00:53.8, 10th of 25.
| SiteOne CrawlerGUI · Default threads | 00:14.916th of 25 | 00:14.816th of 25 | 02:56.613th of 25 | 02:55.410th of 24 |
|---|---|---|---|---|
| SiteOne Crawler (High)GUI · 8 threads | 00:06.412th of 25 | 00:11.912th of 25 | 00:53.810th of 25 | 03:07.211th of 24 |
Site Crawls
Two SiteOne Crawler setups on the Rust Book: 114 pages (176 files) served from a web server on the same Mac, replayed on one clock.
Crawl Results
00:14.911461114 / 61367 MB114 pages · 61 assets · 367 MB peak
00:06.411461114 / 61380 MB114 pages · 61 assets · 380 MB peak
Resource Usage
Watch in Realtime
Every crawl starts together at 0:07. Click a screen to enlarge it.
0:00.0 / 0:25.6
Setups
| SiteOne Crawler | 1.0.8 | GUI | Default threads | Report upload off |
| SiteOne Crawler (High) | 1.0.8 | GUI | 8 threads | 8 threads; 50 requests/s; report upload off |
How This Was Measured
- Test Machine
Mac mini (M1/Tahoe)8 GB of RAM · macOS 26.6.2
Each crawler ran by itself. Crawl time, counts, memory, and CPU are captured consistently, per OS. See more on how it’s measured.
- Sites
Rust Bookrust-book-local.crawlerbench.com114 HTML pages176 files
GIMP 3.0 Manualgimp-local.crawlerbench.com711 HTML pages2,616 files
Served from a local web server on the test machine.