SiteOne Crawler
Crawler/Analyzer · GUI/CLI
The free Swiss-Army-knife for your website.
- Version Tested
- 1.0.8 (released Aug 24, 2024)
- Website
- crawler.siteone.io
- Source Code
- github.com/janreges/siteone-crawler
- Vendor
- SiteOne
- Platforms
- Windows, macOS, and Linux
- Interfaces
- GUI/CLI (both included)
- JavaScript Rendering
- No
- Type
- Crawler/Analyzer
- Test Machine
- Mac mini (M1/Tahoe), 8 GB of RAM, macOS 26.6.2
Across the Test Sites
Tested in two setups on four sites. Its best finish: SiteOne Crawler (High) on the GIMP 3.0 Manual (remote server), 00:55.6, 10th of 22.
| Setup | Rust Book (local) | Rust Book (remote) | GIMP 3.0 Manual (local) | GIMP 3.0 Manual (remote) |
|---|---|---|---|---|
| SiteOne CrawlerGUI · Default threads | 00:15.116th of 22 | 00:14.416th of 22 | 02:56.013th of 22 | 02:56.413th of 22 |
| SiteOne Crawler (High)GUI · 8 threads | 00:06.412th of 22 | 00:06.412th of 22 | 00:56.010th of 22 | 00:55.610th of 22 |
Site Crawls
Two SiteOne Crawler setups on the Rust Book: 114 pages (176 files) served from a web server on the same Mac, replayed on one clock.
Crawl Results
SiteOne Crawler
00:15.111461114 / 61371 MB114 pages · 61 assets · 371 MB peak
SiteOne Crawler (High)
00:06.411461114 / 61375 MB114 pages · 61 assets · 375 MB peak
Resource Usage
Watch in Realtime
Every crawl starts together at 0:07. Click a screen to enlarge it.
0:00.0 / 0:26.1
Setups
| Crawler | Version | Interface | Threads | Settings |
|---|---|---|---|---|
| SiteOne Crawler | 1.0.8 | GUI | Default threads | Report upload off |
| SiteOne Crawler (High) | 1.0.8 | GUI | 8 threads | 8 threads; 50 requests/s; report upload off |
How This Was Measured
- Test Machine
Mac mini (M1/Tahoe)8 GB of RAM · macOS 26.6.2
Each crawler ran by itself. Crawl time, counts, memory, and CPU are captured consistently, per OS. See more on how it’s measured.
- Sites
Rust Bookrust-book-local.crawlerbench.com114 HTML pages176 files
GIMP 3.0 Manualgimp-local.crawlerbench.com711 HTML pages2,616 files
Served from a local web server on the test machine.