Crawler Bench

GNU Wget

Crawler · CLI

GNU Wget is a free software package for retrieving files using HTTP, HTTPS, FTP and FTPS, the most widely used Internet protocols.

In their wordswww.gnu.org/software/wget
Version Tested
1.25.0 (released Nov 10, 2024)
Vendor
GNU Project
Platforms
Windows, macOS, and Linux
Interfaces
CLI (command line interface)
JavaScript Rendering
No
Type
Crawler
Test Machine
Mac mini (M1/Tahoe), 8 GB of RAM, macOS 26.6.2

Across the Test Sites

Tested in one setup on four sites. Its best finish: GNU Wget on the Rust Book (local server), 00:00.1, 1st of 25.

GNU WgetCLI · Single thread00:00.11st of 2500:02.03rd of 2500:01.41st of 2500:30.29th of 24

Site Crawls

One GNU Wget setup on the Rust Book: 114 pages (176 files) served from a web server on the same Mac, replayed on one clock.

Filter Runs

Site
Server
Filters
Interface
Rendering
Price

Crawl Results

00:00.111462114 / 623.4 MB114 pages · 62 assets · 3.4 MB peak

Resource Usage

Watch in Realtime

Every crawl starts together at 0:07. Click a screen to enlarge it.

0:00.0 / 0:10.8

Setups

GNU Wget1.25.0CLISingle threadMirror preset

How This Was Measured

Test Machine
  • Mac mini (M1/Tahoe)8 GB of RAM · macOS 26.6.2

Each crawler ran by itself. Crawl time, counts, memory, and CPU are captured consistently, per OS. See more on how it’s measured.

Sites
  • The Rust Book homepageRust Bookrust-book-local.crawlerbench.com114 HTML pages176 files
  • The GIMP 3.0 Manual homepageGIMP 3.0 Manualgimp-local.crawlerbench.com711 HTML pages2,616 files

Served from a local web server on the test machine.