Crawler BenchWebsite Crawlers
Ten website crawlers, each screen-recorded crawling the same test sites.
- Beam Us Up SEO CrawlerCompletely free, no limits, and easy interface, automatic error detection.
- GNU WgetGNU Wget is a free software package for retrieving files using HTTP, HTTPS, FTP and FTPS, the most widely used Internet protocols.
- HTTrack Website CopierHTTrack is a free (GPL, libre/free software) and easy-to-use offline browser utility.
- InterroBotInterroBot provides the website crawler, search, and analysis you need to understand your website.
- KatanaA next-generation crawling and spidering framework
- Screaming Frog SEO SpiderThe Screaming Frog SEO Spider is a website crawler that audits your site for over 300 SEO issues to help improve both SEO, and UX.
- SEO MacroscopeSEO Macroscope for Microsoft Windows is a free and open source website broken link checking and scanning tool, with some technical SEO functionality for common website problems.
- SiteOne CrawlerThe free Swiss-Army-knife for your website.
- Visual SEO Studio
- Xenu's Link Sleuth