Crawler Bench

Website Crawlers

Nine website crawlers, each screen-recorded crawling the same test sites.

  • Beam Us Up SEO CrawlerCrawler/Analyzer · GUI/CLICompletely free, no limits, and easy interface, automatic error detection.Tested in four setups on four sites.FreeWindowsmacOSLinux
  • GNU WgetCrawler · CLIGNU Wget is a free software package for retrieving files using HTTP, HTTPS, FTP and FTPS, the most widely used Internet protocols.Tested in one setup on four sites.FreeWindowsmacOSLinux
  • InterroBotCrawler/Analyzer · GUI/CLIInterroBot provides the website crawler, search, and analysis you need to understand your website.Tested in eight setups on four sites.FreeWindowsmacOSLinux
  • KatanaCrawler · CLIA next-generation crawling and spidering frameworkTested in one setup on four sites.FreeWindowsmacOSLinux
  • Screaming Frog SEO SpiderCrawler/Analyzer · GUI/CLIThe Screaming Frog SEO Spider is a website crawler that audits your site for over 300 SEO issues to help improve both SEO, and UX.Tested in two setups on four sites.FreeWindowsmacOSLinux
  • SEO MacroscopeCrawler/Analyzer · GUIA free and open source website broken link checking and scanning tool, with some technical SEO functionality for common website problems.Tested in two setups on four sites.FreeWindows
  • SiteOne CrawlerCrawler/Analyzer · GUI/CLIThe free Swiss-Army-knife for your website.Tested in two setups on four sites.FreeWindowsmacOSLinux
  • Visual SEO StudioCrawler/Analyzer · GUISEO audit tool for web agencies, freelancers, in-house SEOs and e-commerce owners.Tested in one setup on four sites.FreeWindows
  • Xenu's Link SleuthCrawler/Analyzer · GUIXenu's Link Sleuth (TM) checks Web sites for broken links.Tested in one setup on four sites.FreeWindows