Crawler Bench

Add a Crawler

Crawler Bench lists open-source crawlers and free proprietary ones. This is what it takes to be added, and where paid crawlers stand.

Who Gets Listed

Mature products only. A crawler should have:

  • At least two years of releases, or if less, an impressive adoption story.
  • A mature website with documentation.
  • A free way to run it: open source, or a free edition of a proprietary product.

Anything that seems fly-by-night, buggy, or untrustworthy will not be included. Listing is at my discretion, and no crawler is owed a place.

With exception to InterroBot, crawlers without a free way to run them are not included yet. I’m open to it, but haven’t fleshed out how to bring them in, and what level of sponsorship will be required.

For other paid options, I am considering a fee system with updates outside the standard publishing cycle, access to historical data, and more. Nothing is bolted down and nothing is implemented yet. Stay tuned.