Crawl your website like a spider: Use these SEO tools

SEO Tools

So, you are conducting an SEO audit. Maybe you were inspired by the article we posted on SEO audits a couple of days ago.

You did your research. You found some checklists and made a list of things you want to check to get started. Your goal is to get a basic SEO health checkup for your client’s website.

So, you started accumulated a list of tools you need to get started. Does that list include a website crawler?

Well, today we are going to take a look at a few website crawlers to get on your way.

Screaming Frog

Screenshot: /
Screenshot: /

Screaming Frog is almost everyone’s favorite choice for a web crawler. It allows you to do a ton of things. While you can download the crawler for free, you are able to crawl an unlimited number of pages for £149.00 per year.

There are many other things included in the premium, but if you are working with smaller sites, the free is more than enough. You are limited to crawling 500 URLs.

The tool is a desktop application. It is actually called The Screaming Frog SEO Spider. Screaming Frog is the name of the company behind the tool and they offer other various digital marketing services.

If you want to see more about how to use the spider, Seer Interactive has created a fantastic guide you can see here:

Screaming Frog Guide To Doing Almost Anything: 55+ Ways Of Looking At A Tool.

Manage all your WordPress sites with the MainWP Dashboard

WordPress Management for Professionals

Are you ready to go Pro?

All MainWP Pro Extensions are available through one of our convenient bundled packages.


Another guide for learning Screaming Frog is this one at Big Leap:

The Ultimate Beginner’s Guide to Using Screaming Frog


Vary is a very good website crawler that provides basic information about your page including the SEO, the speed, and the mobile SEO. The interface is really nice and clean.

See more tools:

I learned about Vary from John Locke. Vary is a great tool to provide a great starting point for your audits.

Originally built and owned by Patrick Sexton, there are several tools on the site. In the past couple of years, it was acquired by the Internet Marketing Ninjas. IMN has its own suite of tools, so this is another in a really nice portfolio of SEO tools.

According to a representative at IMN, they haven’t touched the site since it was acquired.


Other tools

Internet Marketing Ninjas

The longtime SEO company has several tools available for free in addition to owning Vary (see above). Taking advantage of these tools allow you to find many of the apps you need in the same location.

SEO Tools

Site Crawl Tool – Sitemap Generator

Beam Us Up

This one had potential. Many recommended this tool as an alternative to Screaming Frog. It is also a desktop based app. Unfortunately, according to their Twitter account, the tool has been discontinued.

It looks like you can still download the tool as the site is still online, but will have no support. I decided not to download the app because of that and the fact that the browser lists it as “Not Secure.”

SEO Crawler by Ron Hammond

SEO Optimizer


SEO Site Check

SEO Tools Resources

Search Engine Optimization Tools – SEO Book

29 Best Free SEO Tools (Tried & Tested) – Ahrefs

13 of the Best SEO Tools for Auditing & Monitoring Your Website in 2019 – Hubspot

Google’s Top 5 SEO Tools – Search Engine Journal

Wrapping it up

There are lots of tools available for SEO. We just mentioned a few of the crawler-based tools. I believe these tools are best if you can compare the results to other tools and verify the information based on your own testing.

What SEO tools do you like to use?

2 thoughts on “Crawl your website like a spider: Use these SEO tools”

Leave a Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Looking for something?

Your Download Is Just One Click Away

…or just download the plugin.

By entering your email, you agree to our Terms of Service and Privacy Policy.