Posted on Leave a comment

Crawl your website like a spider: Use these SEO tools

SEO Tools

So, you are conducting an SEO audit. Maybe you were inspired by the article we posted on SEO audits a couple of days ago.

You did your research. You found some checklists and made a list of things you want to check to get started. Your goal is to get a basic SEO health checkup for your client’s website.

So, you started accumulated a list of tools you need to get started. Does that list include a website crawler?

Well, today we are going to take a look at a few website crawlers to get on your way.

Screaming Frog

Screenshot: /www.screamingfrog.co.uk/seo-spider/
Screenshot: /www.screamingfrog.co.uk/seo-spider/

Screaming Frog is almost everyone’s favorite choice for a web crawler. It allows you to do a ton of things. While you can download the crawler for free, you are able to crawl an unlimited number of pages for £149.00 per year.

There are many other things included in the premium, but if you are working with smaller sites, the free is more than enough. You are limited to crawling 500 URLs.

The tool is a desktop application. It is actually called The Screaming Frog SEO Spider. Screaming Frog is the name of the company behind the tool and they offer other various digital marketing services.

If you want to see more about how to use the spider, Seer Interactive has created a fantastic guide you can see here:

Screaming Frog Guide To Doing Almost Anything: 55+ Ways Of Looking At A Tool.

Or,

Another guide for learning Screaming Frog is this one at Big Leap:

The Ultimate Beginner’s Guide to Using Screaming Frog

Vary

Vary is a very good website crawler that provides basic information about your page including the SEO, the speed, and the mobile SEO. The interface is really nice and clean.

See more tools: https://varvy.com/toolbox/

I learned about Vary from John Locke. Vary is a great tool to provide a great starting point for your audits.

Originally built and owned by Patrick Sexton, there are several tools on the site. In the past couple of years, it was acquired by the Internet Marketing Ninjas. IMN has its own suite of tools, so this is another in a really nice portfolio of SEO tools.

According to a representative at IMN, they haven’t touched the site since it was acquired.

Screenshot Vary.com
Screenshot Vary.com

Other tools

Internet Marketing Ninjas

The longtime SEO company has several tools available for free in addition to owning Vary (see above). Taking advantage of these tools allow you to find many of the apps you need in the same location.

SEO Tools

Site Crawl Tool – Sitemap Generator

Beam Us Up

This one had potential. Many recommended this tool as an alternative to Screaming Frog. It is also a desktop based app. Unfortunately, according to their Twitter account, the tool has been discontinued.

It looks like you can still download the tool as the site is still online, but will have no support. I decided not to download the app because of that and the fact that the browser lists it as “Not Secure.”

SEO Crawler by Ron Hammond

https://robhammond.co/tools/seo-crawler

SEO Optimizer

https://www.seoptimer.com/

Siteliner

http://www.siteliner.com/

SEO Site Check

https://seositecheckup.com

SEO Tools Resources

Search Engine Optimization Tools – SEO Book

29 Best Free SEO Tools (Tried & Tested) – Ahrefs

13 of the Best SEO Tools for Auditing & Monitoring Your Website in 2019 – Hubspot

Google’s Top 5 SEO Tools – Search Engine Journal

Wrapping it up

There are lots of tools available for SEO. We just mentioned a few of the crawler-based tools. I believe these tools are best if you can compare the results to other tools and verify the information based on your own testing.

What SEO tools do you like to use?

Get MainWP News and Notifications

Todd Jones
Along with being the resident writer for MainWP and content hacker at Copyflight, I specialize in writing about startups, entrepreneurs, social media, WordPress and inbound marketing topics.
Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.