When you’re working on improving your SEO system, you are going to be introduced to many different tools that can help you follow better SEO practices. Each tool has its own set of capabilities, and they can all be used in unique ways to get you different types of data.

With so many tools from, how can you know the best ways to use each one?

Thankfully, there are a number of guides and tutorials out there that can make it simpler to figure the tools out. Today, we’ll be going over how to use Screaming Frog, one of the most well-known SEO tools out there. Marketers have long been talking about how they make good use of Screaming Frog, and now you can, too.

Screaming Frog has both free and paid tools that are incredibly effective for any level marketing campaign. If you haven’t learned how to use this efficient tool yet, it’s time to find out how to bring it into your next plan.

What Is Screaming Frog?

Let’s get into the Screaming Frog guide so that you can learn about why this tool might help you improve your SEO practices and how to use it efficiently.

Screaming Frog is a company that offers a variety of marketing tools and services. Their most well-known tool is the Screaming Frog SEO Spider, and this is the tool that we will be giving you a complete tutorial for using today.

This is not the only thing that the UK-based company offers, however.

Tools Available On Screaming Frog

First, let’s talk about the various services that Screaming Frog offers. As a marketing company, they offer a variety of services to clients who want to get help from Screaming Frog to improve their website or business results.

In particular, they offer the following:

• Search Engine Marketing (SEO optimization and PPC advertising management)
• Social Media Marketing
• Technical SEO Audits
• Content Marketing and Production
• Digital PR
• Link Building
• Conversion Rate Optimization
• Analytics Consulting Work

These are all the areas that the very best SEO-based marketing companies excel in, and Screaming Frog is happy to offer services in all of these areas. If there is one particular area that you would prefer to have an expert company handle rather than learning to do yourself, they may be able to help.

Log File Analyzer

Another tool that is offered by Screaming Frog is the Log File Analyzer. This light tool, which is available in a limited form free or a full form for an annual fee, runs log files through the software to find out which bots ran which pages and what happened when they did.

Through this type of log analysis, pages can be better optimized for searches from Googlebot and other search spiders. With the results from a log file analyzer like this one from Screaming Frog, you can fix broken links, bad redirects, slow pages, and underperforming pages with ease.

SEO Spider

Screaming Frog’s SEO Spider is software that uses a website crawler, also known as a spider, to gather information from the website in a specific way. The results can help you improve your site and SEO practices so that your website performs better in both searches and conversions.

We’ll learn more about this tool (and how to use Screaming Frog) in the upcoming sections.

What Is The Screaming Frog SEO Spider?

Now that we know a bit more about Screaming Frog and all that they offer, let’s narrow our focus in on the SEO tool from their portfolio that is most important to learn more about: the Screaming Frog SEO Spider.

“Spider” is a term used to describe software that harvests data from the internet and then processes it in a specific way. Google, for example, uses spider software to “crawl” every website on the internet for data that they can then index and use to create better search results.

Spiders are used for lots of different search websites, and websites that don’t have their own spider software running typically have some type of access to the index developed by another site.

Screaming Frog’s Spider

The spider on Screaming Frog is one that you can download and run yourself. This spider was created specifically to crawl for important SEO information:

• Categories
• Metadata
• URL
• Text content
• …and much more!

Benefits Of Using This SEO Spider

As mentioned, there are quite a few different ways that you can get access to SEO tools and search indexes. Not all of them, however, give you the ability to search, filter, and customize your own SEO crawls in the way that Screaming Frog does.

Whether you have been working on SEO for more than a decade or you’re still learning the basics, there are benefits that come from using Screaming Frog.

Here are just a few of the awesome benefits that the tools offer:

• What is and isn’t working on the technological side
• Alerts you to any missing metadata
• Highlights bad redirects
• Shows duplicate pages
• Filters how pages show up
• … and so much more!

When you run the tool, you can change the specific data that it collects, but most crawls will be set up to include the following:

• Robots.txt errors
• Redirects
• Link errors
• Server errors
• External links
• Internal links
• Link rankings
• Security of links
• Page title problems
• Missing metadata
• Response times
• Word count for all page text
• Anchor text
• Canonicalization of URLs
• Image hosting information

If you are hoping to do a site or content audit, you can get all of the information that is needed to effectively do this with Screaming Frog’s SEO Spider. Every crawl run by Screaming Frog follows the Googlebot (a.k.a. Google’s primary search spider) standards so that you can get the best SEO results.

It is, however, possible to change up what the spider looks at to get even more information on your own site if you want to. Depending on your goals, it could be beneficial to look at some areas that Googlebot doesn’t care about when indexing your site’s pages.

Screaming Frog SEO Spider Products

This tool is available in a lite version and a full license version.

The Lite version has limitations in place. It’s free, but you are restricted to crawling only 500 URLs. There are also some restrictions on the amount of integration and customization that you can do, so you will find that some aspects of the tool are limited.

If you want full access to the tool or you want to be able to run the spider on sites with more URLs, you will want to go for the full license instead. This license has an annual fee, but it is well worth the price if you need those unlocked capabilities.

One of the biggest benefits of the full license version is the ability to integrate with Google Analytics, Google Search Console, Link Metrics, and PageSpeed Insights. With all of this data linked together, the SEO Spider tool becomes even stronger. Plus, you can save and reupload the same crawls more simply with the full license.

How To Use Screaming Frog For SEO

It’s time to start our official Screaming Frog tutorial on how to use the SEO Spider tool for SEO research, optimization, and more! There is a lot that can be done with this tool, so we’ll only be going over a few key techniques at first.

Once you’ve mastered the basics that we cover today, head to Screaming Frog’s User Guide to get more details about other techniques that you would like to try with their tools.

Note: Throughout these mini-guides, we’ve included suggestions to modify the spider configuration before you run a crawl, which is an option only available with a full license. If you are using the lite version, simply run the crawl without customization and follow the guide from there.

Start With A Total Site Crawl

The first thing that we’re going to learn how to do is a total site crawl. Doing this type of crawl is important for site audits and content audits and to just get a better idea of what you are working with. Plus, it’s a great way to start testing out the SEO Spider tool without needing to do too much customization.

Here’s how to do a basic site crawl:

1. Click “Configuration” menu at the top of the page.
2. Click “Spider” from the drop-down menu.
3. Click “Crawl all subdomains” if you want to get a more complete view of your site, but we recommend leaving the CSS, images, Javascript, and other boxes unclicked at first to do a faster initial crawl.
4. Start the crawl; its speed will depend on how much you have clicked and what your computer can offer in terms of processing power and memory.
5. Once it’s complete, click “Internal” tab.
6. You can choose how to filter the results; we recommended starting with HTML and then exporting the data to a CSV.
7. Browse the data using the tabs (pictured below) to find issues, and you’ll be ready to start improving your site for SEO!

Checking Links With Screaming Frog’s Spider

Another way that you can use this tool to improve your SEO practices is to check your links with it. How many do you have on each page? Are the links working? Finding all of this information can take ages if you try to do it manually, but you can do it more simply with the SEO Spider.

Here is how to do a basic link audit:

1. Click “Configuration menu”
2. Click “Spider”
3. Uncheck CSS, images, Javascript, Flash, and everything other than subdomains.
4. You can decide whether or not you want to include links that are labeled as nofollows in your code; that’s up to you!
5. Start the crawl and wait until it’s complete.
6. Once the crawl is done, click “All Links”
7. From here, you’ll be able to export a CSV of the links as well as important link information.

Now that you have had this link data crawled by the spider, what should you do with it?

There are lots of ways that you could put it into action, but these are some of the most common techniques used by SEO experts to do more with this link information.

Identify Pages With Too Many Links

Use the list to find any pages that have too many outlinks. Spiders like Googlebot will typically flag pages that have too many outlinks, so you will want to avoid having pages with more than 100 outlinks if possible.

To find out which pages have the most outlinks on your site, click “Internal” tab and then sort by outlinks, as pictured below from our limited sample.

Find Pages With Broken Links, Bad Redirects, Etc.

There are a number of different types of status reports that can cause pages to result in problems for a website crawler. 404 broken links, 301 bad redirects, and other status codes are all signs of a problem on the page.

You can find these pages easily after running a website crawl. Click the “Internal” tab and then click “Status Code.” You can find reports of each status code type here, and that will help you find opportunities to improve the way that your site is functioning.

Try A Complete Content Audit

Now that you’ve had a chance to familiarize yourself with some of the data that can be provided by doing a crawl with this SEO Spider, let’s talk about another area that you can improve: content. A huge part of SEO ranking is based on how content is structured and delivered, so you will want to check into that as well.

So, you need to set up your SEO Spider to do another crawl, but this time you want to make sure that you check as many boxes as possible on the configuration menu. The more data that is gathered in this crawl, the better! More data will help to ensure that you have as much information to work with as possible.

More data gives better results, and that is the main goal here!

Here is how to set up your content-focused crawl:

1. Click “Configuration menu”
2. Click “Spider”
3. Include everything that you can in this crawl: CSS, images, scripts, and everything else.
4. Run the crawl.
5. This crawl is likely to take a bit longer because there is a lot more data being included than in the previous crawls, and this is OK. Be patient, and remember that the data gathered is going to help your site perform better than ever before.

Once the crawl is complete, it’s time to put the data to work. For most sites, these are the primary areas to focus on for SEO improvement.

Find Thin Content That Needs To Be Removed Or Improved

Pages that have fewer than 1,000 words on them are very likely to be detrimental to your SEO search results unless you are an e-commerce site with on-site product listings. Pages with thin content can either be nonindexed, improved, or deleted entirely. It’s up to you to decide which pages should get which treatment.

To find these pages, here is what to do:

1. Click the “Internal” tab.
2. Filter by HTML.
3. Sort the word count from low to high.
4. Investigate the underperforming pages.

Fix Problematic Titles: Long Titles

Another way to use your results is to find titles that are too long to show up properly on Google search results. Titles that are too long aren’t going to attract the right kind of attention, so you either want to make sure these titles cut off in an attractive way or shorten them.

To find long titles, follow these steps:

1. Click the “Page titles” tab.
2. Filter down to titles longer than 70 characters.
3. Modify the problematic titles from there.

Fix Problematic Titles: Duplicate Meta Titles

Another type of title problem that can cause issues for your page’s SEO results is when you have duplicate meta titles. Even if the content under these titles is unique and the duplicate title was accidental, it’s very likely that duplicate meta titles will cause you to be penalized in the search results.

To find duplicate pages, follow these steps:

1. Click the “Page titles” tab.
2. Filter by occurrences.
3. Identify duplicate pages if there are pages with an occurrence of more than one.
4. For unique pages, change the titles appropriately.
5. For true duplicates, remove one page and redirect as necessary.

Fix Problematic Descriptions: Duplicate Meta-Description

If your site has pages that have the same meta description on multiple pages, your SEO rankings are being affected more than you might realize, so you want to fix this quickly.

It’s easy to identify what pages need new meta-descriptions by following these steps:

1. Click the “meta description” tab.
2. Use the occurrence filter to find duplicates; duplicates will have an occurrence of more than one.
3. Edit duplicate descriptions so that they are unique.

Alternatively, you can use the “overview” panel on the right side to identify pages that suffer from common SEO problems with titles and descriptions:

Fill In Empty Alt Text And Remove Long Alt Text

You can improve your SEO results by making sure that all of your images have what is known as alt text associated with them. Alt text is a way for Googlebot and other website crawlers to get an idea of what the image is without being able to see it, so they are very important.

Ideally, this alt text will be under 100 characters for the best experience and SEO results. You can check for alt text that is too long and those images that don’t have alt text at all in a few easy steps:

1. Click the “Images” tab.
2. To find images missing text, use the “missing alt text” filter option at the top of the page.
3. To find images with alt text that is too long, filter the alt text to show you anything over 100 characters.

Get Rid Of URL Problems

Another tab that can show you a lot of useful information about any potential URL issues is the URL tab. Once you’re in this tab, you can use the filters to find a few interesting things.

First, check that there aren’t any duplicate URLs using the duplicate filter. Sometimes, a page will have multiple URLs that are very similar, all leading to the same page. This can be as simple as some pages not having a www in the URL. These pages should be canonicalized so that Google knows which page is the primary page.

Next, try out multiple filters to see if you have any pages with URLs that aren’t easy for a page visitor to understand. An easy way to do this is to scroll down in the “Overview” section on the right-hand side until you see the URL submenu, pictured below.

You can see in the primary sections that SEO Spider recommends that you check for your URLs. For example, a URL that has a lot of underscores in it might look very strange and be less likely to be clicked. Find and fix these URLs.

It’s Time To Test For Yourself

Are you ready to give Screaming Frog a hands-on try? With today’s Screaming Frog tutorial, you should be able to figure out how to use Screaming Frog with their free SEO spider free trial. 

Downloading the free version of the SEO Spider is all that you need to do to try out the following tactics to see if they would help improve your marketing efforts:

• Before you try our mini-guides above, simply run a crawl and then look at the “Overview” panel on the right hand to see common SEO problems.
• Then, try out our mini-guides to follow some specific directions to fix specific problems.
• If you are still confused or have something more that you want to learn how to do, head over to the User Guide created by Screaming Frog to learn more about SEO Spider’s capabilities.

If you love what you see after you give Screaming Frog’s SEO Spider a try, remember that they also offer a full license version. If you have found that the SEO Spider doesn’t fit well with your current goals, and you want to focus more on keyword exploration, try Long Tail Pro’s Keyword Research tools instead.