If you’re an SEO, you’ve likely heard of Screaming Frog – a widely used website crawling tool that acts like a swiss army knife for SEOs. I use Screaming Frog on a daily basis, so here are some of my favorite aspects about the tool as well as features that you might not be aware of.
Full disclosure, many of these features require a paid license. I am also not being asked by Screaming Frog to write any of this – they are just one of many paid website crawlers available to the public, most of which I have not tried.
List Mode is Practically Unlimited
As I mentioned earlier, crawl mode is limited to 500 URLs in the free version of Screaming Frog. However, you can crawl up 10,000 URLs using List mode.
For example, if you wanted to check the HTTP statuses of URLs listed pages that have backlinks. Go to Google Search Console > Links > Total External Links > Export. Then you can upload that file to Screaming Frog’s list mode. Make sure all pages on your website that are receiving links from outside sources have 200 or 3xx statuses – all for free!
Two List Mode Export Options
Speaking of list mode, you have two options for exporting crawled data.
- Export consolidated list of uploaded URLs
- Export list of uploaded URLs in exact order
Screaming Frog removes duplicates and URLs entered incorrectly, so if you want a clean and concise version of URLs then option 1 makes more sense. The option 2 exports the URLs in the same order in which they were uploaded, so if you’re looking to supplement an existing list of URLs in an Excel document with data from Screaming Frog, then option 2 makes more sense.
XML Sitemap Audit
Most website owners check to see if there are any issues with their XML Sitemap by checking the Coverage report in Google Search Console. If there are issues with your Sitemap, Google may flag some examples but they may not report all issues – especially if you have a sizeable sitemap.
If you download your XML Sitemap, it can be uploaded to Screaming Frog, where you’ll get a much better idea of what needs to be cleaned up.
Search Console Integration
Screaming Frog supports several APIs including Google Analytics, PageSpeed Insights, Majestic, Ahrefs, and Mozscape. However my favorite integration has to be Google Search Console.
After you link and connect your account, search result performance data will appear when you crawl your site – specifically total clicks, impressions, average CTR, and average position. Use this in tandem with other metrics like meta data and word count and you can quickly find content that needs just a little extra love to get a whole lot more organic search traffic.
Content Tab
Speaking of content improvements, Screaming Frog recently released a new Content tab in version 13.0. You can now quickly identify duplicate pages, “near duplicate” pages, low content pages, and pages with spelling or grammatical errors. You can also exclude items from your nav and footer, which are likely duplicates on all of your pages.
Check out Screaming Frog’s announcement of v13.0 for more details.
Change Configurations
It’s clear that Screaming Frog can be used for many different purposes. That will often require making frequent changes in the configuration settings.
Instead of changing the configuration settings each time, you can easily save specific configurations and set one as “default”. Go to File > Configuration to set this up. I personally have configurations set up to:
- Only crawl external links for a list of URLs
- Only crawl images
- Extract PageSpeed Insight data using custom extraction
- Default – crawling an entire site
Change User Agents
Having trouble crawling a specific website? Try changing the User Agent from Screaming Frog SEO Spider to something like Googlebot or Bingbot. Certain websites trust some crawlers but not others, so switching between user agents will often allow you to crawl particularly stubborn websites.
Always Follow Redirects
Whether you are analyzing a website or doing broken link building, knowing that a page redirects isn’t really valuable information. What is valuable is seeing the final destination of the redirect and if it arrives at a live page or a broken link.
Many websites have pages that follow more than one redirect, so changing the settings to follow redirects will help you find more broken links for BLB and catch internal links that lead to a dead end. Screaming Frog will follow a maximum of five redirects, but you can change that setting to any number you like.
Change the configuration setting to Always Follow Redirects by going to Configuration > Spider > Advanced > Always Follow Redirects. I always have this as my default setting.
Crawl Analysis
While crawling a site, you may have seen certain sections of the Overview report that have (Crawl Analysis Required) messages. This is an excellent way to find potential issues with your website that could be easily remedied.
These items are excluded from the initial crawl as it is better to run the analysis once Screaming Frog has collected the required data. For example, Screaming Frog can identify pages that are not listed in your sitemap through the crawl analysis.
After you’ve crawled your website, go to Crawl Analysis > Start to get access to this additional data. You can also set this up to run automatically every time after you run an initial crawl by going to Crawl Analysis > Configure.
Check out this helpful video detailing crawl analysis:
Did I miss anything? What are your favorite Screaming Frog features? Be sure to let me know!
Image Credit
Leave a Reply