Screaming Frog SEO Spider is a desktop crawler that fetches a website the way a search engine bot follows links, then lays out what it found: status codes, titles, headings, canonicals, redirects, images and more. It is the standard tool for technical SEO audits, but its settings decide whether you see the truth or a distorted picture.
In this guide we explain what it is and what it does, whether it is free (the free version crawls up to 500 URLs), how to install it, the interface, crawl modes, checks to make before crawling, storage mode, spider, limits and rendering settings, advanced settings, include and exclude, robots.txt and user-agent, crawl speed, HTTP headers, crawling password-protected or staging sites, custom search and extraction, and Search Console integration.
What Is Screaming Frog?
A website crawler used for technical SEO that scans a site by following links and reports on each URL.
What Is It For?
Finding broken links, redirect chains, missing or duplicate titles and descriptions, heading problems, canonical issues, indexability, slow or large pages and more. See how to do an SEO analysis and technical SEO.
Is It Free?
The free version checks at most 500 URLs per crawl. Larger sites and advanced configuration options need the licensed version.
| Free | Licensed | |
|---|---|---|
| URL limit | 500 | No fixed limit |
| Advanced configuration | Limited | Full |
How to Install It
Download the installer for your operating system and install it like any desktop app.
How to Use the Interface
- URL entry field.
- Start, Pause and Clear buttons.
- Top tabs (Internal, External, Response Codes, Page Titles and so on).
- Bottom information panel.
Crawl Modes
- Spider mode: crawls by following links from a start URL.
- List mode: crawls a list of URLs you provide.
- Compare mode: compares two crawls.
- SERP mode: previews titles and descriptions.
- API mode: connects data sources.
Checks Before Crawling
Confirm which section you want to crawl, whether the site needs JavaScript rendering, whether robots.txt allows crawling, the server’s capacity, and whether you need to include or exclude certain URLs.
Storage Mode
- Database storage mode: saves to disk; suits large crawls.
- Memory storage mode: keeps data in RAM; faster for small sites.
Choose database mode for big sites and memory mode for small ones.
Spider Crawl Settings
Decide what to store and crawl: HTML pages, images, CSS and JavaScript, canonical links, hreflang, meta refresh, iframes, whether to follow nofollow links, whether to crawl subdomains and whether to crawl outside the starting folder. Note the difference between store (record the data) and crawl (follow it).
Spider Limits
Total crawl limit, crawl depth, folder depth, query string limit, redirects to follow and maximum page size.
Rendering Settings
- Text only mode: fetches the raw HTML.
- JavaScript mode: renders pages in a browser engine.
Use JavaScript rendering if the main content, links, title or canonical tags are created only after JavaScript runs.
Advanced Settings
Response timeout, 5XX response retries, HTML validation and crawling fragment identifiers.
How to Use Include and Exclude
Include restricts the crawl to matching URLs; Exclude skips matching URLs. Typical exclusions: calendar pages, endless filters, session IDs and search results.
Robots.txt Settings
Choose whether to respect robots.txt, ignore it or only report it. See what is robots.txt.
User-Agent and Crawl Speed
Set the user-agent (including Googlebot Smartphone to emulate mobile) and a polite crawl speed. Very high crawl rates can load weak servers, so coordinate with the server administrator on large projects. See what does a server error mean and what is hosting.
HTTP Header Settings
Send custom headers when needed, for example to test cookies or language.
Crawling Password-Protected and Staging Sites
Supply authentication details, and be careful not to expose a staging site to search engines.
Custom Search and Custom Extraction
- Custom search finds pages containing (or not containing) certain text or code, such as missing analytics tags.
- Custom extraction pulls specific data from pages using XPath or CSSPath, such as prices or schema fields.
Google Search Console Integration
Combine crawl data with Search Console clicks, impressions and positions to prioritize fixes. See Google Search Console.
Typical Uses
- Check redirects. See what is a 301 redirect.
- Check canonicals. See what is a canonical URL.
- Check meta tags and headings. See what is a meta tag and what is an H1 tag.
- Check images. See image SEO.
- Generate or verify an XML sitemap. See what is a sitemap. Check indexability, canonicals and status codes before publishing a generated file.
- Find internal links. See on-page SEO.
Related: Google PageSpeed Insights, Largest Contentful Paint, WordPress SEO plugins, e-commerce SEO, what are backlinks, the best Chrome extensions for SEO and our SEO consulting service.
Frequently Asked Questions
How many URLs does the free version crawl?
The free version can check up to 500 URLs in one crawl. The licensed version is needed for larger sites and advanced configuration options.
Does Screaming Frog work like Googlebot?
It works like a search engine bot in following links, but it is not the real Googlebot and doesn’t reproduce Google’s indexing decisions.
Does it show rankings?
It is primarily a technical SEO and crawling tool, not a standard rank tracker. Search Console integration can bring average positions.
Can it do backlink analysis?
It can crawl a site’s internal and external links, but it isn’t designed as a backlink data tool for discovering backlinks across the web.
Why doesn’t it find all my pages?
Possible reasons: no internal link to the page, a robots.txt block, access only through a nofollow link, or content created by JavaScript while the crawl ran in text-only mode.
When should I use JavaScript mode?
If the page’s main content, links, title or canonical tags are created after JavaScript runs.
Can it harm my server?
Very high crawl speeds can load weak servers. Set the speed to match the site’s capacity.
Can it create an XML sitemap?
Yes, from properly crawled URLs, but check indexability, canonicals and status codes before publishing.
Can it crawl a mobile site?
Yes. Set the user-agent to Googlebot Smartphone or another mobile browser and adjust the rendering screen size.
