Search engines crawl websites to discover content and store it in databases — a process known as indexation. Once content has been indexed, it can be served up on search engine results pages (SERPs) for relevant search queries. Site audits are an essential tool to evaluate how easily a search engine can discover, crawl, and index everything from individual elements of a webpage all the way up to an entire domain.
This page is your home base for education on crawling and site audits. You'll find content here that's suited for learners of all skill levels.
The Free Technical SEO (and Beyond) Site Audit Checklist
In this step-by-step checklist, you'll learn the tools and tactics needed to pinpoint technical and on-page issues for your site and turn them into SEO wins.
The One-Hour Guide to SEO
Can you learn SEO in an hour? Surprisingly, the answer is yes, at least when it comes to the fundamentals! From SEO expert Rand Fishkin, we present you a six-part series of roughly ten-minute-long videos designed to deliver core SEO concepts.
20 Years in Search & I Don't Trust My Gut or Google
Presented by Wil Reynolds (he/him) at Mozcon 2019
What would your reaction be if you were told that one of Wil's clients got more conversions from zero-volume search terms than search terms with 1000+ searches per month? It's true. Wil found this out in seconds, leading him to really look at his whole client strategy through a new lens. It also made him question company-wide strategies. How prevalent is this across all clients? Don't they all deserve to get these insights? It required him to dig into the long tail, deep. To use big data and see PPC data as insights, not just marketing. What would your reaction be if you were told that Google's "bad click" business could be generating as much annually as Starbucks or McDonalds? Wil will be making the case for big data, agencies, and why building systems that looking at every single search term you get matched to is the future of search marketing.
What All Marketers Can Do about Site Speed
Presented by Emily Grossman at MozCon 2018
At this point, we should all have some idea of how important site speed is to our performance in search. The recently announced "speed update" underscored that fact yet again. It isn't always easy for marketers to know where to start improving their site's speed, though, and a lot of folks mistakenly believe that site speed should only be a developer's problem. Emily will clear that up with an actionable tour of just how much impact our own work can have on getting our sites to load quickly enough for today's standards.
Machine Learning for SEOs
Presented at MozCon 2018
People generally react to machine learning in one of two ways: either with a combination of fascination and terror brought on by the possibilities that lie ahead, or with looks of utter confusion and slight embarrassment at not really knowing much about it. With the advent of RankBrain, not even higher-ups at Google can tell us exactly how some things rank above others, and the impact of machine learning on SEO is only going to increase from here. Fear not: Moz's own senior SEO scientist, Britney Muller, will talk you through what you need to know.
Technical Site Audits for Local SEO
Presented by Lindsay Wassell at MozCon Local 2016
Onsite SEO success lays in the technical details, but extensive SEO audits can be too expensive and impractical. Lindsay shows you the most important onsite elements for local search optimization and outlines an efficient path for improved performance.
The Technical Site Audit Checklist
This checklist should help you put together a thorough site audit and determine what is holding back the organic performance of your site, so you can then take action to get the most SEO benefit for your efforts.
Winning Value Propositions for Crawlers and Consumers
Presented by Dawn Anderson at MozCon 2017
In an evolving mobile-first web, we can utilize pre-empting solutions to create winning value propositions, which are designed to attract and satisfy search engine crawlers and keep consumers happy. I'll outline a strategy and share tactics that help ensure increased organic reach, in addition to highlighting smart ways to view data, intent, consumer choice theory and crawl optimization.
5 Secrets: How to Execute Lean SEO to Increase Qualified Leads
Presented at MozCon 2017
I invite you to steal some of the ideas I've gleaned from managing SEO for the behemoth bad-ass Moz.com. Learn what it takes to move the needle on qualified leads, execute quick wins, and keep your head above water. I will go over my biggest Moz.com successes, failures, tests, and lessons.
The Truth About Mobile-First Indexing
Presented by Cindy Krum at MozCon 2017
Mobile-first design has been a best practice for a while, and Google is finally about to support it with mobile-first indexing. Learn how mobile-first indexing will give digital marketers their first real swing at influencing Google’s new AI (Artificial Intelligence) landscape. Marketers who embrace an accurate understanding of mobile-first indexing could see a huge first-mover advantage, similar to the early days of the web, and we all need to be prepared.
Facing the Future: 5 Simple Tactics for 5 Scary Changes
Presented by Dr. Pete Meyers at MozCon 2017
We've seen big changes to SEO recently, from an explosion in SERP features to RankBrain to voice search. These fundamental changes to organic search marketing can be daunting, and it's hard to know where to get started. Dr. Pete will walk you through five big changes and five tactics for coping with those changes today.
A Site Migration: Redirects, Resources & Reflection
Presented by Jayna Grassel at MozCon 2017
Site. Migration. No two words elicit more fear, joy, or excitement to a digital marketer. When the idea was shared three years ago, the company was excited. They dreamed of new features and efficiency. But as SEOs we knew better. We knew there would be midnight strategy sessions with IT. More UAT environments than we could track. Deadlines, requirements, and compromises forged through hallway chats. ... The result was a stable transition with minimal dips in traffic. What we didn't know, however, was the amount of cross-functional coordination that was required to pull it off. Learn more in this video!
Faster Sites: Beyond PageSpeed Insights
PageSpeed Insights can be useful, but a perfect score doesn’t guarantee a fast site. There are smarter ways to assess and improve site speed.
Pop-Ups, Overlays, Modals, Interstitials, and How They Interact with SEO
This week we're chatting about pop-ups, overlays, modals, interstitials, etc. They have specific kinds of interactions with SEO. In addition to Google's guidelines around them, they also change how people interact with your site, which can adversely or positively affect you accomplishing your goals, SEO and otherwise.
Robots Meta Directives
Whereas robots.txt file directives give bots suggestions for how to crawl a website's pages, robots meta directives provide more firm instructions on how to crawl and index a page's content. Learn more about what robots meta directives are all about and how to use them in this article.
Launching a New Website: Your SEO Checklist
Hovering your finger over the big red "launch" button for your new website? Hold off for just a second. From a keyword-to-URL content map to sweeping for crawl errors to setting up proper tracking, today's Whiteboard Friday covers 5 boxes you need to check off before finally setting your site live.
How to Do a Content Audit [Updated for 2017]
Learn how to do content audits for SEO in this Moz guide by Everett Sizemore, including tips for crawling large websites, rendering JavaScript content, and auditing dynamic mobile content.
Google Search Operators
Google search operators are special characters and commands (sometimes called “advanced operators”) that extend the capabilities of regular text searches. Search operators can be useful for everything from content research to technical SEO audits. Learn more about them in this article.
Schema.org Markup
Learn what schema.org structured data is and why it matters for search engine optimization.
Robots.txt
What is robots.txt, and how does it impact a website's search visibility? How do you create SEO-friendly robots.txt files? Learn in this article.
HTTP Status Codes
Hypertext Transfer Protocol (or HTTP) response status codes are returned whenever search engines or website visitors make a request to your server. This article covers the SEO best practices for using and interpreting these status codes. Dive in!
Domains
Domain names are the unique, human-readable Internet addresses of websites. Learn about the different types of domains and SEO best practices for optimizing domains in this article.
URLs
A URL (Uniform Resource Locator), more commonly known as a "web address", specifies the location of a resource (such as a web page) on the internet. Learn how URLs work and why they matter in this article.
Canonicalization
What is canonicalization? How do you implement the canonical tag on a webpage, and why does it matter for search engine optimization? Learn in this article.
Redirects
Redirection isn't just a matter of using the right HTTP status code. Learn about the SEO implications of redirection and best practices for success in this article.
I've Optimized My Site, But I'm Still Not Ranking—Help!
When all your hard work seems to be going nowhere, there might be a reason. Jo Cameron walks you through a list of things to check and accomplish to make sure your site can start ranking ASAP.
Server Log Files & Technical SEO Audits: What You Need to Know
Presented by Samuel Scott at MozCon 2016
Server log files contain the only data that is 100% accurate in terms of how Google and other search engines crawl your website. Sam will show you what and where to check and what problems you may to need to fix to maximize your rankings and organic traffic.
Be Your Site's Hero: An Audit Manifesto – Next Level
SEO comes with a lot of "whys" that can be difficult to answer. Jo Cameron is here to remove some of the mystery and level up your site audit skills with the help of ace tips, Bonnie Tyler, and a hefty toolkit in the newest installment of our Next Level educational series!
Controlling Search Engine Crawlers for Better Indexation and Rankings
Robots.txt, meta robots, and the nofollow tag are all ways to tell search engines how to crawl and index your site. In today's Whiteboard Friday, Rand covers four common use cases of these tools, offering guidance on when you should use one over another.
Should I Use Relative or Absolute URLs?
Should we use relative or absolute URLs on our sites? It's a simple question with a nuanced answer, and in today's Whiteboard Friday, Ruth Burr Reedy guides us through.
How to Perform the Ultimate Local SEO Audit
Are you looking to dominate local search? This guide is all you will need to learn how to perform The Ultimate Local SEO Audit for your business or your clients. Learn what key areas you need to pay attention to, which ones carry the most weight, and determine what you need to do to start beating your competition today.
Link Audit Guide for Effective Link Removals & Risk Mitigation
This step-by-step guide aims to help users with the link auditing process relying on own judgment, without blindly relying on automation. Because links are still a very important ranking factor, link audits should be carried out by experienced link auditors rather than third party automated services. A flawed link audit can have detrimental implications.
How to Perform the World's Greatest SEO Audit
Now that tax season is over, it's once again safe to say my favorite A-word... audit! That's right. My name is Steve, and I'm an SEO audit junkie.
An SEO's Guide to HTTP Status Codes (An Infographic)
A visual guide to the HTTP status codes that really matter to SEO (200, 301, 302, 404, 501, 503). Half infographic, half cheat-sheet, half man, half bear, half pig.
Technical SEO Certification
Looking to level up your technical skills?
The Technical SEO Certification combines video lessons with tasks and knowledge to level up your technical SEO know-how. Covering the fundamentals, crawling, indexing, accessibility, and site performance, you’ll receive three hours of on-demand instructor-led content. Following the completion of a final exam at the end of the series, you’ll be awarded a certificate and a shiny new badge to share with your professional network.
Super-Practical Tips for Improving Your Site's E-A-T
Presented by Dr. Marie Haynes at Mozcon 2019
Google has admitted that they measure the concept of "Expertise, Authoritativeness, and Trustworthiness" in their algorithms. If your site is categorized under YMYL (Your Money or Your Life), you absolutely must have good E-A-T in order to rank well. In this talk, you'll learn how Google measures E-A-T and what changes you can make both on site and off in order to outrank your competitors. Using real-life examples, Marie will answer what E-A-T is and how Google measures it, what changes you can make on your site to improve how E-A-T is displayed, and what you can do off-site to improve E-A-T.
Fixing the Indexability Challenge: A Data-Based Framework
Presented by Areej AbuAli (she/her) at Mozcon 2019
How do you turn an unwieldy 2.5 million-URL website into a manageable and indexable site of just 20,000 pages? Areej will share the methodology and takeaways used to restructure a job aggregator site which, like many large websites, had huge problems with indexability and the rules used to direct robot crawl. This talk will tackle tough crawling and indexing issues, diving into the case study with flow charts to explain the full approach and how to implement it.
How to Audit for Inclusive Content
Presented by Emily Triplett Lentz at Mozcon 2019
Digital marketers have a responsibility to learn to spot the biases that frequently find their way into online copy, replacing them with alternatives that lead to stronger, clearer messaging and that cultivate wider, more loyal and enthusiastic audiences. Last year, Help Scout audited several years of content for unintentionally exclusionary language that associated physical disabilities or mental illness with negative-sounding terms, resulting in improved writing clarity and a stronger brand. You'll learn what inclusive content is, how it helps to engage a larger and more loyal audience, how to conduct an audit of potentially problematic language on a site, and how to optimize for inclusive, welcoming language.
Killer E-commerce CRO and UX Wins Using an SEO Crawler
Presented by Luke Carthy at Mozcon 2019
Flex those note-taking muscles, because you'll need them for this high-powered talk! Luke will share solid, profitable, and actionable tips on just how to identify juicy e-commerce revenue wins using the unlikely duo of an SEO crawler and custom extraction. You'll learn how to uncover impactful low-hanging fruit to both increase your conversions and improve your UX using the tool that's already in every SEO's arsenal. You'll also glean in-depth insight into some of the world's biggest names in e-commerce, sharing major missed CRO opportunities, costly oversights, and rookie mistakes hidden in plain sight.
How to Kickstart an SEO Audit for Your Startup
It's incredibly important to consider SEO when launching a startup. In this Whiteboard Friday, Rand shares how to kickstart a comprehensive SEO audit for your new business's site.
The What, Why, and How of Pruning Your Website for SEO
Pruning or merging pages on your website is proving to be a valuable practice post-Google Hummingbird. In this webinar, Bill Sebald explains how this exercise can improve your traffic and boost your conversions, and shares the steps for performing it yourself.
10 Link Removal Pro-Tips That Will Change Your Life
Presented by Sha Menz at MozCon 2013
Many people think link removal is tedious and thankless work, high on effort and low on results. Sha will change your view with what she's learned while working with hundreds of clients. Laced with a heavy dose of TAGFEE, every one of these highly actionable tips will help you eliminate wasted effort, improve cleanup rates, and even generate new business and new friends while running your link removal campaigns.
Web Spam Research: Good Robots vs Bad Robots
Presented by Matt Peters at MozCon 2012
Matt goes on a wild ride to find the interesting features necessary to algorithmically classify a site as spam or non-spam. Will the good robots finally win?
MozCon Video Bundle
Did you miss this year's MozCon? Now is your chance to experience what everyone has been talking about! We have over 19 hours of actionable inbound marketing knowledge.
JavaScript & SEO: Making Your Bot Experience As Good As Your User Experience
More and more, we're realizing it's incredibly important for us as SEOs to understand JavaScript's impact on search experience. Can search engines see your content and experience your site the way a user does? If not, what solutions can you use to fix it?
The Mobile SEO Stack: Tools to Develop a Mobile-First SEO Process
With mobile usage on the rise, it's more important than ever to optimize for on-the-go users. But which tools support mobile SEO? Aleyda's compiled 28 tools that will help you answer all your mobile SEO questions, plus a handy graphic to download for at-a-glance reference.
XML Sitemaps: The Most Misunderstood Tool in the SEO's Toolbox
XML sitemaps are a powerful tool for SEOs, but are often misunderstood and misused. Michael Cottam explains how to leverage XML sitemaps to identify and resolve indexation problems.
Mastering Google Search Operators in 67 Easy Steps
Google search operators are like chess – knowing how the pieces move doesn't make you a master. Dive into 67 examples, from content research to site audits, and level up your search operator game.
How to Find and Fix 14 Technical SEO Problems That Can Be Damaging Your Site Now
Are there technical issues plaguing your site beneath the surface? Learn about some of the most common and damaging technical SEO problems and how to begin fixing them.
How to Fix Crawl Errors in Google Search Console
Search Console is one of the most powerful tools we have for diagnosing website errors. Learn how to prioritize and resolve your site's crawl errors in Google Search Console — those hundreds of 404 errors may not be so daunting after all.
Why is my redirected page not appearing in Google?
Every time this website owner 301-redirects their site to a new location, traffic disappears almost completely. This Q&A post troubleshoots the cause of these traffic drops and looks at a few potential solutions.
How do I noindex duplicate (non-unique) content?
How does "noindexing" a page impact how duplicate content is crawled? This Q&A post explores hypotheses and a few other solutions to preventing content from appearing in Google's search index.
How to implement pagination and other search parameters
Using pagination in conjunction with other URL parameters can be confusing. Check out this Q&A post for a discussion of how to best implement both.
Why are my pages cached but not indexed by Google?
Why might Google not be caching pages that they've previously indexed? Explore a few potential reasons in this post from the Moz Community Q&A.
Blocking blog links from being crawled
Should website owners block blog category and author links from being crawled? Why or why not? Learn more in this Q&A post.
How to block "print" pages from indexing
This Q&A post explores techniques for preventing "print" pages from being indexed by search engines.
Are pages with a canonical tag indexed?
How does using the canonical tag impact indexation and search visibility? This Q&A post explores how search engines interpret the use of this tag on a webpage.
How does a search engine bot navigate past a .PDF link?
Can search engines crawl PDFs? How can website owners get the most SEO value out of linked-to .PDF files? This Q&A post looks at a few hypotheses and solutions.