What is Robots.txt?
Robots.txt is a text file at a website's root directory that instructs search engine crawlers which pages or sections they are allowed or not allowed to visit.
At a Glance
Robots.txt
Robots.txt is a text file at a website's root directory that instructs search engine crawlers which pages or sections they are allowed or not allowed to visit.
Full Definition
Every website can include a robots.txt file at domain.com/robots.txt. Directives like Disallow: /inventory/sold/ tell Googlebot to skip those URLs, preserving crawl budget for valuable pages. Conversely, an overly aggressive robots.txt can accidentally block entire sections of a site from being crawled. This file is public, so competitors (and anyone curious) can read your crawl configuration at any time.
Why It Matters for Your Dealership
A misconfigured robots.txt can block Google from crawling your service pages or model landing pages, removing your most important content from search results entirely. This is one of the more common and harder-to-diagnose causes of unexpected traffic drops at dealerships.
In Practice
A misconfigured robots.txt can block Google from crawling your service pages or model landing pages, removing your most important content from search results entirely.
See How Your Dealership Ranks.
Get a free Competitor DNA Report showing how your store compares across Google and AI search.