developers.google.com/search/docs/crawling-indexing/robots/intro

Robots.txt Introduction and Guide | Google Search Central | Documentation | Goog…

Robots.txt is used to manage crawler traffic. Explore this robots.txt introduction guide to learn what robot.txt files are and how to use them.