Robots.txt
The robots.txt file, placed at the domain root, tells crawling robots which areas of the site are open or closed to them. It governs crawling, not indexation — a distinction that matters more than most people assume.
In practice
Blocking a URL from crawling does not prevent it from being indexed if links point to it. To remove a page from results, noindex is the correct directive. Since 2023, the file also regulates AI data collectors — GPTBot, ClaudeBot, and others — which each site owner can admit or exclude based on AI visibility strategy.