Robots.txt

robots.txt vs noindex

robots.txt prevents crawlers from visiting a URL. The noindex meta tag, placed in the page's HTML, allows visiting but prevents indexing. For pages with sensitive content you want kept private, use both. For pages that can be visited but shouldn't appear in search results, use noindex alone.