How to check robots.txt and noindex before launch

Preview settings can accidentally remain in production. Check crawling and indexing controls before launch, and keep them distinct from access protection.

Review the public robots file

Ask the developer to inspect the deployed robots.txt, not only the source copy. Look for a broad rule left from development that prevents access to public pages or important resources. Confirm the sitemap address is correct. Google's documentation describes robots.txt as a crawler-access control, not a reliable way to keep a page out of search results.

Inspect page-level restrictions

Check the robots meta tag and relevant response headers on the homepage, service pages and articles. A noindex rule can be useful for intentionally excluded pages, but should not remain on content intended for search. Record the reason for every deliberate exclusion.

Understand the interaction

Google must be able to crawl a page to see its noindex rule. Blocking the same page in robots.txt can prevent that. A robots file also does not hide private information from a person who knows the address. Confidential content needs actual access protection.

Keep a production checklist

Use columns for URL or rule, purpose, expected state and observed state. Example: “Checkout draft; exclude from search; noindex; confirmed.” Compare public content with the sitemap so an excluded page is not accidentally advertised as a preferred indexable URL.

Recheck after deployment changes

Some platforms set headers by environment or hostname. Verify the actual live response with appropriate tools rather than assuming a local preview has the same rules. If Search Console is available to an authorised owner later, its inspection tools can help investigate how Google sees the page.

Only mark controls as verified when they have been observed in the relevant environment. A successful browser render is not proof that Google can crawl or index it. Keep the checked states with the launch record so a future deployment can be compared with the intended configuration.

Practical guidance with illustrative examples. Read our editorial standards or send a correction.