A noindex rule asks supporting search engines to exclude a resource from their index. First decide whether the URL should appear in search. Then inspect the actual response, rather than relying on a CMS checkbox or a successful deployment.
01Choose the intended audience
Build a short inventory of affected URLs and explain why each should be excluded or included. A campaign thank-you page, a public article and a password-protected preview have different purposes. Noindex is not security: a person with the address may still access public content. Put confidential material behind appropriate authentication.
- Record the page purpose
- Assign the content owner
- Use access controls for private information
02Inspect both places a rule can appear
For HTML, inspect the head for a robots or Googlebot meta tag containing noindex. Also check the HTTP response for X-Robots-Tag, especially for a PDF or another non-HTML resource. An apparently clean template can still receive an exclusion header from a CDN or hosting rule.
- Save the exact URL and time
- Inspect the final response after redirects
- Check HTML and response headers together
03Keep the rule visible to the crawler
Google needs to crawl a resource to discover its noindex instruction. Blocking the same URL in robots.txt can prevent that discovery. Do not add a noindex line to robots.txt and assume it is supported. Review the crawl rule and indexing rule as separate settings with separate effects.
- Inspect the applicable robots.txt rules
- Locate the setting that generated the directive
- Avoid conflicting template and edge configurations
04Trace unexpected exclusions to their owner
If a production article unexpectedly contains noindex, compare a second article, the preview deployment and the published template. This can reveal whether the problem affects one page or an entire route family. Record the smallest reproducible case before changing a global rule that may intentionally protect other environments.
- Compare one affected and one unaffected URL
- Separate preview from production settings
- Name the application or infrastructure owner
05Verify the change at the public URL
After the responsible team changes the setting, request the public URL again and save its headers and initial HTML. Confirm that intended exclusions remain on the pages that need them. A build artifact alone cannot show what a production proxy adds; the published response is the release acceptance check.
- Check more than the edited template
- Verify assets such as PDFs separately
- Record the commit or configuration version
06Track search status separately
With authorized Search Console access, inspect the URL and compare the last crawl with the release date. Removing an accidental noindex makes a page eligible for further consideration; it does not establish that indexing has occurred. Keep the technical fix, observed crawl and index status as separate entries in the record.
- Retain before-and-after evidence
- Do not promise an indexing deadline
- Recheck using the agreed measurement access