The launch that disappeared from Google
A Thiruvananthapuram interior design firm launched a new website before the Onam season. Two weeks later, enquiries from Google had stopped completely.
The developer had built the site on a staging copy with one line in robots.txt: Disallow: /. It kept Google out while the site was being built. Nobody removed it on launch day.
Search Console showed the home page as “Blocked by robots.txt”. One deleted line and a recrawl request brought the pages back over the next few days, but the firm lost its busiest fortnight.
Now every launch checklist we use has the same first line: open yoursite.in/robots.txt and read it.
Illustrative example based on a common pattern in Kerala e-commerce accounts. Numbers rounded.
What robots.txt does
Every well-behaved crawler, from Googlebot to ChatGPT’s search bot, reads yoursite.in/robots.txt before it visits other pages. The file lists user agents (which crawler a rule is for), Disallow lines (paths it may not visit), Allow lines (exceptions) and the location of your sitemap.
Two things trip people up. First, robots.txt stops crawling, not indexing: a blocked page can still show in Google as a bare link if other sites link to it. Second, it is public and voluntary. Anyone can read it, and bad bots ignore it, so never use it to hide private pages.
User-agent: * · Disallow: /wp-admin/ · Allow: /wp-admin/admin-ajax.php · Sitemap: https://www.yoursite.in/sitemap_index.xml
Create and test your robots.txt
3. Your robots.txt
User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Disallow: /?s= Disallow: /search/ Sitemap: https://www.yoursite.in/sitemap_index.xml
- Yoast and Rank Math sitemaps live at /sitemap_index.xml. If you use another plugin, change the Sitemap line to match.
4. Test a URL
Blocked: this crawler may not visit the page.Matched rule: Disallow: /wp-admin/
A typical WordPress robots.txt, line by line
| Line | What it does |
|---|---|
| User-agent: * | The rules below apply to every crawler. |
| Disallow: /wp-admin/ | Keep crawlers out of the admin area. |
| Allow: /wp-admin/admin-ajax.php | Exception: some themes load content through this file, so Google must reach it. |
| Disallow: /?s= | Block internal search results, which create endless thin pages. |
| Sitemap: https://www.yoursite.in/sitemap_index.xml | Tell crawlers where your list of pages is. Must be a full URL. |
When to use robots.txt (and when not to)
1. Stop crawlers wasting time on endless URLs
Internal search results, filter combinations (?color=red&size=m) and calendar pages can create thousands of near-empty URLs. Blocking them helps Google spend its time on pages that matter. This is worth doing on stores with large catalogues; a 30-page site rarely needs it.
2. Keep a staging site out of Google
Disallow: / on a staging or test copy keeps crawlers out while you build. Password protection is safer still. Remove both on launch day and check yoursite.in/robots.txt within an hour of going live.
3. Decide what AI crawlers can read
You can block AI training bots such as GPTBot and Google-Extended while still allowing search bots such as OAI-SearchBot and PerplexityBot, so AI assistants can cite and link to you. For most businesses that want leads, allowing AI search bots is the right call.
4. Don’t use it to remove pages from Google
If a thank-you page, an old offer or a duplicate page is showing in Google, add noindex and leave it crawlable. Blocking it in robots.txt stops Google seeing the noindex, so the page can stay in results.
Robots.txt vs noindex vs password
| Tool | Use it for | Why |
|---|---|---|
| robots.txt Disallow | Admin areas, internal search, cart and checkout, endless filter URLs | Saves crawl time. Does not guarantee the URL stays out of search. |
| noindex meta tag | Thank-you pages, thin tag pages, duplicate landing pages | Removes the page from results. Google must be able to crawl it to see the tag. |
| Password or login | Staging sites, client portals, anything private | The only option that keeps people and bots out. |
How to check it is working
- Open the file yourself
- Visit yoursite.in/robots.txt in a browser. It must load at the root of the domain (not in a folder) and return a normal page, not an error.
- Search Console robots.txt report
- In Search Console, go to Settings → robots.txt to see the version Google last fetched, when, and any errors. You can ask Google to recrawl it after a fix.
- URL Inspection on important pages
- Inspect your home page and top service pages. “Crawl allowed? Yes” is what you want. “Blocked by robots.txt” on a page that should rank needs fixing today.
- Page indexing report
- Check Pages → “Blocked by robots.txt” and “Indexed, though blocked by robots.txt”. The second one usually means you blocked a page you should have noindexed instead.
Mistakes to avoid
Leaving Disallow: / after launch
The whole site drops out of Google within days. The most expensive one-line mistake in SEO.
Blocking a page to remove it from Google
Google can’t see the noindex, so the page stays in results as “Indexed, though blocked by robots.txt”.
Blocking CSS, JavaScript or /wp-content/
Google can’t render the page properly and may judge it as broken or not mobile-friendly.
Listing private URLs in robots.txt
The file is public, so you are handing out a map of what you want to hide. Use a password instead.
Blocking all AI bots without deciding why
ChatGPT, Perplexity and Claude can’t read your pages to recommend you, while competitors stay visible.
Related terms
Questions people ask about robots.txt
Where is the robots.txt file?
Always at the root of the domain: https://www.yoursite.in/robots.txt. Each subdomain needs its own, so blog.yoursite.in/robots.txt is a separate file.
Does robots.txt stop a page from appearing in Google?
No. It stops Google crawling the page, but the URL can still appear if other pages link to it. Use a noindex tag, and keep the page crawlable so Google can see it.
How do I edit robots.txt in WordPress?
Yoast SEO: Yoast → Tools → File editor. Rank Math: Rank Math → General Settings → Edit robots.txt. Also check Settings → Reading and make sure “Discourage search engines from indexing this site” is not ticked.
Can I edit robots.txt in Shopify?
Yes, through the robots.txt.liquid theme template, but Shopify’s default file already blocks cart, checkout and admin pages. Most stores don’t need to change it.
Should I block GPTBot and other AI crawlers?
It depends on your goal. Blocking GPTBot or Google-Extended stops your content being used for AI training. Blocking AI search bots such as OAI-SearchBot or PerplexityBot also stops those assistants citing and linking to you. Most lead-focused businesses allow the search bots.
Does Google follow Crawl-delay?
No. Google ignores Crawl-delay. If Googlebot is overloading your server, fix hosting or caching first; Bing and some other bots do respect it.
Put it into practice
- SOP 209: Find and Fix Page Indexing Issues in Search Console →
- SOP 207: Find Your XML Sitemap and Submit It to Google Search Console →
Prefer to talk first? Book a free counselling call.