Log In Sign Up

tibetsun.com
robots.txt

Robots Exclusion Standard data for tibetsun.com

Resource Scan

Scan Details

Site Domain tibetsun.com
Base Domain tibetsun.com
Scan Status Ok
Last Scan2025-06-11T20:29:12+00:00
Next Scan 2025-07-11T20:29:12+00:00

Last Scan

Scanned2025-06-11T20:29:12+00:00
URL https://tibetsun.com/robots.txt
Redirect https://www.tibetsun.com/robots.txt
Redirect Domain www.tibetsun.com
Redirect Base tibetsun.com
Domain IPs 138.197.133.2
Redirect IPs 138.197.133.2
Response IP 138.197.133.2
Found Yes
Hash 446bdc7125179f8b8cb8b23978feeba39942b66add8a39ad9a18f8598fbe04d1
SimHash 32061971cf23

Groups

*

Rule Path
Disallow /cgi-bin
Disallow /

chatgpt

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

aspiegelbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

obot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /
Back to top

Comments

  • robots.txt for TIBETSUN.COM
  • 11 aug 2023 - Made it for tibetsun.com, copy of lw file.
  • 15 nov 2021 - edit and add to exiletibetans.com
  • 19 may 2021 - add bots found in awstats.
  • 1 apr 2021 - the bots.
  • Info at http://www.robotstxt.org
  • -- ChatGPT
  • -- Common Crawl
  • Bad bots:
  • -- Facebook:
  • you may want to allow the facebook crawler if your site is active on fb.
  • %% facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
  • %% Disallow: /
  • %% facebookexternalhit/1.1
  • %% Disallow: /
  • %% Facebook
  • %% Disallow: /
  • -- Show location of site map:
  • %% Sitemap: https://www.DOMAIN.TLD/my-sitemap.xml
  • -- E O F
Back to top

Warnings

  • 3 invalid lines.
Back to top