Log In Sign Up

serpentstail.com
robots.txt

Robots Exclusion Standard data for serpentstail.com

Resource Scan

Scan Details

Site Domain serpentstail.com
Base Domain serpentstail.com
Scan Status Ok
Last Scan2025-06-18T17:21:55+00:00
Next Scan 2025-07-18T17:21:55+00:00

Last Scan

Scanned2025-06-18T17:21:55+00:00
URL https://serpentstail.com/robots.txt
Domain IPs 104.21.112.1, 104.21.16.1, 104.21.32.1, 104.21.48.1, 104.21.64.1, 104.21.80.1, 104.21.96.1, 2606:4700:3030::6815:1001, 2606:4700:3030::6815:2001, 2606:4700:3030::6815:3001, 2606:4700:3030::6815:4001, 2606:4700:3030::6815:5001, 2606:4700:3030::6815:6001, 2606:4700:3030::6815:7001
Response IP 104.21.96.1
Found Yes
Hash 03e413dabba9b18b389f3f4f559626c53b640c4e423e3fab2f6624ed57c8adb9
SimHash 5c32c9d2d7b1

Groups

*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 7

*

Rule Path
Disallow /wp-admin/

gptbot
ccbot
chatgpt-user

Rule Path
Disallow /

tinytestbot
semrushbot
ahrefsbot
petalbot
mj12bot
dotbot
mauibot
yandexbot
baiduspider
barkrowler
bytespider
whatstuffwherebot
applebot
sogou pic spider/3.0( http://www.sogou.com/docs/help/webmasters.htm
sogou head spider/3.0( http://www.sogou.com/docs/help/webmasters.htm
sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm
user-agent: sogou orion spider/3.0( http://www.sogou.com/docs/help/webmasters.htm
sogou-test-spider/4.0 (compatible; msie 5.5; windows 98)
mozilla/5.0 (compatible; konqueror/3.5; linux) khtml/3.5.5 (like gecko) (exabot-thumbnails)
mozilla/5.0 (compatible; exabot/3.0; +http://www.exabot.com/go/robot)
swiftbot
slurp
ccbot/2.0 (https://commoncrawl.org/faq/)
ccbot/2.0
ccbot/2.0 (http://commoncrawl.org/faq/)

Product Comment
sogou pic spider/3.0( http://www.sogou.com/docs/help/webmasters.htm 07)
sogou head spider/3.0( http://www.sogou.com/docs/help/webmasters.htm 07)
sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm 07)
user-agent: sogou orion spider/3.0( http://www.sogou.com/docs/help/webmasters.htm 07)
Rule Path
Disallow /

googlebot
bingbot
duckduckbot

Rule Path
Disallow
Back to top

Comments

  • START BOOKSWARM ROBOTS.TXT TEMPLATE
  • -----------------------------------
  • Set the crawl delay to 7 seconds - not all search engines will honour this
  • Tell all user agents to ignore wp-admin
  • Block AI bots
  • Block these other bots
  • Allow Googlebot and other specific bots
  • ---------------------------------
  • END BOOKSWARM ROBOTS.TXT TEMPLATE
  • _
  • [ ]
  • ( )
  • |>|
  • __/===\__
  • //| o=o |\\
  • <] | o=o | [>
  • \=====/
  • / / | \ \
  • <_________>
Back to top