Log In Sign Up

thefishtwincities.com
robots.txt

Robots Exclusion Standard data for thefishtwincities.com

Resource Scan

Scan Details

Site Domain thefishtwincities.com
Base Domain thefishtwincities.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan2024-11-01T08:47:53+00:00
Next Scan 2024-12-01T08:47:53+00:00

Last Successful Scan

Scanned2024-10-03T08:12:54+00:00
URL https://thefishtwincities.com/robots.txt
Domain IPs 104.21.31.78, 172.67.175.72, 2606:4700:3036::6815:1f4e, 2606:4700:3037::ac43:af48
Response IP 172.67.175.72
Found Yes
Hash 9eb6da8f847fba5be5b2921fb282a62fcb9e475b2273c305310b02b30b08180b
SimHash 303cd090c0b5

Groups

googlebot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

twitterbot

Rule Path
Disallow

mj12bot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

architextspider

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

slurp.so/1.0

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

slurp/2.0j

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

slurp/2.0-kite-hourly

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

slurp/2.0-owlweekly

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

slurp/3.0-au

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

teoma

Product Comment
teoma ASK.com

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

scooter

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

lycos_spider_(t-rex)

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

infoseek sidewinder/9.0

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

*

Rule Path Comment
Disallow /newsletter/ disallow this directory

Other Records

Field Value
crawl-delay 5

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /
Back to top

Comments

  • User-agent: *
  • Disallow: /
  • User-Agent: SemrushBot
  • Disallow: /
  • Disallow: /
  • AI Bots
  • OpenAI bots
  • Google AI bots - Bard, Gemini and VertexAI
  • commoncrawl AI
  • Perplexity AI
  • End AI Bots
Back to top