Log In Sign Up

charliekirkshow.com
robots.txt

Robots Exclusion Standard data for charliekirkshow.com

Resource Scan

Scan Details

Site Domain charliekirkshow.com
Base Domain charliekirkshow.com
Scan Status Ok
Last Scan2024-06-27T00:18:14+00:00
Next Scan 2024-07-04T00:18:14+00:00

Last Scan

Scanned2024-06-27T00:18:14+00:00
URL https://charliekirkshow.com/robots.txt
Redirect https://thecharliekirkshow.com/robots.txt
Redirect Domain thecharliekirkshow.com
Redirect Base thecharliekirkshow.com
Domain IPs 104.21.72.76, 172.67.176.101, 2606:4700:3034::ac43:b065, 2606:4700:3037::6815:484c
Redirect IPs 104.26.0.62, 104.26.1.62, 172.67.73.177, 2606:4700:20::681a:13e, 2606:4700:20::681a:3e, 2606:4700:20::ac43:49b1
Response IP 172.67.73.177
Found Yes
Hash 9eb6da8f847fba5be5b2921fb282a62fcb9e475b2273c305310b02b30b08180b
SimHash 303cd090c0b5

Groups

googlebot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

twitterbot

Rule Path
Disallow

mj12bot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

architextspider

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

slurp.so/1.0

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

slurp/2.0j

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

slurp/2.0-kite-hourly

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

slurp/2.0-owlweekly

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

slurp/3.0-au

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

teoma

Product Comment
teoma ASK.com

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

scooter

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

lycos_spider_(t-rex)

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

infoseek sidewinder/9.0

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

*

Rule Path Comment
Disallow /newsletter/ disallow this directory

Other Records

Field Value
crawl-delay 5

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /
Back to top

Comments

  • User-agent: *
  • Disallow: /
  • User-Agent: SemrushBot
  • Disallow: /
  • Disallow: /
  • AI Bots
  • OpenAI bots
  • Google AI bots - Bard, Gemini and VertexAI
  • commoncrawl AI
  • Perplexity AI
  • End AI Bots
Back to top