myenglishguide.com
robots.txt

Robots Exclusion Standard data for myenglishguide.com

Resource Scan

Scan Details

Site Domain myenglishguide.com
Base Domain myenglishguide.com
Scan Status Ok
Last Scan2024-06-09T22:10:15+00:00
Next Scan 2024-06-16T22:10:15+00:00

Last Scan

Scanned2024-06-09T22:10:15+00:00
URL https://myenglishguide.com/robots.txt
Domain IPs 104.21.0.179, 172.67.128.43, 2606:4700:3032::ac43:802b, 2606:4700:3033::6815:b3
Response IP 104.21.0.179
Found Yes
Hash 6036370a66610720608d1548500e67526f1c865fe888bf5cdf591d6f9c3cec96
SimHash 6149838147b1

Groups

*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

mediapartners-google

Rule Path
Disallow

googlebot

Rule Path
Disallow

adsbot-google

Rule Path
Disallow

googlebot-image

Rule Path
Disallow

googlebot-mobile

Rule Path
Disallow

msnbot

Rule Path
Disallow

bingbot

Rule Path
Disallow

slurp

Rule Path
Disallow

yahoo! slurp

Rule Path
Disallow

yandexbot

Rule Path
Disallow

*

Rule Path
Disallow /

*

Rule Path
Disallow /wp-admin/

mj12bot

Rule Path
Disallow /

Comments

  • Allow only major search spiders
  • Block all other spiders
  • Block Directories for all spiders