Log In Sign Up

avav.jp
robots.txt

Robots Exclusion Standard data for avav.jp

Resource Scan

Scan Details

Site Domain avav.jp
Base Domain avav.jp
Scan Status Ok
Last Scan2024-09-19T20:16:40+00:00
Next Scan 2024-10-19T20:16:40+00:00

Last Scan

Scanned2024-09-19T20:16:40+00:00
URL https://avav.jp/robots.txt
Domain IPs 13.58.161.171, 3.134.208.171, 3.139.14.167
Response IP 13.58.161.171
Found Yes
Hash 239defc390d6de63fa2f95ce5c7b4e7db920176d18c49f2e8498e7bac32ca01a
SimHash 321f4d81e058

Groups

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

bytespider

Rule Path
Disallow /

applebot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

amazonbot/0.1

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

blexbot
blexbot/1.0

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

serpstatbot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://avav.jp/sitemap.xml.gz
Back to top

Comments

  • See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file
  • To ban all spiders from the entire site uncomment the next two lines:
  • User-agent: *
  • Disallow: /
  • User-agent: baiduspider
  • Disallow: /
  • Huawei
  • Serpstat
Back to top