Log In Sign Up

petvblog.com
robots.txt

Robots Exclusion Standard data for petvblog.com

Resource Scan

Scan Details

Site Domain petvblog.com
Base Domain petvblog.com
Scan Status Ok
Last Scan2025-08-13T11:53:18+00:00
Next Scan 2025-08-20T11:53:18+00:00

Last Scan

Scanned2025-08-13T11:53:18+00:00
URL https://petvblog.com/robots.txt
Domain IPs 194.1.147.37, 194.1.147.53
Response IP 194.1.147.53
Found Yes
Hash c41d456c46f7793773549fee7de66856ffe2060f83235e18008003df9a3bce01
SimHash 65348a5777c1

Groups

*

Rule Path
Disallow
Disallow /cgi-bin/
Disallow /tmp/
Disallow /private/
Disallow /admin/
Disallow /scripts/
Disallow /backend/
Disallow /secret.html
Disallow /login.html
Disallow /register.html

googlebot-image

Rule Path
Disallow

mediapartners-google

Rule Path
Disallow

googlebot

Rule Path
Disallow

bingbot

Rule Path
Disallow

badbot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.petvblog.com/sitemaps.xml
Back to top

Comments

  • robots.txt file for petvblog.com
  • Author: Your Name
  • Date: 2024-06-03
  • Allow all search engines full access to blog content
  • Block specific folders not intended for public access
  • Block specific files not intended for public access
  • Allow full access to specific bots for images and ads
  • Sitemap location
  • Crawl-delay directive for all bots (remove or update value if necessary)
  • Crawl-delay: 10
  • Specific directives for major search engines
  • Block specific user agents if needed
Back to top