Log In Sign Up

webpro.in
robots.txt

Robots Exclusion Standard data for webpro.in

Resource Scan

Scan Details

Site Domain webpro.in
Base Domain webpro.in
Scan Status Ok
Last Scan2025-11-25T18:45:42+00:00
Next Scan 2025-12-02T18:45:42+00:00

Last Scan

Scanned2025-11-25T18:45:42+00:00
URL https://webpro.in/robots.txt
Domain IPs 204.11.58.187
Response IP 204.11.58.187
Found Yes
Hash a4584b37b3c9a13d2386ad9e4fccf3ca044b73adc2c63e4a4e48c98788a15120
SimHash 423850d00ef7

Groups

*

Rule Path
Allow /

oai-searchbot
chatgpt-user
perplexitybot
firecrawlagent
andibot
exabot
phindbot
youbot

Rule Path
Allow /

gptbot
ccbot
google-extended

Rule Path
Disallow /

googlebot
bingbot

Rule Path
Allow /

*

Rule Path
Disallow /admin/
Disallow /internal/
Back to top

Other Records

Field Value
sitemap https://www.webpro.in/sitemap_index.xml
Back to top

Comments

  • START YOAST BLOCK
  • ---------------------------
  • Allow AI search and agent use
  • Disallow AI training data collection
  • Allow traditional search indexing
  • Disallow access to admin areas for all bots
  • ---------------------------
  • END YOAST BLOCK
Back to top

Warnings

  • 32 invalid lines.
  • `https` is not a known field.
Back to top