Log In Sign Up

defharo.com
robots.txt

Robots Exclusion Standard data for defharo.com

Resource Scan

Scan Details

Site Domain defharo.com
Base Domain defharo.com
Scan Status Ok
Last Scan2024-06-11T21:04:15+00:00
Next Scan 2024-06-18T21:04:15+00:00

Last Scan

Scanned2024-06-11T21:04:15+00:00
URL https://defharo.com/robots.txt
Domain IPs 172.66.40.143, 172.66.43.113, 2606:4700:3108::ac42:288f, 2606:4700:3108::ac42:2b71
Response IP 172.66.40.143
Found Yes
Hash db98dce8e37c25bb33c3a52834cb786148f61924cd086c71d1a334ef710a7b19
SimHash 4248d4408635

Groups

*
wprocketbot

Rule Path
Allow /
Allow /wp-content/uploads/
Disallow /stock/
Disallow /febles/
Disallow /awstats/
Disallow /stats/

msiecrawler

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

httrack

Rule Path
Disallow /

microsoft.url.control

Rule Path
Disallow /

libwww

Rule Path
Disallow /

yandex

Rule Path
Disallow /

baiduspider

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

noxtrumbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 50

mj12bot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

*

Rule Path
Disallow /wp-content/uploads/wp-import-export-lite/
Back to top

Other Records

Field Value
sitemap https://defharo.com/sitemap_index.xml
Back to top

Comments

  • WP Import Export Rule
Back to top