Log In Sign Up

mansion-market.com
robots.txt

Robots Exclusion Standard data for mansion-market.com

Resource Scan

Scan Details

Site Domain mansion-market.com
Base Domain mansion-market.com
Scan Status Ok
Last Scan2024-05-10T11:50:57+00:00
Next Scan 2024-05-17T11:50:57+00:00

Last Scan

Scanned2024-05-10T11:50:57+00:00
URL https://mansion-market.com/robots.txt
Domain IPs 151.101.130.110, 151.101.194.110, 151.101.2.110, 151.101.66.110
Response IP 151.101.2.110
Found Yes
Hash b0b76b3c0f869a5fadd0ba938fb4d4f2fca51c6ea2210cdd61751d3a55f03bcf
SimHash 6e1cdc55e6d0

Groups

mj12bot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

seekportbot

Rule Path
Disallow /

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

*

Rule Path
Disallow /simplified_sign_up
Disallow /sign_up
Disallow /sign_in
Disallow /mansions/search?*
Disallow /mansions/search
Back to top

Other Records

Field Value
sitemap https://mansion-market.com/sitemap.xml
Back to top

Warnings

  • 2 invalid lines.
Back to top