webservice.homesandland.com
robots.txt

Robots Exclusion Standard data for webservice.homesandland.com

Resource Scan

Scan Details

Site Domain webservice.homesandland.com
Base Domain homesandland.com
Scan Status Ok
Last Scan2024-10-05T07:07:32+00:00
Next Scan 2024-10-19T07:07:32+00:00

Last Scan

Scanned2024-10-05T07:07:32+00:00
URL http://webservice.homesandland.com/robots.txt
Redirect https://www.homesandland.com//robots.txt
Redirect Domain www.homesandland.com
Redirect Base homesandland.com
Domain IPs 51.222.249.8
Redirect IPs 51.222.249.8
Response IP 51.222.249.8
Found Yes
Hash 764364510d7233e3f5c16a9545b6749c0fe06724ceab1b179546bbe0cbf39674
SimHash e21699d0cb77

Groups

*

Rule Path
Disallow

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 3

amazonadbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

baiduspider
yisouspider
petalbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

sogou inst spider

Rule Path
Disallow /

owler@ows.eu/1

Rule Path
Disallow /

mojeekbot

Rule Path
Disallow /

obot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.homesandland.com/sitemaps/dyna/sitemap_root_index.xml

Comments

  • Bard and Vertex AI generative APIs
  • Non Market

Warnings

  • 1 invalid line.