abg.baidu.com
robots.txt

Robots Exclusion Standard data for abg.baidu.com

Resource Scan

Scan Details

Site Domain abg.baidu.com
Base Domain baidu.com
Scan Status Ok
Last Scan2025-02-23T09:52:33+00:00
Next Scan 2025-03-25T09:52:33+00:00

Last Scan

Scanned2025-02-23T09:52:33+00:00
URL https://abg.baidu.com/robots.txt
Domain IPs 180.76.11.174, 180.76.11.177
Response IP 180.76.11.177
Found Yes
Hash 07b094d481c085462f70c77a65636eb5a355378d8c6a951fd4cc6f98f8a0c943
SimHash 7e655f772b16

Groups

baiduspider

Rule Path
Disallow /wbusiness/vipcashier

googlebot

Rule Path
Disallow /wbusiness/vipcashier

msnbot

Rule Path
Disallow /wbusiness/vipcashier

baiduspider-image

Rule Path
Disallow /wbusiness/vipcashier

baiduspider-ads

Rule Path
Disallow /wbusiness/vipcashier

youdaobot

Rule Path
Disallow /wbusiness/vipcashier

sogou web spider

Rule Path
Disallow /wbusiness/vipcashier

Other Records

Field Value
crawl-delay 2

sogou inst spider

Rule Path
Disallow /wbusiness/vipcashier

Other Records

Field Value
crawl-delay 2

sogou spider2

Rule Path
Disallow /wbusiness/vipcashier

Other Records

Field Value
crawl-delay 2

sogou blog

Rule Path
Disallow /wbusiness/vipcashier

Other Records

Field Value
crawl-delay 2

sogou news spider

Rule Path
Disallow /wbusiness/vipcashier

Other Records

Field Value
crawl-delay 2

sogou orion spider

Rule Path
Disallow /wbusiness/vipcashier

Other Records

Field Value
crawl-delay 2

chinasospider

Rule Path
Disallow /wbusiness/vipcashier

sosospider

Rule Path
Disallow /wbusiness/vipcashier

yisouspider

Rule Path
Disallow /wbusiness/vipcashier

Other Records

Field Value
crawl-delay 10

easouspider

Rule Path
Disallow /wbusiness/vipcashier

bingbot

Rule Path
Disallow /wbusiness/vipcashier

toutiaospider

Rule Path
Disallow /wbusiness/vipcashier

Warnings

  • 6 invalid lines.
  • `request-rate` is not a known field.