thebige.com
robots.txt

Robots Exclusion Standard data for thebige.com

Resource Scan

Scan Details

Site Domain thebige.com
Base Domain thebige.com
Scan Status Ok
Last Scan2024-11-15T16:38:31+00:00
Next Scan 2024-11-29T16:38:31+00:00

Last Scan

Scanned2024-11-15T16:38:31+00:00
URL https://thebige.com/robots.txt
Redirect https://www.thebige.com/robots.txt
Redirect Domain www.thebige.com
Redirect Base thebige.com
Domain IPs 13.88.18.55
Redirect IPs 13.88.18.55
Response IP 13.88.18.55
Found Yes
Hash a58dc014e3c08312ce722796d56f522755faa701f0fcd760ad9d04ab7c3f02ea
SimHash 4a74dc748203

Groups

baiduspider

Rule Path
Disallow /

dotbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

*

Rule Path
Disallow /undefined*
Disallow *I_map.aspx*
Disallow *I_emailevent.aspx*
Disallow *I_emailbusiness.aspx*

Other Records

Field Value
sitemap https://www.thebige.com/sitemap.xml