bestpregnant.com
robots.txt

Robots Exclusion Standard data for bestpregnant.com

Resource Scan

Scan Details

Site Domain bestpregnant.com
Base Domain bestpregnant.com
Scan Status Ok
Last Scan2024-05-21T05:49:46+00:00
Next Scan 2024-06-20T05:49:46+00:00

Last Scan

Scanned2024-05-21T05:49:46+00:00
URL https://bestpregnant.com/robots.txt
Redirect http://www.bestpregnant.com/robots.txt
Redirect Domain www.bestpregnant.com
Redirect Base bestpregnant.com
Domain IPs 104.21.234.64, 104.21.234.65, 2606:4700:3038::6815:ea40, 2606:4700:3038::6815:ea41
Redirect IPs 104.21.234.64, 104.21.234.65, 2606:4700:3038::6815:ea40, 2606:4700:3038::6815:ea41
Response IP 104.21.234.64
Found Yes
Hash bb3ba3785e66d96c2da0e168092b20a0b839720eaaab84936dd4d78786fb27be
SimHash 7a3fdc037e88

Groups

petalbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

cloudfind

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

riddler

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

webmeupbot

Rule Path
Disallow /

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

googlebot

Rule Path
Disallow /search/
Disallow /go/

Other Records

Field Value
sitemap http://www.bestpregnant.com/sitemap.xml