Log In Sign Up

awesomebooks.com
robots.txt

Robots Exclusion Standard data for awesomebooks.com

Resource Scan

Scan Details

Site Domain awesomebooks.com
Base Domain awesomebooks.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-05-02T21:25:04+00:00
Next Scan 2024-07-01T21:25:04+00:00

Last Successful Scan

Scanned2024-02-10T19:36:06+00:00
URL https://awesomebooks.com/robots.txt
Redirect https://www.awesomebooks.com/robots.txt
Redirect Domain www.awesomebooks.com
Redirect Base awesomebooks.com
Domain IPs 104.22.72.118, 104.22.73.118, 172.67.22.23, 2606:4700:10::6816:4876, 2606:4700:10::6816:4976, 2606:4700:10::ac43:1617
Redirect IPs 104.22.72.118, 104.22.73.118, 172.67.22.23, 2606:4700:10::6816:4876, 2606:4700:10::6816:4976, 2606:4700:10::ac43:1617
Response IP 104.22.73.118
Found Yes
Hash baf66240116e593316f3af6638fe0ff50e40d05576cfb16f03a10d23af8f6c82
SimHash 5e0cd160e2d9

Groups

*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 3

*

Rule Path
Disallow /myaccount/
Disallow /newsletter
Disallow /basket/
Disallow /checkout
Disallow /search*
Disallow /search/
Disallow /changecurrency
Disallow /changeLocation

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

applebot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

adidxbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

gptbot

Rule Path
Disallow /search/

Other Records

Field Value
crawl-delay 8

amazonbot

Rule Path
Disallow /search/

Other Records

Field Value
crawl-delay 8

ahrefsbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

bingpreview

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

tinytestbot

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

pinterest

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 15

yandex

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

neevabot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

semrushbot

Rule Path
Disallow /

ccbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

semrushbot-sa

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

yisouspider

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

sogou inst spider

Rule Path
Disallow /

dataforseobot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

adsbot-google-mobile

Rule Path
Allow /*?keyword=*
Allow /*?search=*
Back to top

Other Records

Field Value
sitemap https://www.awesomebooks.com/sitemap/sitemap.xml
Back to top

Comments

  • Slow down bots
  • http://webmeup-crawler.com
Back to top

Warnings

  • 4 invalid lines.
Back to top