everysize.com
robots.txt

Robots Exclusion Standard data for everysize.com

Resource Scan

Scan Details

Site Domain everysize.com
Base Domain everysize.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-03-31T22:59:50+00:00
Next Scan 2024-06-29T22:59:50+00:00

Last Successful Scan

Scanned2023-02-13T13:11:13+00:00
URL https://everysize.com/robots.txt
Redirect https://www.everysize.com/robots.txt
Redirect Domain www.everysize.com
Redirect Base everysize.com
Domain IPs 172.66.40.57, 172.66.43.199, 2606:4700:3108::ac42:2839, 2606:4700:3108::ac42:2bc7
Redirect IPs 172.66.40.57, 172.66.43.199, 2606:4700:3108::ac42:2839, 2606:4700:3108::ac42:2bc7
Response IP 172.66.43.199
Found Yes
Hash cf670a6085612e9c050b680523fb1671e642a9b9b54033e25592c42a2db22cc7
SimHash 756800307c13

Groups

*

Rule Path
Disallow /go/
Disallow /ping/
Disallow /_data/
Disallow /_data-cached/
Disallow /checkout/
Disallow /*?item=*
Disallow /*%26item%3D*
Disallow /*?utm_campaign=*
Disallow /*%26utm_campaign%3D*
Disallow /*?utm_medium=*
Disallow /*%26utm_medium%3D*
Disallow /*?utm_source=*
Disallow /*%26utm_source%3D*
Disallow /*?utm_content=*
Disallow /*%26utm_content%3D*
Disallow /*?utm_term=*
Disallow /*%26utm_term%3D*

semrushbot
sistrix
optimizer
blexbot
dotbot
mj12bot
barkrowler
seekbot
petalbot
screaming frog seo spider

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.everysize.com/sitemap.xml

Comments

  • www.robotstxt.org/
  • Allow crawling of all content
  • Block crawler
  • User-agent: AhrefsBot