Log In Sign Up

shiftcrypto.io
robots.txt

Robots Exclusion Standard data for shiftcrypto.io

Resource Scan

Scan Details

Site Domain shiftcrypto.io
Base Domain shiftcrypto.io
Scan Status Ok
Last Scan2025-07-12T23:22:01+00:00
Next Scan 2025-08-11T23:22:01+00:00

Last Scan

Scanned2025-07-12T23:22:01+00:00
URL https://shiftcrypto.io/robots.txt
Redirect https://bitbox.swiss/robots.txt
Redirect Domain bitbox.swiss
Redirect Base bitbox.swiss
Domain IPs 142.132.241.138
Redirect IPs 104.26.0.40, 104.26.1.40, 172.67.68.208, 2606:4700:20::681a:128, 2606:4700:20::681a:28, 2606:4700:20::ac43:44d0
Response IP 104.26.0.40
Found Yes
Hash 16b729ed78028aa38b553e59f6dc9d9c9733612014f36166b0304ebb7f802559
SimHash 0d80bb82c93d

Groups

*

Rule Path
Disallow /blog/teaser-*

adsbot-google
adsbot-google-mobile
adsbot-google-mobile-apps
mediapartners-google

Rule Path
Disallow /blog/teaser-*

yandex

No rules defined. All paths allowed.

Back to top

Comments

  • This applies to all but google's ads bots, unfortunately:
  • https://developers.google.com/search/docs/advanced/robots/create-robots-txt
  • Duplicated entries from above specifically for google's ads bots.
  • https://developers.google.com/search/docs/advanced/crawling/overview-google-crawlers
  • Duplicate pages with GET parameters were found
Back to top

Warnings

  • `clean-param` is not a known field.
Back to top