Log In Sign Up

cyclingstage.com
robots.txt

Robots Exclusion Standard data for cyclingstage.com

Resource Scan

Scan Details

Site Domain cyclingstage.com
Base Domain cyclingstage.com
Scan Status Ok
Last Scan2024-11-14T07:19:19+00:00
Next Scan 2024-11-21T07:19:19+00:00

Last Scan

Scanned2024-11-14T07:19:19+00:00
URL https://cyclingstage.com/robots.txt
Redirect https://www.cyclingstage.com/robots.txt
Redirect Domain www.cyclingstage.com
Redirect Base cyclingstage.com
Domain IPs 213.108.110.118
Redirect IPs 213.108.110.118
Response IP 213.108.110.118
Found Yes
Hash 209ed53f0e3ca7e72972650fdb7ffa6bfbc326eb6bb4c61b77c0744485faab01
SimHash 0f56ba1245e0

Groups

mj12bot

Rule Path
Disallow /

ias_crawler

Rule Path
Allow /
Disallow /cgi-bin
Disallow /comments
Disallow /images
Disallow /wp-admin
Disallow /wp-includes
Disallow /*trackback

twitterbot

Rule Path
Allow /

adsbot-google

Rule Path
Allow /

mediapartners-google

Rule Path
Allow /
Disallow /cgi-bin
Disallow /comments
Disallow /images
Disallow /wp-admin
Disallow /wp-includes
Disallow /*trackback
Disallow /*/contact
Disallow /*/*/contact
Disallow /*map-iframe*
Disallow /*function.include
Disallow /wp-content/
Allow /wp-content/uploads/
Disallow /*.php$
Disallow /*.js$
Disallow /*.inc$
Disallow /*.css$
Disallow /*.wmv$
Disallow /*.cgi$
Disallow /*.xhtml$
Disallow /*.doc$
Disallow /*.pdf$
Disallow /*.zip$
Disallow /*.gz$
Disallow /*?*

slurp

Rule Path
Allow /
Disallow /cgi-bin
Disallow /comments
Disallow /images
Disallow /wp-admin
Disallow /wp-includes
Disallow /*trackback
Disallow /*/contact
Disallow /*/*/contact
Disallow /*map-iframe*
Disallow /*function.include
Disallow /wp-content/
Allow /wp-content/uploads/
Disallow /*.php$
Disallow /*.js$
Disallow /*.inc$
Disallow /*.css$
Disallow /*.wmv$
Disallow /*.cgi$
Disallow /*.xhtml$
Disallow /*.doc$
Disallow /*.pdf$
Disallow /*.zip$
Disallow /*.gz$
Disallow /*?*

bingbot

Rule Path
Allow /
Disallow /cgi-bin
Disallow /comments
Disallow /images
Disallow /wp-admin
Disallow /wp-includes
Disallow /*trackback
Disallow /*/contact
Disallow /*/*/contact
Disallow /*map-iframe*
Disallow /*function.include
Disallow /wp-content/
Allow /wp-content/uploads/

googlebot

Rule Path
Allow /
Disallow /cgi-bin
Disallow /comments
Disallow /images
Disallow /wp-admin
Disallow /wp-includes
Disallow /*trackback
Disallow /*/contact
Disallow /*/*/contact
Disallow /*map-iframe*
Disallow /*function.include
Disallow /wp-content/
Allow /wp-content/uploads/
Disallow /*.php$
Disallow /*.js$
Disallow /*.inc$
Disallow /*.css$
Disallow /*.wmv$
Disallow /*.cgi$
Disallow /*.xhtml$
Disallow /*.doc$
Disallow /*.pdf$
Disallow /*.zip$
Disallow /*.gz$
Disallow /tour-down-under-2014/stages-route-australia-2014/
Disallow /vuelta-206-route/
Disallow /vuelta-2014-contenders/
Disallow /cycling/tour-of-dubai-2015/
Disallow /amstel-gold-race-2014/route-agr-2014/_1/
Disallow /performgroup.htm/
Disallow /slideshow-/
Disallow /DARTIFrame_/
Disallow /paris-nice-2022/zweeler-pool-pn/
Disallow /strade-bianche-2022/zweeler-pool-sb/
Disallow /volta-a-catalunya-2022/zweeler-pool-cat/
Disallow /milan-san-remo-2022/zweeler-pool-msr/
Disallow /giro-2022/zweeler-pool-italy-2022/
Disallow /tirreno-adriatico-2023/zweeler-pool-ta
Disallow /paris-nice-2023/zweeler-pool-pn
Disallow /strade-bianche-2023/zweeler-pool-sb
Disallow /milan-san-remo-2023/zweeler-pool-msr/
Disallow /tour-of-flanders-2023/zweeler-pool-rvv/
Disallow /giro-2023/zweeler-pool-italy/
Disallow /tour-de-france-2023/zweeler-pool-tdf/
Disallow /vuelta-2023/zweeler-pool-vuelta-2023/
Back to top

Other Records

Field Value
sitemap https://www.cyclingstage.com/sitemap_index.xml
sitemap https://www.cyclingstage.com/sitemap_index.xml
sitemap https://www.cyclingstage.com/sitemap_index.xml
Back to top

Comments

  • Deny all robots that we do not specifically want to allow
  • User-agent: *
  • Disallow: /
  • Block Majestic crawler
  • Allow ias_crawler
  • Allow Twitterbot
  • Allow Adsbot Google
  • Allow Mediapartner Google
  • Yahoo
  • bingbot
  • google bot
Back to top

Warnings

  • 7 invalid lines.
Back to top