Log In Sign Up

listadejugadores.es
robots.txt

Robots Exclusion Standard data for listadejugadores.es

Resource Scan

Scan Details

Site Domain listadejugadores.es
Base Domain listadejugadores.es
Scan Status Ok
Last Scan2024-11-12T08:10:21+00:00
Next Scan 2024-11-19T08:10:21+00:00

Last Scan

Scanned2024-11-12T08:10:21+00:00
URL https://listadejugadores.es/robots.txt
Domain IPs 54.154.91.134
Response IP 54.154.91.134
Found Yes
Hash eab892c7ea639a395aca294c88198b04ae0846182e2d2ea578280af0897ba12e
SimHash f2552bc5cc72

Groups

*

Rule Path
Disallow /main/team_name_url
Disallow */wp-admin/*
Disallow */wp-login.php
Disallow */wp-register.php
Disallow /password_forgot
Disallow /common/*
Disallow /advert_click
Disallow /advert/
Disallow /private_team_files
Disallow /map/static

mediapartners-google

Rule Path
Disallow
Back to top

Other Records

Field Value
sitemap /sitemap.xml
Back to top

Comments

  • robots.txt
  • See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file
  • This url is for ajax calls only
  • Blog related
  • Disallow common, e.g. advert clicks
  • Never end up crawling private files
  • No indexing of static maps
  • Allow adsense crawler
Back to top