Log In Sign Up

amp.mediotiempo.com
robots.txt

Robots Exclusion Standard data for amp.mediotiempo.com

Resource Scan

Scan Details

Site Domain amp.mediotiempo.com
Base Domain mediotiempo.com
Scan Status Ok
Last Scan2024-05-25T09:41:09+00:00
Next Scan 2024-06-24T09:41:09+00:00

Last Scan

Scanned2024-05-25T09:41:09+00:00
URL https://amp.mediotiempo.com/robots.txt
Domain IPs 13.33.88.36, 13.33.88.47, 13.33.88.76, 13.33.88.82
Response IP 13.33.88.76
Found Yes
Hash b22c3e3d9de05f9e8ec7c8e140ee86e1bef649958d9593cbc1a9c03994e77df2
SimHash bc369d126e64

Groups

*

Rule Path
Disallow /node/
Disallow /media/
Disallow /cdb/
Disallow /wp-content/
Disallow /sites/
Disallow /Topicos/
Disallow /7198/
Disallow /noticias/

Other Records

Field Value
crawl-delay 10

genio

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

scooperbot

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

flamingo_searchengine

Rule Path
Disallow /

facebot

Rule Path
Disallow /

luminatebot

Rule Path
Disallow /

vagabondo

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

r6_commentreader

Rule Path
Disallow /

yeti

Rule Path
Disallow /

heritrix

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

showyoubot

Rule Path
Disallow /

gozaikbot

Rule Path
Disallow /

python-requests

Rule Path
Disallow /

queryseekerspider

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

yandeximages

Rule Path
Disallow /

grapeshot

Rule Path
Disallow
Back to top

Comments

  • robots.txt
  • This file is to prevent the crawling and indexing of certain parts
  • of your site by web crawlers and spiders run by sites like Yahoo!
  • and Google. By telling these "robots" where not to go on your site,
  • you save bandwidth and server resources.
  • This file will be ignored unless it is at the root of your host:
  • Used: http://example.com/robots.txt
  • Ignored: http://example.com/site/robots.txt
  • For more information about the robots.txt standard, see:
  • http://www.robotstxt.org/wc/robots.html
  • For syntax checking, see:
  • http://www.sxw.org.uk/computing/robots/check.html
Back to top