Log In Sign Up

premarathon.com
robots.txt

Robots Exclusion Standard data for premarathon.com

Resource Scan

Scan Details

Site Domain premarathon.com
Base Domain premarathon.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-11-09T06:53:24+00:00
Next Scan 2026-02-07T06:53:24+00:00

Last Successful Scan

Scanned2025-04-13T08:22:25+00:00
URL https://premarathon.com/robots.txt
Domain IPs 178.33.116.125
Response IP 178.33.116.125
Found Yes
Hash 960effba30bf8d2f578199e70db4595934af45f55caa79cf88793aff2d823140
SimHash 68985943c064

Groups

*

Rule Path
Allow /wp-content/uploads/*
Allow /wp-content/*.js
Allow /wp-content/*.css
Allow /wp-includes/*.js
Allow /wp-includes/*.css
Allow /wp-admin/admin-ajax.php
Disallow /cgi-bin
Disallow /wp-admin/
Disallow /wp-content/themes/Newspaper/
Disallow /wp-content/themes/
Disallow /wp-includes/
Disallow /*/attachment/
Disallow /tag/*/page/
Disallow /tag/*/feed/
Disallow /page/
Disallow /xmlrpc.php
Disallow /?attachment_id*
Disallow /*?

*

Rule Path
Disallow /h/

*

Rule Path
Disallow /trackback
Disallow /*trackback
Disallow /*trackback*
Disallow /*/trackback

*

Rule Path
Allow /feed/$
Disallow /feed/
Disallow /comments/feed/
Disallow /*/feed/$
Disallow /*/feed/rss/$
Disallow /*/trackback/$
Disallow /*/*/feed/$
Disallow /*/*/feed/rss/$
Disallow /*/*/trackback/$
Disallow /*/*/*/feed/$
Disallow /*/*/*/feed/rss/$
Disallow /*/*/*/trackback/$

noxtrumbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

facebookexternalhit
meta-externalagent

Rule Path
Allow /

microsoft.url.control
libwww
orthogaffe
ubicrawler
doc
zao
sitecheck.internetseer.com
zealbot
msiecrawler
sitesnagger
webstripper
webcopier
fetch
offline explorer
teleport
teleportpro
webzip
linko
httrack
microsoft.url.control
xenu
larbin
libwww
zyborg
download ninja
wget
grub-client
k2spider
npbot
webreaper

Rule Path
Disallow /

baiduspider
gurujibot
hl_ftien_spider
sogou spider
yeti
yodaobot

Rule Path
Disallow /

amazonbot
applebot
applebot-extended
bytespider
ccbot
chatgpt-user
claude-web
claudebot
diffbot
facebookbot
friendlycrawler
gptbot
google-extended
googleother
googleother-image
googleother-video
icc-crawler
imagesiftbot
meta-externalagent
meta-externalfetcher
oai-searchbot
perplexitybot
petalbot
scrapy
timpibot
velenpublicwebcrawler
youbot
anthropic-ai
cohere-ai
img2dataset
omgili
omgilibot

Rule Path
Disallow /

googlebot

Rule Path
Allow /*.css$
Allow /*.js$
Back to top

Other Records

Field Value
sitemap https://www.premarathon.com/sitemap_index.xml
sitemap https://www.premarathon.com/category-sitemap.xml
sitemap https://www.premarathon.com/page-sitemap.xml
sitemap https://www.premarathon.com/post-sitemap.xml
sitemap https://www.premarathon.com/post_tag-sitemap.xml
Back to top

Comments

  • Bloqueo basico para todos los bots y crawlers 2
  • puede dar problemas por bloqueo de recursos en GWT
  • Bloqueo de las URL dinamicas
  • Bloqueo para SPAM Semalt SEO
  • Bloqueo de trackbacks
  • Bloqueo de feeds para crawlers
  • Ralentizamos algunos bots que se suelen volver locos
  • Habilitar Bot Facebook Threads
  • Bloqueo de bots y crawlers poco utiles
  • Bloqueo de bots y crawlers poco utiles (WEBEMPRESA)
  • Example: Block IA
  • Previene problemas de recursos bloqueados en Google Webmaster Tools
  • Si utilizas Yoast SEO estos son los sitemaps principales
Back to top