vegasinc.lasvegassun.com
robots.txt

Robots Exclusion Standard data for vegasinc.lasvegassun.com

Resource Scan

Scan Details

Site Domain vegasinc.lasvegassun.com
Base Domain lasvegassun.com
Scan Status Ok
Last Scan2024-11-12T06:36:23+00:00
Next Scan 2024-12-12T06:36:23+00:00

Last Scan

Scanned2024-11-12T06:36:23+00:00
URL https://vegasinc.lasvegassun.com/robots.txt
Domain IPs 104.19.177.74, 104.19.178.74, 2606:4700::6813:b14a, 2606:4700::6813:b24a
Response IP 104.19.177.74
Found Yes
Hash 710256f71f6b06afd80e4fc620936234c4d441d62c2a5ad2bd25e802ef0917c1
SimHash c0b1d010d6d5

Groups

*

Rule Path
Disallow /*?
Disallow r/
Disallow *reminder/
Disallow *ufcsn
Disallow /%3A
Disallow /%3A/
Disallow /*rawhtml*
Disallow /702show*
Disallow /accounts*
Disallow /accounts/login*
Disallow /admin/
Disallow /blogs/robin-leachs-las-vegas-celebrity-watch*
Disallow /cgi-bin/
Disallow /comments*
Disallow /compare/
Disallow /compare/*
Disallow /contact/
Disallow /content/
Disallow /dossier*
Disallow /events/search/?category=*
Disallow /events/search/*
Disallow /feedback/
Disallow /fileadmin/
Disallow /flag/
Disallow /mailfriend*
Disallow /mailfriend/
Disallow /mma-sn/
Disallow /r/
Disallow /search/*
Disallow /slideshow_xml/*
Disallow /sun/dossier*
Disallow /sunbin*
Disallow /sunbin/*
Disallow /ufc-sn/
Disallow /ufc-video-sn/
Disallow /users/
Disallow /wec-sn/
Disallow /xml*
Disallow */cdn-cgi/l/email-protection*
Disallow */slideshow_xml/*
Disallow */xml/*
Disallow *inlines/*
Disallow *cdn-cgi/*
Disallow */drudged/*

directcrawler

Rule Path
Disallow /

voilabot

Rule Path
Disallow /

java/1.5.0_11

Rule Path
Disallow /

java/1.4.1_04

Rule Path
Disallow /

gsa-crawler

Rule Path
Disallow /

shopwiki

Rule Path
Disallow /

googlebot-image

Rule Path
Disallow /*?
Disallow /*_t12*
Disallow /*_t18*
Disallow /*_t19*
Disallow /*_t20*
Disallow /*_t27*
Disallow /*_t30*
Disallow /*_t37*
Disallow /*_t60*
Disallow /*_t61*
Disallow /*_t65*
Disallow /*_t96*
Disallow /*_r45x*
Disallow /*_r50x*
Disallow /*_r90x*
Disallow /*_r60x*
Disallow /*_r100x*
Disallow /*_r104x*
Disallow /*_r180x*
Disallow /*_r340x*
Disallow /*_r415x*
Disallow /*_tx50*

twitterbot

Rule Path
Disallow

Other Records

Field Value
sitemap https://vegasinc.lasvegassun.com/sitemap.xml

Comments

  • specific urls to block
  • other user agents
  • User-agent: ia_archiver-web.archive.org
  • Disallow: /
  • images
  • Twitter allow

Warnings

  • `dissallow` is not a known field.