lasvegasweekly.com
robots.txt

Robots Exclusion Standard data for lasvegasweekly.com

Resource Scan

Scan Details

Site Domain lasvegasweekly.com
Base Domain lasvegasweekly.com
Scan Status Ok
Last Scan2024-10-30T17:17:14+00:00
Next Scan 2024-11-06T17:17:14+00:00

Last Scan

Scanned2024-10-30T17:17:14+00:00
URL https://lasvegasweekly.com/robots.txt
Domain IPs 104.17.81.18, 104.17.82.18, 2606:4700::6811:5112, 2606:4700::6811:5212
Response IP 104.17.82.18
Found Yes
Hash dde728bbb6307c6d00ba14da8b7fc00c60f358f2a73b3a53260a85f6c257f4b5
SimHash e4c0c051d6d4

Groups

*

Rule Path
Disallow r/
Disallow *reminder/
Disallow *ufcsn
Disallow /%3A
Disallow /%3A/
Disallow /?*
Disallow /*rawhtml*
Disallow /702show*
Disallow /accounts*
Disallow /accounts/login*
Disallow /admin/
Disallow /blogs/robin-leachs-las-vegas-celebrity-watch*
Disallow /cgi-bin/
Disallow /comments*
Disallow /compare/
Disallow /compare/*
Disallow /contact/
Disallow /content/
Disallow /dossier*
Disallow /events/search/?category=*
Disallow /events/search/*
Disallow /feedback/
Disallow /fileadmin/
Disallow /flag/
Disallow /mailfriend*
Disallow /mailfriend/
Disallow /mma-sn/
Disallow /r/
Disallow /slideshow_xml/*
Disallow /sun/dossier*
Disallow /sunbin*
Disallow /sunbin/*
Disallow /ufc-sn/
Disallow /ufc-video-sn/
Disallow /users/
Disallow /wec-sn/
Disallow /xml*
Disallow */cdn-cgi/l/email-protection*
Disallow */slideshow_xml/*
Disallow */xml/*
Disallow *inlines/*
Disallow *cdn-cgi/*
Disallow */drudged/*
Disallow */typo3/*
Disallow */bbcom/*

directcrawler

Rule Path
Disallow /

voilabot

Rule Path
Disallow /

java/1.5.0_11

Rule Path
Disallow /

java/1.4.1_04

Rule Path
Disallow /

gsa-crawler

Rule Path
Disallow /

shopwiki

Rule Path
Disallow /

directcrawler

Rule Path
Disallow /

voilabot

Rule Path
Disallow /

java/1.5.0_11

Rule Path
Disallow /

java/1.4.1_04

Rule Path
Disallow /

gsa-crawler

Rule Path
Disallow /

shopwiki

Rule Path
Disallow /

*

Rule Path
Disallow /accounts*
Disallow /blogs/luxe-life*
Disallow /comments*
Disallow /content*
Disallow /departments*
Disallow /features*
Disallow /mailfriend*
Disallow /sunbin*
Disallow /distribution*
Disallow /events/search*

Other Records

Field Value
crawl-delay 0.5

googlebot

Rule Path
Disallow /events/search*

twitterbot

Rule Path
Disallow

Other Records

Field Value
sitemap https://lasvegasweekly.com/sitemap.xml

Comments

  • go away
  • User-agent: ia_archiver-web.archive.org
  • Disallow: /
  • go away
  • User-agent: ia_archiver-web.archive.org
  • Disallow: /
  • to stop scraping of old events
  • force googlebot to obey
  • Twitter allow