Log In Sign Up

cityofwestwego.com
robots.txt

Robots Exclusion Standard data for cityofwestwego.com

Resource Scan

Scan Details

Site Domain cityofwestwego.com
Base Domain cityofwestwego.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan2024-09-17T00:31:03+00:00
Next Scan 2024-10-17T00:31:03+00:00

Last Successful Scan

Scanned2024-07-27T00:22:19+00:00
URL https://cityofwestwego.com/robots.txt
Redirect https://www.visitwestwego.com/robots.txt
Redirect Domain www.visitwestwego.com
Redirect Base visitwestwego.com
Domain IPs 52.86.153.249
Redirect IPs 23.185.0.3, 2620:12a:8000::3, 2620:12a:8001::3
Response IP 23.185.0.3
Found Yes
Hash 0b148172ef3ea294af520e8d1e703bbbd76e5dfe532a5d6bb7785f0c7869f6c6
SimHash 3896bd5367c8

Groups

*

Rule Path
Allow /core/*.css$
Allow /core/*.css?
Allow /core/*.js$
Allow /core/*.js?
Allow /core/*.gif
Allow /core/*.jpg
Allow /core/*.jpeg
Allow /core/*.png
Allow /core/*.svg
Allow /profiles/*.css$
Allow /profiles/*.css?
Allow /profiles/*.js$
Allow /profiles/*.js?
Allow /profiles/*.gif
Allow /profiles/*.jpg
Allow /profiles/*.jpeg
Allow /profiles/*.png
Allow /profiles/*.svg
Disallow /core/
Disallow /profiles/
Disallow /README.md
Disallow /composer/Metapackage/README.txt
Disallow /composer/Plugin/ProjectMessage/README.md
Disallow /composer/Plugin/Scaffold/README.md
Disallow /composer/Plugin/VendorHardening/README.txt
Disallow /composer/Template/README.txt
Disallow /modules/README.txt
Disallow /sites/README.txt
Disallow /themes/README.txt
Disallow /web.config
Disallow /admin/
Disallow /comment/reply/
Disallow /filter/tips
Disallow /node/add/
Disallow /search/
Disallow /user/register
Disallow /user/password
Disallow /user/login
Disallow /user/logout
Disallow /media/oembed
Disallow /*/media/oembed
Disallow /index.php/admin/
Disallow /index.php/comment/reply/
Disallow /index.php/filter/tips
Disallow /index.php/node/add/
Disallow /index.php/search/
Disallow /index.php/user/password
Disallow /index.php/user/register
Disallow /index.php/user/login
Disallow /index.php/user/logout
Disallow /index.php/media/oembed
Disallow /index.php/*/media/oembed

serpstatbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

rogerbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

linguee bot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

zoominfobot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

ia_archiver

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

linkdexbot
linkdexbot/2.1

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

gigabot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

ccbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

dotbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

semrushbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

amazonbot
amazonbot/0.1

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

scrapy
scrapy/2.6.2

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

monsidobot
monsidobot/2.2

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

siteauditbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

archive.org_bot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

blexbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

velenpublicwebcrawler

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

dataforseobot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

sogou blog
sogou inst spider
sogou news spider
sogou orion spider
sogou spider2
sogou web spider

Rule Path
Disallow /

buck

Rule Path
Disallow /

coccoc
coccocbot-image
coccocbot-web

Rule Path
Disallow /

seokicks

Rule Path
Disallow /

ioncrawl

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

awariorssbot
awariosmartbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

megaindex.ru
megaindex.ru/2.0

Rule Path
Disallow /

seekportbot

Rule Path
Disallow /

megaindex.com

Rule Path
Disallow /

yeti

Rule Path
Disallow /

haosou 360 spider

Rule Path
Disallow /

dotmic dotbot

Rule Path
Disallow /

baiduspider
baiduspider-image
baiduspider-news
baiduspider-video

Rule Path
Disallow /

yandexbot
yandex
yandexmobilebot
yandexbot/3.0

Rule Path
Disallow /

ecoresearchcrawler

Rule Path
Disallow /

ecoresearch

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

zeus 32297 webster pro v2.9 win32

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

claudebot
claude-web
anthropic-ai

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /
Back to top

Comments

  • robots.txt
  • This file is to prevent the crawling and indexing of certain parts
  • of your site by web crawlers and spiders run by sites like Yahoo!
  • and Google. By telling these "robots" where not to go on your site,
  • you save bandwidth and server resources.
  • This file will be ignored unless it is at the root of your host:
  • Used: http://example.com/robots.txt
  • Ignored: http://example.com/site/robots.txt
  • For more information about the robots.txt standard, see:
  • http://www.robotstxt.org/robotstxt.html
  • CSS, JS, Images
  • Directories
  • Files
  • Paths (clean URLs)
  • Paths (no clean URLs)
Back to top