gaslicht.com
robots.txt

Robots Exclusion Standard data for gaslicht.com

Resource Scan

Scan Details

Site Domain gaslicht.com
Base Domain gaslicht.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-04-19T07:05:40+00:00
Next Scan 2024-07-18T07:05:40+00:00

Last Successful Scan

Scanned2023-11-29T05:10:54+00:00
URL https://gaslicht.com/robots.txt
Redirect https://www.gaslicht.com/robots.txt
Redirect Domain www.gaslicht.com
Redirect Base gaslicht.com
Domain IPs 52.157.220.210
Redirect IPs 13.107.213.73, 13.107.246.73, 2620:1ec:46::59, 2620:1ec:bdf::59
Response IP 13.107.213.57
Found Yes
Hash ce52f203626774ae7ae357ed2fd5de1d753b6e4dc76c676f55d125e7bf3e3838
SimHash 131c65648891

Groups

ahrefsbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

webcrawler

Rule Path
Disallow /

bubing

Rule Path
Disallow /

proximic

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

msnbot-media

Rule Path
Disallow /

vagabondo

Rule Path
Disallow /

bilbo

Rule Path
Disallow /

yandex

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

yandeximages

Rule Path
Disallow /

yandexvideo

Rule Path
Disallow /

yandexmedia

Rule Path
Disallow /

yandexfavicons

Rule Path
Disallow /

linkdexbot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

slurp

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

niki-bot

Rule Path
Disallow /

compspybot

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

beetlebot

Rule Path
Disallow /

mail.ru_bot

Rule Path
Disallow /

istellabot

Rule Path
Disallow /

plukkie

Rule Path
Disallow /

melodibot

Rule Path
Disallow /

spbot

Rule Path
Disallow /

queryseekerspider

Rule Path
Disallow /

buddhabot

Rule Path
Disallow /

imrbot

Rule Path
Disallow /

acoonbot

Rule Path
Disallow /

yacybot

Rule Path
Disallow /

gufybot

Rule Path
Disallow /

nesotebot

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

solomonobot

Rule Path
Disallow /

wbsearchbot

Rule Path
Disallow /

noyabot

Rule Path
Disallow /

flappybot

Rule Path
Disallow /

siteexplorer

Rule Path
Disallow /

proximic

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

crawler4j

Rule Path
Disallow /

nutch

Rule Path
Disallow /

berlin-fu-cow

Rule Path
Disallow /

riddler

Rule Path
Disallow /

python-requests

Rule Path
Disallow /

python-requests/2.8.1

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.gaslicht.com/sitemap_index.xml

Warnings

  • 4 invalid lines.