sciencephoto.com
robots.txt
Robots Exclusion Standard data for sciencephoto.com
Resource Scan
Scan Details
| Site Domain | sciencephoto.com |
| Base Domain | sciencephoto.com |
| Scan Status | Ok |
| Last Scan | 2024-06-20T22:57:39+00:00 |
| Next Scan | 2024-07-20T22:57:39+00:00 |
Last Scan
| Scanned | 2024-06-20T22:57:39+00:00 |
| URL | https://sciencephoto.com/robots.txt |
| Redirect | https://www.sciencephoto.com/robots.txt |
| Redirect Domain | www.sciencephoto.com |
| Redirect Base | sciencephoto.com |
| Domain IPs | 144.76.242.34 |
| Redirect IPs | 144.76.242.34 |
| Response IP | 144.76.242.34 |
| Found | Yes |
| Hash | aee5e954d6f2a7d1dd5dfadafdb53acd3c56beacd4a449a2189bdef149a13ecd |
| SimHash | 53024fd0ec80 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /admin |
| Disallow | /api |
| Disallow | /cms |
| Disallow | /login |
| Disallow | /ping |
| Disallow | /public/basket/ |
| Disallow | /public/login/ |
| Disallow | /sales |
| Disallow | /sciencephoto/ |
| Disallow | /user |
| Disallow | /_assets/ |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
adbeat_bot
adsbot
ahc
ahrefsbot
aihitbot
aiohttp
amazonbot
anthropic-ai
applebot
awariobot
awariorssbot
awariosmartbot
barkrowler
blexbot
brandverity
buck
ccbot
chatgpt-user
cincraw
claudebot
claude-web
cohere-ai
crystalsemantics
dataforseobot
daum
dataprovider
deepcrawl
diffbot
domcopbot
dotbot
ev-crawler
exabot
facebookbot
gptbot
go-http-client
google-extended
grapeshot
httrack
img2dataset
imagesiftbot
lcc
linespider
ltx71 - (http://ltx71.com/)
magellan
magpie-crawler
mail.ru_bot
mauibot
megaindex
metajobbot
mj12bot
neevabot
netpeakcheckerbot
omgili
omgilibot
owler@ows.eu/x
owler@ows.eu/1
owler@ows.eu/2
panscient.com
perplexitybot
petalbot
piplbot
proximic
rainbot
riddler
rogerbot
scrapy
screaming frog seo spider
semanticbot
semanticscholarbot
semrushbot
semrushbot-ba
semrushbot-coub
semrushbot-ct
semrushbot-si
semrushbot-swa
sentibot
serpstatbot
seekportbot
seokicks
siteauditbot
sitecheckerbotcrawler
splitsignalbot
stormcrawler
the knowledge ai
trendictionbot
velenpublicwebcrawler
webprosbot
wellknownbot
wrtnbot
xovibot
yak
yaosoubot
yepbot
yeti
yisouspider
youbot
zoominfobot
| Rule | Path |
|---|---|
| Disallow | / |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.sciencephoto.com/sitemap.xml |
Comments