freeforcommercialuse.net
robots.txt

Robots Exclusion Standard data for freeforcommercialuse.net

Resource Scan

Scan Details

Site Domain freeforcommercialuse.net
Base Domain freeforcommercialuse.net
Scan Status Ok
Last Scan2024-10-31T23:12:48+00:00
Next Scan 2024-11-07T23:12:48+00:00

Last Scan

Scanned2024-10-31T23:12:48+00:00
URL https://freeforcommercialuse.net/robots.txt
Domain IPs 104.21.17.202, 172.67.178.67, 2606:4700:3034::6815:11ca, 2606:4700:3037::ac43:b243
Response IP 104.21.17.202
Found Yes
Hash ff5f26f9c213b7f5413860a8acd5c42452e5ed75de684241c0cd1d9361c4611d
SimHash 281e9f001e12

Groups

*

Rule Path
Allow /
Disallow /istock/*
Disallow /designbundles/download/*
Disallow /design-bundles/download/*
Disallow /s.php*
Disallow /shutterstock.php?q*

semrush
semrushbot
ahrefsbot
mj12bot
sitebot
dotbot
ocelli
sistrix
shopwiki
wbsearchbot
riddlerbot
linguatools
www.integromedb.org/crawler
ccbot
brandverity
scrapy/2.4.1 (+https://scrapy.org)

Rule Path
Disallow /

Warnings

  • 1 invalid line.