oilandgas360.com
robots.txt
Robots Exclusion Standard data for oilandgas360.com
Resource Scan
Scan Details
| Site Domain | oilandgas360.com |
| Base Domain | oilandgas360.com |
| Scan Status | Ok |
| Last Scan | 2024-05-29T15:32:16+00:00 |
| Next Scan | 2024-06-28T15:32:16+00:00 |
Last Scan
| Scanned | 2024-05-29T15:32:16+00:00 |
| URL | https://oilandgas360.com/robots.txt |
| Redirect | https://www.oilandgas360.com/robots.txt |
| Redirect Domain | www.oilandgas360.com |
| Redirect Base | oilandgas360.com |
| Domain IPs | 141.193.213.10, 141.193.213.11 |
| Redirect IPs | 141.193.213.10, 141.193.213.11 |
| Response IP | 141.193.213.11 |
| Found | Yes |
| Hash | 07d29fc3079c5b6b387496184981739086ea8a1f61f8a8fade02779321aa0105 |
| SimHash | 282b7475ac3a |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /wp-admin/ |
| Disallow | /cgi-bin/ |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 4 |
baiduspider
marketwirebot
marketwirebot/1.0
rogerbot
scrapy
megaindex
ccbot
riddler
mj12bot
domain re-animator bot
vegebot
vegi bot
yahoo! slurp
rogerbot
spbot
yandexbot
yandexbot/3.0
ahrefsbot/5.2
ahrefsbot
grouphigh/1.0
No rules defined. All paths allowed.
Other Records
| Field | Value |
|---|---|
| crawl-delay | 600 |
ahrefsbot
go-http-client/1.1
commoncrawl.org/faq
qwantify
qiosk bot
xydo
rogerbot/1.1
paperlibot
istellabot/t.1.13
bubing
dotbot
python-requests/2.13.0
semantic-visions
trendiction
r6_commentreader
grouphigh/1.0
booglebot
megaindex
openvas
grapeshotcrawler
semrushbot
proximic
marketwirebot
nutch
scoutjet
mj12bot
mcafee agent
btcrawler
panscient.com
httpcomponents/1.1
riddler
apache-httpclient/4.5
domain re-animator
muckrack
python/3.4 aiohttp/1.0.5
pcore-http/v0.24.5
dts agent
blp_bbot
blexbot
spbot
sgrunt
synthesio
kraken
synaptive
acunetix
chinaclaw
extract
extractorpro
feedfinder
flashget
flicky
g00g1e
go!zilla
grabnet
jakarta
java
joc web spider
netspider
octopus
reget
riddler
wget
sucker
web sucker
heritrix
snoopy
http::lite
phpcrawl
sbider
zing-bottabot/2.0
zing-bottabot/1.0
scooperbot
nutch
| Rule | Path |
|---|---|
| Disallow | / |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 600 |
Other Records
| Field | Value |
|---|---|
| sitemap | http://www.oilandgas360.com/sitemap_index.xml |