cdn.oilprice.com
robots.txt
Robots Exclusion Standard data for cdn.oilprice.com
Resource Scan
Scan Details
| Site Domain | cdn.oilprice.com |
| Base Domain | oilprice.com |
| Scan Status | Ok |
| Last Scan | 2024-11-15T11:27:32+00:00 |
| Next Scan | 2024-11-22T11:27:32+00:00 |
Last Scan
| Scanned | 2024-11-15T11:27:32+00:00 |
| URL | http://cdn.oilprice.com/robots.txt |
| Redirect | https://oilprice.com:443/robots.txt |
| Redirect Domain | oilprice.com |
| Redirect Base | oilprice.com |
| Domain IPs | 2600:9000:2721:5000:10:4f52:7800:21, 2600:9000:2721:8a00:10:4f52:7800:21, 2600:9000:2721:8c00:10:4f52:7800:21, 2600:9000:2721:9600:10:4f52:7800:21, 2600:9000:2721:a00:10:4f52:7800:21, 2600:9000:2721:ac00:10:4f52:7800:21, 2600:9000:2721:de00:10:4f52:7800:21, 2600:9000:2721:fa00:10:4f52:7800:21, 3.165.91.112, 3.165.91.175, 3.165.91.208, 3.165.91.25 |
| Redirect IPs | 18.217.139.70, 3.131.62.18, 3.22.9.67 |
| Response IP | 3.131.62.18 |
| Found | Yes |
| Hash | 43e6ce485951499942abbd8b39361ed6ee9508a649e9bd4fed1c033281bbed14 |
| SimHash | 0b12cd50e6c2 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /cgi-bin/ |
| Disallow | /search |
| Disallow | /goto/ |
| Disallow | /commodity-price-charts?1* |
| Disallow | /_private/ |
| Disallow | /_vti_bin/ |
| Disallow | /_vti_cnf/ |
| Disallow | /_vti_log/ |
| Disallow | /_vti_pvt/ |
| Disallow | /_vti_txt/ |
| Disallow | /newsletters/ |
| Disallow | /feed/3 |
| Disallow | /newsfeeds/redirect/ |
| Disallow | /salestool |
| Disallow | /salestool/ |
| Disallow | /articles/sponsored |
| Disallow | /articles/sponsored/ |
| Disallow | /auth/* |
| Disallow | /account/* |
| Disallow | /administrator/ |
| Disallow | /cache/ |
| Disallow | /components/ |
| Disallow | /component/ |
| Disallow | /includes/ |
| Disallow | /installation/ |
| Disallow | /language/ |
| Disallow | /libraries/ |
| Disallow | /media/ |
| Disallow | /modules/ |
| Disallow | /plugins/ |
| Disallow | /templates/ |
| Disallow | /tmp/ |
| Disallow | /xmlrpc/ |
| Disallow | /ajax/csrf |
| Disallow | /ajax/add_view/* |
| Disallow | /ajax/return_country |
| Disallow | /widgets/* |
| Disallow | /freewidgets/* |
| Disallow | /opc/* |
| Disallow | /ajax/get_banner/* |
| Disallow | /ajax/get_user_blends |
| Disallow | /ajax/add_pushcrew |
| Disallow | /ajax/* |
| Disallow | /ajax/get_chart_prices/ |
| Disallow | /commodity-widgets.php |
| Disallow | /iframes/barcharts |
| Disallow | /market-intelligence/signup_success/6 |
| Disallow | /newwidgets.php?3 |
| Disallow | /specialreports/8_Mega_Trends.html |
| Disallow | /widgets/energyproduction.html |
| Disallow | /*data.json$ |
| Disallow | /*data.json?t=* |
| Disallow | /*last.json$ |
| Disallow | /*last.json?t=* |
| Disallow | /Energy/Energy-General/The-Gas-Find-That-Could-Transform-Europes-Energy-Future.html |
| Disallow | /Energy/Energy-General/This-Could-Be-A-Gamechanger-For-Natural-Gas-In-Europe.html |
| Disallow | /Energy/Energy-General/How-To-Profit-From-Europes-800-Billion-Energy-Crisis.html |
| Disallow | /Energy/Energy-General/The-No1-Energy-Stock-for-2024.html |
| Disallow | /Energy/Energy-General/Forgotten-Gas-Reserves-Could-Be-A-Gamechanger-For-European-Energy.html |
| Disallow | /Energy/Energy-General/Meet-the-Man-Using-AI-To-Revive-An-Oil-Gas-Play-Supermajors-Left-For-Dead.html |
| Disallow | /Energy/Natural-Gas/How-Artificial-Intelligence-Could-Trigger-a-Natural-Gas-Boom-in-Europe.html |
Other Records
| Field | Value |
|---|---|
| sitemap | https://oilprice.com/sitemap_index.xml |
| sitemap | https://oilprice.com/sitemap.xml |
| sitemap | https://oilprice.com/googlenews.xml |
Comments