diffen.com
robots.txt
Robots Exclusion Standard data for diffen.com
Resource Scan
Scan Details
| Site Domain | diffen.com |
| Base Domain | diffen.com |
| Scan Status | Ok |
| Last Scan | 2024-09-17T22:07:05+00:00 |
| Next Scan | 2024-10-17T22:07:05+00:00 |
Last Scan
| Scanned | 2024-09-17T22:07:05+00:00 |
| URL | https://diffen.com/robots.txt |
| Redirect | https://www.diffen.com/robots.txt |
| Redirect Domain | www.diffen.com |
| Redirect Base | diffen.com |
| Domain IPs | 104.26.8.153, 104.26.9.153, 172.67.69.158, 2606:4700:20::681a:899, 2606:4700:20::681a:999, 2606:4700:20::ac43:459e |
| Redirect IPs | 151.101.130.132, 151.101.194.132, 151.101.2.132, 151.101.66.132 |
| Response IP | 199.232.46.132 |
| Found | Yes |
| Hash | 0e8ec121fb9648821389e727d269cb666498c6f43a616b5ef837624dd99bb5fe |
| SimHash | d64d1850c8b7 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /*Special%3AContribu |
| Disallow | /*Special%3AWhatlinkshere |
| Disallow | /difference/ajaks.php* |
| Disallow | /difference/Image%3A* |
| Disallow | /*action%3Dhistory |
| Disallow | /*action%3Dedit |
| Disallow | /*oldid%3D |
| Disallow | /*diff%3D |
| Disallow | /difference/Special%3AInformation/ |
| Disallow | /*Special%3AEditTable |
| Disallow | /*Special%3AUserlogin |
| Disallow | /*Special%3AFBLogin |
| Disallow | /difference/2008* |
| Disallow | /difference/2009* |
| Disallow | /difference/2010* |
| Disallow | /difference/2011* |
| Disallow | /*imagesapi.php |
| Disallow | /nc/ |
googlebot
| Rule | Path |
|---|---|
| Disallow | /*Special%3AContribu |
| Disallow | /*Special%3AWhatlinkshere |
| Disallow | /difference/ajaks.php* |
| Disallow | /*action%3Dedit |
| Disallow | /Intel |
| Disallow | /AMD |
| Disallow | /*Special%3AEditTable |
| Disallow | /*Special%3AUserlogin |
| Disallow | /*wikiget/ |
| Disallow | /*imagesapi.php |
| Disallow | /*chat.php |
| Disallow | /nc/ |
petalbot
mj12bot
wellknownbot
ahrefsbot
gptbot
ccbot
google-extended
yaanibot
meltwaternews www.meltwater.com
meltwaternews
facebookbot
googleother
omigilibot
awariorssbot
awariosmartbot
bytespider
ccbot
chatgpt-user
claudebot
claude-web
cohere-ai
dataforseobot
diffbot
friendlycrawler
google-extended
googleother
gptbot
img2dataset
imagesiftbot
magpie-crawler
meltwater
omgili
omgilibot
peer39_crawler
peer39_crawler/1.0
perplexitybot
piplbot
scoop.it
seekr
youbot
| Rule | Path |
|---|---|
| Disallow | / |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.diffen.com/sitemaps/sitemapindex.xml |