databasesets.com
robots.txt
Robots Exclusion Standard data for databasesets.com
Resource Scan
Scan Details
Site Domain | databasesets.com |
Base Domain | databasesets.com |
Scan Status | Ok |
Last Scan | 2024-06-18T21:26:57+00:00 |
Next Scan | 2024-06-25T21:26:57+00:00 |
Last Scan
Scanned | 2024-06-18T21:26:57+00:00 |
URL | https://databasesets.com/robots.txt |
Domain IPs | 104.26.10.79, 104.26.11.79, 172.67.75.116, 2606:4700:20::681a:a4f, 2606:4700:20::681a:b4f, 2606:4700:20::ac43:4b74 |
Response IP | 104.26.11.79 |
Found | Yes |
Hash | 3ef301bcb24c0d15092b08c79e168aa7778fac7926d829674f4612b5d8583c97 |
SimHash | 3c169f08c505 |
Groups
*
Rule | Path |
---|---|
Allow | /sites/default/files/ |
Disallow | /cdn-cgi/ |
Disallow | /*comment/reply/* |
Disallow | /*image_captcha* |
Disallow | /node/26 |
Disallow | /node/52 |
Disallow | /node/51 |
Disallow | /node/27 |
Disallow | /node/32 |
Disallow | /node/28 |
Disallow | /node/34 |
Disallow | /node/29 |
Disallow | /node/30 |
Allow | /*misc/*.css$ |
Allow | /*misc/*.css? |
Allow | /*misc/*.js$ |
Allow | /*misc/*.js? |
Allow | /*misc/*.gif |
Allow | /*misc/*.jpg |
Allow | /*misc/*.jpeg |
Allow | /*misc/*.png |
Allow | /*modules/*.css$ |
Allow | /*modules/*.css? |
Allow | /*modules/*.js$ |
Allow | /*modules/*.js? |
Allow | /*modules/*.gif |
Allow | /*modules/*.jpg |
Allow | /*modules/*.jpeg |
Allow | /*modules/*.png |
Allow | /*profiles/*.css$ |
Allow | /*profiles/*.css? |
Allow | /*profiles/*.css? |
Allow | /*profiles/*.js$ |
Allow | /*profiles/*.js? |
Allow | /*profiles/*.gif |
Allow | /*profiles/*.jpg |
Allow | /*profiles/*.jpeg |
Allow | /*profiles/*.png |
Allow | /*themes/*.css$ |
Allow | /*themes/*.css? |
Allow | /*themes/*.js$ |
Allow | /*themes/*.js? |
Allow | /*themes/*.gif |
Allow | /*themes/*.jpg |
Allow | /*themes/*.jpeg |
Allow | /*themes/*.png |
Disallow | /*includes/ |
Disallow | /*misc/ |
Disallow | /*modules/ |
Disallow | /*profiles/ |
Disallow | /*scripts/ |
Disallow | /*themes/ |
Disallow | /*CHANGELOG.txt |
Disallow | /*cron.php |
Disallow | /*INSTALL.mysql.txt |
Disallow | /*INSTALL.pgsql.txt |
Disallow | /*INSTALL.sqlite.txt |
Disallow | /*install.php |
Disallow | /*INSTALL.txt |
Disallow | /*LICENSE.txt |
Disallow | /*MAINTAINERS.txt |
Disallow | /*update.php |
Disallow | /*UPGRADE.txt |
Disallow | /*xmlrpc.php |
Disallow | /*admin/ |
Disallow | /*comment/reply/ |
Disallow | /*filter/tips/ |
Disallow | /*node/add/ |
Disallow | /*search/ |
Disallow | /*user/register/ |
Disallow | /*user/password/ |
Disallow | /*user/login/ |
Disallow | /*user/logout/ |
Disallow | /*?q=admin%2F |
Disallow | /*?q=comment%2Freply%2F |
Disallow | /*?q=filter%2Ftips%2F |
Disallow | /*?q=node%2Fadd%2F |
Disallow | /*?q=search%2F |
Disallow | /*?q=user%2Fpassword%2F |
Disallow | /*?q=user%2Fregister%2F |
Disallow | /*?q=user%2Flogin%2F |
Disallow | /*?q=user%2Flogout%2F |
Disallow | /*/user/login/ |
Other Records
Field | Value |
---|---|
sitemap | https://databasesets.com/sitemap.xml |
sitemap | https://databasesets.com/rss.xml |
sitemap | https://databasesets.com/sitemap_national-company_country.xml |
sitemap | https://databasesets.com/sitemap_national-company_id.xml |
sitemap | https://databasesets.com/sitemap_national-company.xml |
Comments