crash.net
robots.txt
Robots Exclusion Standard data for crash.net
Resource Scan
Scan Details
Site Domain | crash.net |
Base Domain | crash.net |
Scan Status | Ok |
Last Scan | 2024-04-20T01:17:06+00:00 |
Next Scan | 2024-04-27T01:17:06+00:00 |
Last Scan
Scanned | 2024-04-20T01:17:06+00:00 |
URL | https://crash.net/robots.txt |
Redirect | https://www.crash.net/robots.txt |
Redirect Domain | www.crash.net |
Redirect Base | crash.net |
Domain IPs | 35.227.195.152 |
Redirect IPs | 35.227.195.152 |
Response IP | 35.227.195.152 |
Found | Yes |
Hash | d505fc8eb5a633e1abf51d3976376dfd50ecc62748d7e2caf975b91283cb99ba |
SimHash | 3d94dd08ef68 |
Groups
*
Rule | Path |
---|---|
Allow | /core/*.css$ |
Allow | /core/*.css? |
Allow | /core/*.js$ |
Allow | /core/*.js? |
Allow | /core/*.gif |
Allow | /core/*.jpg |
Allow | /core/*.jpeg |
Allow | /core/*.png |
Allow | /core/*.svg |
Allow | /misc/*.css$ |
Allow | /misc/*.css? |
Allow | /misc/*.js$ |
Allow | /misc/*.js? |
Allow | /misc/*.gif |
Allow | /misc/*.jpg |
Allow | /misc/*.jpeg |
Allow | /misc/*.png |
Allow | /modules/*.css$ |
Allow | /modules/*.css? |
Allow | /modules/*.js$ |
Allow | /modules/*.js? |
Allow | /modules/*.gif |
Allow | /modules/*.jpg |
Allow | /modules/*.jpeg |
Allow | /modules/*.png |
Allow | /profiles/*.css$ |
Allow | /profiles/*.css? |
Allow | /profiles/*.js$ |
Allow | /profiles/*.js? |
Allow | /profiles/*.gif |
Allow | /profiles/*.jpg |
Allow | /profiles/*.jpeg |
Allow | /profiles/*.png |
Allow | /profiles/*.svg |
Allow | /themes/*.css$ |
Allow | /themes/*.css? |
Allow | /themes/*.js$ |
Allow | /themes/*.js? |
Allow | /themes/*.gif |
Allow | /themes/*.jpg |
Allow | /themes/*.jpeg |
Allow | /themes/*.png |
Disallow | /core/ |
Disallow | /includes/ |
Disallow | /misc/ |
Disallow | /modules/ |
Disallow | /profiles/ |
Disallow | /scripts/ |
Disallow | /themes/ |
Disallow | /CHANGELOG.txt |
Disallow | /cron.php |
Disallow | /INSTALL.mysql.txt |
Disallow | /INSTALL.pgsql.txt |
Disallow | /INSTALL.sqlite.txt |
Disallow | /install.php |
Disallow | /INSTALL.txt |
Disallow | /LICENSE.txt |
Disallow | /MAINTAINERS.txt |
Disallow | /update.php |
Disallow | /UPGRADE.txt |
Disallow | /README.txt |
Disallow | /web.config |
Disallow | /xmlrpc.php |
Disallow | /admin/ |
Disallow | /comment/reply/ |
Disallow | /comment/* |
Disallow | /id/comment/* |
Disallow | /filter/tips/ |
Disallow | /node/add/ |
Disallow | /search/ |
Disallow | /user/* |
Disallow | /id/user/* |
Disallow | /?q=admin%2F |
Disallow | /?q=comment%2Freply%2F |
Disallow | /?q=filter%2Ftips%2F |
Disallow | /?q=node%2Fadd%2F |
Disallow | /?q=search%2F |
Disallow | /?q=user%2Fpassword%2F |
Disallow | /?q=user%2Fregister%2F |
Disallow | /?q=user%2Flogin%2F |
Disallow | /?q=user%2Flogout%2F |
Disallow | /index.php/admin/ |
Disallow | /index.php/comment/reply/ |
Disallow | /index.php/filter/tips |
Disallow | /index.php/node/add/ |
Disallow | /index.php/search/ |
Disallow | /index.php/user/password |
Disallow | /index.php/user/register |
Disallow | /index.php/user/login |
Disallow | /index.php/user/logout |
Disallow | /news/articlePrint.aspx |
Disallow | /*/print.html |
Disallow | /*/email.html |
Disallow | /contact.html |
Disallow | /login.html |
Disallow | /register.html |
Disallow | /*/tags/ |
Disallow | /*/newcomment/ |
Disallow | /*/newComment/ |
Disallow | /*/have_your_say/ |
Disallow | /*/have_your_say_author/ |
Disallow | /*/have_your_say_comments/ |
Disallow | /*/reportComment/ |
Disallow | /*/picturearchive/ |
Disallow | /*/grid_girls/ |
Disallow | /*/top_grid_girls/ |
Disallow | /*/kw/ |
Disallow | /*/igallery/ |
Disallow | /*/services/ |
Disallow | /*/radio_archive/ |
Disallow | /radio.asp |
Disallow | /radio.htm |
Disallow | /radioplayer/ |
Disallow | /*/video_archive/ |
Disallow | /*/audio_archive/ |
Disallow | /video.asp |
Disallow | /video.htm |
Disallow | /*/vote/ |
Disallow | /*/poll/ |
Disallow | /*/poll_results/ |
Disallow | /*/shop/ |
Disallow | /*/picture_search/ |
Disallow | /*/free_podcasts/ |
Disallow | /pictures/igallery.aspx |
Disallow | /videos/eplayer.aspx |
Disallow | /PublicServices/ |
Disallow | /ScriptResource.axd |
Disallow | /WebResource.axd |
Disallow | /videos/eplayer.aspx |
Disallow | /1037673/* |
Disallow | /21100889/* |
Disallow | /f1/related_news* |
Disallow | /f1/team* |
Disallow | /s3/files/styles/* |
Disallow | /node/* |
Disallow | /id/node/* |
Disallow | /node?* |
Disallow | /id/node?* |
Disallow | /files/* |
Disallow | /id/files/* |
Disallow | /search/* |
Disallow | /id/search/* |
Disallow | /content/* |
Disallow | /id/content/* |
Disallow | /countries/* |
Disallow | /id/countries/* |
Disallow | /images/* |
Disallow | /*/images/* |
Disallow | /*/large_picture/* |
Disallow | /*/youtube/* |
Allow | /sites/all/themes/crash/images/* |
Allow | /themes/custom/crash_theme/images/* |
Other Records
Field | Value |
---|---|
sitemap | https://www.crash.net/sitemap.xml |
sitemap | https://www.crash.net/image-sitemap.xml |
sitemap | https://www.crash.net/video-sitemap.xml |
sitemap | https://www.crash.net/googlenews.xml |
Comments