Log In Sign Up

media.fstoppers.com
robots.txt

Robots Exclusion Standard data for media.fstoppers.com

Resource Scan

Scan Details

Site Domain media.fstoppers.com
Base Domain fstoppers.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-10-15T10:52:28+00:00
Next Scan 2025-01-13T10:52:28+00:00

Last Successful Scan

Scanned2023-08-29T18:33:23+00:00
URL https://media.fstoppers.com/robots.txt
Domain IPs 54.230.112.114, 54.230.112.122, 54.230.112.50, 54.230.112.63
Response IP 18.66.97.10
Found Yes
Hash 2564b735ec79e38884edefa9dac5621a5efb251cf945584c314d9c7af6a425a2
SimHash 11947791c6d8

Groups

goolgebot-news

Rule Path
Disallow /profile/
Disallow /user/
Disallow /photo/
Disallow /list/
Disallow /popular/
Disallow /latest/
Disallow /newest/
Disallow /top/
Disallow /community/
Disallow /product/
Disallow /sites/
Disallow /media/
Disallow /email/
Disallow /invoice/
Disallow /store
Disallow /community
Disallow /group/
Disallow /groups/
Disallow /search
Disallow /achievement/
Disallow /license/

*

Rule Path
Allow /page/
Allow /misc/*.css$
Allow /misc/*.css?
Allow /misc/*.js$
Allow /misc/*.js?
Allow /misc/*.gif
Allow /misc/*.jpg
Allow /misc/*.jpeg
Allow /misc/*.png
Allow /modules/*.css$
Allow /modules/*.css?
Allow /modules/*.js$
Allow /modules/*.js?
Allow /modules/*.gif
Allow /modules/*.jpg
Allow /modules/*.jpeg
Allow /modules/*.png
Allow /themes/*.css$
Allow /themes/*.css?
Allow /themes/*.js$
Allow /themes/*.js?
Allow /themes/*.gif
Allow /themes/*.jpg
Allow /themes/*.jpeg
Allow /themes/*.png
Disallow /includes/
Disallow /misc/
Disallow /modules/
Disallow /profiles/
Disallow /scripts/
Disallow /themes/
Disallow /sites/all/modules/contrib/radioactivity/
Disallow /p/
Disallow /CHANGELOG.txt
Disallow /cron.php
Disallow /INSTALL.mysql.txt
Disallow /INSTALL.pgsql.txt
Disallow /INSTALL.sqlite.txt
Disallow /install.php
Disallow /INSTALL.txt
Disallow /LICENSE.txt
Disallow /MAINTAINERS.txt
Disallow /update.php
Disallow /UPGRADE.txt
Disallow /xmlrpc.php
Disallow /expire
Disallow /admin/
Disallow /ajax_error_behavior/
Disallow /comment/
Disallow /filter/
Disallow /comment/reply/
Disallow /filter/tips/
Disallow /node/add/
Disallow /search
Disallow /search/
Disallow /user/password/
Disallow /user/login
Disallow /user/login/
Disallow /user/logout/
Disallow /message/
Disallow /messages/
Disallow /flag/
Disallow /views/
Disallow /worksafe/
Disallow /file/
Disallow /pin/
Disallow /morecomments/
Disallow /toboggan/
Disallow /email/
Disallow /invoice/
Disallow /ajax/
Disallow /achievement/
Disallow /system/
Disallow /license/
Disallow /json/
Disallow *?destination=*
Disallow *?rate=*
Disallow /?s=*
Disallow /?r=*
Disallow /?rate=*
Disallow /?q=*
Disallow /*?*&t=*
Disallow /*?*&k=*
Disallow /*?*&0=*
Disallow /*?*&rate=*
Disallow /*?*&destination=*
Disallow /*?*&view=*
Disallow /*?*&js=*
Disallow /*?*&item=*
Disallow /*?vip%2F*
Disallow /*?u=*
Disallow /?q=admin%2F
Disallow /?q=comment%2Freply%2F
Disallow /?q=filter%2Ftips%2F
Disallow /?q=node%2Fadd%2F
Disallow /?q=search%2F
Disallow /?q=user%2Fpassword%2F
Disallow /?q=user%2Fregister%2F
Disallow /?q=user%2Flogin%2F
Disallow /?q=user%2Flogout%2F
Disallow /*?quicktabs_*
Disallow /*%26quicktabs_*
Back to top

Other Records

Field Value
sitemap http://www.example.com/sitemap.xml
Back to top

Comments

  • robots.txt
  • Old 301ed homepages
  • CSS, JS, Images
  • Directories
  • Files
  • Paths (clean URLs)
  • Disallow: /node/
  • Disallow: /user
  • Disallow: /user/
  • Disallow: /media/
  • Paths (no clean URLs)
  • Disallow: /*?/*
  • No access for quicktabs in the URL
  • Disallow: *?quicktabs_*
  • Disallow: *&quicktabs_*
Back to top