Log In Sign Up

fda.gov
robots.txt

Robots Exclusion Standard data for fda.gov

Resource Scan

Scan Details

Site Domain fda.gov
Base Domain fda.gov
Scan Status Ok
Last Scan2025-02-16T14:32:10+00:00
Next Scan 2025-03-18T14:32:10+00:00

Last Scan

Scanned2025-02-16T14:32:10+00:00
URL https://www.fda.gov/robots.txt
Domain IPs 184.85.114.4, 2600:1413:b000:681::308a, 2600:1413:b000:696::308a
Response IP 184.87.105.159
Found Yes
Hash b28f1d0132f966047fe40018266dbff647024fa01c020b80e44b5c3f167114ed
SimHash 15d69f439800

Groups

vspider

Rule Path
Disallow /

usasearch

Rule Path
Allow /core/*.css$
Allow /core/*.css?
Allow /core/*.js$
Allow /core/*.js?
Allow /core/*.gif
Allow /core/*.jpg
Allow /core/*.jpeg
Allow /core/*.png
Allow /core/*.svg
Allow /profiles/*.css$
Allow /profiles/*.css?
Allow /profiles/*.js$
Allow /profiles/*.js?
Allow /profiles/*.gif
Allow /profiles/*.jpg
Allow /profiles/*.jpeg
Allow /profiles/*.png
Allow /profiles/*.svg
Disallow /core/
Disallow /profiles/
Disallow /README.txt
Disallow /web.config
Disallow /admin/
Disallow /comment/reply/
Disallow /filter/tips
Disallow /node/
Disallow /file/
Disallow /taxonomy/
Disallow /search/
Disallow /user/register/
Disallow /user/password/
Disallow /user/login/
Disallow /user/logout/
Disallow /index.php/admin/
Disallow /index.php/comment/reply/
Disallow /index.php/filter/tips
Disallow /index.php/node/add/
Disallow /index.php/search/
Disallow /index.php/user/password/
Disallow /index.php/user/register/
Disallow /index.php/user/login/
Disallow /index.php/user/logout/

Other Records

Field Value
crawl-delay 2

*

Rule Path Comment
Disallow /health don't crawl healthcheck
Allow /core/*.css$ -
Allow /core/*.css? -
Allow /core/*.js$ -
Allow /core/*.js? -
Allow /core/*.gif -
Allow /core/*.jpg -
Allow /core/*.jpeg -
Allow /core/*.png -
Allow /core/*.svg -
Allow /profiles/*.css$ -
Allow /profiles/*.css? -
Allow /profiles/*.js$ -
Allow /profiles/*.js? -
Allow /profiles/*.gif -
Allow /profiles/*.jpg -
Allow /profiles/*.jpeg -
Allow /profiles/*.png -
Allow /profiles/*.svg -
Disallow /core/ -
Disallow /profiles/ -
Disallow /README.txt -
Disallow /web.config -
Disallow /admin/ -
Disallow /comment/reply/ -
Disallow /filter/tips -
Disallow /node/ -
Disallow /file/ -
Disallow /taxonomy/ -
Disallow /search/ -
Disallow /user/register/ -
Disallow /user/password/ -
Disallow /user/login/ -
Disallow /user/logout/ -
Disallow /index.php/admin/ -
Disallow /index.php/comment/reply/ -
Disallow /index.php/filter/tips -
Disallow /index.php/node/add/ -
Disallow /index.php/search/ -
Disallow /index.php/user/password/ -
Disallow /index.php/user/register/ -
Disallow /index.php/user/login/ -
Disallow /index.php/user/logout/ -

Other Records

Field Value Comment
crawl-delay 30 wait 30 seconds before starting a new URL request default=30
Back to top

Other Records

Field Value
sitemap https://www.fda.gov/sitemap.xml
Back to top

Comments

  • Added for Bristol-Myers on Sept 2005
  • Search.gov
  • CSS, JS, Images
  • Directories
  • Files
  • Paths (clean URLs)
  • Paths (no clean URLs)
  • For all other crawlers
  • CSS, JS, Images
  • Directories
  • Files
  • Paths (clean URLs)
  • Paths (no clean URLs)
Back to top