edf.org
robots.txt

Robots Exclusion Standard data for edf.org

Resource Scan

Scan Details

Site Domain edf.org
Base Domain edf.org
Scan Status Ok
Last Scan5/13/2025, 8:15:35 AM
Next Scan 6/12/2025, 8:15:35 AM

Last Scan

Scanned5/13/2025, 8:15:35 AM
URL https://edf.org/robots.txt
Redirect https://www.edf.org/robots.txt
Redirect Domain www.edf.org
Redirect Base edf.org
Domain IPs 151.101.194.216
Redirect IPs 151.101.130.216, 151.101.194.216, 151.101.2.216, 151.101.66.216
Response IP 151.101.66.216
Found Yes
Hash fb3671795fc0fc72fe79f840c8a440d699e4a57e66ca7777ecec79484b7ec76c
SimHash 39903559c544

Groups

*

Rule Path
Allow /core/*.css$
Allow /core/*.css?
Allow /core/*.js$
Allow /core/*.js?
Allow /core/*.gif
Allow /core/*.jpg
Allow /core/*.jpeg
Allow /core/*.png
Allow /core/*.svg
Allow /profiles/*.css$
Allow /profiles/*.css?
Allow /profiles/*.js$
Allow /profiles/*.js?
Allow /profiles/*.gif
Allow /profiles/*.jpg
Allow /profiles/*.jpeg
Allow /profiles/*.png
Allow /profiles/*.svg
Disallow /core/
Disallow /profiles/
Disallow /includes/
Disallow /misc/
Disallow /modules/
Disallow /scripts/
Disallow /themes/
Disallow /sites/default/files/private/
Disallow /README.txt
Disallow /README.md
Disallow /web.config
Disallow /LICENSE.txt
Disallow /update.php
Disallow /admin/
Disallow /comment/reply/
Disallow /filter/tips
Disallow /node/add/
Disallow /search/
Disallow /user/register
Disallow /user/password
Disallow /user/login
Disallow /user/logout
Disallow /media/oembed
Disallow /*/media/oembed
Disallow /user
Disallow /admin
Disallow /comment/reply
Disallow /filter/tip
Disallow /node/add
Disallow /search
Disallow /index.php/admin/
Disallow /index.php/comment/reply/
Disallow /index.php/filter/tips
Disallow /index.php/node/add/
Disallow /index.php/search/
Disallow /index.php/user/password
Disallow /index.php/user/register
Disallow /index.php/user/login
Disallow /index.php/user/logout
Disallow /index.php/media/oembed
Disallow /index.php/*/media/oembed
Disallow /index.php/user
Disallow /index.php/admin
Disallow /index.php/comment/reply
Disallow /index.php/filter/tip
Disallow /index.php/node/add
Disallow /index.php/search
Disallow /sites/default/files/Oceans-Update-Feb-20-2013.pdf
Disallow /sites/default/files/Three-short-updates-for-the-Oceans-Committee.pdf
Disallow /sites/default/files/Inside-the-BP-Trial-March-5-2013.pdf
Disallow /sites/default/files/Safer-chemical-principles-Feb-22-2013.pdf
Disallow /sites/default/files/EDF-ICP_fact-sheet.pdf
Disallow /sites/default/files/EDF-PA-Comments-11-12-15.pdf
Disallow /sites/default/files/documents/North-Carolina-Community-Impacts-Agricultural-Methane-Biogas.pdf
Disallow /sites/edfclimatecorps.org/files/2016_Climate_Corps_Training_Agenda.pdf
Disallow /sites/edfclimatecorps.org/files/Fellowship_Training_Prework.pdf
Disallow /sites/edfclimatecorps.org/files/EDFCC-training-travel-information.pdf
Disallow /sites/edfclimatecorps.org/files/EDFCC-pathway-to-training.pdf
Disallow /sites/edfclimatecorps.org/files/EDFCC-climate-corps-agenda.pdf
Disallow /sites/edfclimatecorps.org/files/RFP-EDF-Climate-Corps-OnlineTraining.pdf
Disallow /sites/edfclimatecorps.org/files/elective-sign-up.pdf
Disallow /sites/edfclimatecorps.org/files/ESE-Sustainability-Tours.pdf
Disallow /sites/edfclimatecorps.org/files/ese-2017-agenda.pdf
Disallow /sites/default/files/energy-conference-bingo-cards.pdf
Disallow /sites/edfclimatecorps.org/files/summit-tours.pdf
Disallow /sites/edfclimatecorps.org/files/EDF_Climate_Corps_Summit_Agenda.pdf
Disallow /contact/email/*
Disallow /related_blog_posts/*
Disallow /slideshow/
Disallow /less-carbon-more-jobs/map/
Disallow /less-carbon-more-jobs/district/
Disallow /our-work-fighting-climate-change
Disallow /how-you-can-help-2
Disallow /climate/cap-and-trade-case-studies
Disallow /sites/default/files/CT_*
Disallow /living-style-guide/*
Disallow /george-shultz-edfs-approach-natural-gas-gains-respect
Disallow /george-shultz-edfs-approach-natural-gas-gains-respect-2
Disallow /style-guide
Disallow /home-test
Disallow /health/myexposome-wristband
Disallow /50th-anniversary-celebration
Disallow /methane-moment-event
Disallow /PPTrsvp
Disallow /response-forbes-michael-shellenberger
Disallow /node/23208
Disallow /methanesat-materials
Disallow /greenwire
Disallow /mission-methane
Disallow /mission-methane-revealed
Disallow /climatetech
Disallow /climate-week/turning-climate-risk-climate-progress
Disallow /people/feed/*
Disallow /aggregator/sources
Disallow /aggregator/sources/*
Disallow /sites/default/files/AR2019/EDF_Annual_Report_2019.pdf
Disallow /Zi7B
Disallow /Zi7X
Disallow /Zi78
Disallow /Zi72

Other Records

Field Value
crawl-delay 10

Other Records

Field Value
sitemap https://www.edf.org/sitemap.xml

Comments

  • robots.txt
  • This file is to prevent the crawling and indexing of certain parts
  • of your site by web crawlers and spiders run by sites like Yahoo!
  • and Google. By telling these "robots" where not to go on your site,
  • you save bandwidth and server resources.
  • This file will be ignored unless it is at the root of your host:
  • Used: http://example.com/robots.txt
  • Ignored: http://example.com/site/robots.txt
  • For more information about the robots.txt standard, see:
  • http://www.robotstxt.org/robotstxt.html
  • Limit time window to every 10 seconds (not followed by Google)
  • CSS, JS, Images
  • Directories
  • Directories: Added 2017-12-01
  • Files
  • Files: Added 2017-12-01
  • Paths (clean URLs)
  • Paths (clean URLs, trailing slash removed)
  • Paths (no clean URLs)
  • Paths (no clean URLs, trailing slash removed)
  • Custom files
  • Custom paths
  • Custom: AR2019
  • Custom: Sitemap