Log In Sign Up

expertclick.com
robots.txt

Robots Exclusion Standard data for expertclick.com

Resource Scan

Scan Details

Site Domain expertclick.com
Base Domain expertclick.com
Scan Status Ok
Last Scan2024-09-25T18:56:53+00:00
Next Scan 2024-10-02T18:56:53+00:00

Last Scan

Scanned2024-09-25T18:56:53+00:00
URL https://expertclick.com/robots.txt
Domain IPs 162.254.209.219
Response IP 162.254.209.219
Found Yes
Hash 3afaff87beef31b5e1fce648a2a13e92260e08912d83de5367c3c748a3fab2e5
SimHash 3018115149f5

Groups

*

Rule Path
Disallow /Admin/
Disallow /DBPickUpFiles/
Disallow /radiotour/

yahoo-slurp

Rule Path
Disallow /newsreleasewire/

msnbot

Rule Path
Disallow /newsreleasewire/

slurp

Rule Path
Disallow /newsreleasewire/

twitterbot

Rule Path
Allow /images

facebookexternalhit

Rule Path
Allow /images

*

Rule Path
Allow /
Back to top

Other Records

Field Value
sitemap https://www.expertclick.com/GoogleNewsSiteMap.xml
Back to top

Comments

  • Exclude Files From All Robots:
  • Certain social media sites are whitelisted to allow crawlers to access page markup when links to /images are shared.
  • End robots.txt file
Back to top