experts-exchange.com
robots.txt

Robots Exclusion Standard data for experts-exchange.com

Resource Scan

Scan Details

Site Domain experts-exchange.com
Base Domain experts-exchange.com
Scan Status Ok
Last Scan2024-11-08T09:44:53+00:00
Next Scan 2024-11-15T09:44:53+00:00

Last Scan

Scanned2024-11-08T09:44:53+00:00
URL https://experts-exchange.com/robots.txt
Redirect https://www.experts-exchange.com/robots.txt
Redirect Domain www.experts-exchange.com
Redirect Base experts-exchange.com
Domain IPs 104.22.4.165, 104.22.5.165, 172.67.36.241, 2606:4700:10::6816:4a5, 2606:4700:10::6816:5a5, 2606:4700:10::ac43:24f1
Redirect IPs 104.22.4.165, 104.22.5.165, 172.67.36.241, 2606:4700:10::6816:4a5, 2606:4700:10::6816:5a5, 2606:4700:10::ac43:24f1
Response IP 172.67.36.241
Found Yes
Hash 2cbadf9d3eeba005b9966a65eef68bc7acf72139203d19601455c9d01f9a36c1
SimHash d21c9c7a8e90

Groups

*

Rule Path
Disallow /login.jsp
Disallow /checkout.jsp
Disallow /signup.jsp
Disallow /forgotPassword.jsp
Disallow /searchResults.jsp
Disallow /advancedSearch.jsp
Disallow /search.jsp
Disallow /cookiePage.jsp
Disallow /viewCodeSnippet.jsp
Disallow /getCaptcha/
Disallow /async/
Disallow /async/nb.jsp
Disallow /*.async$
Disallow /questions/_next/
Allow /questions/_next/static/css/*

dotbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

moatbot

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

omgili

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.experts-exchange.com/siteMapXML.jsp