Log In Sign Up

fodlan.com
robots.txt

Robots Exclusion Standard data for fodlan.com

Resource Scan

Scan Details

Site Domain fodlan.com
Base Domain fodlan.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't establish SSL connection.
Last Scan2024-11-15T12:11:20+00:00
Next Scan 2024-11-22T12:11:20+00:00

Last Successful Scan

Scanned2024-11-07T12:10:46+00:00
URL http://fodlan.com/robots.txt
Redirect https://fodlan.jcink.net/robots.txt
Redirect Domain fodlan.jcink.net
Redirect Base jcink.net
Domain IPs 157.7.174.95
Redirect IPs 104.161.46.138
Response IP 104.161.46.138
Found Yes
Hash eeca35d468b39b17cadeed746ade25c72687a80b143e869b08470098313bef4d
SimHash 5a16551bcfc0

Groups

*

Rule Path
Disallow /index.php?act=calendar*
Disallow /index.php?act=daffiliates*
Disallow /index.php?act=Post*
Disallow /index.php?act=Forward*
Disallow /index.php?act=Track*
Disallow /index.php?act=Msg*
Disallow /index.php?act=Search*

Other Records

Field Value
crawl-delay 10

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

spiderling

Rule Path
Disallow /

slurp

Rule Path
Disallow /

spbot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

neevabot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

seekportbot

Rule Path
Disallow /

seekport crawler

Rule Path
Disallow /

blexbot/1.0

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

crawler4j

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

a6-indexer

Rule Path
Disallow /

alphaseobot

Rule Path
Disallow /

alphaseobot-sa

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

sogou inst spider

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

the knowledge ai

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

ltx71 - (http://ltx71.com/)

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

awariorssbot

Rule Path
Disallow /

awariosmartbot

Rule Path
Disallow /

applebot

Rule Path
Disallow /

seokicks

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

piplbot

Rule Path
Disallow /

adsbot

Rule Path
Disallow /

domaincrawler

Rule Path
Disallow /

checkmarknetwork

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

istellabot

Rule Path
Disallow /

pimeyes.com crawler

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

oai-searchbot

Rule Path
Disallow /

twitterbot

Rule Path
Disallow
Back to top

Comments

  • Extremely aggressive and support unhelpful
  • User-agent: bingbot
  • Disallow: /
  • Really didn't want to do this but Applebot's crawling
  • is insanely, overwhelmingly aggressive for some reason.
  • NGINX cached robots.txt (8/29/2024)
Back to top