Log In Sign Up

librarything.nl
robots.txt

Robots Exclusion Standard data for librarything.nl

Resource Scan

Scan Details

Site Domain librarything.nl
Base Domain librarything.nl
Scan Status Ok
Last Scan2024-09-06T14:09:42+00:00
Next Scan 2024-10-06T14:09:42+00:00

Last Scan

Scanned2024-09-06T14:09:42+00:00
URL https://librarything.nl/robots.txt
Redirect https://www.librarything.com/robots.txt
Redirect Domain www.librarything.com
Redirect Base librarything.com
Domain IPs 74.201.105.36
Redirect IPs 104.22.32.65, 104.22.33.65, 172.67.7.204, 2606:4700:10::6816:2041, 2606:4700:10::6816:2141, 2606:4700:10::ac43:7cc
Response IP 172.67.7.204
Found Yes
Hash 5b7944c330df9fdde34b59576c500221fd4914c6fd4d842e6c956ff6dfe1f280
SimHash 7d1869584f9e

Groups

twitterbot

Rule Path
Disallow
Allow /work/book/*

sosospider

Rule Path
Disallow /

accompanybot

Rule Path
Disallow /

*

Rule Path
Disallow /ajax_*
Disallow /ajaxinc_*
Disallow /api_*
Disallow /api-*
Disallow /api-*
Disallow /api/
Disallow /api/*
Disallow /catalog_bottom
Disallow /catalog_bottom*
Disallow /catalog_top
Disallow /catalog_top*
Disallow /catalog/
Disallow /catalog.php
Disallow /combine.php*
Disallow /combine.php
Disallow /author_split.php
Disallow /author_split.php*
Disallow /author_claim.php
Disallow /author_cloud.php
Disallow /authormerge.php
Disallow /tagcloud.php
Disallow /dotimer.php
Disallow /files/
Disallow /inc_*
Disallow /ajaxinc_*
Disallow /ajax_*
Disallow /isbn/
Disallow /ISBN/
Disallow /ltw/
Disallow /ltw*
Disallow /m/
Disallow /MDS/
Disallow /readmessage.php*
Disallow /search_works*
Disallow /search*
Disallow /title/
Disallow /work/book/
Disallow /work/*/book
Disallow /work/*/edit
Disallow /work/*/covers
Disallow /work/*/workdetails
Disallow /work/*/details
Disallow /work/*/reviews
Disallow /work/*/recommendations
Disallow /work/*/members
Disallow /work/*/descriptions
Disallow /work/*/members
Disallow /work/*/summary
Disallow /work/*/conversations
Disallow /work/*/commonknowledge
Disallow /work/*/popularity
Disallow /work/*/editions
Disallow /work/book/*
Disallow /work/*/getlocal
Disallow /work/*/quicklinks
Disallow /work/*/get
Disallow /work/*/stats
Disallow /author/*/stats
Disallow /author/*/readalikes
Disallow /author/*/names
Disallow /profile/*/recommendations
Disallow /profile/*/readalikes
Disallow /profile/*/recommendations/member
Disallow /profile/*/recommendations/memberbooks
Disallow /profile/*/recommendations/bymember
Disallow /profile/*/recommendations/unsuggester
Disallow /profile_reviews.php
Disallow /profile/*/stats/library
Disallow /profile/*/stats/*
Disallow /profile/tagcloud/*
Disallow /profile/authorgallery/*
Disallow /profile/tagmirror/*
Disallow /review/
Disallow /connect
Disallow /widget
Disallow /tagcloud/*
Disallow /authorgallery.php
Disallow /home/*
Disallow /work-info.php
Disallow /work-info/
Disallow /rss/tags/
Disallow /commonknowledge/search.php*
Disallow /matrix/*
Disallow /la/*
Disallow /forlibraries/*
Disallow /profile/gravlady
Disallow /work/12275901
Disallow /work/15929672
Disallow /work/11923187
Disallow /work/14228188
Disallow /work/14228203
Disallow /work/14228209
Disallow /work/14228213
Disallow /work/14228218
Disallow /work/14228220
Disallow /work/14228224
Disallow /work/14228228
Disallow /work/14228229
Disallow /author/strietelmeiernathan
Disallow /author/caitlinradano
Disallow /author/samarinatatiana
Disallow /signup_twitter.php
Disallow /signup_fb.php
Disallow /signup_verify.php
Disallow /local/map/*

Other Records

Field Value
crawl-delay 2

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 0

petalbot

Rule Path
Disallow /

accompanybot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.librarything.com/siteindex_eng.xml
Back to top

Comments

  • robots_txt_lt_english.php
Back to top

Warnings

  • 2 invalid lines.
Back to top