librarything.com
robots.txt
Robots Exclusion Standard data for librarything.com
Resource Scan
Scan Details
| Site Domain | librarything.com |
| Base Domain | librarything.com |
| Scan Status | Ok |
| Last Scan | 2024-08-28T15:59:57+00:00 |
| Next Scan | 2024-09-27T15:59:57+00:00 |
Last Scan
| Scanned | 2024-08-28T15:59:57+00:00 |
| URL | https://librarything.com/robots.txt |
| Redirect | https://www.librarything.com/robots.txt |
| Redirect Domain | www.librarything.com |
| Redirect Base | librarything.com |
| Domain IPs | 104.22.32.65, 104.22.33.65, 172.67.7.204, 2606:4700:10::6816:2041, 2606:4700:10::6816:2141, 2606:4700:10::ac43:7cc |
| Redirect IPs | 104.22.32.65, 104.22.33.65, 172.67.7.204, 2606:4700:10::6816:2041, 2606:4700:10::6816:2141, 2606:4700:10::ac43:7cc |
| Response IP | 104.22.33.65 |
| Found | Yes |
| Hash | 41f3dae21feac2096d4c673bb77f88f3899989b4175add5bc69df636fc2b932e |
| SimHash | 7d387959cf9c |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /ajax_* |
| Disallow | /ajaxinc_* |
| Disallow | /api_* |
| Disallow | /api-* |
| Disallow | /api-* |
| Disallow | /api/ |
| Disallow | /api/* |
| Disallow | /catalog_bottom |
| Disallow | /catalog_bottom* |
| Disallow | /catalog_top |
| Disallow | /catalog_top* |
| Disallow | /catalog/ |
| Disallow | /catalog.php |
| Disallow | /combine.php* |
| Disallow | /combine.php |
| Disallow | /author_split.php |
| Disallow | /author_split.php* |
| Disallow | /author_claim.php |
| Disallow | /author_cloud.php |
| Disallow | /authormerge.php |
| Disallow | /tagcloud.php |
| Disallow | /dotimer.php |
| Disallow | /files/ |
| Disallow | /inc_* |
| Disallow | /ajaxinc_* |
| Disallow | /ajax_* |
| Disallow | /isbn/ |
| Disallow | /ISBN/ |
| Disallow | /ltw/ |
| Disallow | /ltw* |
| Disallow | /m/ |
| Disallow | /MDS/ |
| Disallow | /readmessage.php* |
| Disallow | /search_works* |
| Disallow | /search* |
| Disallow | /title/ |
| Disallow | /work/book/ |
| Disallow | /work/*/book |
| Disallow | /work/*/edit |
| Disallow | /work/*/covers |
| Disallow | /work/*/workdetails |
| Disallow | /work/*/details |
| Disallow | /work/*/reviews |
| Disallow | /work/*/recommendations |
| Disallow | /work/*/members |
| Disallow | /work/*/descriptions |
| Disallow | /work/*/members |
| Disallow | /work/*/summary |
| Disallow | /work/*/conversations |
| Disallow | /work/*/commonknowledge |
| Disallow | /work/*/popularity |
| Disallow | /work/*/editions |
| Disallow | /work/book/* |
| Disallow | /work/*/getlocal |
| Disallow | /work/*/quicklinks |
| Disallow | /work/*/get |
| Disallow | /work/*/stats |
| Disallow | /author/*/stats |
| Disallow | /author/*/readalikes |
| Disallow | /author/*/names |
| Disallow | /profile/*/recommendations |
| Disallow | /profile/*/readalikes |
| Disallow | /profile/*/recommendations/member |
| Disallow | /profile/*/recommendations/memberbooks |
| Disallow | /profile/*/recommendations/bymember |
| Disallow | /profile/*/recommendations/unsuggester |
| Disallow | /profile_reviews.php |
| Disallow | /profile/*/stats/library |
| Disallow | /profile/*/stats/* |
| Disallow | /profile/tagcloud/* |
| Disallow | /profile/authorgallery/* |
| Disallow | /profile/tagmirror/* |
| Disallow | /review/ |
| Disallow | /connect |
| Disallow | /widget |
| Disallow | /tagcloud/* |
| Disallow | /authorgallery.php |
| Disallow | /home/* |
| Disallow | /work-info.php |
| Disallow | /work-info/ |
| Disallow | /rss/tags/ |
| Disallow | /commonknowledge/search.php* |
| Disallow | /matrix/* |
| Disallow | /la/* |
| Disallow | /forlibraries/* |
| Disallow | /profile/gravlady |
| Disallow | /work/12275901 |
| Disallow | /work/15929672 |
| Disallow | /work/11923187 |
| Disallow | /author/strietelmeiernathan |
| Disallow | /author/caitlinradano |
| Disallow | /signup_twitter.php |
| Disallow | /signup_fb.php |
| Disallow | /signup_verify.php |
| Disallow | /local/map/* |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 2 |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.librarything.com/siteindex_eng.xml |
Warnings
- 2 invalid lines.
Comments