librarything.fr
robots.txt
Robots Exclusion Standard data for librarything.fr
Resource Scan
Scan Details
| Site Domain | librarything.fr |
| Base Domain | librarything.fr |
| Scan Status | Ok |
| Last Scan | 2024-11-04T21:01:46+00:00 |
| Next Scan | 2024-12-04T21:01:46+00:00 |
Last Scan
| Scanned | 2024-11-04T21:01:46+00:00 |
| URL | https://librarything.fr/robots.txt |
| Redirect | https://www.librarything.com/robots.txt |
| Redirect Domain | www.librarything.com |
| Redirect Base | librarything.com |
| Domain IPs | 74.201.105.36 |
| Redirect IPs | 104.22.32.65, 104.22.33.65, 172.67.7.204, 2606:4700:10::6816:2041, 2606:4700:10::6816:2141, 2606:4700:10::ac43:7cc |
| Response IP | 104.22.32.65 |
| Found | Yes |
| Hash | db37465e696d02f2d5c22de99ecd5e8b9a9116a637e205e28c26b6bf22dfc3f8 |
| SimHash | 791829194f9e |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /ajax_* |
| Disallow | /ajaxinc_* |
| Disallow | /api_* |
| Disallow | /api-* |
| Disallow | /api-* |
| Disallow | /api/ |
| Disallow | /api/* |
| Disallow | /catalog_bottom |
| Disallow | /catalog_bottom* |
| Disallow | /catalog_top |
| Disallow | /catalog_top* |
| Disallow | /catalog/ |
| Disallow | /catalog.php |
| Disallow | /combine.php* |
| Disallow | /combine.php |
| Disallow | /author_split.php |
| Disallow | /author_split.php* |
| Disallow | /author_claim.php |
| Disallow | /author_cloud.php |
| Disallow | /authormerge.php |
| Disallow | /tagcloud.php |
| Disallow | /dotimer.php |
| Disallow | /files/ |
| Disallow | /inc_* |
| Disallow | /ajaxinc_* |
| Disallow | /ajax_* |
| Disallow | /isbn/ |
| Disallow | /ISBN/ |
| Disallow | /ltw/ |
| Disallow | /ltw* |
| Disallow | /m/ |
| Disallow | /MDS/ |
| Disallow | /readmessage.php* |
| Disallow | /search_works* |
| Disallow | /search* |
| Disallow | /title/ |
| Disallow | /work/book/ |
| Disallow | /work/*/book |
| Disallow | /work/*/edit |
| Disallow | /work/*/covers |
| Disallow | /work/*/workdetails |
| Disallow | /work/*/details |
| Disallow | /work/*/reviews |
| Disallow | /work/*/recommendations |
| Disallow | /work/*/members |
| Disallow | /work/*/descriptions |
| Disallow | /work/*/members |
| Disallow | /work/*/summary |
| Disallow | /work/*/conversations |
| Disallow | /work/*/commonknowledge |
| Disallow | /work/*/popularity |
| Disallow | /work/*/editions |
| Disallow | /work/book/* |
| Disallow | /work/*/getlocal |
| Disallow | /work/*/quicklinks |
| Disallow | /work/*/get |
| Disallow | /work/*/stats |
| Disallow | /author/*/stats |
| Disallow | /author/*/readalikes |
| Disallow | /author/*/names |
| Disallow | /profile/*/recommendations |
| Disallow | /profile/*/readalikes |
| Disallow | /profile/*/recommendations/member |
| Disallow | /profile/*/recommendations/memberbooks |
| Disallow | /profile/*/recommendations/bymember |
| Disallow | /profile/*/recommendations/unsuggester |
| Disallow | /profile_reviews.php |
| Disallow | /profile/*/stats/library |
| Disallow | /profile/*/stats/* |
| Disallow | /profile/tagcloud/* |
| Disallow | /profile/authorgallery/* |
| Disallow | /profile/tagmirror/* |
| Disallow | /review/ |
| Disallow | /connect |
| Disallow | /widget |
| Disallow | /tagcloud/* |
| Disallow | /authorgallery.php |
| Disallow | /home/* |
| Disallow | /work-info.php |
| Disallow | /work-info/ |
| Disallow | /rss/tags/ |
| Disallow | /commonknowledge/search.php* |
| Disallow | /matrix/* |
| Disallow | /la/* |
| Disallow | /forlibraries/* |
| Disallow | /profile/gravlady |
| Disallow | /work/12275901 |
| Disallow | /work/15929672 |
| Disallow | /work/11923187 |
| Disallow | /work/14228188 |
| Disallow | /work/14228203 |
| Disallow | /work/14228209 |
| Disallow | /work/14228213 |
| Disallow | /work/14228218 |
| Disallow | /work/14228220 |
| Disallow | /work/14228224 |
| Disallow | /work/14228228 |
| Disallow | /work/14228229 |
| Disallow | /author/strietelmeiernathan |
| Disallow | /author/caitlinradano |
| Disallow | /author/samarinatatiana |
| Disallow | /signup_twitter.php |
| Disallow | /signup_fb.php |
| Disallow | /signup_verify.php |
| Disallow | /local/map/* |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 2 |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.librarything.com/siteindex_eng.xml |
Warnings
- 2 invalid lines.
Comments