Log In Sign Up

thetwinning.com
robots.txt

Robots Exclusion Standard data for thetwinning.com

Resource Scan

Scan Details

Site Domain thetwinning.com
Base Domain thetwinning.com
Scan Status Failed
Failure ReasonScan timed out.
Last Scan2024-09-24T07:17:29+00:00
Next Scan 2024-12-23T07:17:29+00:00

Last Successful Scan

Scanned2022-11-09T15:12:00+00:00
URL http://thetwinning.com/robots.txt
Redirect https://twinning.bandcamp.com/robots.txt
Redirect Domain twinning.bandcamp.com
Redirect Base bandcamp.com
Response IP 151.101.1.28, 151.101.193.28, 151.101.65.28, 151.101.129.28
Found Yes
Hash 38fbd96b559f9845be5bf4354f907467c9bfe5dfb533ea9ef4cc2563b1c8b146
SimHash 0238fa51df73

Groups

*

Rule Path
Disallow /tools
Disallow /checkout
Disallow /download_check
Disallow /cart/
Disallow /corpbanner/
Disallow /stream
Disallow /api/
Allow /api/currency_data/
Disallow /*_cb$

nextgensearchbot

Rule Path
Disallow /

edisterbot

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

swebot

Rule Path
Disallow /

discobot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

grapeshot

Rule Path
Disallow /

bubing

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://twinning.bandcamp.com/sitemap.xml
Back to top

Comments

  • the currency data endpoint is required to render pages
  • pattern matching known to work only with Google and Yahoo
  • badly-behaving bots
  • unwanted bots
Back to top