Log In Sign Up

simplydating.com
robots.txt

Robots Exclusion Standard data for simplydating.com

Resource Scan

Scan Details

Site Domain simplydating.com
Base Domain simplydating.com
Scan Status Ok
Last Scan2024-08-31T03:43:06+00:00
Next Scan 2024-09-30T03:43:06+00:00

Last Scan

Scanned2024-08-31T03:43:06+00:00
URL https://simplydating.com/robots.txt
Redirect https://www.simplydating.com/robots.txt
Redirect Domain www.simplydating.com
Redirect Base simplydating.com
Domain IPs 104.26.0.65, 104.26.1.65, 172.67.72.116, 2606:4700:20::681a:141, 2606:4700:20::681a:41, 2606:4700:20::ac43:4874
Redirect IPs 104.26.0.65, 104.26.1.65, 172.67.72.116, 2606:4700:20::681a:141, 2606:4700:20::681a:41, 2606:4700:20::ac43:4874
Response IP 104.26.0.65
Found Yes
Hash b2772638954e2a244d53b7c90fa85701ec73f2c7648ae4d126fe1611e809e130
SimHash 2829d1600954

Groups

*

Rule Path
Disallow /admin
Disallow /agency
Disallow /partner
Disallow /blocks
Disallow /chatbookings
Disallow /chathistory
Disallow /contactdetailsrequests
Disallow /conversations
Disallow /favourites/
Disallow /introductions
Disallow /meetings
Disallow /memberchildren/
Disallow /members
Allow /members/details/
Disallow /payments
Disallow /statistics/
Disallow /shoppingcart/
Disallow /tasks/
Disallow /tours
Disallow /webhooks/
Disallow /ru/admin
Disallow /ru/agency
Disallow /ru/partner
Disallow /ru/blocks
Disallow /ru/chatbookings
Disallow /ru/chathistory
Disallow /ru/contactdetailsrequests
Disallow /ru/conversations
Disallow /ru/favourites/
Disallow /ru/introductions
Disallow /ru/meetings
Disallow /ru/memberchildren/
Disallow /ru/members
Allow /ru/members/details/
Disallow /ru/payments
Disallow /ru/statistics/
Disallow /ru/shoppingcart/
Disallow /ru/tasks/
Disallow /ru/tours
Disallow /ru/webhooks/
Disallow /en-gb/admin
Disallow /en-gb/agency
Disallow /en-gb/partner
Disallow /en-gb/blocks
Disallow /en-gb/chatbookings
Disallow /en-gb/chathistory
Disallow /en-gb/contactdetailsrequests
Disallow /en-gb/conversations
Disallow /en-gb/favourites/
Disallow /en-gb/introductions
Disallow /en-gb/meetings
Disallow /en-gb/memberchildren/
Disallow /en-gb/members
Allow /en-gb/members/details/
Disallow /en-gb/payments
Disallow /en-gb/statistics/
Disallow /en-gb/shoppingcart/
Disallow /en-gb/tasks/
Disallow /en-gb/tours
Disallow /en-gb/webhooks/
Disallow /en-ca/admin
Disallow /en-ca/agency
Disallow /en-ca/partner
Disallow /en-ca/blocks
Disallow /en-ca/chatbookings
Disallow /en-ca/chathistory
Disallow /en-ca/contactdetailsrequests
Disallow /en-ca/conversations
Disallow /en-ca/favourites/
Disallow /en-ca/introductions
Disallow /en-ca/meetings
Disallow /en-ca/memberchildren/
Disallow /en-ca/members
Allow /en-ca/members/details/
Disallow /en-ca/payments
Disallow /en-ca/statistics/
Disallow /en-ca/shoppingcart/
Disallow /en-ca/tasks/
Disallow /en-ca/tours
Disallow /en-ca/webhooks/
Disallow /en-au/admin
Disallow /en-au/agency
Disallow /en-au/partner
Disallow /en-au/blocks
Disallow /en-au/chatbookings
Disallow /en-au/chathistory
Disallow /en-au/contactdetailsrequests
Disallow /en-au/conversations
Disallow /en-au/favourites/
Disallow /en-au/introductions
Disallow /en-au/meetings
Disallow /en-au/memberchildren/
Disallow /en-au/members
Allow /en-au/members/details/
Disallow /en-au/payments
Disallow /en-au/statistics/
Disallow /en-au/shoppingcart/
Disallow /en-au/tasks/
Disallow /en-au/tours
Disallow /en-au/webhooks/
Allow /sv/$
Disallow /sv/
Disallow /zh/

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.simplydating.com/sitemap.xml
Back to top

Comments

  • robotstxt.org/
  • Allow all bots by default
  • Don't use trailing / when disallowing as then Index actions won't be blocked. Some controllers don't have Index
  • actions or otherwise do need that Index action to be indexable. In those cases we have left a trailing / on the
  • disallow directive
  • Also block the russian versions of these sites
  • Also block the english (UK) versions of these sites
  • Also block the english (CA) versions of these sites
  • Also block the english (AU) versions of these sites
  • Block all Swedish pages but the homepage
  • We used to have a Chinese version of the site, but don't any longer
  • Block SemrushBot as they cause exceptions by visiting corrupted links (e.g. url parameters without values)
  • https://www.semrush.com/bot/
Back to top