Log In Sign Up

sex-lexis.com
robots.txt

Robots Exclusion Standard data for sex-lexis.com

Resource Scan

Scan Details

Site Domain sex-lexis.com
Base Domain sex-lexis.com
Scan Status Ok
Last Scan2025-03-07T03:34:27+00:00
Next Scan 2025-03-14T03:34:27+00:00

Last Scan

Scanned2025-03-07T03:34:27+00:00
URL https://sex-lexis.com/robots.txt
Domain IPs 104.21.112.1, 104.21.16.1, 104.21.32.1, 104.21.48.1, 104.21.64.1, 104.21.80.1, 104.21.96.1, 2606:4700:3030::6815:1001, 2606:4700:3030::6815:2001, 2606:4700:3030::6815:3001, 2606:4700:3030::6815:4001, 2606:4700:3030::6815:5001, 2606:4700:3030::6815:6001, 2606:4700:3030::6815:7001
Response IP 104.21.80.1
Found Yes
Hash 1f69678a7de92b1baa7ccf77ed331329e741caabd48b4cb4223d171622660d44
SimHash 021140082de1

Groups

bender

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

msiecrawler

Rule Path
Disallow /

npbot

Rule Path
Disallow /

msrbot

Rule Path
Disallow /

open*

Rule Path
Disallow /

twiceler

Rule Path
Disallow /

scooter*

Rule Path
Disallow /

websearch*

Rule Path
Disallow /

zyborg*

Rule Path
Disallow /

wget*

Rule Path
Disallow /

turnitinbot*

Rule Path
Disallow /

slysearch*

Rule Path
Disallow /

seekbot*

Rule Path
Disallow /

netresearchserver*

Rule Path
Disallow /

seekbot*

Rule Path
Disallow /

mozdex*

Rule Path
Disallow /

link*

Rule Path
Disallow /

larbin*

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

charlotte

Rule Path
Disallow /

mediapartners-google*

Rule Path
Disallow

*

Rule Path
Disallow /_/access/
Back to top