/.well-known/

Log In Sign Up

erotic4u.com
robots.txt

Robots Exclusion Standard data for erotic4u.com

Archived Snapshots

Resource Scan

Scan Details

Site Domain	erotic4u.com
Base Domain	erotic4u.com
Scan Status	Ok
Last Scan	2024-09-08T18:52:27+00:00
Next Scan	2024-10-08T18:52:27+00:00

Last Scan

Scanned	2024-09-08T18:52:27+00:00
URL	https://erotic4u.com/robots.txt
Redirect	https://www.erotic4u.com/robots.txt
Redirect Domain	www.erotic4u.com
Redirect Base	erotic4u.com
Domain IPs	92.112.194.247
Redirect IPs	92.112.194.247
Response IP	92.112.194.247
Found	Yes
Hash	eb382ebe7d42503e99b3dab0094d48024824b25a7f4b378e0d3798ced4fc86b9
SimHash	6a44d71fcfe8

Groups

*

Rule

Path

Disallow

/_cgi-bin/

Disallow

/_errorpages/

Disallow

/members/

Disallow

/MembersArea/

Disallow

/THELIST/

Disallow

/_blue/SSL.php

Other Records

Field

Value

Comment

crawl-delay

10

10 seconds between page requests

cazoodlebot

Rule

Path

Disallow

/

mj12bot

Rule

Path

Disallow

/

dotbot/1.0

Rule

Path

Disallow

/

dotbot

Rule

Path

Disallow

/

gigabot

Rule

Path

Disallow

/

sogou spider

Rule

Path

Disallow

/

sogou blog

Rule

Path

Disallow

/

sogou inst spider

Rule

Path

Disallow

/

sogou news spider

Rule

Path

Disallow

/

sogou orion spider

Rule

Path

Disallow

/

sogou spider2

Rule

Path

Disallow

/

sogou web spider

Rule

Path

Disallow

/

ahrefsbot

Rule

Path

Disallow

/

baiduspider
baiduspider-video
baiduspider-image

Rule

Path

Disallow

/

yandex

Rule

Path

Disallow

/

bubing

Rule

Path

Disallow

/

slurp

No rules defined. All paths allowed.

Other Records

Field

Value

crawl-delay

5

seznambot

Rule

Path

Disallow

/

blexbot

Rule

Path

Disallow

/

cliqzbot

Rule

Path

Disallow

/

adscanner

Rule

Path

Disallow

/

ccbot

Rule

Path

Disallow

/

semrushbot

Rule

Path

Disallow

/

semrushbot-sa

Rule

Path

Disallow

/

petalbot

Rule

Path

Disallow

/

Back to top

Other Records

Field

Value

sitemap

https://www.ep.de/sitemap.xml

Back to top

Comments

For all robots
Block access to specific groups of pages
Allow search crawlers to discover the sitemap
Block CazoodleBot as it does not present correct accept content headers
Block MJ12bot as it is just noise
Block dotbot as it cannot parse base urls properly
dotbot without version
Block Gigabot
SoGou (CN) Info: http://www.sogou.com/docs/help/webmasters.htm#07
https://ahrefs.com/robot
http://law.di.unimi.it/BUbiNG.html
slow down Yahoo

Back to top

Warnings

`request-rate` is not a known field.
`visit-time` is not a known field.

Back to top