gaywebsite.com
robots.txt

Robots Exclusion Standard data for gaywebsite.com

Resource Scan

Scan Details

Site Domain gaywebsite.com
Base Domain gaywebsite.com
Scan Status Ok
Last Scan2024-09-17T04:05:35+00:00
Next Scan 2024-10-17T04:05:35+00:00

Last Scan

Scanned2024-09-17T04:05:35+00:00
URL https://gaywebsite.com/robots.txt
Redirect https://www.gaywebsite.com/robots.txt
Redirect Domain www.gaywebsite.com
Redirect Base gaywebsite.com
Domain IPs 84.247.170.157
Redirect IPs 84.247.170.157
Response IP 84.247.170.157
Found Yes
Hash c3f744e26fd2426a4b694769c12e579acd0d22343605f275c2d5b1e5400e0cba
SimHash de1f0121bc9b

Groups

amazonbot

Rule Path
Disallow *

yandex

Rule Path
Disallow *

blexbot

Rule Path
Disallow *

dataforseobot

Rule Path
Disallow *

dotbot

Rule Path
Disallow *

ahrefsbot

Rule Path
Disallow *

mj12bot

Rule Path
Disallow *

semrushbot

Rule Path
Disallow *

barkrowler

Rule Path
Disallow *

gptbot

Rule Path
Disallow *

claudebot

Rule Path
Disallow *

seekportbot

Rule Path
Disallow *

timpibot

Rule Path
Disallow *

metajobbot

Rule Path
Disallow *

awariobot

Rule Path
Disallow *

applebot

Rule Path
Disallow *

mediatoolkitbot

Rule Path
Disallow *

bitsightbot

Rule Path
Disallow *

serpstatbot

Rule Path
Disallow *

ccbot

Rule Path
Disallow *

seznambot

Rule Path
Disallow *

domainstatsbot

Rule Path
Disallow *

friendly_crawler

Rule Path
Disallow *

facebookexternalhit

Rule Path
Disallow *

go-http-client

Rule Path
Disallow *

yandexbot

Rule Path
Disallow *

yandex

Rule Path
Disallow *

amazonbot

Rule Path
Disallow *

*

Rule Path
Disallow /wp-admin/*
Disallow /wp-json/*
Disallow /wp-login.php
Disallow /*/feed/
Disallow /my-account/
Disallow /login/
Disallow /cart/
Disallow /registrierung/
Disallow /sitemap/
Disallow /impressum/
Disallow /checkout/
Disallow /mein-konto/
Disallow /wp-admin/
Disallow /registrierung/
Disallow /impressum/
Disallow /mein-konto/
Disallow /kasse/
Disallow /warenkorb/
Disallow /shop/
Allow /media/
Allow /media/bildergalerien/
Allow /media/models
Allow /media/videos
Allow /sexkontakte/kontaktanzeigen
Allow /sexkontakte/prostituierte
Allow /trends
Allow /verzeichnis/locations
Allow /verzeichnis/events
Allow /*

Other Records

Field Value
sitemap https://www.gaywebsite.com/sitemap_index.xml

Comments

  • This is technically redundant but can be kept for clarity if you prefer