pornxpert.com
robots.txt

Robots Exclusion Standard data for pornxpert.com

Resource Scan

Scan Details

Site Domain pornxpert.com
Base Domain pornxpert.com
Scan Status Ok
Last Scan2024-09-19T15:01:01+00:00
Next Scan 2024-10-19T15:01:01+00:00

Last Scan

Scanned2024-09-19T15:01:01+00:00
URL https://pornxpert.com/robots.txt
Domain IPs 84.247.2.194
Response IP 84.247.2.194
Found Yes
Hash d54dbad4282f2d0d1eed7d5514b99591561291a3cac31e44eaaf0bf6c471cf2f
SimHash 501d1810f7a0

Groups

*

Rule Path
Disallow /player/float.php
Disallow /player/html.php
Disallow /redirect_cs.php
Disallow /*?action=*$
Disallow /*mode%3Dasync*$
Disallow /*function%3Dget_block*$
Disallow /404.php
Disallow /logout/
Disallow /email/
Disallow /payments/
Disallow /login/
Disallow /login-required/
Disallow /signup/
Disallow /upgrade/
Disallow /reset-password/
Disallow /resend-confirmation/
Disallow /feedback/
Disallow /categories-selector/
Disallow /models-selector/
Disallow /captcha/*
Disallow /link/*
Disallow /related_videos_html/*
Disallow /embed/*

googlebot

Rule Path
Allow /

bingbot

Rule Path
Allow /

duckduckbot

Rule Path
Allow /

perplexitybot

Rule Path
Allow /

anthropic-ai

Rule Path
Allow /

chatgpt-user

Rule Path
Allow /

claudebot

Rule Path
Allow /

claude-web

Rule Path
Allow /

gptbot

Rule Path
Allow /

ccbot

Rule Path
Allow /

cohere-ai

Rule Path
Allow /

applebot

Rule Path
Allow /

applebot-extended

Rule Path
Allow /

google-extended

Rule Path
Allow /

facebookbot

Rule Path
Allow /

bytespider

Rule Path
Allow /

diffbot

Rule Path
Allow /

friendlycrawler

Rule Path
Allow /

ahrefsbot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

censysinspect

Rule Path
Disallow /

expanse

Rule Path
Disallow /

internet-measurement

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

dataprovider

Rule Path
Disallow /

dalvik/2.1.0

Rule Path
Disallow /

go-http-client

Rule Path
Disallow /

googleother

Rule Path
Disallow /

ioncrawl

Rule Path
Disallow /

isscyberriskcrawler

Rule Path
Disallow /

java

Rule Path
Disallow /

mozlila

Rule Path
Disallow /

news-please

Rule Path
Disallow /

orbbot

Rule Path
Disallow /

python-requests

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

bw/1.1

Rule Path
Disallow /

wp_is_mobile

Rule Path
Disallow /

zoominfobot

Rule Path
Disallow /

image2dataset

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.pornxpert.com/sitemap.xml

Comments

  • Search Engines
  • AI and Language Model Bots
  • Apple Bots
  • Other potentially AI-related bots
  • Disallow other bots