Log In Sign Up

xxxtranaughty.com
robots.txt

Robots Exclusion Standard data for xxxtranaughty.com

Resource Scan

Scan Details

Site Domain xxxtranaughty.com
Base Domain xxxtranaughty.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-08-16T09:43:59+00:00
Next Scan 2024-11-14T09:43:59+00:00

Last Successful Scan

Scanned2023-12-28T09:01:52+00:00
URL https://xxxtranaughty.com/robots.txt
Domain IPs 104.21.234.80, 104.21.234.81, 2606:4700:3038::6815:ea50, 2606:4700:3038::6815:ea51
Response IP 104.21.234.80
Found Yes
Hash 5cbc9eef51ab21d79580325e90137cb575119d8fdf2f1ea0ec5c1bd0ad37aa9a
SimHash 4ab4d16066fe

Groups

*

Rule Path
Disallow

googlebot-video

Rule Path
Allow /
Allow /wp-content/uploads/
Allow /wp-content/uploads/*
Allow /wp-content/videoaudio/
Allow /wp-content/videoaudio/*
Allow *.js
Disallow /*?*wordfence_lh=*

googlebot-image

Rule Path
Allow /
Allow /wp-content/uploads/
Allow /wp-content/uploads/*
Allow /wp-content/videoaudio/
Allow /wp-content/videoaudio/*
Allow *.js

twitterbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 3

bingbot

Rule Path
Allow /
Disallow /cgi-bin
Disallow /wp-admin
Disallow /xmlrpc.php
Disallow /xmlrpc.php?rsd
Disallow /trackback
Disallow /cgi-bin
Disallow /search/*/feed*
Allow /wp-admin/admin-ajax.php
Allow /wp-admin/admin-ajax.php*
Allow *.js

semrushbot-sa

Rule Path
Allow /
Disallow /cgi-bin
Disallow /wp-admin
Disallow /xmlrpc.php
Disallow /xmlrpc.php?rsd
Disallow /trackback
Disallow /cgi-bin/
Disallow /wp-admin/
Disallow /trackback/
Disallow /*/page
Allow /wp-admin/admin-ajax.php
Allow /wp-admin/admin-ajax.php*
Allow *.js

Other Records

Field Value
crawl-delay 2

semrushbot-si

Rule Path
Allow /
Disallow /cgi-bin
Disallow /wp-admin
Disallow /xmlrpc.php
Disallow /xmlrpc.php?rsd
Disallow /trackback
Disallow /cgi-bin/
Disallow /wp-admin/
Disallow /trackback/
Disallow /*/page
Allow /wp-admin/admin-ajax.php
Allow /wp-admin/admin-ajax.php*
Allow *.js

Other Records

Field Value
crawl-delay 2

semrushbot

Rule Path
Allow /
Disallow /cgi-bin
Disallow /wp-admin
Disallow /xmlrpc.php
Disallow /xmlrpc.php?rsd
Disallow /trackback
Disallow /cgi-bin/
Disallow /wp-admin/
Disallow /trackback/
Disallow /*/page
Allow /wp-admin/admin-ajax.php
Allow /wp-admin/admin-ajax.php*
Allow *.js

Other Records

Field Value
crawl-delay 3

ravencrawler

Rule Path
Allow /

Other Records

Field Value
crawl-delay 3

rogerbot

Rule Path
Allow /
Disallow /cgi-bin
Disallow /wp-admin
Disallow /xmlrpc.php
Disallow /xmlrpc.php?rsd
Disallow /trackback
Disallow /cgi-bin/
Disallow /wp-admin/
Disallow /trackback/
Disallow /search
Disallow /*/page
Allow /wp-admin/admin-ajax.php
Allow /wp-admin/admin-ajax.php*
Allow *.js

Other Records

Field Value
crawl-delay 3

dotbot

Rule Path
Allow /
Disallow /cgi-bin
Disallow /wp-admin
Disallow /xmlrpc.php
Disallow /xmlrpc.php?rsd
Disallow /trackback
Disallow /cgi-bin/
Disallow /wp-admin/
Disallow /trackback/
Disallow /search
Disallow /*/page
Allow /wp-admin/admin-ajax.php
Allow /wp-admin/admin-ajax.php*
Allow *.js

Other Records

Field Value
crawl-delay 4

ninjabot

Rule Path
Allow /
Disallow /cgi-bin
Disallow /wp-admin
Disallow /xmlrpc.php
Disallow /xmlrpc.php?rsd
Disallow /trackback
Disallow /cgi-bin/
Disallow /wp-admin/
Disallow /trackback/
Disallow /search
Disallow /*/page
Allow /wp-admin/admin-ajax.php
Allow /wp-admin/admin-ajax.php*
Allow *.js

Other Records

Field Value
crawl-delay 3

petalbot

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

seo-powersuite-bot

Rule Path
Allow /
Disallow /wp-json/

blexbot

Rule Path
Allow /
Disallow /cgi-bin
Disallow /wp-admin
Disallow /xmlrpc.php
Disallow /xmlrpc.php?rsd
Disallow /trackback
Disallow /cgi-bin/
Disallow /wp-admin/
Disallow /trackback/
Disallow /search
Disallow /*/page
Allow *.js

Other Records

Field Value
crawl-delay 5

ccbot

Rule Path
Disallow /cgi-bin
Disallow /wp-admin
Disallow /xmlrpc.php
Disallow /xmlrpc.php?rsd
Disallow /trackback
Disallow /cgi-bin/
Disallow /wp-admin/
Disallow /trackback/
Disallow /search
Disallow /*/page

Other Records

Field Value
crawl-delay 3

gptbot

Rule Path
Disallow /search/
Disallow /search
Disallow /cgi-bin
Disallow /wp-admin
Disallow /xmlrpc.php
Disallow /xmlrpc.php?rsd
Disallow /trackback
Disallow /cgi-bin/
Disallow /wp-admin/
Disallow /trackback/
Disallow /*/page
Disallow /*/page/*
Allow /wp-admin/admin-ajax.php
Allow /wp-admin/admin-ajax.php*
Allow *.js

Other Records

Field Value
crawl-delay 3

ia_archiver

Rule Path
Disallow /

archive.org_bot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://xxxtranaughty.com/sitemap_index.xml
Back to top