Log In Sign Up

paybito.com
robots.txt

Robots Exclusion Standard data for paybito.com

Resource Scan

Scan Details

Site Domain paybito.com
Base Domain paybito.com
Scan Status Ok
Last Scan2025-12-16T07:02:31+00:00
Next Scan 2026-01-15T07:02:31+00:00

Last Scan

Scanned2025-12-16T07:02:31+00:00
URL https://paybito.com/robots.txt
Domain IPs 54.215.68.252
Response IP 54.215.68.252
Found Yes
Hash 4d1815478d4c359e138183984ae6934a476a453ab39027694e7e75444d49aab0
SimHash 55148a227233

Groups

*

Rule Path
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php

baiduspider

Rule Path
Disallow /

bingbot

Rule Path
Allow /

duckduckbot

Rule Path
Allow /

slurp

Rule Path
Allow /

googlebot

Rule Path
Allow /

gptbot

Rule Path
Allow /

chatgpt-user

Rule Path
Allow /

chatgpt-user/2.0

Rule Path
Allow /

oai-searchbot

Rule Path
Allow /

claudebot

Rule Path
Allow /

claude-web

Rule Path
Allow /

claude-user

Rule Path
Allow /

anthropic-ai

Rule Path
Allow /

perplexitybot

Rule Path
Allow /

perplexity-user

Rule Path
Allow /

cohere-ai

Rule Path
Allow /

youbot

Rule Path
Allow /

bytespider

Rule Path
Allow /

xai-bot

Rule Path
Allow /

ai2bot

Rule Path
Allow /

ccbot

Rule Path
Allow /

diffbot

Rule Path
Allow /

petalbot

Rule Path
Allow /

google-extended

Rule Path
Allow /

google-cloudvertexbot

Rule Path
Allow /

together-bot

Rule Path
Allow /

replicate-bot

Rule Path
Allow /

runpod-bot

Rule Path
Allow /

timpibot

Rule Path
Allow /

webzio-extended

Rule Path
Allow /

terracotta

Rule Path
Allow /

*

Rule Path
Allow /

ahrefsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

megaindex

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

spbot

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

emailcollector

Rule Path
Disallow /

emailsiphon

Rule Path
Disallow /

emailwolf

Rule Path
Disallow /

extractorpro

Rule Path
Disallow /

webscraper

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

python-requests

Rule Path
Disallow /

curl

Rule Path
Disallow /

wget

Rule Path
Disallow /

java

Rule Path
Disallow /

libwww-perl

Rule Path
Disallow /

sqlmap

Rule Path
Disallow /

acunetix

Rule Path
Disallow /

wpscan

Rule Path
Disallow /

nikto

Rule Path
Disallow /

nessus

Rule Path
Disallow /

baiduspider-image

Rule Path
Disallow /

baiduspider-video

Rule Path
Disallow /

sogou

Rule Path
Disallow /

yandeximages

Rule Path
Disallow /

seekportbot

Rule Path
Disallow /

netcraftsurveyagent

Rule Path
Disallow /

nimbostratus-bot

Rule Path
Disallow /

archivebot

Rule Path
Disallow /

linkpadbot

Rule Path
Disallow /

python

Rule Path
Disallow /

go-http-client

Rule Path
Disallow /

okhttp

Rule Path
Disallow /

lua-resty-http

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.paybito.com/sitemap_index.xml
Back to top

Comments

  • -----------------------------------------------------
  • Default WordPress Rules
  • -----------------------------------------------------
  • --- Allow OpenAI bots ---
  • --- Allow Anthropic / Claude bots ---
  • --- Allow Perplexity bots ---
  • --- Allow Cohere ---
  • --- Allow You.com bots ---
  • --- Allow ByteDance AI crawlers ---
  • --- Allow xAI ---
  • --- Allow AI2 (Allen Institute) ---
  • --- Allow Common Crawl ---
  • --- Allow Diffbot ---
  • --- Allow DotBot & PetalBot ---
  • Note: DotBot later blocked (as per your final rules)
  • --- Allow Google AI crawlers ---
  • --- Allow emerging AI bots (2025 lists) ---
  • --- Default for all other bots ---
  • --- Aggressive scrapers / harvesters ---
  • --- Email harvesters / spam bots ---
  • --- Known content scrapers ---
  • --- Security scanners ---
  • --- Known Chinese / RU bulk crawlers ---
  • --- Suspicious / resource-draining bots ---
Back to top

Warnings

  • 2 invalid lines.
Back to top