prepaidunion.com
robots.txt

Robots Exclusion Standard data for prepaidunion.com

Resource Scan

Scan Details

Site Domain prepaidunion.com
Base Domain prepaidunion.com
Scan Status Ok
Last Scan2024-11-15T21:37:26+00:00
Next Scan 2024-12-15T21:37:26+00:00

Last Scan

Scanned2024-11-15T21:37:26+00:00
URL https://prepaidunion.com/robots.txt
Redirect https://www.prepaidunion.com/robots.txt
Redirect Domain www.prepaidunion.com
Redirect Base prepaidunion.com
Domain IPs 185.111.121.112, 2a06:5dc0:3::112
Redirect IPs 185.111.121.112
Response IP 185.111.121.112
Found Yes
Hash 20ee7dd86a363662426530568a0bfedbb67b815b01fe39e3a7eda23f7db4e7d1
SimHash 1a5ec567dab1

Groups

node/simplecrawler

Rule Path
Disallow /

anglesharp/0.9.9

Rule Path
Disallow /

anglesharp

Rule Path
Disallow /

cannabot/2.0

Rule Path
Disallow /

bubing

Rule Path
Disallow /

dnyzbot/1.0

Rule Path
Disallow /

barkrowler/0.5.1

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

megaindex.ru/2.0

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

semrushbot/1.2~bl

Rule Path
Disallow /

obot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

pinterest/0.2

Rule Path
Disallow /

exabot-thumbnails

Rule Path
Disallow /

woobot

Rule Path
Disallow /

woobot/2.0

No rules defined. All paths allowed.

mj12bot

Rule Path
Disallow /

twittercrawler

Rule Path
Disallow /

twitterbot/1.0

Rule Path
Disallow /

meanpathbot

Rule Path
Disallow /

expertsearchspider

Rule Path
Disallow /

finbot

Rule Path
Disallow /

spbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

wevikabot

Rule Path
Disallow /

archive.org_bot

Rule Path
Disallow /

*

Rule Path
Disallow /prepay/*
Disallow /de/icepay/*
Disallow /en/icepay/*
Disallow /es/icepay/*
Disallow /fr/icepay/*
Disallow /nl/icepay/*
Disallow /pl/icepay/*
Disallow /tl/icepay/*
Disallow /default/icepay/*
Disallow /de/adyen/*
Disallow /en/adyen/*
Disallow /es/adyen/*
Disallow /fr/adyen/*
Disallow /nl/adyen/*
Disallow /pl/adyen/*
Disallow /tl/adyen/*
Disallow /default/adyen/*
Disallow /de/iframe/*
Disallow /en/iframe/*
Disallow /es/iframe/*
Disallow /fr/iframe/*
Disallow /nl/iframe/*
Disallow /pl/iframe/*
Disallow /tl/iframe/*
Disallow /default/iframe/*
Disallow /en/iframe/*
Disallow /default/auth/*
Disallow /de/auth/*
Disallow /en/auth/*
Disallow /es/auth/*
Disallow /fr/auth/*
Disallow /nl/auth/*
Disallow /pl/auth/*
Disallow /tl/auth/*
Disallow /plugins/*
Disallow /maintenance/*

Other Records

Field Value
crawl-delay 300

Other Records

Field Value
sitemap https://www.prepaidunion.com/sitemap.xml

Warnings

  • 3 invalid lines.