Log In Sign Up

archiveuk.biz
robots.txt

Robots Exclusion Standard data for archiveuk.biz

Resource Scan

Scan Details

Site Domain archiveuk.biz
Base Domain archiveuk.biz
Scan Status Ok
Last Scan5/28/2025, 4:28:20 PM
Next Scan 6/4/2025, 4:28:20 PM

Last Scan

Scanned5/28/2025, 4:28:20 PM
URL https://archiveuk.biz/robots.txt
Domain IPs 104.21.49.32, 172.67.140.132, 2606:4700:3036::ac43:8c84, 2606:4700:3037::6815:3120
Response IP 172.67.140.132
Found Yes
Hash d7f4614ae3971e3f9ec35877ff44ea5798a0b372c6725d13e04301e42b6a7ed4
SimHash 501d0142f560

Groups

daumoa

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

crawler4j

Rule Path
Disallow /

getintent

Rule Path
Disallow /

coccoc

Rule Path
Disallow /

proximic

Rule Path
Disallow /

grapeshot

Rule Path
Disallow /

ltx71

Rule Path
Disallow /

jamesbot

Rule Path
Disallow /

smtbot

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

meta-externalagent

Rule Path
Disallow /

facebookexternalhit

Rule Path
Disallow /

serpstatbot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

*

Rule Path
Disallow /search?
Disallow /edit/
Disallow /cdn-cgi/
Disallow /dynjs/
Disallow /dyn/actions/
Disallow /en/search?
Allow /
Back to top

Other Records

Field Value
sitemap https://archiveuk.biz/sitemaps/sitemap_index.xml
Back to top