Log In Sign Up

archlinux.pkgs.org
robots.txt

Robots Exclusion Standard data for archlinux.pkgs.org

Resource Scan

Scan Details

Site Domain archlinux.pkgs.org
Base Domain pkgs.org
Scan Status Ok
Last Scan2024-05-18T07:23:25+00:00
Next Scan 2024-06-17T07:23:25+00:00

Last Scan

Scanned2024-05-18T07:23:25+00:00
URL https://archlinux.pkgs.org/robots.txt
Domain IPs 138.201.217.61
Response IP 138.201.217.61
Found Yes
Hash 5a976ad3ceeff110fec47d3ac567c7a36fb680fc52710342e7e537066321e63e
SimHash 5b44954847e9

Groups

amazonadbot

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

buck

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

clickagy intelligence bot v2

Rule Path
Disallow /

criteobot/0.1

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

drupal

Rule Path
Disallow /

embedly

Rule Path
Disallow /

femtosearchbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

getintent crawler

Rule Path
Disallow /

gnowitnewsbot

Rule Path
Disallow /

gozlebot

Rule Path
Disallow /

ioncrawl

Rule Path
Disallow /

leikibot

Rule Path
Disallow /

linespider

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

maxpointcrawler

Rule Path
Disallow /

mediatoolkitbot

Rule Path
Disallow /

monsidobot

Rule Path
Disallow /

quantcastbot

Rule Path
Disallow /

riddler

Rule Path
Disallow /

smtbot

Rule Path
Disallow /

seekport crawler

Rule Path
Disallow /

semanticbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

senutobot

Rule Path
Disallow /

serendeputybot

Rule Path
Disallow /

sirdatabot

Rule Path
Disallow /

siteauditbot

Rule Path
Disallow /

splitsignalbot

Rule Path
Disallow /

surdotlybot

Rule Path
Disallow /

ttd-content

Rule Path
Disallow /

the knowledge ai

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

webwikibot

Rule Path
Disallow /

yeti

Rule Path
Disallow /

yisouspider

Rule Path
Disallow /

zoombot

Rule Path
Disallow /

adbeat_bot

Rule Path
Disallow /

archive.org_bot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

fluid

Rule Path
Disallow /

grapeshot

Rule Path
Disallow /

heritrix

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

ias_crawler

Rule Path
Disallow /

ltx71

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

proximic

Rule Path
Disallow /

*

Rule Path
Allow /
Back to top

Comments

  • block robots
  • default
Back to top