agfbrakpan.com
robots.txt
Robots Exclusion Standard data for agfbrakpan.com
Resource Scan
Scan Details
| Site Domain | agfbrakpan.com |
| Base Domain | agfbrakpan.com |
| Scan Status | Failed |
| Failure Stage | Fetching resource. |
| Failure Reason | Server returned a client error. |
| Last Scan | 2026-01-15T19:31:29+00:00 |
| Next Scan | 2026-04-15T19:31:29+00:00 |
Last Successful Scan
| Scanned | 2023-06-07T13:30:58+00:00 |
| URL | https://agfbrakpan.com/robots.txt |
| Domain IPs | 104.21.86.225, 172.67.137.106, 2606:4700:3030::6815:56e1, 2606:4700:3036::ac43:896a |
| Response IP | 104.21.86.225 |
| Found | Yes |
| Hash | 2286280b6ffb0dd834f55d751115afe352be115fac9fb6d8b839ea461d948285 |
| SimHash | e37173574407 |
Groups
twitterbot
mediapartners-google
googlebot
adsbot-google
googlebot-image
googlebot-news
googlebot-video
googlebot-mobile
adsbot-google-mobile-apps
adsbot-google-mobile
apis-google
msnbot
msnbot-media
applebot
bingbot
bingpreview
facebookexternalhit/1.1
facebookexternalhit/1.1 (+https://www.facebook.com/externalhit_uatext.php)
adidxbot
slurp
yahoo! slurp
yahoo-blogs/v3.9
ia_archiver
duckduckbot
aolbuild
teoma
gigabot
robozilla
nutch
yahoo-mmcrawler
psbot
| Rule | Path |
|---|---|
| Allow | / |
yandex
moget
moget/2.1
naverbot
yeti
baiduspider
baiduspider-video
baiduspider-image
sogou spider
youdaobot
| Rule | Path |
|---|---|
| Disallow | /API/ |
| Disallow | /App_Data/ |
| Disallow | /aspnet_client/ |
| Disallow | /bin/ |
| Disallow | /cgi-bin/ |
| Disallow | /CMS/ |
| Disallow | /client_scripts/ |
| Disallow | /Components/ |
| Disallow | /extra/ |
| Disallow | /js/ |
| Disallow | /MailTemplates/ |
| Disallow | /spaw/ |
| Disallow | /obj/ |
| Disallow | /Procs/ |
| Disallow | /Properties/ |
| Disallow | /src/ |
| Disallow | /UserControls/ |
| Disallow | /mediaplayers/ |
| Disallow | /events.aspx |
| Disallow | /log.aspx |
| Disallow | /logcty.aspx |
| Disallow | /loglang.aspx |
| Disallow | /logspk.aspx |
| Disallow | /logsourceurl.aspx |
asterias
backdoorbot/1.0
black hole
blackwidow
blexbot
blowfish/1.0
botalot
builtbottough
bullseye/1.0
bunnyslippers
cegbfeieh
cheesebot
cherrypicker
cherrypickerelite/1.0
cherrypickerse/1.0
chinaclaw
copyrightcheck
cosmos
crescent
crescent internet toolpak http ole control v.1.0
custo
disco
dittospyder
download\ demon
ecatch
eirgrabber
emailcollector
emailsiphon
emailwolf
erocrawler
exabot
express\ webpictures
extractorpro
eyenetie
flashget
foobot
getright
getweb!
gigabot
go!zilla
go-ahead-got-it
grabnet
grafula
harvest/1.5
hloader
hmview
httplib
httrack
httrack 3.0
humanlinks
image\ stripper
image\ sucker
indy\ library
infonavirobot
interget
internet\ ninja
jennybot
jetcar
joc\ web\ spider
kenjin spider
keyword density/0.9
larbin
leechftp
lexibot
libweb/clshttp
linkextractorpro
linkscan/8.1a unix
linkwalker
lnspiderguy
lwp-trivial
lwp-trivial/1.34
mass\ downloader
mata hari
microsoft url control - 5.01.4511
microsoft url control - 6.00.8169
midown\ tool
miixpc
miixpc/4.2
mister pix
mister\ pix
mj12bot
mozilla/4
mozilla/4.0 (compatible; bullseye; windows 95)
mozilla/4.0 (compatible; msie 4.0; windows 2000)
mozilla/4.0 (compatible; msie 4.0; windows 95)
mozilla/4.0 (compatible; msie 4.0; windows 98)
mozilla/4.0 (compatible; msie 4.0; windows me)
mozilla/4.0 (compatible; msie 4.0; windows nt)
mozilla/4.0 (compatible; msie 4.0; windows xp)
mozilla/5
mozilla/5.0
navroad
nearsite
net\ vampire
netants
netspider
netzip
nicerspro
octopus
offline explorer
offline\ explorer
offline\ navigator
openfind
openfind data gathere
pagegrabber
papa\ foto
pavuk
pcbrowser
propowerbot/2.14
prowebwalker
queryn metasearch
realdownload
reget
repomonkey
repomonkey bait & tackle/v1.01
rma
sistrix
sitesnagger
smartdownload
spankbot
spanner
superbot
superhttp
surfbot
suzuran
szukacz/1.4
takeout
teleport
teleport\ pro
teleportpro
telesoft
the intraformant
thenomad
tighttwatbot
titan
tocrawl/urldispatcher
true_robot
true_robot/1.0
turingos
turnitinbot
urly warning
vci
vci webviewer vci webviewer win32
voideye
web image collector
web\ image\ collector
web\ sucker
webauto
webbandit
webbandit/3.50
webcopier
webenhancer
webfetch
webgo\ is
webleacher
webmasterworldforumbot
webreaper
websauger
website quester
website\ extractor
website\ quester
webster pro
webstripper
webwhacker
webzip
webzip/4.0
wget
wget/1.5.3
wget/1.6
widow
www-collector-e
wwwoffle
xaldon\ webspider
zeus
zeus 32297 webster pro v2.9 win32
| Rule | Path |
|---|---|
| Disallow | / |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 600 |
Other Records
| Field | Value |
|---|---|
| sitemap | https://agfbrakpan.com/sitemap.xml |
Warnings
- `host` is not a known field.
Comments