ahexp.com
robots.txt
Robots Exclusion Standard data for ahexp.com
Resource Scan
Scan Details
| Site Domain | ahexp.com |
| Base Domain | ahexp.com |
| Scan Status | Ok |
| Last Scan | 2024-11-12T19:17:39+00:00 |
| Next Scan | 2024-11-19T19:17:39+00:00 |
Last Scan
| Scanned | 2024-11-12T19:17:39+00:00 |
| URL | https://ahexp.com/robots.txt |
| Domain IPs | 209.177.156.219 |
| Response IP | 209.177.156.219 |
| Found | Yes |
| Hash | 1b8f9b931f03fdd02d3e430e3ddb168242ee7059ac6ffa90713b2851f17b8226 |
| SimHash | 5a1569505391 |
Groups
amazonbot
googleother
| Rule | Path |
|---|---|
| Disallow | /member/ |
| Disallow | /modelinfo/ |
| Disallow | /registry/ |
| Disallow | /forum/ |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 5 |
baiduspider
ccbot
cliqzbot
criteobot
crsspxlbot
dataforseobot
discoverybot
domaincrawler
dotbot
extlinksbot
ezooms
grapeshot
istellabot
jikespider
linkpadbot
magpie-crawler
mauibot
megaindex.ru
mj12bot
proximic
seokicks-robot
sistrix
sogou
spbot
velenpublicwebcrawler
wbsearchbot
weborama
weborama-fetcher
wesee
yandex
yyspider
zumbot
| Rule | Path |
|---|---|
| Disallow | /journal/ |
| Disallow | /member/ |
| Disallow | /modelinfo/ |
| Disallow | /phorum/ |
| Disallow | /forum/ |
| Disallow | /registry/ |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 5 |
ahrefsbot
alphabot
alphaseobot
alphaseobot-sa
anthropic-ai
awariorssbot
awariosmartbot
barkrowler
blexbot
bomborabot
bytespider
careerbot
chatgpt-user
cohere-ai
friendlycrawler
google-extended
gptbot
gumgum
gumgum-bot
ip-web-crawler.com
linguee
meta-externalagent
oai-searchbot
perplexitybot
petalbot
seekport
semrushbot
semrushbot-ba
semrushbot-sa
serpstatbot
seznambot
timpibot
trendictionbot
turnitinbot
yak
zoominfobot
| Rule | Path |
|---|---|
| Disallow | / |
bingbot
msnbot
msnbot-media
No rules defined. All paths allowed.
Other Records
| Field | Value |
|---|---|
| crawl-delay | 3 |
weborama
weborama-fetcher
No rules defined. All paths allowed.
Other Records
| Field | Value |
|---|---|
| crawl-delay | 5 |
*
| Rule | Path |
|---|---|
| Disallow | /ads/click.php |
| Disallow | /calendar/attend.php |
| Disallow | /desktop |
| Disallow | /forum/newer/* |
| Disallow | /forum/older/* |
| Disallow | /member/activate |
| Disallow | /member/map.php |
| Disallow | /member/password_reset.php |
| Disallow | /member/reset |
| Disallow | /member/pp_result.php |
| Disallow | /modelinfo/ |
| Disallow | /newsletter/ |
| Disallow | /phorum/control.php |
| Disallow | /phorum/feed.php |
| Disallow | /phorum/login.php |
| Disallow | /phorum/moderation.php |
| Disallow | /phorum/pm.php |
| Disallow | /phorum/post.php |
| Disallow | /phorum/posting.php |
| Disallow | /phorum/random.php |
| Disallow | /phorum/read.php?*%2Cprintview%2C* |
| Disallow | /phorum/read.php?*%2Cquote=1 |
| Disallow | /phorum/read.php?*%2Cnewer |
| Disallow | /phorum/read.php?*%2Colder |
| Disallow | /phorum/report.php |
| Disallow | /phorum/search.php |
| Disallow | /registry/imgr_reset_pass.php |
| Disallow | /registry/browse.php |
| Disallow | /registry/transfer.php |
| Disallow | /toggle_comment.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 5 |
Comments