mgexp.com
robots.txt
Robots Exclusion Standard data for mgexp.com
Resource Scan
Scan Details
| Site Domain | mgexp.com |
| Base Domain | mgexp.com |
| Scan Status | Ok |
| Last Scan | 2024-06-07T17:58:51+00:00 |
| Next Scan | 2024-06-14T17:58:51+00:00 |
Last Scan
| Scanned | 2024-06-07T17:58:51+00:00 |
| URL | https://mgexp.com/robots.txt |
| Domain IPs | 209.177.156.219 |
| Response IP | 209.177.156.219 |
| Found | Yes |
| Hash | 3d76333c8c3ff0dcd2147673f9008c309ee48cdfcc519993508966a28bf4e261 |
| SimHash | 4a55691057d0 |
Groups
baiduspider
ccbot
cliqzbot
criteobot
crsspxlbot
discoverybot
domaincrawler
dotbot
extlinksbot
ezooms
grapeshot
istellabot
jikespider
linkpadbot
magpie-crawler
mauibot
megaindex.ru
mj12bot
proximic
seokicks-robot
sistrix
sogou
spbot
velenpublicwebcrawler
wbsearchbot
weborama
weborama-fetcher
wesee
yandex
yyspider
zumbot
| Rule | Path |
|---|---|
| Disallow | /journal/ |
| Disallow | /member/ |
| Disallow | /modelinfo/ |
| Disallow | /phorum/ |
| Disallow | /forum/ |
| Disallow | /registry/ |
ahrefsbot
alphabot
alphaseobot
alphaseobot-sa
barkrowler
blexbot
bomborabot
careerbot
chatgpt-user
friendlycrawler
gptbot
gumgum
gumgum-bot
linguee
ip-web-crawler.com
petalbot
semrushbot
semrushbot-ba
semrushbot-sa
seznambot
turnitinbot
trendictionbot
zoominfobot
| Rule | Path |
|---|---|
| Disallow | / |
bingbot
msnbot
msnbot-media
No rules defined. All paths allowed.
Other Records
| Field | Value |
|---|---|
| crawl-delay | 3 |
seekport
weborama
weborama-fetcher
No rules defined. All paths allowed.
Other Records
| Field | Value |
|---|---|
| crawl-delay | 5 |
*
| Rule | Path |
|---|---|
| Disallow | /ads/click.php |
| Disallow | /calendar/attend.php |
| Disallow | /desktop |
| Disallow | /forum/newer/* |
| Disallow | /forum/older/* |
| Disallow | /member/activate |
| Disallow | /member/map.php |
| Disallow | /member/password_reset.php |
| Disallow | /member/reset |
| Disallow | /member/pp_result.php |
| Disallow | /modelinfo/ |
| Disallow | /newsletter/ |
| Disallow | /phorum/control.php |
| Disallow | /phorum/feed.php |
| Disallow | /phorum/login.php |
| Disallow | /phorum/moderation.php |
| Disallow | /phorum/pm.php |
| Disallow | /phorum/post.php |
| Disallow | /phorum/posting.php |
| Disallow | /phorum/random.php |
| Disallow | /phorum/read.php?*%2Cprintview%2C* |
| Disallow | /phorum/read.php?*%2Cquote=1 |
| Disallow | /phorum/read.php?*%2Cnewer |
| Disallow | /phorum/read.php?*%2Colder |
| Disallow | /phorum/report.php |
| Disallow | /phorum/search.php |
| Disallow | /registry/imgr_reset_pass.php |
| Disallow | /registry/browse.php |
| Disallow | /registry/transfer.php |
| Disallow | /toggle_comment.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 5 |
Comments