cyclekartclub.com
robots.txt
Robots Exclusion Standard data for cyclekartclub.com
Resource Scan
Scan Details
Site Domain | cyclekartclub.com |
Base Domain | cyclekartclub.com |
Scan Status | Ok |
Last Scan | 2024-09-29T21:39:39+00:00 |
Next Scan | 2024-10-06T21:39:39+00:00 |
Last Scan
Scanned | 2024-09-29T21:39:39+00:00 |
URL | https://cyclekartclub.com/robots.txt |
Domain IPs | 209.177.156.219 |
Response IP | 209.177.156.219 |
Found | Yes |
Hash | 709467ee120de3e599a71dcc0b9ccf907ec0c00f7b966752bc5b95cc44d1e99e |
SimHash | 4a15695053d1 |
Groups
amazonbot
googleother
Rule | Path |
---|---|
Disallow | /member/ |
Disallow | /modelinfo/ |
Disallow | /registry/ |
Disallow | /forum/ |
Other Records
Field | Value |
---|---|
crawl-delay | 5 |
baiduspider
ccbot
cliqzbot
criteobot
crsspxlbot
dataforseobot
discoverybot
domaincrawler
dotbot
extlinksbot
ezooms
grapeshot
istellabot
jikespider
linkpadbot
magpie-crawler
mauibot
megaindex.ru
mj12bot
proximic
seokicks-robot
sistrix
sogou
spbot
velenpublicwebcrawler
wbsearchbot
weborama
weborama-fetcher
wesee
yandex
yyspider
zumbot
Rule | Path |
---|---|
Disallow | /journal/ |
Disallow | /member/ |
Disallow | /modelinfo/ |
Disallow | /phorum/ |
Disallow | /forum/ |
Disallow | /registry/ |
Other Records
Field | Value |
---|---|
crawl-delay | 5 |
ahrefsbot
alphabot
alphaseobot
alphaseobot-sa
anthropic-ai
barkrowler
blexbot
bomborabot
careerbot
chatgpt-user
cohere-ai
friendlycrawler
google-extended
gptbot
gumgum
gumgum-bot
linguee
meta-externalagent
ip-web-crawler.com
oai-searchbot
panscient.com
petalbot
semrushbot
semrushbot-ba
semrushbot-sa
seznambot
turnitinbot
trendictionbot
zoominfobot
Rule | Path |
---|---|
Disallow | / |
bingbot
msnbot
msnbot-media
No rules defined. All paths allowed.
Other Records
Field | Value |
---|---|
crawl-delay | 3 |
seekport
weborama
weborama-fetcher
No rules defined. All paths allowed.
Other Records
Field | Value |
---|---|
crawl-delay | 5 |
*
Rule | Path |
---|---|
Disallow | /ads/click.php |
Disallow | /calendar/attend.php |
Disallow | /desktop |
Disallow | /forum/newer/* |
Disallow | /forum/older/* |
Disallow | /member/activate |
Disallow | /member/map.php |
Disallow | /member/password_reset.php |
Disallow | /member/reset |
Disallow | /member/pp_result.php |
Disallow | /modelinfo/ |
Disallow | /newsletter/ |
Disallow | /phorum/control.php |
Disallow | /phorum/feed.php |
Disallow | /phorum/login.php |
Disallow | /phorum/moderation.php |
Disallow | /phorum/pm.php |
Disallow | /phorum/post.php |
Disallow | /phorum/posting.php |
Disallow | /phorum/random.php |
Disallow | /phorum/read.php?*%2Cprintview%2C* |
Disallow | /phorum/read.php?*%2Cquote=1 |
Disallow | /phorum/read.php?*%2Cnewer |
Disallow | /phorum/read.php?*%2Colder |
Disallow | /phorum/report.php |
Disallow | /phorum/search.php |
Disallow | /registry/imgr_reset_pass.php |
Disallow | /registry/browse.php |
Disallow | /registry/transfer.php |
Disallow | /toggle_comment.php |
Other Records
Field | Value |
---|---|
crawl-delay | 5 |
Comments