Log In Sign Up

singletrackworld.com
robots.txt

Robots Exclusion Standard data for singletrackworld.com

Resource Scan

Scan Details

Site Domain singletrackworld.com
Base Domain singletrackworld.com
Scan Status Ok
Last Scan2024-11-09T20:27:16+00:00
Next Scan 2024-11-16T20:27:16+00:00

Last Scan

Scanned2024-11-09T20:27:16+00:00
URL https://singletrackworld.com/robots.txt
Domain IPs 2406:da18:9d0:143f:2124:4e9c:36a9:d9de, 52.221.42.138
Response IP 52.221.42.138
Found Yes
Hash 1bef04795553098874f8b4f0fd478c76ae6822448095a491bb9fe820a02b20b4
SimHash 628cd8a48b93

Groups

*

Rule Path
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php
Disallow /private-directory/
Allow /public-directory/

*

Rule Path
Disallow /singletrackworld-registration
Disallow /forum/topic/debt-recovery-companies
Disallow /forum/topic/a-shambolic-week-with-sierra-cycling
Disallow /forum/topic/a-shambolic-week-with-sierra-cycling/page/2
Disallow /forum/topic/a-shambolic-week-with-sierra-cycling/page/3
Disallow /forum/topic/a-shambolic-week-with-sierra-cycling/page/4
Disallow /forum/topic/a-shambolic-week-with-sierra-cycling/page/5
Disallow /forum/topic/a-shambolic-week-with-sierra-cycling/page/6
Disallow /forum/topic/a-shambolic-week-with-sierra-cycling/page/7
Disallow /forum/topic/a-shambolic-week-with-sierra-cycling/page/8
Disallow /forum/topic/a-shambolic-week-with-sierra-cycling/page/9
Disallow /forum/topic/a-shambolic-week-with-sierra-cycling/page/10
Disallow /forum/topic/a-shambolic-week-with-sierra-cycling/page/11
Disallow /forum/topic/a-shambolic-week-with-sierra-cycling/page/12
Disallow /forum/topic/a-shambolic-week-with-sierra-cycling/page/13
Disallow /forum/topic/a-shambolic-week-with-sierra-cycling/page/14
Disallow /magarchivedb/

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

msnbot-media

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

yandex

No rules defined. All paths allowed.

Other Records

Field Value Comment
crawl-delay 20 sets a 2-second timeout

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

magpie-crawler

Rule Path
Disallow /

exabot

Rule Path
Disallow /

the knowledge ai

Rule Path
Disallow /wp-json/oembed/1.0/embed

amazonbot

Rule Path
Disallow /wp-json/oembed/1.0/embed

gptbot

Rule Path
Disallow /

ias_crawler

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://singletrackworld.com/sitemap.xml
sitemap https://singletrackworld.com/news-sitemap.xml
Back to top