Log In Sign Up

carlist.my
robots.txt

Robots Exclusion Standard data for carlist.my

Resource Scan

Scan Details

Site Domain carlist.my
Base Domain carlist.my
Scan Status Ok
Last Scan2024-04-25T18:53:49+00:00
Next Scan 2024-05-02T18:53:49+00:00

Last Scan

Scanned2024-04-25T18:53:49+00:00
URL https://carlist.my/robots.txt
Redirect https://www.carlist.my/robots.txt
Redirect Domain www.carlist.my
Redirect Base carlist.my
Domain IPs 35.186.244.59
Redirect IPs 35.186.244.59
Response IP 35.186.244.59
Found Yes
Hash 3dfaff896be31af8a5e65e3574857de5e621610fdbb249983431cbf1b64cd3d9
SimHash 6b14d8f5e391

Groups

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

adidxbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

bingpreview

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

dataforseobot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

serpstatbot

Rule Path
Disallow /

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

teracent-feed-processing

Rule Path
Disallow /

*

Rule Path
Disallow /ajax/
Back to top

Other Records

Field Value
sitemap https://www.carlist.my/sitemap.xml
Back to top