sgcarmart.com
robots.txt

Robots Exclusion Standard data for sgcarmart.com

Resource Scan

Scan Details

Site Domain sgcarmart.com
Base Domain sgcarmart.com
Scan Status Ok
Last Scan2024-11-14T17:37:43+00:00
Next Scan 2024-11-21T17:37:43+00:00

Last Scan

Scanned2024-11-14T17:37:43+00:00
URL https://sgcarmart.com/robots.txt
Redirect https://www.sgcarmart.com/robots.txt
Redirect Domain www.sgcarmart.com
Redirect Base sgcarmart.com
Domain IPs 104.18.20.216, 104.18.21.216
Redirect IPs 104.18.20.216, 104.18.21.216
Response IP 104.18.20.216
Found Yes
Hash 4b1cad2ee81ed187fdf1dd944bc51ddae137944cc096fdb4cfee5b190b7ee75f
SimHash 61234865c71f

Groups

*

Rule Path
Disallow /articles/listing*?q=
Disallow /car-loan/info.php*?q=
Disallow /used-cars/listing?*page=
Allow /used-cars/listing?*page=1
Disallow /search*?q=

Other Records

Field Value
crawl-delay 30

ia_archiver

Rule Path
Disallow /

yahoo-mmcrawler

Rule Path
Disallow /

paracrawl

Rule Path
Disallow /

psbot

Rule Path
Disallow /

sentibot

Rule Path
Disallow /

sentibot www.sentibot.eu (compatible with googlebot)

Rule Path
Disallow /

applebot

Rule Path
Disallow /cgi-bin/
Disallow /images/
Disallow /mail/
Disallow /dealer/
Disallow /directory/premium/
Disallow /includes/
Disallow /phpads/
Disallow /update/
Disallow /upload/
Disallow /used_cars/valuation.php
Disallow /api
Disallow /used-cars/api
Disallow /shop/api

Other Records

Field Value
crawl-delay 5

grapeshot

Rule Path
Disallow /cgi-bin/
Disallow /images/
Disallow /mail/
Disallow /dealer/
Disallow /directory/premium/
Disallow /includes/
Disallow /phpads/
Disallow /update/
Disallow /upload/
Disallow /used_cars/valuation.php
Disallow /api
Disallow /used-cars/api
Disallow /shop/api

Other Records

Field Value
crawl-delay 0

petalbot

Rule Path
Allow /rss/

googlebot-desktop

Rule Path
Disallow /cgi-bin/
Disallow /images/
Disallow /mail/
Disallow /dealer/
Disallow /directory/premium/
Disallow /includes/
Disallow /phpads/
Disallow /update/
Disallow /upload/
Disallow /checkout/
Disallow /used_cars/valuation.php
Disallow /api
Disallow /used-cars/api
Disallow /shop/api

siteauditbot-desktop

Rule Path
Disallow /cgi-bin/
Disallow /images/
Disallow /mail/
Disallow /dealer/
Disallow /directory/premium/
Disallow /includes/
Disallow /phpads/
Disallow /update/
Disallow /upload/
Disallow /checkout/
Disallow /used_cars/valuation.php
Disallow /api
Disallow /used-cars/api
Disallow /shop/api

googlebot-mobile

Rule Path
Disallow /cgi-bin/
Disallow /images/
Disallow /mail/
Disallow /dealer/
Disallow /directory/premium/
Disallow /includes/
Disallow /phpads/
Disallow /update/
Disallow /upload/
Disallow /checkout/
Disallow /used_cars/valuation.php
Disallow /api
Disallow /used-cars/api
Disallow /shop/api

siteauditbot-mobile

Rule Path
Disallow /cgi-bin/
Disallow /images/
Disallow /mail/
Disallow /dealer/
Disallow /directory/premium/
Disallow /includes/
Disallow /phpads/
Disallow /update/
Disallow /upload/
Disallow /checkout/
Disallow /used_cars/valuation.php
Disallow /api
Disallow /used-cars/api
Disallow /shop/api

*

Rule Path
Disallow /cgi-bin/
Disallow /images/
Disallow /mail/
Disallow /dealer/
Disallow /directory/premium/
Disallow /includes/
Disallow /phpads/
Disallow /update/
Disallow /upload/
Disallow /used_cars/valuation.php
Disallow /api
Disallow /used-cars/api
Disallow /shop/api

Other Records

Field Value
crawl-delay 5

Other Records

Field Value
sitemap https://www.sgcarmart.com/sitemap_index.xml