Log In Sign Up

industry.co.id
robots.txt

Robots Exclusion Standard data for industry.co.id

Resource Scan

Scan Details

Site Domain industry.co.id
Base Domain industry.co.id
Scan Status Ok
Last Scan2024-10-01T22:43:38+00:00
Next Scan 2024-10-08T22:43:38+00:00

Last Scan

Scanned2024-10-01T22:43:38+00:00
URL https://industry.co.id/robots.txt
Redirect https://www.industry.co.id/robots.txt
Redirect Domain www.industry.co.id
Redirect Base industry.co.id
Domain IPs 104.21.1.221, 172.67.128.26, 2606:4700:3031::6815:1dd, 2606:4700:3031::ac43:801a
Redirect IPs 104.21.1.221, 172.67.128.26, 2606:4700:3031::6815:1dd, 2606:4700:3031::ac43:801a
Response IP 172.67.128.26
Found Yes
Hash 4ebc3b483d2b48b9dcf9568797d1f8f64077bab119e90edc1c10d1bc32c88765
SimHash 8914c010c6d1

Groups

mediapartners-google*

Rule Path
Allow /
Allow /garuda/uploads/
Disallow /cgi-bin/
Disallow /system/

googlemediabot*

Rule Path
Allow /
Allow /garuda/uploads/
Disallow /cgi-bin/
Disallow /system/

googlebot

Rule Path
Allow /
Allow /garuda/uploads/
Disallow /cgi-bin/
Disallow /system/

gulliver

Rule Path
Allow /
Allow /garuda/uploads/
Disallow /cgi-bin/
Disallow /system/

slurp

Rule Path
Allow /
Allow /garuda/uploads/
Disallow /cgi-bin/
Disallow /system/

fast

Rule Path
Allow /
Allow /garuda/uploads/
Disallow /cgi-bin/
Disallow /system/

scooter

Rule Path
Allow /
Allow /garuda/uploads/
Disallow /cgi-bin/
Disallow /system/

vscooter

Rule Path
Allow /
Allow /garuda/uploads/
Disallow /cgi-bin/
Disallow /system/

ia_archiver

Rule Path
Allow /
Allow /garuda/uploads/
Disallow /cgi-bin/
Disallow /system/

nutch

Rule Path
Allow /
Allow /garuda/uploads/
Disallow /cgi-bin/
Disallow /system/

fast-webcrawler

Rule Path
Allow /
Allow /garuda/uploads/
Disallow /cgi-bin/
Disallow /system/

teoma

Rule Path
Allow /
Allow /garuda/uploads/
Disallow /cgi-bin/
Disallow /system/

lycos_spider_(t-rex)

Rule Path
Allow /
Allow /garuda/uploads/
Disallow /cgi-bin/
Disallow /system/

bingbot

Rule Path
Allow /
Allow /garuda/uploads/
Disallow /cgi-bin/
Disallow /system/

msnbot

Rule Path
Allow /
Allow /garuda/uploads/
Disallow /cgi-bin/
Disallow /system/

msnptc

Rule Path
Allow /
Allow /garuda/uploads/
Disallow /cgi-bin/
Disallow /system/

msnbot-media/1.0

Rule Path
Allow /
Allow /garuda/uploads/
Disallow /cgi-bin/
Disallow /system/

architextspider

Rule Path
Allow /
Allow /garuda/uploads/
Disallow /cgi-bin/
Disallow /system/

zyborg

Rule Path
Allow /
Allow /garuda/uploads/
Disallow /cgi-bin/
Disallow /system/

surveybot

Rule Path
Allow /
Allow /garuda/uploads/
Disallow /cgi-bin/
Disallow /system/

alexabot

Rule Path
Allow /
Allow /garuda/uploads/
Disallow /cgi-bin/
Disallow /system/

baiduspider

Rule Path
Allow /
Allow /garuda/uploads/
Disallow /cgi-bin/
Disallow /system/

yandex

Rule Path
Allow /
Allow /garuda/uploads/
Disallow /cgi-bin/
Disallow /system/

jikespider

Rule Path
Allow /
Allow /garuda/uploads/
Disallow /cgi-bin/
Disallow /system/

naverbot

Rule Path
Allow /
Allow /garuda/uploads/
Disallow /cgi-bin/
Disallow /system/

yeti

Rule Path
Allow /
Allow /garuda/uploads/
Disallow /cgi-bin/
Disallow /system/

coccoc

Rule Path
Allow /
Allow /garuda/uploads/
Disallow /cgi-bin/
Disallow /system/

*

Rule Path
Allow /
Allow /garuda/uploads/
Disallow /cgi-bin/
Disallow /system/
Back to top

Other Records

Field Value
sitemap https://www.industry.co.id/sitemap.xml
sitemap https://www.industry.co.id/news-sitemap-indeks.xml
sitemap https://www.industry.co.id/photos-sitemap.xml
Back to top

Comments

  • The crawlers listed below are allowed on the Industry.co.id site.
  • Industry.co.id allows other crawlers on a case by case basis.
  • Setup Other crawlers
Back to top