Log In Sign Up

autoindustriya.com
robots.txt

Robots Exclusion Standard data for autoindustriya.com

Resource Scan

Scan Details

Site Domain autoindustriya.com
Base Domain autoindustriya.com
Scan Status Ok
Last Scan2024-05-30T15:46:38+00:00
Next Scan 2024-06-06T15:46:38+00:00

Last Scan

Scanned2024-05-30T15:46:38+00:00
URL https://autoindustriya.com/robots.txt
Domain IPs 172.66.40.111, 172.66.43.145, 2606:4700:3108::ac42:286f, 2606:4700:3108::ac42:2b91
Response IP 172.66.40.111
Found Yes
Hash f5ec5af5ff9688721e30437e676c246b61b2fab8f40c3454dcb6eedaaf66a550
SimHash 201b1a824171

Groups

*

Rule Path
Disallow /talkboard/*action
Disallow /talkboard/*PHPSESSID
Disallow /talkboard/*%3B
Allow /talkboard/*board%2C*.0.html$
Allow /talkboard/*topic%2C*.0.html$

facebookexternalhit

Rule Path
Allow /

facebookbot

Rule Path
Allow /images/

twitterbot/1.0

Rule Path
Allow /images/

googlebot-image

Rule Path
Allow /images/

googlebot

Rule Path
Allow /images/
Disallow /cron.php?ts=*
Disallow /talkboard/*action
Disallow /talkboard/*PHPSESSID
Disallow /talkboard/*%3B
Allow /talkboard/*board%2C*.0.html$
Allow /talkboard/*topic%2C*.0.html$

facebookexternalhit

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

facebookbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

spider

Rule Path
Disallow /

bot-

Rule Path
Disallow /

linkchecker

Rule Path
Disallow /

microsoft url control

Rule Path
Disallow /

irlbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

java

Rule Path
Disallow /

nicebot

Rule Path
Disallow /

nutch

Rule Path
Disallow /

python-urllib

Rule Path
Disallow /

powermarks

Rule Path
Disallow /

missigua_locator

Rule Path
Disallow /

web downloader

Rule Path
Disallow /

lanshanbot

Rule Path
Disallow /

custo

Rule Path
Disallow /

cfnetwork

Rule Path
Disallow /

httrack off-line browser

Rule Path
Disallow /

nutchcvs

Rule Path
Disallow /

t-h-u-n-d-e-r-s-t-o-n-e

Rule Path
Disallow /

jakarta commons-httpclient

Rule Path
Disallow /

htmlparser

Rule Path
Disallow /

crawl

Rule Path
Disallow /

sitecheck.internetseer.com

Rule Path
Disallow /

zealbot

Rule Path
Disallow /

msiecrawler

Rule Path
Disallow /

sitesnagger

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

fetch

Rule Path
Disallow /

offline explorer

Rule Path
Disallow /

teleport

Rule Path
Disallow /

teleportpro

Rule Path
Disallow /

webzip

Rule Path
Disallow /

linko

Rule Path
Disallow /

httrack

Rule Path
Disallow /

microsoft.url.control

Rule Path
Disallow /

larbin

Rule Path
Disallow /

zyborg

Rule Path
Disallow /

download ninja

Rule Path
Disallow /

k2spider

Rule Path
Disallow /

webreaper

Rule Path
Disallow /

npbot

Rule Path
Disallow /

serpstatbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

*

Rule Path
Disallow /cron.php
Back to top

Other Records

Field Value
sitemap https://www.autoindustriya.com/sitemap.xml
Back to top