Log In Sign Up

tengemarket.uz
robots.txt

Robots Exclusion Standard data for tengemarket.uz

Resource Scan

Scan Details

Site Domain tengemarket.uz
Base Domain tengemarket.uz
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a server error.
Last Scan2025-04-07T13:16:37+00:00
Next Scan 2025-07-06T13:16:37+00:00

Last Successful Scan

Scanned2024-11-16T10:48:42+00:00
URL https://tengemarket.uz/robots.txt
Domain IPs 194.93.25.216
Response IP 194.93.25.216
Found Yes
Hash 26014a6bc235989bc1d9b411c364a6ade3dc4443bde6df2a026853b378df681b
SimHash 34bedf7246b7

Groups

*

Rule Path
Disallow /search
Disallow /profile
Disallow /checkout
Disallow /api
Disallow /forms
Disallow /order
Disallow /login
Disallow /cms

Other Records

Field Value
crawl-delay 2

googlebot

Rule Path
Allow /
Disallow /search
Disallow /profile
Disallow /checkout
Disallow /api
Disallow /forms
Disallow /order
Disallow /login
Disallow /cms

googlebot-image

Rule Path
Allow /
Disallow /search
Disallow /profile
Disallow /checkout
Disallow /api
Disallow /forms
Disallow /order
Disallow /login
Disallow /cms

googlebot-mobile

Rule Path
Allow /
Disallow /search
Disallow /profile
Disallow /checkout
Disallow /api
Disallow /forms
Disallow /order
Disallow /login
Disallow /cms

yandex

Rule Path
Allow /
Disallow /search
Disallow /profile
Disallow /checkout
Disallow /api
Disallow /forms
Disallow /order
Disallow /login
Disallow /cms

ahrefs

Rule Path
Allow /
Disallow /search
Disallow /profile
Disallow /checkout
Disallow /api
Disallow /forms
Disallow /order
Disallow /login
Disallow /cms

semrush

Rule Path
Allow /
Disallow /search
Disallow /profile
Disallow /checkout
Disallow /api
Disallow /forms
Disallow /order
Disallow /login
Disallow /cms

ahrefsbot

Rule Path
Allow /
Disallow /search
Disallow /profile
Disallow /checkout
Disallow /api
Disallow /forms
Disallow /order
Disallow /login
Disallow /cms

bingbot

Rule Path
Disallow /

cazoodlebot

Rule Path
Disallow /

dotbot/1.0

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

sputnikbot

Rule Path
Disallow /

spbot

Rule Path
Disallow /

digext

Rule Path
Disallow /

sogou

Rule Path
Disallow /

mj12

Rule Path
Disallow /

majestic12

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

crowsnest

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

talktalk

Rule Path
Disallow /

paperlibot

Rule Path
Disallow /

peerindex

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

slurp

Rule Path
Disallow /

aport

Rule Path
Disallow /

ning

Rule Path
Disallow /

js-kit

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

twiceler

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

java

Rule Path
Disallow /

commentreader

Rule Path
Disallow /

yeti

Rule Path
Disallow /

discobot

Rule Path
Disallow /

btwebclient

Rule Path
Disallow /

tagoobot

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

igdespyder

Rule Path
Disallow /

teleport

Rule Path
Disallow /

offline

Rule Path
Disallow /

disco

Rule Path
Disallow /

netvampire

Rule Path
Disallow /

copier

Rule Path
Disallow /

httrack

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

webcollector

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://tengemarket.uz/sitemap-service/sitemap.xml
sitemap https://tengemarket.uz/sitemap-service/sitemap.xml
sitemap https://tengemarket.uz/sitemap-service/sitemap.xml
sitemap https://tengemarket.uz/sitemap-service/sitemap.xml
sitemap https://tengemarket.uz/sitemap-service/sitemap.xml
sitemap https://tengemarket.uz/sitemap-service/sitemap.xml
sitemap https://tengemarket.uz/sitemap-service/sitemap.xml
sitemap https://tengemarket.uz/sitemap-service/sitemap.xml
Back to top

Comments

  • Crawlers Setup
  • Directories
  • Paths
  • Website Sitemap
Back to top

Warnings

  • 2 invalid lines.
  • `clean-param` is not a known field.
  • `request-rate` is not a known field.
  • `visit-time` is not a known field.
Back to top