Log In Sign Up

butikgent.rs
robots.txt

Robots Exclusion Standard data for butikgent.rs

Resource Scan

Scan Details

Site Domain butikgent.rs
Base Domain butikgent.rs
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan2025-07-17T04:49:11+00:00
Next Scan 2025-10-15T04:49:11+00:00

Last Successful Scan

Scanned2023-09-03T06:37:09+00:00
URL http://butikgent.rs/robots.txt
Redirect http://www.butikgent.rs/static/robots.txt
Redirect Domain www.butikgent.rs
Redirect Base butikgent.rs
Domain IPs 146.70.66.55
Redirect IPs 146.70.66.55
Response IP 146.70.66.55
Found Yes
Hash f38283d6f0eac0711171294005acd2e324b12047f85eac944159f8f16222f38f
SimHash be3df9328bf8

Groups

*

Rule Path
Disallow /feed/
Disallow /html/
Disallow /scripts/
Disallow /remote
Disallow /invoice
Disallow /*?f=*
Disallow /*?fb_xd_fragment
Disallow /*?qty=*
Disallow /*?mobile=*

Other Records

Field Value
crawl-delay 5

ahrefsbot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

etaospider

Rule Path
Disallow /

npbot

Rule Path
Disallow /

synthesio

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap http://www.butikgent.rs/sitemap.xml
Back to top

Comments

  • No need to index this stuff
  • Avoid duplicate content as it may do more harm than good
  • Disallow some bots we do not care about
Back to top

Warnings

  • 2 invalid lines.
Back to top