Log In Sign Up

auto-treff.com
robots.txt

Robots Exclusion Standard data for auto-treff.com

Resource Scan

Scan Details

Site Domain auto-treff.com
Base Domain auto-treff.com
Scan Status Ok
Last Scan2024-09-28T01:36:12+00:00
Next Scan 2024-10-05T01:36:12+00:00

Last Scan

Scanned2024-09-28T01:36:12+00:00
URL https://auto-treff.com/robots.txt
Domain IPs 144.76.174.212
Response IP 144.76.174.212
Found Yes
Hash b103d7373a068e23d5330d5b99b14e864e2da263b53b93dda2c1bd8711202319
SimHash 7b5dd052dd02

Groups

*

Rule Path
Disallow /forum/index.php?board%2F130-politik%2F

*

Rule Path
Allow *
Disallow /*/acp/
Disallow /*/attachments/
Disallow /*/cache/
Disallow /*/language/
Disallow /*/lib/
Disallow /*/log/
Disallow /*/templates/
Disallow /*/tmp/
Disallow /*/user/
Disallow /*/register/
Disallow /*/login/
Disallow /*/lost-password/
Disallow /*/github-auth/
Disallow /*/twitter-auth/
Disallow /*/google-auth/
Disallow /*/facebook-auth/
Disallow /*/search-result/
Disallow /*/search/
Disallow /*/tag-search/
Disallow /*/tagged/
Disallow /*/user-search/
Disallow /*/members-list/
Disallow /*?*sortField=
Disallow /*?*sortOrder=
Disallow /*?*filterByDate=
Disallow /*?*filterByStatus=
Disallow /*?*highlight=
Disallow /*?*objectType=
Disallow /*?*utm_source=
Disallow /*?*utm_medium=
Disallow /*?*utm_campaign=
Disallow /*/impressum/
Disallow /*/legal-notice/
Disallow /*/allgemeine-geschaeftsbedingungen/
Disallow /*/widerrufsbelehrung/
Disallow /*/datenschutzerklaerung/
Disallow /*/privacy-policy/
Disallow /*/nutzungsbedingungen/
Disallow /*/kontakt/
Disallow /*/contact/
Disallow /*/product-terms-and-conditions-download/
Disallow /*/268120-was-man-noch-so-sagen-darf/
Disallow /*/266538-pandemie-durch-corona-virus/

my-tiny-bot
fidget-spinner-bot
ahrefsbot
semrushbot
blexbot
dotbot
mj12bot
barkrowler
yandexbot
petalbot
sentibot
dataforseobot
megaindex
neticle
censysinspect
um-ic
researchbot
omgili
bytespider
mojeekbot
seocompany
serpstatbot
trendiction-bot
petalbot

Rule Path
Disallow /

amazonbot
pinterestbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 60

claudebot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.auto-treff.com/sitemap.xml
Back to top

Comments

  • sitemap
  • block internal directories
  • block user profiles
  • block registration and login
  • block post links
  • Disallow: /*?*postID=
  • Disallow: /*?*tagID=
  • Disallow: /*?*pageNo=1$
  • Disallow: /*?*action=firstNew
  • Disallow: /*?*action=lastPost
  • block search
  • block filtered / ordered pages
  • block utm
  • block legal pages and contact form
  • block crawler
  • limit crawler
Back to top

Warnings

  • 4 invalid lines.
Back to top