Log In Sign Up

ataberkestate.com
robots.txt

Robots Exclusion Standard data for ataberkestate.com

Resource Scan

Scan Details

Site Domain ataberkestate.com
Base Domain ataberkestate.com
Scan Status Ok
Last Scan2025-12-01T03:30:55+00:00
Next Scan 2025-12-31T03:30:55+00:00

Last Scan

Scanned2025-12-01T03:30:55+00:00
URL https://ataberkestate.com/robots.txt
Redirect https://www.ataberkestate.com/robots.txt
Redirect Domain www.ataberkestate.com
Redirect Base ataberkestate.com
Domain IPs 104.21.65.183, 172.67.165.77, 2606:4700:3032::ac43:a54d, 2606:4700:3035::6815:41b7
Redirect IPs 104.21.65.183, 172.67.165.77, 2606:4700:3032::ac43:a54d, 2606:4700:3035::6815:41b7
Response IP 172.67.165.77
Found Yes
Hash b95a46984f7969a8addc5c1dd963c269befd9b58aaa3d2c8f7632d78b9218a65
SimHash 700c5f409f49

Groups

*

Rule Path
Allow /core/*.css$
Allow /core/*.css?
Allow /core/*.js$
Allow /core/*.js?
Allow /core/*.gif
Allow /core/*.jpg
Allow /core/*.jpeg
Allow /core/*.png
Allow /core/*.svg
Allow /core/*.webp
Allow /profiles/*.css$
Allow /profiles/*.css?
Allow /profiles/*.js$
Allow /profiles/*.js?
Allow /profiles/*.gif
Allow /profiles/*.jpg
Allow /profiles/*.jpeg
Allow /profiles/*.png
Allow /profiles/*.svg
Allow /profiles/*.webp
Allow /libraries/*.css$
Allow /libraries/*.css?
Allow /libraries/*.js$
Allow /libraries/*.js?
Allow /libraries/*.gif
Allow /libraries/*.jpg
Allow /libraries/*.jpeg
Allow /libraries/*.png
Allow /libraries/*.svg
Allow /libraries/*.webp
Allow /modules/*.css$
Allow /modules/*.css?
Allow /modules/*.js$
Allow /modules/*.js?
Allow /modules/*.gif
Allow /modules/*.jpg
Allow /modules/*.jpeg
Allow /modules/*.png
Allow /modules/*.svg
Allow /modules/*.webp
Allow /themes/*.css$
Allow /themes/*.css?
Allow /themes/*.js$
Allow /themes/*.js?
Allow /themes/*.gif
Allow /themes/*.jpg
Allow /themes/*.jpeg
Allow /themes/*.png
Allow /themes/*.svg
Allow /themes/*.webp
Allow /sites/default/files/*.css$
Allow /sites/default/files/*.css?
Allow /sites/default/files/*.js$
Allow /sites/default/files/*.js?
Allow /sites/default/files/*.gif$
Allow /sites/default/files/*.gif?
Allow /sites/default/files/*.jpg$
Allow /sites/default/files/*.jpg?
Allow /sites/default/files/*.jpeg$
Allow /sites/default/files/*.jpeg?
Allow /sites/default/files/*.png$
Allow /sites/default/files/*.png?
Allow /sites/default/files/*.svg$
Allow /sites/default/files/*.svg?
Allow /sites/default/files/*.webp$
Allow /sites/default/files/*.webp?
Disallow /*?*sort_bef_combine=
Disallow /*%26sort_bef_combine%3D
Disallow /manager/search/
Disallow /core*$
Disallow /profiles*$
Disallow /libraries*$
Disallow /modules*$
Disallow /themes*$
Disallow /*node_modules*$
Disallow /README.txt
Disallow /INSTALL.txt
Disallow /web.config
Disallow /update.php
Disallow /index.php*$
Disallow /cron.php
Disallow /*.php$
Disallow /*.yml$
Disallow /*.po$
Disallow /*.theme$
Disallow /*.md$
Disallow /*.module$
Disallow /*.install$
Disallow /*.twig$
Disallow /*.gitignore$
Disallow /*.git
Disallow /form*$
Disallow /admin*$
Disallow /filter*$
Disallow /search*$
Disallow /user*$
Disallow /user/register*$
Disallow /login*$
Disallow *login
Disallow *register
Disallow /register*$
Disallow /reset-password*$
Disallow /taxonomy*$
Disallow /comment*$
Disallow /node*$
Disallow /modules/
Disallow /*?sy0p8r
Disallow /admin/
Disallow /comment/reply/
Disallow /filter/tips
Disallow /node/add/
Disallow /search/
Disallow /user/register/
Disallow /user/password/
Disallow /user/login/
Disallow /user/logout/
Disallow /index.php/admin/
Disallow /index.php/comment/reply/
Disallow /index.php/filter/tips
Disallow /index.php/node/add/
Disallow /index.php/search/
Disallow /index.php/user/password/
Disallow /index.php/user/register/
Disallow /index.php/user/login/
Disallow /index.php/user/logout/
Disallow /noindex
Disallow /noindex/*
Disallow /noindex/
Disallow /favorites
Disallow /*?*page=
Disallow /*%26page%3D
Disallow /*?*sort=
Disallow /*%26sort%3D
Disallow *?marquiz*
Disallow /flag/
Disallow /flag/flag/
Disallow /flag/flag/favorites
Disallow *?token=
Disallow *?destination=
Disallow /book/export/
Disallow /top-rated-
Disallow /messages/
Disallow /book/export/
Disallow /user2userpoints/
Disallow /myuserpoints/
Disallow /referral/
Disallow /aggregator/
Disallow /files/pin/
Disallow /your-votes
Disallow /comments/recent
Disallow /*/edit/
Disallow /*/delete/
Disallow /*/export/html/
Disallow /*/edit$
Disallow /*/outline$
Disallow /*/revisions$
Disallow /*/contact$
Disallow /taxonomy/
Disallow /node
Disallow /node/
Disallow /*?page=*&*
Disallow /*?page=0*
Disallow /*?field
Disallow /*?field*
Disallow */taxonomy*
Disallow *?field*
Disallow /mobile*
Disallow /selection*
Disallow /selection/*
Disallow /selection
Disallow /podpiska
Disallow /*?page=0
Disallow /*%
Disallow /*%26
Disallow /*?
Disallow /print-page
Disallow /print-page/*
Disallow /flag/*
Disallow /flag
Disallow /news/denezhnye-perevody-zolotaya-korona-v-turcii-alaniya

Other Records

Field Value
crawl-delay 20

googlebot-image

Rule Path
Allow /*.gif
Allow /*.jpg
Allow /*.jpeg
Allow /*.png
Allow /*.svg
Allow /*.webp

googlebot

Rule Path
Allow /sites/default/files/*.css
Allow /sites/default/files/*.js
Allow /sites/default/files/*.png
Allow /sites/default/files/*.jpg
Disallow /*?sy0p8r
Disallow /flag/

Other Records

Field Value
crawl-delay 10

yandex

Rule Path
Disallow /*?sy0p8r
Disallow /flag/

Other Records

Field Value
crawl-delay 20

yandeximages

Rule Path
Allow /*.gif
Allow /*.jpg
Allow /*.jpeg
Allow /*.png
Allow /*.svg
Allow /*.webp

mediapartners-google

Rule Path
Disallow /

yadirectbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

riddler

Rule Path
Disallow /

aihitbot

Rule Path
Disallow /

trovitbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

flipboardproxy

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.ataberkestate.com/sitemap.xml?page=1
sitemap https://www.ataberkestate.com/sitemap.xml?page=2
sitemap https://www.ataberkestate.com/sitemap.xml?page=3
sitemap https://www.ataberkestate.com/sitemap.xml?page=4
sitemap https://www.ataberkestate.com/sitemap.xml?page=5
Back to top

Comments

  • Разрешаем все CSS, JS, Images
  • Разрешаем индексацию минифицированных файлов скриптов
  • и изображений-стилей в папке files
  • Запрещаем директории
  • Запрещаем файлы
  • Запрещаем пути (чистые URLs)
  • Paths (clean URLs)
  • Paths (no clean URLs)
  • Пагинация и дубли страниц
Back to top