Log In Sign Up

pikabu.ru
robots.txt

Robots Exclusion Standard data for pikabu.ru

Resource Scan

Scan Details

Site Domain pikabu.ru
Base Domain pikabu.ru
Scan Status Ok
Last Scan2024-11-11T11:07:39+00:00
Next Scan 2024-11-18T11:07:39+00:00

Last Scan

Scanned2024-11-11T11:07:39+00:00
URL https://pikabu.ru/robots.txt
Domain IPs 185.178.210.201
Response IP 185.178.210.201
Found Yes
Hash f5051627da6a0a5312b1d91d2f0a0b542b6f19fdc8e643ff8e9ff79303fd3982
SimHash 3a46c1cb6ab4

Groups

*

Rule Path
Allow */community/edits/new
Allow /search?n=16&r=3
Allow /original-stories
Disallow /generate_xml.php
Disallow /restore.php
Disallow /ajax/*
Disallow /api/*
Disallow /api2/*
Disallow /assets/*
Disallow /cron/*
Disallow /data_vars/*
Disallow /external/*
Disallow /files/*
Disallow /staff/*
Disallow /pds/*
Disallow *username%3D*
Disallow /offer-redirect/
Disallow /*?r=
Disallow /*?q=*
Disallow */edits/*
Disallow /*?n=*
Disallow /admitad/finance/redirect
Disallow /page/misc/adstat.php*
Disallow /.well-known/*
Disallow /search?*
Disallow /tag/*st%3D*
Disallow /tag/*n%3D*
Disallow /tag/*et%3D*
Disallow /tag/*d%3D*
Disallow /tag/*cl%3D*
Disallow /tag/*vh%3D*
Disallow /%40*/*
Disallow /community/*/*
Disallow /communities/*
Disallow /communities?t=*
Disallow /communities?q=*
Disallow /add?community=*
Disallow /information/docs
Disallow *etm_term%3D*
Disallow *ysclid%3D*
Disallow *loc%3D*

yandex

Rule Path
Allow */community/edits/new
Allow /search?n=16&r=3
Allow /original-stories
Disallow /generate_xml.php
Disallow /restore.php
Disallow /ajax/*
Disallow /api/*
Disallow /api2/*
Disallow /assets/*
Disallow /cron/*
Disallow /data_vars/*
Disallow /external/*
Disallow /files/*
Disallow /rabbit_workers/*
Disallow /staff/*
Disallow /pds/*
Disallow *username%3D*
Disallow /offer-redirect/
Disallow /*?r=
Disallow /*?q=*
Disallow */edits/*
Disallow /*?n=*
Disallow /admitad/finance/redirect
Disallow /page/misc/adstat.php*
Disallow /.well-known/*
Disallow /search?*
Disallow /tag/*st%3D*
Disallow /tag/*n%3D*
Disallow /tag/*et%3D*
Disallow /tag/*d%3D*
Disallow /tag/*cl%3D*
Disallow /tag/*vh%3D*
Disallow /%40*/*
Disallow /community/*/*
Disallow /communities/*
Disallow /communities?t=*
Disallow /communities?q=*
Disallow /add?community=*

Other Records

Field Value
crawl-delay 3

mail.ru_bot

Rule Path
Allow */community/edits/new
Allow /search?n=16&r=3
Allow /original-stories
Disallow /generate_xml.php
Disallow /restore.php
Disallow /WT4fgrvixo2F/*
Disallow /ajax/*
Disallow /api/*
Disallow /api2/*
Disallow /assets/*
Disallow /cron/*
Disallow /data_vars/*
Disallow /external/*
Disallow /files/*
Disallow /rabbit_workers/*
Disallow /staff/*
Disallow /pds/*
Disallow *username%3D*
Disallow /offer-redirect/
Disallow /*?r=
Disallow /*?q=*
Disallow */edits/*
Disallow /*?n=*
Disallow /admitad/finance/redirect
Disallow /page/misc/adstat.php*
Disallow /.well-known/*
Disallow /search?*
Disallow /tag/*st%3D*
Disallow /tag/*n%3D*
Disallow /tag/*et%3D*
Disallow /tag/*d%3D*
Disallow /tag/*cl%3D*
Disallow /tag/*vh%3D*
Disallow /%40*/*
Disallow /community/*/*
Disallow /communities/*
Disallow /communities?t=*
Disallow /communities?q=*
Disallow /add?community=*

Other Records

Field Value
crawl-delay 3
Back to top

Other Records

Field Value
sitemap https://pikabu.ru/sitemap_index.xml
Back to top

Comments

  • _________
  • / \
  • / \
  • / \
  • | #### #### |
  • | / @ | / @ | | .---------------------------------.
  • | \__/ \__/ | ( YOU'RE DEFINITELY NOT A CRAWLER )
  • | | .---------------------------------.
  • \ ___ / _/
  • \ /
  • \________/
  • |
  • /|\
  • / | \
  • / | |
  • / | |
  • |
Back to top

Warnings

  • `clean-param` is not a known field.
Back to top