prokerala.com
robots.txt

Robots Exclusion Standard data for prokerala.com

Resource Scan

Scan Details

Site Domain prokerala.com
Base Domain prokerala.com
Scan Status Ok
Last Scan2024-05-07T19:03:20+00:00
Next Scan 2024-05-14T19:03:20+00:00

Last Scan

Scanned2024-05-07T19:03:20+00:00
URL https://prokerala.com/robots.txt
Redirect https://www.prokerala.com/robots.txt
Redirect Domain www.prokerala.com
Redirect Base prokerala.com
Domain IPs 104.26.14.242, 104.26.15.242, 172.67.69.237, 2606:4700:20::681a:ef2, 2606:4700:20::681a:ff2, 2606:4700:20::ac43:45ed
Redirect IPs 23.52.171.201, 23.59.168.168
Response IP 23.52.171.201
Found Yes
Hash 2edf04c67097c0ed170268ea6d94e19c0acc6740679ed8d37e70057de8d124b2
SimHash cc3ef960a43b

Groups

googlebot-news

Rule Path
Allow /news/
Allow /astrology/
Allow /festivals/
Allow /$
Allow /?amp=1$
Disallow /

mediapartners-google

Rule Path
Allow /
Disallow

adsbot-google

Rule Path
Disallow

proximic

Rule Path
Allow /
Disallow

ahrefsbot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

baiduspider-video

Rule Path
Disallow /

baiduspider-image

Rule Path
Disallow /

discoverybot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

ichiro

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

moget

Rule Path
Disallow /

naverbot

Rule Path
Disallow /

psbot

Rule Path
Disallow /

paperlibot

Rule Path
Disallow /

piplbot

Rule Path
Disallow /

scooperbot

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

sosospider

Rule Path
Disallow /

spinn3r

Rule Path
Disallow /
Allow /news/

yandex

Rule Path
Disallow /

yandeximages

Rule Path
Disallow /

yeti

Rule Path
Disallow /

youdaobot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

*

Rule Path
Allow /
Allow /res/images/
Disallow /account/
Disallow /api/
Disallow /astrology/location-search.php
Disallow /astrology/*panchang*/*.pdf$
Disallow /calendar/*.pdf$
Disallow /automobile/image-update.php
Disallow /automobile/news/
Disallow /automobile/search_details.php
Disallow /banking/ajax-search.php
Disallow /downloads/downloadfile.php
Disallow /education/imgs/email/
Disallow /general/calendar/date.php
Disallow /general/calendar/mal-date.php
Disallow /general/calendar/pdf/
Disallow /general/recipes/recipe.php
Disallow /how-to/call/loc_search.php
Disallow /movies/articles/
Disallow /res/
Disallow /redirect/
Disallow /search/
Disallow /share/
Disallow /travel/airports/search.php
Disallow /travel/indian-railway/search.php
Disallow /travel/hotels/ajax_search.php
Disallow /travel/hotels/imgs/
Disallow /travel/airlines/destination_search.php
Disallow /travel/airlines/airline_search.php
Disallow /travel/search.php
Disallow /what-is/meaning-of/notfound/
Disallow /*/cron/

Other Records

Field Value
sitemap https://www.prokerala.com/banking/sitemap-index.xml.gz
sitemap https://www.prokerala.com/bikes/sitemap-index.xml.gz
sitemap https://www.prokerala.com/news/photos/sitemap-index.xml.gz
sitemap https://www.prokerala.com/pincode/sitemap-index.xml.gz
sitemap https://www.prokerala.com/sitemap/daily.xml.gz
sitemap https://www.prokerala.com/videos/sitemap-index.xml.gz

Comments

  • AI Crawlers

Warnings

  • 2 invalid lines.