roofandfloor.thehindu.com
robots.txt

Robots Exclusion Standard data for roofandfloor.thehindu.com

Resource Scan

Scan Details

Site Domain roofandfloor.thehindu.com
Base Domain thehindu.com
Scan Status Ok
Last Scan2024-05-15T09:55:18+00:00
Next Scan 2024-06-14T09:55:18+00:00

Last Scan

Scanned2024-05-15T09:55:18+00:00
URL https://roofandfloor.thehindu.com/robots.txt
Domain IPs 13.228.143.96, 52.220.150.188
Response IP 52.220.150.188
Found Yes
Hash 4c06fec44dc40cf335d2e2ef5d84f5d08ed7eb6ab8e2277628c630e929bf770d
SimHash 0a55c7a045b1

Groups

adsbot-google

Rule Path
Allow /

alexabot

Rule Path
Allow /

applebot

Rule Path
Allow /

architextspider

Rule Path
Allow /

baiduspider

Rule Path
Allow /

bender

Rule Path
Allow /

bingbot

Rule Path
Allow /

birubot

Rule Path
Allow /

blogpulselive

Rule Path
Allow /

claritybot

Rule Path
Allow /

coccoc

Rule Path
Allow /

daumoa

Rule Path
Allow /

duckduckbot

Rule Path
Allow /

fast

Rule Path
Allow /

feedfetcher-google

Rule Path
Allow /

googlebot

Rule Path
Allow /

googleprober

Rule Path
Allow /

googleproducer

Rule Path
Allow /

googlebot-image

Rule Path
Allow /

googlebot-mobile

Rule Path
Allow /

googlebot-news

Rule Path
Allow /

googlebot-video

Rule Path
Allow /

gulliver

Rule Path
Allow /

ia_archiver

Rule Path
Allow /

iisbot

Rule Path
Allow /

jikespider

Rule Path
Allow /

lycos_spider_(t-rex)

Rule Path
Allow /

mail.ru_bot

Rule Path
Allow /

mediapartners-google

Rule Path
Allow /

msnbot

Rule Path
Allow /

msnbot-media

Rule Path
Allow /

msnptc

Rule Path
Allow /

naverbot

Rule Path
Allow /

nutch

Rule Path
Allow /

quickobot

Rule Path
Allow /

qwantify

Rule Path
Disallow /

rogerbot

Rule Path
Allow /

scooter

Rule Path
Allow /

seznambot

Rule Path
Allow /

semrushbot

Rule Path
Disallow /

slurp

Rule Path
Allow /

stackrambler

Rule Path
Allow /

surveybot

Rule Path
Allow /

teoma

Rule Path
Allow /

trovitbot

Rule Path
Disallow /

twitterbot/1.0

Rule Path
Allow /

vscooter

Rule Path
Allow /

yandex

Rule Path
Disallow /

yeti

Rule Path
Allow /

zyborg

Rule Path
Allow /

ahrefssiteaudit

Rule Path
Allow /

googlebot
googlebot-mobile

Rule Path
Disallow */404*
Disallow *?p=
Disallow *?bot=true
Disallow *?_

*

Rule Path
Disallow /api/
Disallow /v1/
Disallow /v2/
Disallow /geoip/
Disallow /null
Disallow /gihs
Disallow /mynest
Disallow /test
Disallow /Megapropertyshow
Disallow /megapropertyshow
Disallow /obj
Disallow /payments
Disallow /process.php
Disallow /sclite
Disallow /mclite
Disallow /?s=
Disallow /virtualfair
Disallow /lead-purchase
Disallow /undefined
Disallow /usersprofile
Disallow /collections
Disallow /trueview
Disallow /trueprice
Disallow /trueassist
Disallow /viewfound
Disallow /searchwidget
Disallow *?p=
Disallow *?bot=true
Disallow *?_
Disallow /microsite/
Disallow /raf/raf/real-estate-blog/wp-login.php

screaming frog seo spider

Rule Path
Disallow /

Other Records

Field Value
sitemap https://roofandfloor.thehindu.com/sitemap.xml
sitemap https://roofandfloor.thehindu.com/search.xml