Log In Sign Up

dsp.espreso.tv
robots.txt

Robots Exclusion Standard data for dsp.espreso.tv

Resource Scan

Scan Details

Site Domain dsp.espreso.tv
Base Domain espreso.tv
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-03-26T15:49:20+00:00
Next Scan 2024-06-24T15:49:20+00:00

Last Successful Scan

Scanned2023-05-04T00:55:29+00:00
URL https://dsp.espreso.tv/robots.txt
Redirect https://espreso.tv/robots.txt
Redirect Domain espreso.tv
Redirect Base espreso.tv
Domain IPs 104.18.10.148, 104.18.11.148, 2606:4700::6812:a94, 2606:4700::6812:b94
Redirect IPs 104.18.10.148, 104.18.11.148, 2606:4700::6812:a94, 2606:4700::6812:b94
Response IP 104.18.10.148
Found Yes
Hash 72cd2dec58e6985bc0dc3b04077e37b6498d104ece40bed735b74c4fb39eaa43
SimHash 28190f1241e9

Groups

*

Rule Path
Allow /
Disallow /*?q=
Disallow /search-results*
Disallow /streamonline*
Disallow /*?*
Disallow /*?
Allow /*?page*
Allow /*?amp$
Allow /uploads
Allow /uploads/photobank/
Allow /uploads/*.png
Allow /uploads/*.jpg
Allow /uploads/*.jpeg
Allow /uploads/*.gif
Allow /uploads/*.svg
Allow /uploads/*.pdf

googlebot

Rule Path
Disallow /*.php$
Disallow /*.inc$
Disallow /*.gz$

googlebot-image

Rule Path
Allow /*
Back to top

Other Records

Field Value
sitemap https://espreso.tv/sitemap.xml
Back to top

Comments

  • Disalow search :
  • Disallow indexation of URLs having duplicate content parameters
  • Allow to index images
  • Allow images in plugins, cache (check your path!!)
  • Disallow indexation of sensitive files (check yours)
  • Allow Google Image Bot
  • Show to spiders sitemap (create a sitemap with fresh news only - last week, and place on this link)
Back to top