Log In Sign Up

teraspendopo.com
robots.txt

Robots Exclusion Standard data for teraspendopo.com

Resource Scan

Scan Details

Site Domain teraspendopo.com
Base Domain teraspendopo.com
Scan Status Ok
Last Scan2024-10-03T09:46:59+00:00
Next Scan 2024-10-10T09:46:59+00:00

Last Scan

Scanned2024-10-03T09:46:59+00:00
URL https://teraspendopo.com/robots.txt
Domain IPs 194.163.41.33
Response IP 194.163.41.33
Found Yes
Hash c73b891870bc95cb1aac363da7fff02ca01172b9464c694fb2ddfca64ec9abf5
SimHash 2a6cd983ecf0

Groups

*

Rule Path
Allow /
Disallow /author/
Disallow /page/
Disallow /tag/
Disallow /category/
Disallow /alur-cerita/
Disallow /berita-terkini/
Disallow /koran-hiburan/
Disallow /resensi-cerita/
Disallow /sinopsis-spoile/
Disallow /?p=
Disallow /amp/
Disallow /search/
Disallow /wp-login.php
Disallow /license.txt
Disallow /?s
Disallow /*/page/
Disallow /*?page=
Disallow /feed/
Disallow /comments/feed/
Disallow /*?*
Disallow /wp-admin/
Disallow /wp-includes/
Disallow /xmlrpc.php
Disallow /readme.html
Allow /wp-content/uploads/

dotbot

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

httrack

Rule Path
Disallow /

openai

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

exabot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /
Allow /ads.txt

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10
Back to top

Other Records

Field Value
sitemap https://teraspendopo.com/sitemap_index.xml
Back to top

Comments

  • Izinkan semua bot mengakses konten, kecuali yang diblokir
  • Disallow non-article pages to focus crawling on articles
  • Block paginated content and feeds to save crawl budget
  • Block parameterized URLs (query strings)
  • Block admin, login, and internal files
  • Izinkan file media diunggah
  • Block bad bots
  • Block AI and other unwanted bots
  • Block other data scraping bots
  • Allow ads.txt for advertising verification
  • Sitemap location for SEO optimization
  • Crawl-delay for Bingbot
Back to top