thecuriousjalebi.wordpress.com
robots.txt

Robots Exclusion Standard data for thecuriousjalebi.wordpress.com

Resource Scan

Scan Details

Site Domain thecuriousjalebi.wordpress.com
Base Domain wordpress.com
Scan Status Ok
Last Scan2024-05-14T10:19:41+00:00
Next Scan 2024-06-13T10:19:41+00:00

Last Scan

Scanned2024-05-14T10:19:41+00:00
URL https://thecuriousjalebi.wordpress.com/robots.txt
Redirect https://thecuriousjalebi.com/robots.txt
Redirect Domain thecuriousjalebi.com
Redirect Base thecuriousjalebi.com
Domain IPs 192.0.78.12, 192.0.78.13
Redirect IPs 192.0.78.179, 192.0.78.234
Response IP 192.0.78.179
Found Yes
Hash 0235271735868be326f2ab743952c4d24abfd757ce3f2c0938cbd8b4a8475059
SimHash 40008a004f93

Groups

scrapy

Rule Path
Allow /

*

Rule Path
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php

Other Records

Field Value
sitemap https://thecuriousjalebi.com/sitemap.xml
sitemap https://thecuriousjalebi.com/news-sitemap.xml
sitemap https://thecuriousjalebi.com/sitemap_index.xml