smartoptions.io
robots.txt

Robots Exclusion Standard data for smartoptions.io

Resource Scan

Scan Details

Site Domain smartoptions.io
Base Domain smartoptions.io
Scan Status Ok
Last Scan2024-05-29T20:17:24+00:00
Next Scan 2024-06-28T20:17:24+00:00

Last Scan

Scanned2024-05-29T20:17:24+00:00
URL https://smartoptions.io/robots.txt
Domain IPs 172.66.40.85, 172.66.43.171, 2606:4700:3108::ac42:2855, 2606:4700:3108::ac42:2bab
Response IP 172.66.43.171
Found Yes
Hash ceef1eac1ced174499bd81fe7ab485089b3b969adff88f0c42e7c81ad774ff0a
SimHash 6a201293f5b5

Groups

*

Rule Path
Disallow

googlebot
googlebot-image
googlebot-mobile
twitterbot
semrushbot
semrushbot-ct
bingbot
slurp
msnbot
yandex

Rule Path Comment
Allow /wp-admin/admin-ajax.php -
Disallow /wp-admin/ block access to admin section
Disallow /wp-login.php block access to admin section
Disallow /search/ block access to internal search result pages
Disallow *?s=* block access to internal search result pages
Disallow *?p=* block access to pages for which permalinks fails
Disallow *%26p%3D* block access to pages for which permalinks fails
Disallow *%26preview%3D* block access to preview pages
Disallow /tag/ block access to tag pages
Disallow /404-error/ block access to 404 page

Other Records

Field Value
crawl-delay 60

Other Records

Field Value
sitemap https://smartoptions.io/sitemap_index.xml