aniday.com
robots.txt
Robots Exclusion Standard data for aniday.com
Resource Scan
Scan Details
| Site Domain | aniday.com |
| Base Domain | aniday.com |
| Scan Status | Ok |
| Last Scan | 2025-07-20T13:58:03+00:00 |
| Next Scan | 2025-08-19T13:58:03+00:00 |
Last Scan
| Scanned | 2025-07-20T13:58:03+00:00 |
| URL | https://aniday.com/robots.txt |
| Domain IPs | 104.26.8.60, 104.26.9.60, 172.67.71.206, 2606:4700:20::681a:83c, 2606:4700:20::681a:93c, 2606:4700:20::ac43:47ce |
| Response IP | 172.67.71.206 |
| Found | Yes |
| Hash | 4a613666e425a7ba515108171b3684d342d0eca7c8a0e7c3e449fde4d0eb46bb |
| SimHash | a740a8d342e0 |
Groups
*
| Rule | Path | Comment |
|---|---|---|
| Disallow | /rss | - |
| Disallow | /chat | - |
| Disallow | /search | Prevent indexing of internal search results |
| Disallow | /admin | Prevent access to admin areas (if applicable) |
| Disallow | /tmp | Avoid temporary directories being crawled |
| Allow | / | - |
| Allow | /*.js$ | - |
| Allow | /*.css$ | - |
| Allow | /*.jpg$ | - |
| Allow | /*.png$ | - |
| Allow | /*.gif$ | - |
| Allow | /*.webp$ | - |
| Disallow | /*?sort= | - |
| Disallow | /*?filter= | - |
| Disallow | /*?page= | - |
| Disallow | /*?session= | - |
| Disallow | /*.php$ | - |
| Disallow | /*.cgi$ | - |
| Disallow | /*.asp$ | - |
| Disallow | /*.aspx$ | - |
| Allow | /structured-data/ | - |
| Allow | /open-graph/ | - |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 2 |
Other Records
| Field | Value |
|---|---|
| sitemap | https://aniday.com/sitemap_index.xml |
| sitemap | https://aniday.com/sitemap_vi_landingpages.xml |
| sitemap | https://aniday.com/sitemap_vi_blogs.xml |
| sitemap | https://aniday.com/sitemap_vi_jobs.xml |
| sitemap | https://aniday.com/sitemap_vi_employers.xml |
| sitemap | https://aniday.com/sitemap_en_landingpages.xml |
| sitemap | https://aniday.com/sitemap_en_jobs.xml |
| sitemap | https://aniday.com/sitemap_en_blogs.xml |
| sitemap | https://aniday.com/sitemap_en_headhunters.xml |
| sitemap | https://aniday.com/sitemap_en_employers.xml |
| sitemap | https://aniday.com/sitemap_id_landingpages.xml |
| sitemap | https://aniday.com/sitemap_id_blogs.xml |
| sitemap | https://aniday.com/sitemap_ja_landingpages.xml |
| sitemap | https://aniday.com/sitemap_ja_blogs.xml |
Comments