Log In Sign Up

billboardarabia.com
robots.txt

Robots Exclusion Standard data for billboardarabia.com

Resource Scan

Scan Details

Site Domain billboardarabia.com
Base Domain billboardarabia.com
Scan Status Ok
Last Scan2024-10-30T19:54:51+00:00
Next Scan 2024-11-06T19:54:51+00:00

Last Scan

Scanned2024-10-30T19:54:51+00:00
URL https://billboardarabia.com/robots.txt
Redirect https://www.billboardarabia.com/robots.txt
Redirect Domain www.billboardarabia.com
Redirect Base billboardarabia.com
Domain IPs 104.26.6.174, 104.26.7.174, 172.67.73.253, 2606:4700:20::681a:6ae, 2606:4700:20::681a:7ae, 2606:4700:20::ac43:49fd
Redirect IPs 13.227.254.43, 13.227.254.5, 13.227.254.8, 13.227.254.89, 2600:9000:200a:2c00:1d:9cdf:6a40:93a1, 2600:9000:200a:4000:1d:9cdf:6a40:93a1, 2600:9000:200a:4800:1d:9cdf:6a40:93a1, 2600:9000:200a:6400:1d:9cdf:6a40:93a1, 2600:9000:200a:7000:1d:9cdf:6a40:93a1, 2600:9000:200a:b600:1d:9cdf:6a40:93a1, 2600:9000:200a:e800:1d:9cdf:6a40:93a1, 2600:9000:200a:ee00:1d:9cdf:6a40:93a1
Response IP 13.227.254.5
Found Yes
Hash 129716e0de354f6fa97353d921fc07517ae961e4fcd5b2709db5736736953535
SimHash 21a56d70e595

Groups

*

Rule Path
Disallow
Disallow /api/
Disallow /_next/
Disallow /static/
Disallow /images/
Disallow /admin/
Disallow /login/
Disallow /profile/
Back to top

Comments

  • Allow all bots to crawl everything
  • Block crawlers from accessing Next.js internal API routes
  • Block crawlers from accessing dynamic/static assets (optional)
  • Block sensitive pages or paths (optional)
  • Specify the sitemap location (update with the actual sitemap URL)
  • Sitemap: https://billboardarabia.com/sitemap_0.xml
Back to top