Log In Sign Up

vling.net
robots.txt

Robots Exclusion Standard data for vling.net

Resource Scan

Scan Details

Site Domain vling.net
Base Domain vling.net
Scan Status Ok
Last Scan2024-06-12T13:21:59+00:00
Next Scan 2024-06-19T13:21:59+00:00

Last Scan

Scanned2024-06-12T13:21:59+00:00
URL https://vling.net/robots.txt
Domain IPs 13.35.18.104, 13.35.18.119, 13.35.18.3, 13.35.18.49
Response IP 13.35.18.104
Found Yes
Hash ffd5a42f5620ad5ea33a5172eb149107e4a8566648918a8ba89c2ada78d074a6
SimHash 4a2909385ed0

Groups

googlebot

Rule Path
Disallow /video
Disallow /*/video
Disallow /todayvideo/detail
Disallow /*/todayvideo/detail
Disallow /setting
Disallow /*/setting
Disallow /enterprise
Disallow /*/enterprise
Disallow /verifySuccess
Disallow /*/verifySuccess

amazonbot

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

grapeshot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

*

Rule Path
Allow /
Disallow /*.json$
Disallow /*_buildManifest.js$
Disallow /*_middlewareManifest.js$
Disallow /*_ssgManifest.js$
Disallow /*.js$
Back to top

Other Records

Field Value
sitemap https://vling.net/sitemap.xml
Back to top

Comments

  • generated by 2023-08-02
  • https://www.robotstxt.org/robotstxt.html
  • Next.JS Crawl Budget Performance Updates
  • Block files ending in .json, _buildManifest.js, _middlewareManifest.js, _ssgManifest.js, and any other JS files
  • The asterisks allows any file name
  • The dollar sign ensures it only matches the end of an URL and not a oddly formatted url (e.g. /locations.json.html)
Back to top