netwerkmediawijsheid.nl
robots.txt

Robots Exclusion Standard data for netwerkmediawijsheid.nl

Resource Scan

Scan Details

Site Domain netwerkmediawijsheid.nl
Base Domain netwerkmediawijsheid.nl
Scan Status Ok
Last Scan2024-08-31T22:17:24+00:00
Next Scan 2024-09-30T22:17:24+00:00

Last Scan

Scanned2024-08-31T22:17:24+00:00
URL https://netwerkmediawijsheid.nl/robots.txt
Domain IPs 2600:9000:2024:3400:1c:71db:1e00:93a1, 2600:9000:2024:3e00:1c:71db:1e00:93a1, 2600:9000:2024:5000:1c:71db:1e00:93a1, 2600:9000:2024:7e00:1c:71db:1e00:93a1, 2600:9000:2024:800:1c:71db:1e00:93a1, 2600:9000:2024:9600:1c:71db:1e00:93a1, 2600:9000:2024:d600:1c:71db:1e00:93a1, 2600:9000:2024:e400:1c:71db:1e00:93a1, 65.9.112.7, 65.9.112.78, 65.9.112.79, 65.9.112.85
Response IP 52.85.49.52
Found Yes
Hash 79b5fc442fbce5a18a458d49007323a5f4c59b14237e1d012dee58edef96d4c2
SimHash cd081c762793

Groups

scrapy

Rule Path
Allow /

*

Rule Path
Disallow /*?s=
Disallow /*?frm
Disallow wp-content/plugins
Disallow wp-content/themes
Disallow wp-content/languages
Disallow wp-content/cache
Disallow wp-content/maintenance
Disallow wp-content/upgrade
Disallow wp-content/uploads/backwpup-*
Disallow wp-content/uploads/complianz
Disallow wp-content/uploads/formidable
Disallow wp-content/uploads/ithemes-security
Disallow wp-content/uploads/revslider
Disallow wp-content/uploads/sass
Disallow wp-content/uploads/w*
Disallow wp-includes
Disallow wp-admin
Disallow wp-logs

Other Records

Field Value
crawl-delay 10

Other Records

Field Value
sitemap /sitemap_index.xml