firstpage.nz
robots.txt

Robots Exclusion Standard data for firstpage.nz

Resource Scan

Scan Details

Site Domain firstpage.nz
Base Domain firstpage.nz
Scan Status Ok
Last Scan2024-10-25T16:17:08+00:00
Next Scan 2024-11-24T16:17:08+00:00

Last Scan

Scanned2024-10-25T16:17:08+00:00
URL https://firstpage.nz/robots.txt
Domain IPs 104.21.15.2, 172.67.160.218, 2606:4700:3032::6815:f02, 2606:4700:3036::ac43:a0da
Response IP 172.67.160.218
Found Yes
Hash 2b16cfb7cf63222104fe8ef7700e5bc5891ad87b5b1950ed2b0f943913f6f9a0
SimHash ff0bc2426d31

Groups

*

Rule Path
Allow /wp-content/uploads/
Allow /wp-admin/admin-ajax.php
Disallow /wp-admin/
Disallow /wp-content/plugins/
Disallow /readme.html
Disallow /refer/
Disallow /cdn-cgi/
Disallow /*?utm_term=*
Disallow /*?__hstc=*
Disallow /embed
Disallow /?s

dotbot/1.1

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

teleport

Rule Path
Disallow /

teleportpro

Rule Path
Disallow /

emailcollector

Rule Path
Disallow /

emailsiphon

Rule Path
Disallow /

webbandit

Rule Path
Disallow /

webzip

Rule Path
Disallow /

webreaper

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

web downloader

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

offline explorer pro

Rule Path
Disallow /

httrack website copier

Rule Path
Disallow /

offline commander

Rule Path
Disallow /

leech

Rule Path
Disallow /

websnake

Rule Path
Disallow /

blackwidow

Rule Path
Disallow /

http weazel

Rule Path
Disallow /

petalbot

Rule Path
Disallow /