paulnicholson.com
robots.txt

Robots Exclusion Standard data for paulnicholson.com

Resource Scan

Scan Details

Site Domain paulnicholson.com
Base Domain paulnicholson.com
Scan Status Ok
Last Scan2025-04-19T22:46:13+00:00
Next Scan 2025-04-26T22:46:13+00:00

Last Scan

Scanned2025-04-19T22:46:13+00:00
URL https://paulnicholson.com/robots.txt
Redirect https://pnuk.com/robots.txt
Redirect Domain pnuk.com
Redirect Base pnuk.com
Domain IPs 104.21.30.30, 172.67.150.115, 2606:4700:3033::6815:1e1e, 2606:4700:3036::ac43:9673
Redirect IPs 104.21.20.238, 172.67.194.221, 2606:4700:3034::6815:14ee, 2606:4700:3036::ac43:c2dd
Response IP 172.67.194.221
Found Yes
Hash 1bbe19f061c0692fe0d3edaad707cc2a5dbb61833b5deb44bcdf4dbdbc1c368c
SimHash 730c82a2f335

Groups

*

Rule Path
Disallow /private/
Allow /public/

psbot

Rule Path
Disallow /

cfnetwork

Rule Path
Disallow /

microsoft url control

Rule Path
Disallow /

java

Rule Path
Disallow /

httrack off-line browser

Rule Path
Disallow /

mbcrawler

Rule Path
Disallow /

yandex

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

ahrefsbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

linkedinbot

Rule Path
Disallow /infiniterss*

mediapartners-google*

Rule Path
Disallow