Log In Sign Up

farlex.com
robots.txt

Robots Exclusion Standard data for farlex.com

Resource Scan

Scan Details

Site Domain farlex.com
Base Domain farlex.com
Scan Status Ok
Last Scan2025-03-12T17:33:38+00:00
Next Scan 2025-03-19T17:33:38+00:00

Last Scan

Scanned2025-03-12T17:33:38+00:00
URL https://farlex.com/robots.txt
Domain IPs 108.181.102.69
Response IP 108.181.102.69
Found Yes
Hash a2a071d406453b1b1f48336a66ef4960332754fbb6e90b522d570ea43cc90bae
SimHash 0a1144280da3

Groups

msiecrawler

Rule Path
Disallow /

npbot

Rule Path
Disallow /

msrbot

Rule Path
Disallow /

open*

Rule Path
Disallow /

twiceler

Rule Path
Disallow /

scooter*

Rule Path
Disallow /

websearch*

Rule Path
Disallow /

zyborg*

Rule Path
Disallow /

wget*

Rule Path
Disallow /

turnitinbot*

Rule Path
Disallow /

slysearch*

Rule Path
Disallow /

seekbot*

Rule Path
Disallow /

netresearchserver*

Rule Path
Disallow /

seekbot*

Rule Path
Disallow /

mozdex*

Rule Path
Disallow /

link*

Rule Path
Disallow /

larbin*

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

charlotte

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap http://www.farlex.com/sitemap.xml
Back to top