ao3-cn.top
robots.txt

Robots Exclusion Standard data for ao3-cn.top

Resource Scan

Scan Details

Site Domain ao3-cn.top
Base Domain ao3-cn.top
Scan Status Ok
Last Scan2025-09-27T14:00:26+00:00
Next Scan 2025-10-04T14:00:26+00:00

Last Scan

Scanned2025-09-27T14:00:26+00:00
URL https://ao3-cn.top/robots.txt
Domain IPs 212.192.221.136
Response IP 205.198.87.192
Found Yes
Hash 1493972bb388474f4fae4950aca406bd71dd10c17fe4677974c1d8ce989201d3
SimHash ca120d85d1e7

Groups

*

Rule Path Comment
Disallow /works? cruel but efficient
Disallow /autocomplete/ -
Disallow /downloads/ -
Disallow /external_works/ -
Disallow /bookmarks/search? -
Disallow /people/search? -
Disallow /tags/search? -
Disallow /works/search? -

googlebot

Rule Path
Disallow /autocomplete/
Disallow /downloads/
Disallow /external_works/
Disallow /works/*?
Disallow /*search?
Disallow /*?*query=
Disallow /*?*sort_
Disallow /*?*selected_tags
Disallow /*?*view_adult
Disallow /*?*tag_id
Disallow /*?*pseud_id
Disallow /*?*user_id
Disallow /*?*pseud=

ccbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

Comments

  • See https://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file
  • disallow indexing of search results
  • Googlebot is smart and knows pattern matching