pornhub.com
robots.txt

Robots Exclusion Standard data for pornhub.com

Resource Scan

Scan Details

Site Domain pornhub.com
Base Domain pornhub.com
Scan Status Ok
Last Scan2024-10-05T02:31:04+00:00
Next Scan 2024-11-04T02:31:04+00:00

Last Scan

Scanned2024-10-05T02:31:04+00:00
URL https://pornhub.com/robots.txt
Redirect https://www.pornhub.com/robots.txt
Redirect Domain www.pornhub.com
Redirect Base pornhub.com
Domain IPs 66.254.114.41
Redirect IPs 66.254.114.41
Response IP 66.254.114.41
Found Yes
Hash cf611b823d055890a6042b97504f1bd44d2a63b83d73708a0c7792680b17416c
SimHash d10058238e90

Groups

*

Rule Path
Disallow /album/
Disallow /albums*o%3D*
Disallow /albums*page%3D*
Disallow /albums/*-*
Disallow /albums/*/*
Disallow /albums/gay?search=*
Disallow /albums/private*
Disallow /live*
Disallow /live*c%3D*
Disallow /live*f%3D*
Disallow /live*l%3D*
Disallow /live*n%3D*
Disallow /live*page%3D*
Disallow /live*r%3D*
Disallow /live*streamate%3D*
Disallow /live/*
Disallow /language*?p=*
Disallow /language*%26p%3D*
Disallow /language/*?*hd=1*
Disallow /language/*?*p=*
Disallow /language/*?*promo=*
Disallow /model/*/*
Disallow /model/*/live*
Disallow /model/*/videos/paid
Allow /model/*/videos*
Allow /model/*/photos
Disallow /pornstar/*/*achievements*
Disallow /pornstar/*/*photo*%3D*
Disallow /pornstar/*/*subscription*
Disallow /pornstar/*/costars*
Disallow /pornstar/*/friends*
Disallow /pornstar/*/gifs*
Disallow /pornstar/*/live*
Disallow /pornstar/*/playlists*
Disallow /pornstar/*/similar*
Disallow /pornstar/*/stream*
Disallow /pornstar/*/subscribers*
Disallow /pornstar/*/videos/paid
Disallow */pornstars*%3D*performerType%3D*
Disallow */pornstars*l%3D*
Disallow */pornstars*t%3D*
Disallow */pornstars/search*
Disallow /pornstars*?age=*=*
Disallow /pornstars*%26age%3D*
Disallow /pornstars*breasttype%3D*%3D*
Disallow /pornstars*cup%3D*%3D*
Disallow /pornstars*ethnicity%3D*%3D*
Disallow /pornstars*gender%3D*%3D*
Disallow /pornstars*hair%3D*%3D*
Disallow /pornstars*piercings%3D*%3D*
Disallow /pornstars*tattoos%3D*%3D*
Disallow */videos/fanonly*
Disallow */videos/private*
Disallow *search%3D*%2B*%2B*%2B*%2B*page%3D*
Disallow *search%3D*p%3D*
Disallow *search%3D*page%3D*
Disallow *search%3D*p%3Dhomemade*o%3Dmr$
Disallow *search%3D*p%3Dprofessional*o%3Dmr$
Disallow /*search*filter_category%3D*
Disallow /*search*filter_duration%3D*
Disallow /*search*o%3D*
Disallow /*gifs/search*o%3D
Disallow /*o%3D*search*
Disallow /channels/search*
Disallow /gay/gifs/search
Allow *video*search%3D*o%3Dmr$
Allow *video*search%3D*%26page%3D2$
Allow *video*search%3D*%26page%3D3$
Allow *video*search%3D*%26page%3D4$
Allow *video*search%3D*%26page%3D5$
Allow *video*search%3D*%26page%3D6$
Allow *video*search%3D*%26page%3D7$
Allow *video*search%3D*%26page%3D8$
Allow *video*search%3D*%26page%3D9$
Allow *video*search%3D*%26page%3D10$
Disallow *video*search%3D*o%3Dmr%26page%3D2$
Disallow *video*search%3D*o%3Dmr%26page%3D3$
Disallow *video*search%3D*o%3Dmr%26page%3D4$
Disallow *video*search%3D*o%3Dmr%26page%3D5$
Disallow *video*search%3D*o%3Dmr%26page%3D6$
Disallow *video*search%3D*o%3Dmr%26page%3D7$
Disallow *video*search%3D*o%3Dmr%26page%3D8$
Disallow *video*search%3D*o%3Dmr%26page%3D9$
Disallow *video*search%3D*o%3Dmr%26page%3D10$
Disallow *search%3D*%2B*%2B*%2B*%2B*page%3D2$
Disallow *search%3D*%2B*%2B*%2B*%2B*page%3D3$
Disallow *search%3D*%2B*%2B*%2B*%2B*page%3D4$
Disallow *search%3D*%2B*%2B*%2B*%2B*page%3D5$
Disallow *search%3D*%2B*%2B*%2B*%2B*page%3D6$
Disallow *search%3D*%2B*%2B*%2B*%2B*page%3D7$
Disallow *search%3D*%2B*%2B*%2B*%2B*page%3D8$
Disallow *search%3D*%2B*%2B*%2B*%2B*page%3D9$
Disallow *search%3D*%2B*%2B*%2B*%2B*page%3D10$
Disallow */video*id%3D*
Disallow /video/ajax_search_related*
Disallow /video/ajax_video_related*
Disallow /video/manage*
Disallow /video/player_related_datas*
Disallow /video/pornstar_suggestion_list*
Disallow /video/privates*
Disallow /video/propose_pornstar*
Disallow /video/rate*
Disallow /video/search_autocomplete*
Disallow /video/search*p%3D*o%3D*
Disallow /shorties/
Disallow /shorties?cat=*
Disallow /gay/shorties/
Disallow /trans/shorties/
Disallow /lesbian/shorties/
Disallow /gay/shorties?cat=*
Disallow /lesbian/shorties?cat=*
Disallow /trans/shorties?cat=*
Disallow */incategories/*
Disallow /svvt*
Disallow /user/
Disallow /users/
Disallow /front/
Disallow /embedgif/
Disallow /gay/user/
Disallow /gifgenerator?viewkey=*
Disallow /interstitial*
Disallow /photo/
Disallow /phubapi*
Disallow /players/
Disallow /playlist/rate*
Disallow /chat/index*
Disallow /uploading/gifgenerator*
Disallow /watch_stream/*
Disallow /clip/
Disallow /clips?*
Disallow /*max_duration%3D*
Disallow */gif/view*
Disallow */oauth2*
Disallow *ajax%3D1*
Disallow *hd%3D01*
Disallow *join%3D*
Disallow *premium%3D1*
Disallow *promo%3D*
Disallow *token%3D*
Disallow /_xa/ads*
Disallow /*/*/photos/fan*
Disallow /*/*/photos/private
Disallow /*cc%3D*
Disallow /*correction%3D*
Disallow /*exclude_category%3D*
Disallow /*hd%3D1*
Disallow /*login%3D*
Disallow /*min_duration%3D*
Disallow /*orientation%3D*
Disallow /*page*page*
Disallow /*pkey%3D*
Disallow /*player.php
Disallow /*redirect%3D*
Disallow /*showDuration%3D*
Disallow /*token%3D*
Disallow /wall_player_v2.php
Disallow /_i?type=*
Disallow /shemale/gifs/search
Disallow /shemale/gifs/search
Disallow /transgender/gifs/search
Disallow /*7urren*
Disallow /*embed_player*
Disallow /*redirectTo%3D*
Disallow /*redirectto%3D*
Disallow /*video_related*
Disallow /*watch_player*
Disallow /chronopop/*
Disallow /iframe.php*
Disallow /rlock/
Disallow /select/video/*
Disallow /webmasters*
Disallow /xmoov_flv/player/index.php
Disallow */fanclub*
Disallow /*player_v1.6b.swf
Disallow */iframe-mobile*
Disallow *tubes_cms*

twitterbot

Rule Path
Disallow

amazonbot
anthropic-ai
applebot-extended
awariorssbot
awariosmartbot
bytespider
ccbot
chatgpt-user
claudebot
claude-web
cohere-ai
dataforseobot
facebookbot
gptbot
imagesiftbot
magpie-crawler
omgili
omgilibot
peer39_crawler
peer39_crawler/1.0
perplexitybot
youbot

Rule Path
Disallow /
Allow /$
Allow /insights/
Allow /blog/

Other Records

Field Value
sitemap https://www.pornhub.com/sitemaps.xml

Comments

  • Albums
  • Live section - near fully deprecated
  • Porn in Your Language /language
  • Model section
  • Pornstar main page
  • Pornstars individual pages
  • Search
  • Video directory
  • Major Site Area Blocks
  • specific variable and file blocks
  • DEPRECATED terminology search for deletion
  • DEPRECATED variable and directory lines for deletion