Log In Sign Up

agora.co.il
robots.txt

Robots Exclusion Standard data for agora.co.il

Resource Scan

Scan Details

Site Domain agora.co.il
Base Domain agora.co.il
Scan Status Ok
Last Scan2024-11-09T09:10:47+00:00
Next Scan 2024-11-16T09:10:47+00:00

Last Scan

Scanned2024-11-09T09:10:47+00:00
URL https://agora.co.il/robots.txt
Domain IPs 104.21.26.195, 172.67.168.127, 2606:4700:3033::6815:1ac3, 2606:4700:3036::ac43:a87f
Response IP 172.67.168.127
Found Yes
Hash ac6b2f051bcc1bf31e2fd648f230cb778b841f6357cd0a507d1a6d02872d5cb5
SimHash 95e18a32dd75

Groups

facebookexternalhit

Rule Path
Allow /*?src=

gptbot

Rule Path
Allow /texts/
Disallow /

mediapartners-google

Rule Path
Disallow /*toGet.asp*submitValue%3Ddone

*

Rule Path
Disallow /inc/
Disallow /manager/
Disallow /models/
Disallow /tests/
Disallow /texts/templates/
Disallow *texts/midrag.asp
Allow /manager/error404Handler.asp*
Disallow /*userActivation.asp
Disallow /*marketing/coopCounter.asp
Disallow /*models/linkCounter.asp
Disallow /*addAgent.asp
Disallow /*deleteUser.asp
Disallow /*infoAndNotifications.asp
Disallow /*kehilot.asp
Disallow /*myAgents.asp
Disallow /*myFriends.asp
Disallow /*myMessages.asp
Disallow /*myObjects.asp
Disallow /*personalDetails.asp
Disallow /*send2Friend.asp
Disallow /*supporters.asp
Disallow /*toGetCheck.asp
Disallow /*marketing/pop.asp
Disallow /*Experiment.asp
Disallow /*dealtype0
Disallow /*takeCi%26
Disallow /*takeCi0
Disallow /*takeCi*1
Disallow /*takeCity%3D%C3%AF%C2%BF%C2%BD%C3%AF%C2%BF%C2%BD
Disallow /*takeCity%3D%C3%AF%C2%BF%C2%BD%C3%AF%C2%BF%C2%BD%C3%AF%C2%BF%C2%BD
Disallow /*takeCity%3Dc
Disallow /*takeCity%3Dg
Disallow /*takeCity%3Dy
Disallow /showPhoto.asp?*src=ws
Disallow /*?*...
Disallow /*?*categorySearchdealType
Disallow /*?*%2C&
Disallow /*?*yamp%3B
Disallow /*?*subCategorydealType
Disallow /*?*middleCategorydealType
Disallow /*?subCategor&
Disallow /*?*..&
Disallow /*?*1iseek
Disallow /*?*2iseek
Disallow /*?*searchType=s&
Disallow /*?*searchT&
Disallow /*?*searcha&
Disallow /*?*searchA&
Disallow /*?sea&
Disallow /*?dealstatus
Disallow /*gclid
Disallow /*?2iseek
Disallow /*%28
Disallow /*%29
Disallow /*%26src%3D$
Disallow /*fireglass_rsn
Disallow /*subCategor%26
Disallow /*%3C
Disallow /*force_isolation
Disallow /*?*searchType=categorySearch*&category=
Disallow /*?*category=*&searchType=categorySearch
Disallow /*%26category%26
Disallow /*?category&
Disallow /*?*tags&
Disallow /*?*subcategory&
Disallow /*?*condition&
Disallow /*?*dealStatus&
Disallow /*?*dealType&
Disallow /*?*dealtype&
Disallow /*?*iseek&
Disallow /*?*searchType&
Disallow /*?*search&
Disallow /*?*category=&
Disallow /*?*tags=&
Disallow /*?*subcategory=&
Disallow /*?*condition=&
Disallow /*?*dealStatus=&
Disallow /*?*dealType=&
Disallow /*?*dealtype=&
Disallow /*?*iseek=&
Disallow /*?*searchType=&
Disallow /*?*gnt=1
Disallow /*?*token=
Disallow /*%3D%20%26
Disallow /*page%3D*0*0*0*0%26
Disallow /*page%3D*5*5*5*5%26
Disallow /*page%3D*0*0*0*0$
Disallow /*page%3D*5*5*5*5$
Disallow *format%3Datom
Disallow *src%3Dfeed-atom
Disallow /*searchType%3DagentSearch
Disallow /*searchType%3DdigestSearch
Disallow /*searchType%3DuserSearch
Disallow /*src%3DdailyDigest
Disallow /*opened%3D1
Disallow /*device%3D
Disallow /*%26t%3D
Disallow /*?t=
Disallow /*utm_expid%3D
Disallow /*?*submitValue
Disallow /*?*searchType=searchById
Disallow /*showPhoto*.asp?dir=
Disallow /*?*&src=agent
Disallow /*?*&src=ws
Disallow /*?src=
Disallow /*toGet*?*condition
Disallow /*?*&ref=
Disallow /*?*groupByRegDate
Disallow /*?*editcss=1
Disallow /*?*noRedirect
Disallow /*?*iframe=1
Disallow /*categoryContent*.asp?*&dealType=1
Disallow /*?*picOnly
Disallow /*?*orderByRegDate
Disallow /*category%3Da
Disallow /*category%3Db
Disallow /*category%3Dc
Disallow /*category%3Dd
Disallow /*category%3De
Disallow /*category%3Df
Disallow /*category%3Dg
Disallow /*category%3Dh
Disallow /*category%3Di
Disallow /*category%3Dj
Disallow /*category%3Dk
Disallow /*category%3Dl
Disallow /*category%3Dm
Disallow /*category%3Dn
Disallow /*category%3Do
Disallow /*category%3Dp
Disallow /*category%3Dq
Disallow /*category%3Dr
Disallow /*category%3Ds
Disallow /*category%3Dt
Disallow /*category%3Du
Disallow /*category%3Dv
Disallow /*category%3Dw
Disallow /*category%3Dx
Disallow /*category%3Dy
Disallow /*category%3Dz
Disallow /*dealType%3Da
Disallow /*dealType%3Db
Disallow /*dealType%3Dc
Disallow /*dealType%3Dd
Disallow /*dealType%3De
Disallow /*dealType%3Df
Disallow /*dealType%3Dg
Disallow /*dealType%3Dh
Disallow /*dealType%3Di
Disallow /*dealType%3Dj
Disallow /*dealType%3Dk
Disallow /*dealType%3Dl
Disallow /*dealType%3Dm
Disallow /*dealType%3Dn
Disallow /*dealType%3Do
Disallow /*dealType%3Dp
Disallow /*dealType%3Dq
Disallow /*dealType%3Dr
Disallow /*dealType%3Ds
Disallow /*dealType%3Dt
Disallow /*dealType%3Du
Disallow /*dealType%3Dv
Disallow /*dealType%3Dw
Disallow /*dealType%3Dx
Disallow /*dealType%3Dy
Disallow /*dealType%3Dz
Disallow /*dealStatus%3Da
Disallow /*dealStatus%3Db
Disallow /*dealStatus%3Dc
Disallow /*dealStatus%3Dd
Disallow /*dealStatus%3De
Disallow /*dealStatus%3Df
Disallow /*dealStatus%3Dg
Disallow /*dealStatus%3Dh
Disallow /*dealStatus%3Di
Disallow /*dealStatus%3Dj
Disallow /*dealStatus%3Dk
Disallow /*dealStatus%3Dl
Disallow /*dealStatus%3Dm
Disallow /*dealStatus%3Dn
Disallow /*dealStatus%3Do
Disallow /*dealStatus%3Dp
Disallow /*dealStatus%3Dq
Disallow /*dealStatus%3Dr
Disallow /*dealStatus%3Ds
Disallow /*dealStatus%3Dt
Disallow /*dealStatus%3Du
Disallow /*dealStatus%3Dv
Disallow /*dealStatus%3Dw
Disallow /*dealStatus%3Dx
Disallow /*dealStatus%3Dy
Disallow /*dealStatus%3Dz
Back to top

Comments

  • After a Facebook update we had to explicitly allow this rule
  • Don't crawl these directories
  • Don't crawl functional pages
  • Don't crawl logged in user pages
  • dont crawl experiment pages
  • Don't crawl invented pages
  • Don't crawl legit parameters without values (applies also for empty values with =, like category=&)
  • Don't crawl legit parameters without values (seems that googlebot needs the =, like category=&)
  • Don't crawl photos pages of listings generated in giventake
  • Don't crawl pages with tokens
  • dont crawl white space as parameter value (does not work in google bot tester)
  • Block pages with very long numbers like page=62263962746438423892578228988697196068899993695538434585999356338603798545933526103973336853104574584667747101022784584828109577624427598
  • Don't crawl rss pages and links - google does not index them and they consume huge crawl budget
  • Don't crawl pages with user filtering
  • Dont crawl pages with non content changing parameters
  • Dont crawl content with explicit default parameters
  • dont crawl pages with negligable content filtering params
  • Dont crawl pages with invented content (2)
Back to top

Warnings

  • 1 invalid line.
Back to top