java - How to Diagnose ElasticSearch Search Queue Growth -

-

i'm trying diagnose issue our elasticsearch search queue seemingly randomly fills up.

the behavior observe in our monitor on 1 node of our cluster search queue growth (just one) , after search thread pool used start getting timeouts of course. there seems 1 query blocking while thing. way resolve problem @ moment restart node.

you can see below relevant behavior in charts: first queue size, pending cluster tasks (to show no other operations blocking or queing up, e.g. index operations or so) , active threads search thread pool. spike @ 11 o'clock restart of node.

enter image description here

the log files on nodes show no entries during hour before or after issue until restarted node. garbage collection events of around 200 -600ms , 1 on relevant node around 20 minutes before event.

my questions: - how can debug there no information logged anywhere on failing or timing out query? - possible reasons this? don't have dynamic queries or similar - can set query timeout or clear / reset active searches when happens prevent node restart?

some more details don't apply, based on questions far:

  • exactly same hardware (16 cores, 60gb mem)
  • same config, no special nodes
  • no swap enabled
  • nothing noticeable on other metrics io or cpu
  • not master node
  • no special shards, 3 shards per node each node, pertty standard queries, queries getting send es 10 minutes before queries typically finish within 5-10ms, ones timeout on same, no increase in query rate or else
  • we have 5 nodes deployment, accessed round robin
  • we have slow log of 2 seconds on info level, no entries

the hot threads after 1 minute of queue build @ https://gist.github.com/elm-/5ed398054ea6b46522c0, several snapshots of dumps on few moments.

andrei stefan's answer isn't wrong i'd start looking @ hot_threads clogged node rather trying figure out might special node.

i don't know of way inside queue. slowlogs, andrei says, great idea though.


Comments

Post a Comment

Popular posts from this blog

java - Run spring boot application error: Cannot instantiate interface org.springframework.context.ApplicationListener -

-
i have spring project , try make use spring boot , run @ embadded tomcat following : https://spring.io/guides/gs/rest-service/ this application //@configuration //@enableaspectjautoproxy @springbootapplication @componentscan(basepackages = "gux.prome") public class application { public static void main(string[] args) { springapplication.run(application.class, args); } } if use maven command: mvn spring-boot:run , project starts fine, need debug , run main method @ intelij, exception occurs: exception in thread "main" java.lang.illegalargumentexception: cannot instantiate interface org.springframework.context.applicationlistener : org.springframework.boot.logging.classpathloggingapplicationlistener @ org.springframework.boot.springapplication.createspringfactoriesinstances(springapplication.java:414) @ org.springframework.boot.springapplication.getspringfactoriesinstances(springapplication.java:394) @ org.springframework.bo...
Read more

python - pip wont install .WHL files -

-
when try install .whl gohike typing cmd c:\users\owner\downloads>pip install ta_lib-0.4.9-cp27-none-win_amd64.whl i print out: `processing c:\users\owner\downloads\ta_lib-0.4.9-cp27-none-win_amd64.whl complete output command python setup.py egg_info: traceback <most recent call last>: file "<string>", line 1, in <module> ioerror: [errno 2] no such file or directory:'c:\\users\\owner\\appdata\\lo cal\\temp\\pip-9gwk2c-build\\setup.py' ----------------------------------------- command "python setup.py egg_info" failed error code 1 in c:\users\owner\ap pdata\local\temp\pip-9gwk2c-build\` the part between pip- , -build in pip-9gwk2c-build different on same file. i asked python 2.7.11 pip 8.0.3 can't install gohike whl of ta-lib 0.4.9 few days ago , decided ask differently. can me! you have outdated pip not understand wheels. wheels not supposed have setup.py inside. try upgrading pip pip in...
Read more

Excel VBA "Microsoft Windows Common Controls 6.0 (SP6)" Location Changes -

-
i have excel workbook used generate quotations customers. there main page user inputs data, , there buttons generate various pdfs , send them customer. however, 1 user consistently getting error preventing generating , sending pdfs. i beleive "microsoft windows common controls 6.0 (sp6)". it's installed on users' computers. on mine, location "c:\windows\syswow64\mscomctl.ocx" on others "c:\windows\system32\mscomctl.ocx" the user experiencing issues on 32-bit machine, "mscomctl.ocx" file should in system32. have manually put in there, , pointed common controls reference in vba it. issue seems keep reverting looking in syswow64. since folder doesn't exist, can't find it, , can't produce pdfs. i thought perhaps issue user running older version of office (2007) there users on 2013 , 2016. thought perhaps mixture of 64-bit , 32-bit users. user gets sent workbook other users (as quotation process goes further, gets passed ...
Read more