Tags index with filebeat and logstash -

-

i use logstash-forwarder , logstash , create dinamic index tag configuration:

/etc/logstash/conf.d/10-output.conf

output {   elasticsearch {     hosts => "localhost:9200"     manage_template => false     index => "logstash-%{tags}-%{+yyyy.mm.dd}"   } }

/etc/logstash-forwarder.conf

"files": [     {       "paths": [         "/var/log/httpd/ssl_access_log",         "/var/log/httpd/ssl_error_log"        ],       "fields": { "type": "apache", "tags": "mytag" }     },

i convert configuration files filebeat in way:

/etc/filebeat/filebeat.yml

filebeat:   prospectors:     -      paths:        - /var/log/httpd/access_log      input_type: log      document_type: apache      fields:        tags: mytag

now in kibana index, instead of mytag see beats_input_codec_plain_applied

i can see 2 problems mentioned in topic. let me summarize own benefit , other visitors struggling problem too.

  1. format add tag(s) in filebeat prospector (per prospector tags available since 5.0 or 1.2.3 a-j noticed) configuration

bad:

 fields:        tags: mytag

good:

 fields:        tags: ["mytag"]

however, there's more important issue

  1. tags getting concatenated. want tags array, if ship newly added tags logstash we'll see them being concatenated strings in es.

if adding 1 tag, workaround (as per hellb0y77) remove automatic tag filebeat adds, in logstash (central server side): 

filter {     if "beats_input_codec_plain_applied" in [tags] {         mutate {             remove_tag => ["beats_input_codec_plain_applied"]         }     } }

this not work if 1 wanted add multiple tags in filebeat.

one have make logstash split concatenated string , add each item tags. perhaps better in case, put tags on filebeat end custom field, not "tags" field , extract them custom field on logstash.

anyway, there seems no way make work changing filebeat configuration. way doing parsing on receiving logstash filter chain. see https://github.com/elastic/filebeat/issues/220

if can remove logstash solution you. when sending logs filebeat directly elasticsearch, tags appear in es expected.


Comments

Post a Comment

Popular posts from this blog

java - Run spring boot application error: Cannot instantiate interface org.springframework.context.ApplicationListener -

-
i have spring project , try make use spring boot , run @ embadded tomcat following : https://spring.io/guides/gs/rest-service/ this application //@configuration //@enableaspectjautoproxy @springbootapplication @componentscan(basepackages = "gux.prome") public class application { public static void main(string[] args) { springapplication.run(application.class, args); } } if use maven command: mvn spring-boot:run , project starts fine, need debug , run main method @ intelij, exception occurs: exception in thread "main" java.lang.illegalargumentexception: cannot instantiate interface org.springframework.context.applicationlistener : org.springframework.boot.logging.classpathloggingapplicationlistener @ org.springframework.boot.springapplication.createspringfactoriesinstances(springapplication.java:414) @ org.springframework.boot.springapplication.getspringfactoriesinstances(springapplication.java:394) @ org.springframework.bo...
Read more

reactjs - React router and this.props.children - how to pass state to this.props.children -

-
i'm using react-router first time , don't know how think in yet. here's how i'm loading components in nested routes. entry point .js reactdom.render( <router history={hashhistory} > <route path="/" component={app}> <route path="models" component={content}> </route> </router>, document.getelementbyid('app') ); app.js render: function() { return ( <div> <header /> {this.props.children} </div> ); } so child of app content component sent in. i'm using flux , app.js has state , listens changes, don't know how pass state down this.props.children. before using react-router app.js defines children explicitly, passing state natural don't see how now. using couple of react helper methods can add state, props , whatever else this.props.children render: function() { var children = react.ch...
Read more

Excel VBA "Microsoft Windows Common Controls 6.0 (SP6)" Location Changes -

-
i have excel workbook used generate quotations customers. there main page user inputs data, , there buttons generate various pdfs , send them customer. however, 1 user consistently getting error preventing generating , sending pdfs. i beleive "microsoft windows common controls 6.0 (sp6)". it's installed on users' computers. on mine, location "c:\windows\syswow64\mscomctl.ocx" on others "c:\windows\system32\mscomctl.ocx" the user experiencing issues on 32-bit machine, "mscomctl.ocx" file should in system32. have manually put in there, , pointed common controls reference in vba it. issue seems keep reverting looking in syswow64. since folder doesn't exist, can't find it, , can't produce pdfs. i thought perhaps issue user running older version of office (2007) there users on 2013 , 2016. thought perhaps mixture of 64-bit , 32-bit users. user gets sent workbook other users (as quotation process goes further, gets passed ...
Read more