elasticsearch - Lines splitting despite using multiline logstash filter -

elasticsearch - Lines splitting despite using multiline logstash filter -

i used multiline logstash filter.it fails @ few instances.

for illustration tried parsing study multiple blocks.i formed these blocks/paragraphs using multiline filter.

page 1 date title author data................................................................................ .................................................................................... <delimiter>

when tested using 100 blocks, every time doesn't append 1 or 2 blocks correctly. takes

page 1 date title author

as single message , parses , remaining part message. note: observation @ times, 1st part of block say

page 21 date title author

is parsed , followed by

page 20 date title author data................................................................................ .................................................................................... <delimiter>

the prev block , remaining part of incomplete block parsed. kind of multi-threading issue?

edit: config file looks this:

input{ file{ path=>"/home/tudit/input.txt" start_position=>"beginning" } } filter{ multiline{ pattern => "^\f" negate => true => "previous" enable_flush=>"true" } grok{ match=>["message","page%{space}%{number:page_no:int}"] } grok{ match=>["message","(?<date>%{monthday}%{space}%{month}%{space}%{year})"] } grok{ match=>["message","author:%{space}%{word:author}"] } grok{ match=>["message","title:%{space}%{word:title}"] } date{ match => [ "date", "dd mmm yyyy" ] } } output { stdout{ codec=>"rubydebug" } elasticsearch{ embedded=> "true" host=> "localhost" } }

\f delimiter.

elasticsearch multiline logstash