User Tools

Site Tools


elk

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
elk [2016/01/28 19:05]
ben
elk [2016/01/28 22:56] (current)
ben
Line 4: Line 4:
  
   * Define a template mapping for the netflow data, mapping the fields to the correct datatypes: <code bash>   * Define a template mapping for the netflow data, mapping the fields to the correct datatypes: <code bash>
-curl -XPUT http://​localhost:​9200/​_template/​logstash-netflow ​-d '{ +curl -XPUT http://​localhost:​9200/​_template/​logstash-netflow9 ​-d '{ 
-    "​template"​ : "​logstash-netflow-*",+    "​template"​ : "​logstash-netflow9-*",
     "​order":​ 10,     "​order":​ 10,
     "​settings":​ {     "​settings":​ {
Line 49: Line 49:
                        "​src_as":​ { "​index":​ "​analyzed",​ "​type":​ "​integer"​ },                        "​src_as":​ { "​index":​ "​analyzed",​ "​type":​ "​integer"​ },
                        "​l4_dst_port":​ { "​index":​ "​not_analyzed",​ "​type":​ "​long"​ },                        "​l4_dst_port":​ { "​index":​ "​not_analyzed",​ "​type":​ "​long"​ },
-                       "​l4_src_port":​ { "​index":​ "​not_analyzed",​ "​type":​ "​long"​ }+                       "​l4_src_port": { "​index":​ "​not_analyzed",​ "​type":​ "​long"​ }, 
 +                       "​ipv4_dst_addr_postnat":​ { "​index":​ "​analyzed",​ "​type":​ "​ip"​ }, 
 +                       "​ipv4_src_addr_postnat":​ { "​index":​ "​analyzed",​ "​type":​ "​ip"​ }, 
 +                       "​l4_dst_port_postnat":​ { "​index":​ "​not_analyzed",​ "​type":​ "​long"​ }, 
 +                       "​l4_src_port_postnat": { "​index":​ "​not_analyzed",​ "​type":​ "​long"​ }
                    },                    },
                    "​type":​ "​object"​                    "​type":​ "​object"​
Line 57: Line 61:
    }    }
 }' }'
 +</​code>​
 +  * Copy the ''​netflow.yaml''​ definitions from ''/​opt/​logstash/​vendor/​bundle/​jruby/​1.9/​gems/​logstash-codec-netflow-2.0.2/​lib/​logstash/​codecs/​netflow.rb''​ to ''/​etc/​logstash/​mikrotik.netflow9.yaml''​ and patch the following: <​code>​
 +# Changed from 2->4
 +10:
 +- 4
 +- :input_snmp
 +14:
 +- 4
 +- :​output_snmp
 +
 +# Changed from uint24->​uint32
 +31:
 +- :uint32
 +- :​ipv6_flow_label
 +
 +# Add these entries:
 +225:
 +- :ip4_addr
 +- :​ipv4_src_addr_postnat
 +226:
 +- :ip4_addr
 +- :​ipv4_dst_addr_postnat
 +227:
 +- :uint16
 +- :​l4_src_port_postnat
 +228:
 +- :uint16
 +- :​l4_dst_port_postnat
 </​code>​ </​code>​
   * Setup a listening UDP port to receive the UDP data, and feed it into the netflow indexes in elasticsearch:​ <code bash /​etc/​logstash/​conf.d/​50-netflow.conf>​   * Setup a listening UDP port to receive the UDP data, and feed it into the netflow indexes in elasticsearch:​ <code bash /​etc/​logstash/​conf.d/​50-netflow.conf>​
 input { input {
     udp {     udp {
-        port  => 9995+        port  => 9996
         codec => netflow {         codec => netflow {
-            ​# Logstash doesn'​t support importing netflow v9 templates from the netflow device +            ​versions ​=> [9] 
-            # and lacks built-in templates for id=256,257 leading to errors and no data +            ​definitions ​=> "/​etc/​logstash/​mikrotik.netflow9.yaml"​
-            ​versions ​=> [5]+
         }         }
-        type  => "netflow"+        type  => "netflow9"
     }     }
 } }
Line 78: Line 109:
 } }
 </​code>​ </​code>​
-  * In Kibana Settings, add a new index pattern for ''​logstash-netflow-*''​ (if this pattern already exists and the mapping has been changed since, remember to hit the ''​refresh''​ button to ensure the changed datatypes are picked up. Verify all fields have the right type. +  * In Kibana Settings, add a new index pattern for ''​logstash-netflow9-*''​ (if this pattern already exists and the mapping has been changed since, remember to hit the ''​refresh''​ button to ensure the changed datatypes are picked up. Verify all fields have the right type. 
-  * Enable ''​IP->​Traffic Flow''​ on the desired interfaces, and add the logstash host as a netflow ​v5 target +  * Enable ''​IP->​Traffic Flow''​ on the desired interfaces, and add the logstash host as a netflow ​v9 target 
-  * Verify data is being indexed by doing a search on ''​*''​ against the ''​logstash-netflow-*''​ index pattern+  * Verify data is being indexed by doing a search on ''​*''​ against the ''​logstash-netflow9-*''​ index pattern 
 + 
 +The following warnings will show up briefly when logstash first starts. This is because the templates needed to understand the netflow messages are published in-band on a regular basis, and when logstash first starts up it might not have seen a copy of the templates before flow data is received. Once the template message is received (defaulting to 20 packets on the mikrtoik boards), these messages will cease: <code text> 
 +Jan 28 22:40:08 silverhold logstash[9496]:​ {:​timestamp=>"​2016-01-28T22:​40:​08.238000+0000",​ :​message=>"​No matching template for flow id 257", :​level=>:​warn} 
 +Jan 28 22:40:09 silverhold logstash[9496]:​ {:​timestamp=>"​2016-01-28T22:​40:​09.266000+0000",​ :​message=>"​No matching template for flow id 256", :​level=>:​warn} 
 +</​code>​
  
 ====== Bugs ====== ====== Bugs ======
elk.txt · Last modified: 2016/01/28 22:56 by ben