I've created a simple extractor for pfSense firewall logs. Extractors use regex on incoming logs to extract pieces of information that I want separatley and I will then be able to filter on those extractions.
{ "extractors": [ { "condition_type": "regex", "condition_value": "filterlog:.*,(?:pass|block),.*", "converters": [], "cursor_strategy": "copy", "extractor_config": { "regex_value": "filterlog:.*,(pass|block),.*" }, "extractor_type": "regex", "order": 0, "source_field": "message", "target_field": "action", "title": "pfSense Action" }, { "condition_type": "none", "condition_value": ".* (?:pass|block) .* on (?:[^:]+):.* (?:TCP|UDP|IGMP|ICMP) .* (?:\\d+\\.\\d+\\.\\d+\\.\\d+)\\.?(?:\\d*) [<|>] (?:\\d+\\.\\d+\\.\\d+\\.\\d+)\\.?(?:\\d*): (?:.*)", "converters": [], "cursor_strategy": "copy", "extractor_config": { "regex_value": "filterlog:.*[0-9]{1,10},[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3},[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3},(\\d+)," }, "extractor_type": "regex", "order": 4, "source_field": "message", "target_field": "srcport", "title": "pfSense Source Port" }, { "condition_type": "none", "condition_value": ".* (?:pass|block) .* on (?:[^:]+):.* (?:TCP|UDP|IGMP|ICMP) .* (?:\\d+\\.\\d+\\.\\d+\\.\\d+)\\.?(?:\\d*) [<|>] (?:\\d+\\.\\d+\\.\\d+\\.\\d+)\\.?(?:\\d*): (?:.*)", "converters": [], "cursor_strategy": "copy", "extractor_config": { "regex_value": "filterlog:.*[0-9]{1,10},[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3},([0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3})" }, "extractor_type": "regex", "order": 5, "source_field": "message", "target_field": "destip", "title": "pfSense Destination IP" }, { "condition_type": "none", "condition_value": ".* (?:pass|block) .* on (?:[^:]+):.* (?:TCP|UDP|IGMP|ICMP) .* (?:\\d+\\.\\d+\\.\\d+\\.\\d+)\\.?(?:\\d*) [<|>] (?:\\d+\\.\\d+\\.\\d+\\.\\d+)\\.?(?:\\d*): (?:.*)", "converters": [], "cursor_strategy": "copy", "extractor_config": { "regex_value": "filterlog:.*[0-9]{1,10},[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3},[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3},\\d+,(\\d+)" }, "extractor_type": "regex", "order": 6, "source_field": "message", "target_field": "destport", "title": "pfSense Destination Port" }, { "condition_type": "none", "condition_value": ".* (?:pass|block) .* on (?:[^:]+):.* (?:TCP|UDP|IGMP|ICMP) .* (?:\\d+\\.\\d+\\.\\d+\\.\\d+)\\.?(?:\\d*) [<|>] (?:\\d+\\.\\d+\\.\\d+\\.\\d+)\\.?(?:\\d*): (?:.*)", "converters": [], "cursor_strategy": "copy", "extractor_config": { "regex_value": "filterlog: \\d+,\\d+,,\\d+,(.+),match,.*" }, "extractor_type": "regex", "order": 1, "source_field": "message", "target_field": "interface", "title": "pfSense Interface" }, { "condition_type": "none", "condition_value": ".* (?:pass|block) .* on (?:[^:]+):.* (?:TCP|UDP|IGMP|ICMP) .* (?:\\d+\\.\\d+\\.\\d+\\.\\d+)\\.?(?:\\d*) [<|>] (?:\\d+\\.\\d+\\.\\d+\\.\\d+)\\.?(?:\\d*): (?:.*)", "converters": [], "cursor_strategy": "copy", "extractor_config": { "regex_value": "filterlog: .*,(udp|tcp),.*" }, "extractor_type": "regex", "order": 2, "source_field": "message", "target_field": "protocol", "title": "pfSense Protocol" }, { "condition_type": "none", "condition_value": ".* (?:pass|block) .* on (?:[^:]+):.* (?:TCP|UDP|IGMP|ICMP) .* (?:\\d+\\.\\d+\\.\\d+\\.\\d+)\\.?(?:\\d*) [<|>] (?:\\d+\\.\\d+\\.\\d+\\.\\d+)\\.?(?:\\d*): (?:.*)", "converters": [], "cursor_strategy": "copy", "extractor_config": { "regex_value": "filterlog:.*,[0-9]{1,10},([0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3})" }, "extractor_type": "regex", "order": 3, "source_field": "message", "target_field": "srcip", "title": "pfSense Source IP" } ], "version": "1.1.0-beta.2 (e5c1876)" } |>|>|>|>|>|>
Today it only handles IPv4 traffic and some of the regex rules would really need some more attention, but it works. I get Action (pass or block), IPv4 Source Port, IPv4 Destination Port, IPv4 Source IP, IPv4 Destination IP, Interface and Protocol.
You can find a GIST here: https://gist.github.com/linuxprofessor/86b9a67d779b4ded755d