Recent releases for zeek

zeek v2.6.1

2019-02-04T16:14:54+00:00

Bro 2.6.1 updates the embedded SQLite to version 3.26.0 to address the "Magellan" remote code execution vulnerability. The stock Bro configuration/scripts don't use SQLite by default, but custom user scripts/packages may. This release also updates Broker to v1.1.2, which includes a minor bug fix in its Python bindings and improved support for building it as a static library.

zeek v2.6.2

2019-05-30T16:23:37+00:00

This is a **security patch release** to address potential Denial of Service vulnerabilities: - Integer type mismatches in BinPAC-generated parser code and Bro analyzer code may allow for crafted packet data to cause unintentional code paths in the analysis logic to be taken due to unsafe integer conversions causing the parser and analysis logic to each expect different fields to have been parsed. One such example, reported by Maksim Shudrak, causes the Kerberos analyzer to dereference a null pointer. CVE-2019-12175 was assigned for this issue. - The Kerberos parser allows for several fields to be left uninitialized, but they were not marked with an &optional attribute and several usages lacked existence checks. Crafted packet data could potentially cause an attempt to access such uninitialized fields, generate a runtime error/exception, and leak memory. Existence checks and &optional attributes have been added to the relevent Kerberos fields. - BinPAC-generated protocol parsers commonly contain fields whose length is derived from other packet input, and for those that allow for incremental parsing, BinPAC did not impose a limit on how large such a field could grow, allowing for remotely-controlled packet data to cause growth of BinPAC's flowbuffer bounded only by the numeric limit of an unsigned 64-bit integer, leading to memory exhaustion. There is now a generalized limit for how large flowbuffers are allowed to grow, tunable by setting "BinPAC::flowbuffer_capacity_max".

zeek v2.6

2019-07-08T23:06:51+00:00

New Functionality ----------------- - Bro has switched to using the new Broker library for all its communication. Broker's API has been completely redesigned (compared to the version in 2.5), and much of its implementation has been redone. There's a new script-level "broker" framework that supersedes the old "communication" framework, which is now deprecated. All scripts that ship with Bro have been ported to use Broker. BroControl has likewise been ported to use Broker. For more about the new Broker framework, see https://www.bro.org/sphinx-git/frameworks/broker.html. There's also a guide there for porting existing Bro scripts to Broker. For more about Broker itself, including its API for external applications, see https://bro-broker.readthedocs.io/en/stable When using BroControl, the function of proxies has changed with Broker. If you are upgrading and have configured more than one proxy currenty, we recommend going back down to a single proxy node now. That should be fine unless you are using custom scripts doing significant data distribution through the new cluster framework. A side effect of the switch to using Broker is that each Bro node now runs as a single process instead of two. Also, the number of file descriptors being polled in Bro's main event loop has been reduced (1 per worker versus 5). This should increase the number of workers one can use before reaching the common 1024 file descriptor limitation of "select()". - Bro now has new "is" and "as" script operators for dynamic type-checking and casting. - "v as T" casts a value v into a value of type T, assuming that's possible (if not, it triggers a runtime error). - "v is T" returns a boolean indicating whether value v can be casted into type T (i.e., if true then "v as T" will succeed). This casting supports three cases currently: (1) a value of declared type "any" can be casted to its actual underlying type; (2) Broker values can be casted to their corresponding script types; and (3) all values can be casted to their declared types (i.e., a no-op). Example for "any":: # cat a.bro function check(a: any) { local s: string = "default"; if ( a is string ) s = (a as string); print fmt("s=%s", s); } event bro_init() { check("Foo"); check(1); } # bro a.bro s=Foo s=default - The existing "switch" statement got extended to now also support switching by type rather than value. The new syntax supports two type-based versions of "case": - "case type T: ...": Take branch if operand can be casted to type T. - "case type T as x: ... ": Take branch if operand can be casted to type T, and make the casted value available through ID "x". Multiple types can be listed per branch, separated by commas. However, one cannot mix cases with expressions and types inside a single switch statement. Example:: function switch_one(v: any) { switch (v) { case type string: print "It's a string!"; break; case type count as c: print "It's a count!", c; break; case type bool, type addr: print "It's a bool or address!"; break; default: print "Something else!"; break; } } - Bro now comes with a new "configuration framework" that allows updating script options dynamically at runtime. This functionality consists of three larger pieces working together: - Option variables: The new "option" keyword allows variables to be declared as runtime options. Such variables cannot be changed using normal assignments. Instead, they can be changed using the new function "Config::set_value". This function will automatically apply the change to all nodes in a cluster. Note that options can also be changed using the new function "Option::set", but this function will not send the change to any other nodes, so Config::set_value should typically be used instead of Option::set. Various redef-able constants in the standard Bro scripts have been converted to runtime options. This change will not affect any user scripts because the initial value of runtime options can still be redefined with a "redef" declaration. Example:: option testvar = "old value"; redef testvar = "new value"; It is possible to "subscribe" to an option through "Option::set_change_handler", which will trigger a handler callback when an option changes. Change handlers can optionally modify values before they are applied by returning the desired value, or reject updates by returning the old value. Priorities can be specified if there are several handlers for one option. Example script:: option testbool: bool = T; function option_changed(ID: string, new_value: bool): bool { print fmt("Value of %s changed from %s to %s", ID, testbool, new_value); return new_value; } event bro_init() { print "Old value", testbool; Option::set_change_handler("testbool", option_changed); Option::set("testbool", F); print "New value", testbool; } - Script-level configuration framework: The new script framework base/framework/config facilitates reading in new option values from external files at runtime. The format for these files looks like this:: [option name][tab/spaces][new variable value] Configuration files to read can be specified by adding them to "Config::config_files". Usage example:: redef Config::config_files += { "/path/to/config.dat" }; module TestConfig; export { option testbool: bool = F; } The specified file will now be monitored continuously for changes, so that writing "TestConfig::testbool T" into ``/path/to/config.dat`` will automatically update the option's value accordingly. The configuration framework creates a ``config.log`` that shows all value changes that took place. - Config reader: Internally, the configuration framework uses a new type of input reader to read such configuration files into Bro. The reader uses the option name to look up the type that variable has, converts the read value to the correct type, and then updates the option's value. Example script use:: type Idx: record { option_name: string; }; type Val: record { option_val: string; }; global currconfig: table[string] of string = table(); event InputConfig::new_value(name: string, source: string, id: string, value: any) { print id, value; } event bro_init() { Input::add_table([$reader=Input::READER_CONFIG, $source="../configfile", $name="configuration", $idx=Idx, $val=Val, $destination=currconfig, $want_record=F]); } - Support for OCSP and Signed Certificate Timestamp. This adds the following events and BIFs: - Events: - ocsp_request - ocsp_request_certificate - ocsp_response_status - ocsp_response_bytes - ocsp_response_certificate - ocsp_extension - x509_ocsp_ext_signed_certificate_timestamp - ssl_extension_signed_certificate_timestamp - Functions: - sct_verify - x509_subject_name_hash - x509_issuer_name_hash - x509_spki_hash - The SSL scripts provide a new hook "ssl_finishing(c: connection)" to trigger actions after the handshake has concluded. - New functionality has been added to the TLS parser, adding several events. These events mostly extract information from the server and client key exchange messages. The new events are: - ssl_ecdh_server_params - ssl_dh_server_params - ssl_server_signature - ssl_ecdh_client_params - ssl_dh_client_params - ssl_rsa_client_pms Since "ssl_ecdh_server_params" contains more information than the old "ssl_server_curve" event, "ssl_server_curve" is now marked as deprecated. - The "ssl_application_data" event was retired and replaced with "ssl_plaintext_data". - Some SSL events were changed and now provide additional data. These events are: - ssl_client_hello - ssl_server_hello - ssl_encrypted_data If you use these events, you can make your scripts work on old and new versions of Bro by wrapping the event definition in an "@if", for example:: @if ( Version::at_least("2.6") || ( Version::number == 20500 && Version::info$commit >= 944 ) ) event ssl_client_hello(c: connection, version: count, record_version: count, possible_ts: time, client_random: string, session_id: string, ciphers: index_vec, comp_methods: index_vec) @else event ssl_client_hello(c: connection, version: count, possible_ts: time, client_random: string, session_id: string, ciphers: index_vec) @endif - Functions for retrieving files by their ID have been added: - Files::file_exists - Files::lookup_File - New functions in the logging API: - Log::get_filter_names - Log::enable_stream - HTTP now recognizes and skips upgraded/websocket connections. A new event, "http_connection_upgrade", is raised in such cases. - A new hook, HTTP::sqli_policy, may be used to whitelist requests that could otherwise be counted as SQL injection attempts. - Added a MOUNT3 protocol parser - This is not enabled by default (no ports are registered and no DPD signatures exist, so no connections will end up attaching the new Mount analyzer). If it were to be activated by users, the following events are available: - mount_proc_null - mount_proc_mnt - mount_proc_umnt - mount_proc_umnt_all - mount_proc_not_implemented - mount_reply_status - Added new NFS events: - nfs_proc_symlink - nfs_proc_link - nfs_proc_sattr - The SMB scripts in ``policy/protocols/smb`` are now moved into ``base/protocols/smb`` and loaded/enabled by default. If you previously loaded these scripts from their ``policy/`` location (in local.bro or other custom scripts) you may now remove/change those although they should still work since ``policy/protocols/smb`` is simply a placeholder script that redirects to the new ``base/`` location. - Added new SMB events: - smb1_transaction_secondary_request - smb1_transaction2_secondary_request - smb1_transaction_response - Bro can now decrypt Kerberos tickets, and retrieve the authentication from them, given a suitable keytab file. - Added support for bitwise operations on "count" values. '&', '|' and '^' are binary "and", "or" and "xor" operators, and '~' is a unary ones-complement operator. - The '&' and '|' operators can apply to patterns, too. p1 & p2 yields a pattern that represents matching p1 followed by p2, and p1 | p2 yields a pattern representing matching p1 or p2. The p1 | p2 functionality was semi-present in previous versions of Bro, but required constants as its operands; now you can use any pattern-valued expressions. - You can now specify that a pattern matches in a case-insensitive fashion by adding 'i' to the end of its specification. So for example /fOO/i == "Foo" yields T, as does /fOO/i in "xFoObar". You can achieve the same functionality for a subpattern enclosed in parentheses by adding "?i:" to the open parenthesis. So for example /foo|(?i:bar)/ will match "BaR", but not "FoO". For both ways of specifying case-insensitivity, characters enclosed in double quotes remain case-sensitive. So for example /"foo"/i will not match "Foo", but it will match "foo". - "make install" now installs Bro's include headers (and more) into "--prefix" so that compiling plugins no longer needs access to a source/build tree. For OS distributions, this also facilitates creating "bro-devel" packages providing all files necessary to build plugins. - Bro now supports PPPoE over QinQ. - Bro now supports OpenSSL 1.1. - The new connection/conn.log history character 'W' indicates that the originator ('w' = responder) advertised a TCP zero window (instructing the peer to not send any data until receiving a non-zero window). - The connection/conn.log history characters 'C' (checksum error seen), 'T' (retransmission seen), and 'W' (zero window advertised) are now repeated in a logarithmic fashion upon seeing multiple instances of the corresponding behavior. Thus a connection with 2 C's in its history means that the originator sent >= 10 packets with checksum errors; 3 C's means >= 100, etc. - The above connection history behaviors occurring multiple times (i.e., starting at 10 instances, than again for 100 instances, etc.) generate corresponding events: - tcp_multiple_checksum_errors - udp_multiple_checksum_errors - tcp_multiple_zero_windows - tcp_multiple_retransmissions Each has the same form, e.g.:: event tcp_multiple_retransmissions(c: connection, is_orig: bool, threshold: count); - Added support for set union, intersection, difference, and comparison operations. The corresponding operators for the first three are "s1 | s2", "s1 & s2", and "s1 - s2". Relationals are in terms of subsets, so "s1 < s2" yields true if s1 is a proper subset of s2 and "s1 == s2" if the two sets have exactly the same elements. "s1 <= s2" holds for subsets or equality, and similarly "s1 != s2", "s1 > s2", and "s1 >= s2" have the expected meanings in terms of non-equality, proper superset, and superset-or-equal. - An expression of the form "v += e" will append the value of the expression "e" to the end of the vector "v" (of course assuming type-compatibility). "redef v += { a, b, c }" will similarly extend a vector previously declared with &redef by appending the result of expressions "a", "b", and "c" to the vector at initialization-time. - A new "@deprecated" directive was added. It marks a script-file as deprecated. Changed Functionality --------------------- - All communication is now handled through Broker, requiring changes to existing scripts to port them over to the new API. The Broker framework documentation comes with a porting guide. - The DHCP analyzer and its script-layer interface have been rewritten. - Supports more DHCP options than before. - The DHCP log now represents DHCP sessions based on transaction ID and works on Bro cluster deployments. - Removed the ``policy/protocols/dhcp/known-devices-and-hostnames.bro`` script since it's generally less relevant now with the updated log. - Removed the ``base/protocols/dhcp/utils.bro`` script and thus the "reverse_ip" function. - Replaced all DHCP events with the single "dhcp_message" event. The list of removed events includes: - dhcp_discover - dhcp_offer - dhcp_request - dhcp_decline - dhcp_ack - dhcp_nak - dhcp_release - dhcp_inform - A new script, ``policy/protocols/dhcp/deprecated_events.bro``, may be loaded to aid those transitioning away from the list of "removed" events above. The script provides definitions for the old events and automatically generates them from a "dhcp_message" handler, thus providing equivalent functionality to the previous Bro release. Such usage emits deprecation warnings. - Removed ``policy/misc/known-devices.bro`` script and thus ``known_devices.log`` will no longer be created. - The "--with-binpac" configure option has changed to mean "path to the binpac executable" instead of "path to binpac installation root". - The MIME types used to identify X.509 certificates in SSL connections changed from "application/pkix-cert" to "application/x-x509-user-cert" for host certificates and "application/x-x509-ca-cert" for CA certificates. - The "ssl_server_curve" event is considered deprecated and will be removed in the future. See the new "ssl_ecdh_server_params" event for a replacement. - The Socks analyzer no longer logs passwords by default. This brings its behavior in line with the FTP/HTTP analyzers which also do not log passwords by default. To restore the previous behavior and log Socks passwords, use:: redef SOCKS::default_capture_password = T; - The DNS base scripts no longer generate some noisy and annoying weirds: - dns_unmatched_msg - dns_unmatched_msg_quantity - dns_unmatched_reply - The "tunnel_parents" field of ``conn.log`` is now marked ``&optional``, so, in the default configuration of logs, this field will show "-" instead of "(empty)" for connections that lack any tunneling. - SMB event argument changes: - "smb1_transaction_request" now has two additional arguments, "parameters" and "data" strings - "smb1_transaction2_request" now has an additional "args" record argument - The "SMB::write_cmd_log" option has been removed and the corresponding logic moving to ``policy/protocols/smb/log-cmds.bro`` which can simply be loaded to produce the same effect of toggling the old flag on. - SSL event argument changes: - "ssl_server_signature" now has an additional argument "signature_and_hashalgorithm". - The "dnp3_header_block" event no longer has the "start" parameter. - The "string_to_pattern()" and now-deprecated "merge_pattern()" built-ins are no longer restricted to only be called at initialization time. - GeoIP Legacy Database support has been replaced with GeoIP2 MaxMind DB format support. - This updates the "lookup_location" and "lookup_asn" BIFs to use libmaxminddb. The motivation for this is that MaxMind is discontinuing GeoLite Legacy databases: no updates after April 1, 2018, no downloads after January 2, 2019. It's also noted that all GeoIP Legacy databases may be discontinued as they are superseded by GeoIP2. - "Weird" events are now generally suppressed/sampled by default according to some tunable parameters: - Weird::sampling_whitelist - Weird::sampling_threshold - Weird::sampling_rate - Weird::sampling_duration Those options can be changed if one needs the previous behavior of a "net_weird", "flow_weird", or "conn_weird" event being raised for every single event. The original ``weird.log`` may not differ much with these changes, except in the cases where a particular weird type exceeds the sampling threshold. Otherwise, there is a new ``weird_stats.log`` generated via ``policy/misc/weird-stats.bro`` which contains concise summaries of weird counts per type per time period. - Improved DCE-RPC analysis via tracking of context identifier mappings - These DCE-RPC events now contain an additional context-id argument: - dce_rpc_bind - dce_rpc_request - dce_rpc_response - Added new events: - dce_rpc_alter_context - dce_rpc_alter_context_resp - The default value of ``Pcap::snaplen`` changed from 8192 to 9216 bytes to better accommodate jumbo frames. - Improvements to ``ntlm.log`` to fix incorrect reporting of login success/failure. Also, the "status" field was removed and "server_nb_computer_name", "server_dns_computer_name", and "server_tree_name" fields added. - BroControl: The output of the broctl "top" command has changed slightly. The "Proc" column has been removed from the output. This column previously indicated whether each Bro process was the "parent" or "child", but this is no longer relevant because each Bro node now runs as a single process. - The ``DNP3::function_codes`` name for request 0x21 has been corrected from "AUTHENTICATE_ERR" to "AUTHENTICATE_REQ_NR". - The ``DNS::query_types`` names for resource records 41 and 100 have been corrected from "EDNS" to "OPT" and "DINFO" to "UINFO", respectively. Removed Functionality --------------------- - We no longer maintain any Bro plugins as part of the Bro distribution. Most of the plugins that used to be in aux/plugins have been moved over to use the Bro Package Manager instead. See https://packages.bro.org for a list of Bro packages currently available. - The "ocsp_request" event no longer has "requestorName" parameter. - The node-specific ``site/local-*.bro`` scripts have been removed. - BroControl: The "IPv6Comm" and "ZoneID" options are no longer available (though Broker should be able to handle IPv6 automatically). Deprecated Functionality ------------------------ - The old communication system is now deprecated and scheduled for removal with the next Bro release. This includes the "communication" framework, the ``&sychronized`` attributes, and the existing communication-related BiFs. Use Broker instead. - The infrastructure for serializing Bro values into a binary representation is now deprecated and scheduled for removal with the next Bro release. This includes the ``&persistent`` attribute, as well as BIFs like "send_id()". Use Broker data stores and the new configuration framework instead. - Mixing of scalars and vectors, such as "v + e" yielding a vector corresponding to the vector v with the scalar e added to each of its elements, has been deprecated. - The built-in function "merge_pattern()" has been deprecated. It will be replaced by the '&' operator for patterns. - The undocumented feature of using "&&" and "||" operators for patterns has been deprecated. - BroControl: The "update" command is deprecated and scheduled for removal with the next Bro release. Bro's new configuration framework is taking its place.

zeek v2.6.3

2019-08-09T00:41:41+00:00

This is a **security patch release** to address potential Denial of Service vulnerabilities: - Null pointer dereference in the RPC analysis code. RPC analyzers (e.g. MOUNT or NFS) are not enabled in the default configuration. - Signed integer overflow in BinPAC-generated parser code. The result of this is Undefined Behavior with respect to the array bounds checking conditions that BinPAC generates, so it's unpredictable what an optimizing compiler may actually do under the assumption that signed integer overlows should never happen. The specific symptom which lead to finding this issue was with the PE analyzer causing out-of-memory crashes due to large allocations that were otherwise prevented when the array bounds checking logic was changed to prevent any possible signed integer overlow.

zeek v2.6.4

2019-08-29T00:08:43+00:00

This is a **security patch release** to address a potential Denial of Service vulnerability: * The NTLM analyzer did not properly handle AV Pair sequences that were either empty or unterminated, resulting in invalid memory access or heap buffer over-read. The NTLM analyzer is enabled by default and used in the analysis of SMB, DCE/RPC, and GSSAPI protocols. Thanks to Chris Hinshaw for reporting the issue.

zeek v3.0.0

2019-09-23T19:07:47+00:00

New Functionality ----------------- - Added support for DNSSEC resource records RRSIG, DNSKEY, DS, NSEC, and NSEC3. The associated events are: - dns_RRSIG - dns_DNSKEY - dns_DS - dns_NSEC - dns_NSEC3 - Added support for parsing and logging DNS SPF resource records. A new ``dns_SPF_reply`` event is also available. - Zeek's Plugin framework now allows a patch version. If a patch version is not provided, it will default to 0. To specify this, modify the plugin Configuration class in your ``src/Plugin.cc`` and set ``config.version.patch``. Note that the default plugin skeleton includes a unit test whose Baseline has the plugin version number in it and that will now fail due to the version number now including a patch number. For those that want to keep the unit test, simply adapt the unit test/baseline to include the new plugin patch number. - The default http.log not includes a field for the HTTP request Origin header. - Support for decapsulating VXLAN tunnels. - The for-loop syntax now allows for iterating over key-value pairs of tables. Previously, a separate lookup within the loop was required to obtain the value at a given index/key, but now this works:: local t: table[count] of string = table(); t[1] = "hello"; t[55] = "goodbye"; for ( key, value in t ) print key, value; - Added options for controlling the source path/prefix for Input and Intel framework files: - InputAscii::path_prefix - InputBinary::path_prefix - Intel::path_prefix - Support for NFLOG link-layer type. - Support for some SMB 3.x features - An ``smb2_transform_header`` event is raised after parsing TRANSFORM_HEADER structures associated with encrypted messages. - The ``SMB2::NegotiateResponse`` record now contains ``negotiate_context_count`` and ``negotiate_context_values`` fields containing capability information found in an SMB 3.1.1 dialect's negotiation message. - Added a new hook, ``Intel::filter_item``, to assist in filtering and removal of intelligence items that are about to be inserted. - Add support for SMB filenames in the intel framework. - Added a new event for weirdness found via file analysis: ``file_weird``. - The conn.log "history" field supports a new character 'G' or 'g' (capital for originator, lowercase responder) to indicate a content gap in the TCP stream. These are recorded logarithmically. - The ``ZEEK_DNS_RESOLVER`` environment variable now controls the DNS resolver to use by setting it to an IPv4 or IPv6 address. If not set, then the first IPv4 address from /etc/resolv.conf gets used. - The ``//i`` convenience syntax for case-insensitive patterns is now also allowed when specifying patterns used in signature files. - New RDP functionality. - New events: - rdp_client_network_data - rdp_client_security_data - rdp_client_cluster_data - rdp_native_encrypted_data - Add a new "client_channels" field to rdp.log based on data parsed from the Client Network Data (TS_UD_CS_NET) packet. The channel list is also available in the new ``rdp_client_network_data`` event. - Add parsing support for TLS 1.3 pre-shared key extension. This info is available in the events: ``ssl_extension_pre_shared_key_client_hello`` and ``ssl_extension_pre_shared_key_server_hello``. - Added/re-wrote support for NTP. - Parsing support for modes 1-7, with parsed structures available in the ``ntp_message`` event. - An ntp.log is produced by default, containing data extracted from NTP messages with modes 1-5. - Add support for vector slicing operations. For example:: local v = vector(1, 2, 3, 4, 5); v[2:4] = vector(6, 7, 8); # v is now [1, 2, 6, 7, 8, 5] print v[:4]; # prints [1, 2, 6, 7] - Add support for paraglob, a fairly quick data structure for matching a string against a large list of patterns. For example:: local v1 = vector("*", "d?g", "*og", "d?", "d[!wl]g"); local p1 = paraglob_init(v1); print paraglob_match(p1, "dog"); - An ``expire_func`` for a table with a multi-value-index will now unroll the index and take one argument for each index value. For example, for a ``table[string,string] of count`` the expire function signature is: function(t: table[string, string] of count, s: string, s2: string): interval - Zeek's anonymous functions now capture their closures by reference. This means that they can use and modify variables from the scope that they were generated in. For example: local n = 3; local f = function() { n += 1; }; f(); print n; # prints 4 These anonymous functions can also be serialized over Broker with their closures. In order to be serialzed over Broker the receiving script needs to have an identical version of the function declared. For the above example, a receiving script would need to have declared a function local name = function() { n += 1; }; to be able to receive the senders function over Broker. Functions with closures can still use the variables they have captured even after they have left the scope that they were declared in. For example, a simple generator function like the one below works as expected. local make_adder = function(n: count): function(m: count): count { return function (m: count): count { return n + m; }; }; print make_adder(3)(5); # prints 8 local three = make_adder(3); print three(5); # prints 8 - Add ``LogAscii::enable_utf_8`` option to allow valid utf8 sequences to be written directly into the ASCII logs without any escaping. - Add parsing, analysis, and logging support for MQTT protocol v3.1/v3.1.1. This is not enabled by default, use ``@load policy/protocols/mqtt`` to use the new MQTT analysis support. - Zeek now supports duration thresholding on connections, similarly to how it supports byte and packet thresholds. - New events: - ConnThreshold::duration_threshold_crossed - conn_duration_threshold_crossed - New functions: - ConnThreshold::set_duration_threshold - ConnThreshold::delete_duration_threshold - set_current_conn_duration_threshold - get_current_conn_duration_threshold Changed Functionality --------------------- - The following executable names have changed (the old names will continue to work, but emit a deprecation warning): - ``bro`` is now ``zeek`` - ``bro-config`` is now ``zeek-config`` - ``broctl`` is now ``zeekctl`` - ``bro-cut`` is now ``zeek-cut`` - BroControl has been completely renamed to ZeekControl. Many installation directories and files with "broctl" in their name have been changed to use "zeekctl" instead. It's expected this has been done in a way that's backwards compatible with previous Bro installations. E.g. if you made customizations to the ``broctl.cfg`` file of a previous installation, installing the newer Zeek version over it will retain that file and even symlink the new ``zeekctl.cfg`` to it. - The default install prefix is now ``/usr/local/zeek`` instead of ``/usr/local/bro``. If you have an existing installation that used the previous default and are still using the new default when upgrading, we'll crate ``/usr/local/zeek`` as a symlink to ``/usr/local/bro``. Certain subdirectories will also get similar treatment: ``share/bro``, ``include/bro``, and ``lib/bro``. - ``$prefix/share/bro/site/local.bro`` has been renamed to ``local.zeek``. If you have a ``local.bro`` file from a previous installation, possibly with customizations made to it, the new version of Zeek will install a ``local.zeek`` file that is a symlink to the pre-existing ``local.bro``. In that case, you may want to just copy ``local.bro`` into the new ``local.zeek`` location to avoid confusion, but things are otherwise meant to work properly without intervention. - All scripts ending in ``.bro`` that ship with the Zeek source tree have been renamed to ``.zeek``. - The search logic for the ``@load`` script directive now prefers files ending in ``.zeek``, but will fallback to loading a ``.bro`` file if it exists. E.g. ``@load foo`` will first check for a ``foo.zeek`` file to load and then otherwise ``foo.bro``. Note that ``@load foo.bro`` (with the explicit ``.bro`` file suffix) prefers in the opposite order: it first checks for ``foo.bro`` and then falls back to a ``foo.zeek``, if it exists. - The for-loop index variable for vectors has been changed from 'int' to 'count' type. It's unlikely this would alter/break any script behavior unless they were explicitly inspecting the variable's type (and there's typically no reason to do that). - The startup/initialization behavior has changed such that any errors encountered while processing the ``bro_init()`` event will cause the process to terminate rather than continue on the main run loop. - The ``dns_state`` field within ``connection`` records has changed: the ``pending_queries`` and ``pending_replies`` fields are now ``&optional``, and there is a new field ``pending_query`` that is populated before ``pending_queries``. If you have scripts that access the ``pending_queries`` or ``pending_replies`` fields, they will need to be updated. This change was made to improve performance. - The ternary operator (" ? : ") now enforces that if "" and "" are both records, they are of the same type. It was always assumed that they were, but code might have still worked even if not. - The "orig_fuids", "orig_filenames", "orig_mime_types" http.log fields as well as their "resp" counterparts are now limited to having "HTTP::max_files_orig" or "HTTP::max_files_resp" entries, which are 15 by default. The limit can also be ignored case-by-case via the "HTTP::max_files_policy" hook. - The binpac library is now only compiled as a shared library by default. To revert back to compiling only a static library, there's the ``--enable-static-binpac`` configure option. - The Broker C++ API has some breaking changes, see it's own NEWS file for details on how to migrate old code. - Some Weirds associated with generic binpac parsing exceptions in analyzers that didn't otherwise handle them (like syslog, modbus, dnp3) are now a ProtocolViolation instead - An "addl" parameter was added to the ``flow_weird`` and ``net_weird`` events for describing additional information about the weird. The ``conn_weird`` event already had such a parameter. - Weird names that contained variable content and may result in an unbounded number of weird names have been renamed to remove the variable content (which has been made available in the "addl" field of ``conn_weird``, ``flow_weird``, or ``net_weird`` events): - "unknown_dce_rpc_auth_type_%d" -> unknown_dce_rpc_auth_type - "gtp_invalid_info_element_%d" -> gtp_invalid_info_element - "unknown_netbios_type:" 0x%x -> unknown_netbios_type - "excess_netbios_hdr_len" (%d > %d) -> excess_netbios_hdr_len - "deficit_netbios_hdr_len" (%d > %d) -> deficit_netbios_hdr_len - "bad_RPC_program (%d)" -> bad_RPC_program - "unknown_MOUNT_request(%u)" -> unknown_MOUNT_request - "unknown_NFS_request(%u)" -> unknown_NFS_request - "RPC resync: discard %d bytes\n" -> RPC_resync - "RPC_message_too_long (%d64)" -> RPC_message_too_long - "socks5_unsupported_authentication_method_%d" -> socks5_unsupported_authentication_method - "socks5_unsupported_authentication_%d_%d" -> socks5_unsupported_authentication - "ssh_unknown_kex_algorithm=%s" -> ssh_unknown_kex_algorithm - "Encountered unknown type in server name ssl extension: %d" -> ssl_ext_unknown_server_name_type - "UDP_datagram_length_mismatch(%d!=%d)" -> UDP_datagram_length_mismatch - "OPENSSL Could not parse OCSP request (fuid %s)" -> openssl_ocsp_request_parse_error - "OPENSSL Could not parse OCSP response (fuid %s)" -> openssl_ocsp_response_parse_error - "Could not parse X509 certificate (fuid %s)" -> x509_cert_parse_error - "Certificate with invalid BasicConstraint. fuid %s" -> x509_invalid_basic_constraint - "Could not parse subject alternative names. fuid %s" -> x509_san_parse_error - "DNS-field does not contain an IA5String. fuid %s" -> x509_san_non_string - "Weird IP address length %d in subject alternative name. fuid %s" -> x509_san_ip_length - "Could not parse time in X509 certificate (fuid %s) -- UTCTime has wrong length" -> x509_utc_length - "Could not parse UTC time in non-YY-format in X509 certificate (x509 %s)" -> x509_utc_format - "Could not parse time in X509 certificate (fuid %s) -- Generalized time has wrong length" -> x509_gen_time_length - "Invalid time type in X509 certificate (fuid %s)" -> x509_invalid_time_type - "Could not parse time in X509 certificate (fuid %s) -- additional char after time" -> x509_time_add_char - "Could not parse time in X509 certificate (fuid %s) -- not enough bytes remaining for offset" -> x509_time_offset_underflow - "Could not parse time in X509 certificate (fuid %s) -- unknown offset type" -> x509_time_offset_type - "X509::GetExtensionFromBIO: %s" -> x509_get_ext_from_bio - "unknown_mobility_type_%d" -> unknown_mobility_type - "unknown_routing_type_%d" -> unknown_routing_type - "unknown_protocol_%d" -> unknown_protocol - "unknown_gre_version_%d" -> unknown_gre_version - "unknown_gre_protocol_%u16" -> unknown_gre_protocol - The "missed_bytes" field of conn.log can be calculated slightly differently in some cases: ACKs that reveal a content gap, but also come at the end of a connection (in a FIN or RST) are considered unreliable and aren't counted as true gaps. - The Broxygen component, which is used to generate our Doxygen-like scripting API documentation has been renamed to Zeekygen. This likely has no breaking or visible changes for most users, except in the case one used it to generate their own documentation via the ``--broxygen`` flag, which is now named ``--zeekygen``. Besides that, the various documentation in scripts has also been updated to replace Sphinx cross-referencing roles and directives like ":bro:see:" with ":zeek:see:". - The catch-and-release and unified2 scripts are no longer loaded by default. Because there was a performance impact simply from loading them and it's unlikely a majority of user make use of their features, they've been moved from the scripts/base/ directory into scripts/policy/ and must be manually loaded to use their functionality. The "drop" action for the notice framework is likewise moved since it was implemented via catch-and-release. As a result, the default notice.log no longer contains a "dropped" field. If you previously used the catch-and-release functionality add this: @load policy/frameworks/netcontrol/catch-and-release If you previously used Notice::ACTION_DROP add: @load policy/frameworks/notice/actions/drop If you previously used the Unified2 file analysis support add: @load policy/files/unified2 - The default value of ``peer_description`` has changed from "bro" to "zeek". This won't effect most users, except for the fact that this value may appear in several log files, so any external plugins that have written unit tests that compare baselines of such log files may need to be updated. - The "remote_ip" field of "addr" type was removed from radius.log and replaced with a field named "tunnel_client" of "string" type. The reason for this is that the Tunnel-Client-Endpoint RADIUS attribute this data is derived from may also be a FQDN, not just an IP address. - The ``ssl_server_hello`` event's ``server_random`` parameter has been changed to always include the full 32-byte field from the ServerHello. Previously a 4-byte timestamp and 28-byte random data were parsed separately as some TLS protocol versions specified a separate timestamp field as part of the full 32-byte random sequence. - The namespace used by all the builtin plugins that ship with Zeek have changed to use "Zeek::" instead of "Bro::". - Any Broker topic names used in scripts shipped with Zeek that previously were prefixed with "bro/" are now prefixed with "zeek/" instead. In the case where external applications were using a "bro/" topic to send data into a Bro process, a Zeek process still subscribes to those topics in addition to the equivalently named "zeek/" topic. In the case where external applications were using a "bro/" topic to subscribe to remote messages or query data stores, there's no backwards compatibility and external applications must be changed to use the new "zeek/" topic. The thought is this change will have low impact since most data published under "bro/" topic names is intended for use only as a detail of implementing cluster-enabled versions of various scripts. A list of the most relevant/common topic names that could potentially be used in external applications to consume/query remote data that one may need to change: - store names - bro/known/services - bro/known/hosts - bro/known/certs - cluster nodes - bro/cluster/ - bro/cluster/node/ - bro/cluster/nodeid/ - logging - bro/logs/ - The ``resp_ref`` argument was removed from the ``ocsp_response_bytes`` event. ``resp_ref`` was not used by anything in the codebase and could not be passed to any other functions for further processing. The remainder of the ``ocsp_response_bytes`` is unchanged. - For performance reasons, processing of notices is now always performed by the node on which the notice is raised rather than the centralized Manager node. This has potential incompatibilities for those that relied on global state for notice policy processing. It also introduces an expected race condition that may cause multiple notices of the same kind that are generated within a short timespan of each other on separate cluster nodes to all be logged rather than suppressed and de-duplicated into a single notice. - to_json is now a bif, no longer a script. Loading base/utils/json.zeek is no longer necessary and has been deprecated. to_json should yield much better, always valid json. There are some small differences in output; unnecessary spaces are removed and port values are rendered differently, now including the port and the protocol. - The output of the JSON logger now uses an external library to generate json. There are small changes to the output; most visibly double numbers are now rounded slightly differently. The way in which port values are rendered does _not_ change for JSON logs. - The C++-layer List, Queue, and Dict types have changed from using macros to templates as well as some other API changes. - Range-based for-loops are now supported - The loop_over_queue macro is now removed - PList is now a template instead of a macro, so any "PList(T)" usages in external code should now use "PList" - PDict is now a template instead of a macro, so any "PDict(T)" usages in external code should now use "PDict" - Generally some methods used to assume containers were only using integer or pointer types, so semantics may now be slightly different to either avoid copying or unsafely returning arbitrary T types by value. E.g. List::remove_nth and List::get can no longer return a "null" value when the provided index is out of range, so they assert instead, and Queue::pop methods do not return a value at all (one must check for a non-empty container before removing an element). - Google Perftools (tcmalloc) is no longer used by default on Linux systems if it's found during the configuration process. Use the --enable-perftools configuration flag to use tcmalloc. The --disable-perftools flag is also no longer provided since there's no longer any case in which tcmalloc will be used by default. - There now is a maximum number of protocol violations that can be raised by an analyzer before it is disabled; the default is set to 5. This behavior is customizable using ``DPD::max_violations`` and ``DPD::ignore_violations``. - The scan detection script, ``policy/misc/scan``, is no longer loaded by default in ``site/local.zeek`` due to it frequenty causing performance issues. Removed Functionality --------------------- - A number of functions that were deprecated in version 2.6 or below and completely removed from this release. Most of the functions were used for the old communication code. - ``find_ip_addresses`` - ``cat_string_array`` - ``cat_string_array_n`` - ``complete_handshake`` - ``connect`` - ``decode_base64_custom`` - ``disconnect`` - ``enable_communication`` - ``encode_base64_custom`` - ``get_event_peer`` - ``get_local_event_peer`` - ``join_string_array`` - ``listen`` - ``merge_pattern`` - ``request_remote_events`` - ``request_remote_logs`` - ``request_remote_sync`` - ``resume_state_updates`` - ``send_capture_filter`` - ``send_current_packet`` - ``send_id`` - ``send_ping`` - ``set_accept_state`` - ``set_compression_level`` - ``sort_string_array`` - ``split1`` - ``split_all`` - ``split`` - ``suspend_state_updates`` - ``terminate_communication`` - ``split`` - ``send_state`` - ``checkpoint_state`` - ``rescan_state`` - ``log_file_name`` - ``open_log_file`` - ``disable_print_hook`` - The following events were deprecated in version 2.6 or below and are completely removed from this release: - ``ssl_server_curve`` - ``dhcp_ack`` - ``dhcp_decline`` - ``dhcp_discover`` - ``dhcp_inform`` - ``dhcp_nak`` - ``dhcp_offer`` - ``dhcp_release`` - ``dhcp_request`` - ``remote_state_access_performed`` - ``remote_state_inconsistency`` - ``remote_connection_established`` - ``remote_connection_closed`` - ``remote_connection_handshake_done`` - ``remote_event_registered`` - ``remote_connection_error`` - ``remote_capture_filter`` - ``remote_log_peer`` - ``remote_log`` - ``finished_send_state`` - ``remote_pong`` - ``software_version_found`` - ``software_unparsed_version_found`` - ``software_parse_error`` - ``print_hook`` - ``interconn_stats`` - ``interconn_remove_conn`` - ``root_backdoor_signature_found`` - ``napster_signature_found`` - ``kazaa_signature_found`` - ``gaobot_signature_found`` - ``ftp_signature_found`` - ``gnutella_signature_found`` - ``http_signature_found`` - ``irc_signature_found`` - ``telnet_signature_found`` - ``ssh_signature_found`` - ``rlogin_signature_found`` - ``smtp_signature_found`` - ``http_proxy_signature_found`` - ``backdoor_stats`` - ``backdoor_remove_conn`` - ``dns_full_request`` - ``non_dns_request`` - The following types/records were deprecated in version 2.6 or below and are removed from this release: - ``peer_id`` - ``event_peer`` - ``packet`` - ``software`` - ``software_version`` - The following configuration options were deprecated in version 2.6 or below and are removed from this release: - ``max_remote_events_processed`` - ``forward_remote_events`` - ``forward_remote_state_changes`` - ``enable_syslog`` - ``remote_trace_sync_interval`` - ``remote_trace_sync_peers`` - ``remote_check_sync_consistency`` - ``log_rotate_interval`` - ``log_max_size`` - ``log_encryption_key`` - ``state_dir`` - ``state_write_delay`` - ``ssl_ca_certificate`` - ``ssl_private_key`` - ``ssl_passphrase`` - ``suppress_local_output`` - ``irc_servers`` - ``interconn_min_interarrival`` - ``interconn_max_interarrival`` - ``interconn_max_keystroke_pkt_size`` - ``interconn_default_pkt_size`` - ``interconn_stat_period`` - ``interconn_stat_backoff`` - ``interconn_endp_stats`` - ``backdoor_stat_period`` - ``backdoor_stat_backoff`` - ``backdoor_endp_stats`` - ``chunked_io_buffer_soft_cap`` - The following constants were used as part of deprecated functionality in version 2.6 or below and are removed from this release: - ``PEER_ID_NONE`` - ``REMOTE_LOG_INFO`` - ``REMOTE_SRC_CHILD`` - ``REMOTE_SRC_PARENT`` - ``REMOTE_SRC_SCRIPT`` - The deprecated script ``policy/protocols/smb/__load__.bro`` was removed. Instead of ``@load policy/protocols/smb`` use ``@load base/protocols/smb``. - Broccoli, which had been deprecated in version 2.6 and was no longer built by default was removed from the source tree. - Support for the &persistent, &synchronized, &mergeable, &encrypt, &rotate_interval, and &rotate_size attributes, which were deprecated in Bro 2.6, was removed. The ``-g`` command-line option (dump-config) which relied on this functionality was also removed. - Functionality for writing state updates for variables with the &synchronized attribute was removed. This entails the ``-x`` command-line option (print-state) as well as the ``capture_state_updates`` function. - Removed the BroControl ``update`` command, which was deprecated in Bro 2.6. - Functionality for writing/reading binary event streams was removed. This functionality relied on the old communication code anc was basically untested. The ``-R`` command-line option (replay) as well as the ``capture_events`` function were removed. - Removed p0f (passive OS fingerprinting) support. The version of p0f shipped with zeek was ancient, probably did not give any reliable support anymore and did not offer a clear upgrade path. The ``OS_version_found`` event as well as the ``generate_OS_version_event`` configuration option were removed. - Removed the ``max_files_in_cache`` option and the associated "file caching" feature it's associated with. That feature allowed one to open many scripting-layer ``file`` objects and potentially bypass the operating system's resource limits for open files. This is typically not necessary and it's a problem that is more appropriately addressed at the system configuration level. - Removed the InterConn analyzer. - Removed the BackDoor analyzer. - Removed ``List::sortedinsert`` from the C++ API. Deprecated Functionality ------------------------ - The ``str_shell_escape`` function is now deprecated, use ``safe_shell_quote`` instead. The later will automatically return a value that is enclosed in double-quotes. - The ``bro_init``, ``bro_done``, and ``bro_script_loaded`` events are now deprecated, use ``zeek_init``, ``zeek_done``, and ``zeek_script_loaded`` instead. Any existing event handlers for the deprecated versions will automatically alias to the new events such that existing code will not break, but will emit a deprecation warning. - The ``bro_is_terminating`` and ``bro_version`` function are deprecated and replaced by functions named ``zeek_is_terminating`` and ``zeek_version``. - The ``List::insert`` method from the C++ API is deprecated, use ``List::push_front`` instead.

zeek v3.0.1

2019-12-10T21:22:14+00:00

This is a bug-fix release addressing the following: - #598: `to_json()` support for `set` with composite keys - #615: add `print_raw()` BIF as convenience/workaround for #596 - #595: json logging performance regression - #602: external plugins can't find paraglob header - #593: improve sub-microsecond interval printing/descriptions - bc18ca44e6d18c28c92818fdc11dcd645c33d675: ptr_func Xcode compiler warnings - zeek/broker#58: Broker Xcode compiler warnings - #591: `set[enum]` options don't work - #606: ambiguous json format for vectors with null elements - #611: memory leak and "incorrect" field order in RecordVal JSON formatting - #632: fix `redef` of table to use a different `&default` value - https://github.com/zeek/broker/pull/67: potential libc++ mismatch between CAF and Broker - #667: fix signature matching for payload-carrying TCP SYN packets - #678: fix ZEEK_PROFILER_FILE - #649: fix OpenBSD build - #685: fix RPC call parsing

zeek v3.1.0-rc1

2020-02-10T19:23:01+00:00

Release Candidate for Zeek 3.1.0, which is now released: https://github.com/zeek/zeek/releases/tag/v3.1.0

zeek v3.0.2

2020-02-25T21:34:26+00:00

This long-term supported release addresses the following security issues: * Potential Denial of Service due to memory leak in DNS TSIG message parsing. Thanks to Max Kellermann for the report and patch. See https://github.com/zeek/zeek/pull/799 * Potential Denial of Service due to memory leak (or assertion when compiling with assertions enabled) when receiving a second SSH KEX message after a first. Thanks to Max Kellermann for the report and patch. See https://github.com/zeek/zeek/pull/792 * Potential Denial of Service due to buffer read overflow and/or memory leaks in Kerberos analyzer. The buffer read overflow could occur when the Kerberos message indicates it contains an IPv6 address, but does not send enough data to parse out a full IPv6 address. A memory leak could occur when processing KRB_KDC_REQ KRB_KDC_REP messages for message types that do not match a known/expected type. See https://github.com/zeek/zeek/pull/753 * Potential Denial of Service when sending many zero-length SSL/TLS certificate data. Such messages underwent the full Zeek file analysis treatment which is expensive (and meaninguless here) compared to how cheaply one can "create" or otherwise indicate many zero-length contained in an SSL message. See https://github.com/zeek/zeek/pull/748 * Potential Denial of Service due to buffer read overflow in SMB transaction data string handling. The length of strings being parsed from SMB messages was trusted to be whatever the message claimed instead of the actual length of data found in the message. See https://github.com/zeek/zeek/pull/747 * Potential Denial of Service due to null pointer dereference in FTP ADAT Base64 decoding. See https://github.com/zeek/zeek/pull/739 * Potential Denial of Service due buffer read overflow in FTP analyzer word/whitespace handling. This typically won't be a problem in most default deployments of Zeek since the FTP analyzer receives data from a ContentLine (NVT) support analyzer which first null-terminates the buffer used for further FTP parsing. See https://github.com/zeek/zeek/pull/749 Also addressed are the following bug fixes: * Use-after-free in paraglob https://github.com/zeek/paraglob/commit/d65dd0a * Invalid memory read in paraglob https://github.com/zeek/paraglob/commit/ac86ce76ac86f0ca9cd43aad74bb7b6043319fb1 * Improve Broker python binding Event validity checks https://github.com/zeek/broker/commit/2d9f4742618116e8591a37fe6c16ea2eef7269cf * Misleading Broker debug/error output https://github.com/zeek/broker/commit/0d136d187fef2b720ce3a85d7df34ab166f0edfd * malloc/delete mismatch in JSON formatting https://github.com/zeek/zeek/commit/c0d6eb9efbc133c320a963729e99251d4b6952e3 * Plugin API returning reference to temporary https://github.com/zeek/zeek/commit/ae9e79969e9d2a9de659dbc1fc81a2c94c67f54c * Memory leak in OCSP parsing when using OpenSSL 1.1 https://github.com/zeek/zeek/commit/2fbcf23f767797d321d7239f5260d16a25b16eb1 * Memory leak in Kerberos ticket decryption https://github.com/zeek/zeek/commit/53fadb2bb0e9a165dac3899290282236f770be68 * Memory leak when table-based input stream overwrites old entry https://github.com/zeek/zeek/commit/3742e5601c7390aacd1f8ce59b054e0090d7524a * Memory leak in packet filter functions https://github.com/zeek/zeek/commit/a961f0b4c4ad5a365d2078acbacaf6a380997013 * Memory leak in system_env() BIF https://github.com/zeek/zeek/commit/273eb19ff5780d8d823631f906fb2fcf2e81b312 * Memory leak of Log::Filter "config" field https://github.com/zeek/zeek/commit/bf05add5423cdeedddad21c07b08a3a65ce98431 * Memory leak in Reporter::get_weird_sampling_whitelist() BIF https://github.com/zeek/zeek/commit/3b6a2a5f4ebb04dda32f7f35fad4ed603d226a6d * Memory leaks in input framework error-handling cases https://github.com/zeek/zeek/commit/6f5f7df9705e7eff95b88203b2b4ee95d52d55b8 * Memory leak when a plugin prevents a log write operation https://github.com/zeek/zeek/commit/09578c6176e5be2f794ebbfc323ef517e45131e9 * Memory leaks due to reference-counting issues in lambdas/closures https://github.com/zeek/zeek/commit/44d922c4b5e67c32cb8ca87d6f2be5c3e5abdacc * Memory leak when creating input streams which use &type_column https://github.com/zeek/zeek/commit/51970c256b159a2bb16eb70b3200878533bf2bf9 * NTLM field access scripting error: accessing uninitialized https://github.com/zeek/zeek/commit/80469a1fde28ba4413843c43fb9a88c38e651a3c * File analysis scripting error: accessing of an uninitialized field https://github.com/zeek/zeek/commit/d9ed76c90a899b852d9b33adc96cf1a6741b6dba * Inconsistent &priority for Log::create_stream() calls https://github.com/zeek/zeek/commit/7a748526c05f473dd9a1f03db14421c88bf16cb4 * Fix potential for indefinite buffering of logs (e.g. when they are sparse/bursty) https://github.com/zeek/zeek/commit/43e54c7930a1286447d97a7f4696232b54d1491c * Dictionary::Clear() didn't reset number of entries https://github.com/zeek/zeek/commit/1e499b08318ec87afd9956ad518c91c6390e0c21 * Paraglob compile failure due to non-standard VLA usage https://github.com/zeek/paraglob/commit/903b5cf852c6d92cd885fef838e92386d8ee58f2 * Binpac cross-compilation configuration fix https://github.com/zeek/zeek/commit/d33613c2a5e2c76412b2bd7be462a133432f24b9

zeek v3.1.0

2020-02-25T21:37:36+00:00

This release primarily adds new features: configuration options, scripting language functionality, and a new "supervisor" deployment mode (alternative to ZeekControl). Performance is improved, especially the handling of SYN-scans and also JSON logging. The main I/O loop of Zeek is rewritten with better idle behavior and reduced CPU load. It also includes many smaller bug fixes and improvements. Reminder: Zeek 3.0.x is the current Long-Term Support release, receiving bug fixes until at least October 2020. New Functionality ----------------- - Add a new supervisor framework that enables Zeek to operate clusters of processes itself without any external help. The Supervisor framework provides an entirely new deployment mode for Zeek, one that supervises a set of Zeek processes that are meant to be persistent. A Supervisor automatically revives any process that dies or exits prematurely and also arranges for an ordered shutdown of the entire process tree upon its own termination. This Supervisor mode for Zeek provides the basic foundation for process configuration/management that could be used to deploy a Zeek cluster similar to what ZeekControl does, but is also simpler to integrate as a standard system service. This mode is still experimental and will evolve over time. The command-line argument of ``-j`` toggles Zeek to run in "Supervisor mode" to allow for creation and management of child processes. If you're going to test this, please note that you will need some custom script code to configure the processes you want Zeek to run. See the documentation for more information: https://docs.zeek.org/en/current/frameworks/supervisor.html - Add a new option, ``dpd_late_match_stop``, which can be used in conjuction with the option ``dpd_match_only_beginning`` and the new event ``protocol_late_match`` to help annotate the conn.log with a field to speculate on the protocol/service in cases where the DPD buffer was already exhausted and can't analyze the full connection anymore, but where there was still a late signature match. A new script, ``policy/protocols/conn/speculative-service.zeek``, was added as an example of how to perform this tuning and add a "speculative_service" field to conn.log, but it's not loaded by default. - There is now a new ``tcp_options`` event that is raised for each TCP header that contains options. - Added a new option, ``Log::print_to_log`` that can be set to automatically redirect the output from "print" statements to a real log stream (e.g. instead of writing to stdout). - There is now a new ``&on_change`` attribute that can be used to be notified of changes to tables and sets. Changed Functionality --------------------- - A C++17-capable compiler and CMake 3.0+ are now required to compile Zeek - The backwards-compability wrappers & work-arounds introduced in 3.0 for the "Bro to Zeek rename" have either changed their operation, or in some cases been removed. Generally, anything that reported a naming-related warning in 3.0 now aborts with a corresponding error message. In cases where 3.0 silently continued to accept old names, 3.1 now reports warnings. Most importantly, that's loading of scripts with ".bro" endings, which are now flagged and should be renamed. - Broker has switched versions for the underlying CAF communication library from 0.16 to 0.17. CAF has changed its wireformat between those versions, which means that previous Broker and Zeek versions won't be able to connect to the new ones. In other words, all Zeek instances, as well as other Broker clients, need to upgrade at the same time. In case of version mismatches, Broker now reports better error messages to point out the problem. - The key type of ``Known::service_store`` has changed to ``Known::AddrPortServTriplet`` and ``Known::services`` is now a table instead of just a set. - The DNS class name for Hesiod in the ``DNS::classes`` table is now spelled correctly as "C_HESIOD" instead of "C_HESOD". For reference, this class name string may appear in the dns.log file or in any custom script logic that inspects the ``qclass`` field of ``DNS::Info`` after a ``dns_request`` event. - The configuration input reader now ignores trailing spaces at the end of configuration lines. - The tcp_option event is now correctly raised. - The base scripts shipped with Zeek now use the new ``successful_connection_remove`` event instead of ``connection_state_remove`` where possible (when the logic doesn't pertain to unestablished TCP connections). There's a performance benefit to this switch, but it may potentially break custom scripts that depended on accessing or modifying state via specific ``&priority`` ordering of ``connection_state_remove`` event handlers. However, since most of Zeek's base scripts previously just used that event with the intention of writing a finalized log as the last thing executed for a given connection, and the new ``successful_connection_remove`` event handlers all run after ``connection_state_remove``, it's not likely this change to the base scripts causes any incompatibility with user scripts. There's also a new event called ``connection_successful`` and a new ``connection`` record field named "successful" to help indicate this new property of connections. - The JSON output formatters now use the RapidJSON library. This improves their performance considerably over the library that was previously used. Output from the formatters remains nearly identical. - The ``decompose_uri`` function no longer raises an error when parsing URIs with an empty port number (e.g. ``http://example.org:/``). Instead, the ``portnum`` component of the returned ``URI`` value is left uninitialized. - Replace old ``select``-based IO loop with a new architecture that doesn't spin checking for active IO sources. The new architecture now waits for the the sources to actively notify it when activity occurs and only processes data once it's ready. This helps heavily reduce the CPU usage on idle network connections. This includes a couple of breaking changes: - Only a single packet source is allowed to be specified from the command-line now. If you pass combinations of multiple ``-r`` and/or ``-i`` flags, Zeek will return an error at startup. - The IOSource API changed fairly wildly. The ``GetFds()`` and ``NextTimestamp`` methods no longer exist. If you had previously implemented a custom IOSource, you will need to look at the new API and make changes to your code to accomodate it. This does not include packet sources, which should remain functional with little to no changes, since the entirety of the changes should be in ``PktSrc``. - Remove a large number of headers from being included by various files across the entire code base, which leads to a sizeable build time improvement. This set of changes has the potential to cause plugins to not build anymore. The fixes for this potential breakage should just be a matter of including the necessary headers in the plugin code. Removed Functionality --------------------- - Removed the ``current_conns_extern`` field from the ConnStats record type. Zeek only maintains a single timer manager now, and without the manager tags that came with multiple tiemr managers, we don't track whether a connection is external anymore. Deprecated Functionality ------------------------ - The C++ API typedefs for int{8,16,32,64} and uint{8,16,32,64} are deprecated in favor of the real types they alias. E.g. use int8_t instead of int8. - The C++ API functions "safe_snprintf" and "safe_vsnprintf" are deprecated. Use "snprintf" and "vsnprintf" instead.

zeek v3.0.3

2020-03-10T18:58:00+00:00

zeek v3.1.1

2020-03-10T19:00:22+00:00

This release fixes the following bugs: * Potential high CPU load due to race in Broker data stores https://github.com/zeek/broker/pull/97 * Run-time script errors do not pop global frame/call stack (exhausting memory) https://github.com/zeek/zeek/commit/36557f3086c95f079aa3e69c26a34adeb493c73f * Wrong 'bro' symlink in binary packaging mode https://github.com/zeek/zeek/commit/b324fecc0d247830189ef895dae77531af6bc9dd * Install full RapidJSON include tree to allow some external plugins to compile https://github.com/zeek/zeek/commit/0d97d3f0c44fb506942b6df72f75a8613d8164ab Reminder: Zeek 3.0.x is the Long-Term Support release, receiving bug fixes until at least October 2020 while Zeek 3.1.x is the current feature release, receiving bug fixes until approximately July 2020 when the 3.2.x release series begins.

zeek v3.0.4

2020-04-14T19:55:37+00:00

This release fixes the following security issue: * Fix stack overflow in POP3 analyzer https://github.com/zeek/zeek/commit/bb3250c28ea6600ec82dd4d98186d15de1572901 Thanks to Matteo Rizzo (Google) for reporting this issue. Impact: An attacker can crash Zeek remotely via crafted packet sequence. Affected versions: All versions since at least late 2010 and possibly earlier. POP3Analyzer parses the POP3 protocol. The affected code processes AUTH_PLAIN authentication requests, in which the client authenticates to the server by sending its credentials to the server as base64-encoded plaintext. Zeek allocates a variable length array (VLA) on the stack that is as big as the credentials sent by the client. GCC implements VLA allocation by subtracting the size of the VLA from the stack pointer. When the credentials sent by the client are larger than the available stack space, the stack pointer lands into the unmapped memory that the OS places above the stack. When that happens, the call to memcpy immediately below causes Zeek to crash with a segmentation fault because the destination address now points to unmapped memory. An attacker can crash Zeek by sending a very long credential line. This class of bug is normally turned into a memory corruption primitive by moving the stack pointer into a writable region (for example a library) and overwriting that region. However in this case it is not possible to move the stack pointer far enough because ContentLineAnalyzer limits the length of the credentials to slightly less than 16MiB by default. Therefore it should not be possible to gain remote code execution using this bug, but only to crash Zeek. Also fixed are the following bugs: * Fix use-after-free in Zeek lambda functions with uninitialized locals https://github.com/zeek/zeek/issues/845 * Fix buffer overflow due to tables/records created at parse-time not rebuilt on record `redef` https://github.com/zeek/zeek/pull/860 * Fix SMB NegotiateContextList parsing https://github.com/zeek/zeek/pull/869 * Fix binpac flowbuffer frame length parsing doing too much bounds checking https://github.com/zeek/zeek/pull/873 * Fix parsing ERSPAN III optional sub-header https://github.com/zeek/zeek/commit/42dc2906afaa3b07aebb8dd8474d1fc8510e8d50 * Fix bug in intel indicator normalization https://github.com/zeek/zeek/pull/883 * Fix connection duration thresholding https://github.com/zeek/zeek/pull/899 * Fix `X509Common.h` header include for external plugins https://github.com/zeek/zeek/commit/c83567246ee9d86ca695787821172e3ebe00884a * Fix incorrect targeting of node-specific Broker/Cluster messages https://github.com/zeek/zeek/commit/ce9183a2ed776e3815ecb3e7b89eb0a7e43852b2 https://github.com/zeek/broker/commit/0d04bf43131ade7000506711cbe43c068dab2471 Reminder: Zeek 3.0.x is a Long-Term Support release, receiving bug fixes until at least October 2020.

zeek v3.1.2

2020-04-14T19:57:48+00:00

This release fixes the following security issue: * Fix stack overflow in POP3 analyzer https://github.com/zeek/zeek/commit/bb3250c28ea6600ec82dd4d98186d15de1572901 Thanks to Matteo Rizzo (Google) for reporting this issue. Impact: An attacker can crash Zeek remotely via crafted packet sequence. Affected versions: All versions since at least late 2010 and possibly earlier. POP3Analyzer parses the POP3 protocol. The affected code processes AUTH_PLAIN authentication requests, in which the client authenticates to the server by sending its credentials to the server as base64-encoded plaintext. Zeek allocates a variable length array (VLA) on the stack that is as big as the credentials sent by the client. GCC implements VLA allocation by subtracting the size of the VLA from the stack pointer. When the credentials sent by the client are larger than the available stack space, the stack pointer lands into the unmapped memory that the OS places above the stack. When that happens, the call to memcpy immediately below causes Zeek to crash with a segmentation fault because the destination address now points to unmapped memory. An attacker can crash Zeek by sending a very long credential line. This class of bug is normally turned into a memory corruption primitive by moving the stack pointer into a writable region (for example a library) and overwriting that region. However in this case it is not possible to move the stack pointer far enough because ContentLineAnalyzer limits the length of the credentials to slightly less than 16MiB by default. Therefore it should not be possible to gain remote code execution using this bug, but only to crash Zeek. Also fixed are the following bugs: * Fix use-after-free in Zeek lambda functions with uninitialized locals https://github.com/zeek/zeek/issues/845 * Fix buffer overflow due to tables/records created at parse-time not rebuilt on record `redef` https://github.com/zeek/zeek/pull/860 * Fix SMB NegotiateContextList parsing https://github.com/zeek/zeek/pull/869 * Fix binpac flowbuffer frame length parsing doing too much bounds checking https://github.com/zeek/zeek/pull/873 * Fix parsing ERSPAN III optional sub-header https://github.com/zeek/zeek/commit/42dc2906afaa3b07aebb8dd8474d1fc8510e8d50 * Fix bug in intel indicator normalization https://github.com/zeek/zeek/pull/883 * Fix connection duration thresholding https://github.com/zeek/zeek/pull/899 * Fix using patterns as table/set indices https://github.com/zeek/zeek/pull/902 * Fix `X509Common.h` header include for external plugins https://github.com/zeek/zeek/commit/c83567246ee9d86ca695787821172e3ebe00884a * Fix incorrect targeting of node-specific Broker/Cluster messages https://github.com/zeek/zeek/commit/ce9183a2ed776e3815ecb3e7b89eb0a7e43852b2 https://github.com/zeek/broker/commit/0d04bf43131ade7000506711cbe43c068dab2471 Reminder: Zeek 3.0.x is the Long-Term Support release, receiving bug fixes until at least October 2020 while Zeek 3.1.x is the current feature release, receiving bug fixes until approximately July 2020 when the 3.2.x release series begins.

zeek v3.0.5

2020-04-14T21:55:32+00:00

This release is the same as [v3.0.4](https://github.com/zeek/zeek/releases/tag/v3.0.4), but additionally fixes compilation on various platforms with older compiler, e.g. GCC 4.8.x (see patch at https://github.com/zeek/zeek/commit/3ad19762770c567edc3498b3c1f9f216f46970b0) Reminder: Zeek 3.0.x is a Long-Term Support release, receiving bug fixes until at least October 2020.

zeek v3.0.6

2020-05-06T20:04:00+00:00

This release fixes the following security issues: * Fix buffer over-read in Ident analyzer https://github.com/zeek/zeek/pull/925 Thanks to Max Kellermann for reporting and patching. * Fix SSL scripting error leading to uninitialized field access and memory leak https://github.com/zeek/zeek/commit/b749dda5205f1b01aeee03c5874acae7c543f0c5 Thanks to Justin Azoff for reporting. * Fix POP3 analyzer global buffer over-read https://github.com/zeek/zeek/commit/280bf567865d8ff6353f85d8863a6bc85dd9afd1 Thanks to Justin Azoff for reporting and patching. * Fix potential stack overflows due to use of Variable-Length-Arrays Parts of https://github.com/zeek/zeek/pull/912: * BIFs `bytestring_to_hexstr()` and `hexstr_to_bytestring()` * `socks-analyzer.pac`: `array_to_string()` * SMB, NTLM, and RDP analyzers use of `utf16_bytestring_to_utf8_val()` * `smb-strings.pac`: `uint8s_to_stringval()` and `extract_string()` Also fixed are the following bugs: * Fix unusable `subscriber.poll()` method in Broker Python bindings https://github.com/zeek/broker/pull/110 * Fix uninitialized field access in `ssl/log-hostcerts-only.zeek` https://github.com/zeek/zeek/pull/916 * Fix missing default function for Kerberos constant-lookup-tables https://github.com/zeek/zeek/pull/918 * Fix cloning of `TypeType` values https://github.com/zeek/zeek/pull/933 * Remove misleading error message on empty bloomfilter lookup https://github.com/zeek/zeek/pull/930 * Fix `misc/stats.zeek` skipping log entry on termination https://github.com/zeek/zeek/commit/ccdaf5f111936d9c7e6c23995eab1e5f41872894 Reminder: Zeek 3.0.x is a Long-Term Support release, receiving bug fixes until at least October 2020.

zeek v3.1.3

2020-05-06T20:06:53+00:00

This release fixes the following security issues: * Fix buffer over-read in Ident analyzer https://github.com/zeek/zeek/pull/925 Thanks to Max Kellermann for reporting and patching. * Fix SSL scripting error leading to uninitialized field access and memory leak https://github.com/zeek/zeek/commit/b749dda5205f1b01aeee03c5874acae7c543f0c5 Thanks to Justin Azoff for reporting. * Fix POP3 analyzer global buffer over-read https://github.com/zeek/zeek/commit/280bf567865d8ff6353f85d8863a6bc85dd9afd1 Thanks to Justin Azoff for reporting and patching. * Fix potential stack overflows due to use of Variable-Length-Arrays Parts of https://github.com/zeek/zeek/pull/912: * BIFs `bytestring_to_hexstr()` and `hexstr_to_bytestring()` * `socks-analyzer.pac`: `array_to_string()` * SMB, NTLM, and RDP analyzers use of `utf16_bytestring_to_utf8_val()` * `smb-strings.pac`: `uint8s_to_stringval()` and `extract_string()` Also fixed are the following bugs: * Fix unusable `subscriber.poll()` method in Broker Python bindings https://github.com/zeek/broker/pull/110 * Fix uninitialized field access in `ssl/log-hostcerts-only.zeek` https://github.com/zeek/zeek/pull/916 * Fix missing default function for Kerberos constant-lookup-tables https://github.com/zeek/zeek/pull/918 * Fix cloning of `TypeType` values https://github.com/zeek/zeek/pull/933 * Remove misleading error message on empty bloomfilter lookup https://github.com/zeek/zeek/pull/930 * Fix `misc/stats.zeek` skipping log entry on termination https://github.com/zeek/zeek/commit/ccdaf5f111936d9c7e6c23995eab1e5f41872894 * Offline pcap processing no longer initializes `network_time` before first events after `zeek_init` get dispatched https://github.com/zeek/zeek/commit/1b190906c7c2e26dad058176ac969f513e5e391f * Ensure time moves forward when suspending a pcap file IO source https://github.com/zeek/zeek/pull/950 Reminder: Zeek 3.0.x is the Long-Term Support release, receiving bug fixes until at least October 2020 while Zeek 3.1.x is the current feature release, receiving bug fixes until approximately July 2020 when the 3.2.x release series begins.

zeek v3.0.7

2020-06-09T18:50:09+00:00

This release fixes the following security issues: * Fix potential stack overflow in NVT analyzer https://github.com/zeek/zeek/commit/3b51d72aa165fd7e4baac260b7cb8103a0a02822 The NVT_Analyzer (e.g. as instantiated to support the FTP analyzer) uses a recursive parsing function that may only advance one byte at a time and can easily cause a stack overflow as a result. Credit to OSS-Fuzz for discovery https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=22898 * Fix NVT analyzer memory leak from multiple telnet authn name options https://github.com/zeek/zeek/commit/e532335f9718bd902e2c600827c975824c0fd0a6 Credit to OSS-Fuzz for discovery https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=23069 * Fix multiple content-transfer-encoding headers causing a memory leak https://github.com/zeek/zeek/commit/0195880c4842cb2dbb8ea551078fb49f4f1e896f The MIME analyzer leaks memory if it sees many content-transfer-encoding headers or also if it see many multipart boundary parameters. Credit to OSS-Fuzz for discovery https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=22871 * Fix potential leak of Analyzers added to tree during Analyzer::Done https://github.com/zeek/zeek/commit/d2eb701b7e8a2444abcc5d8a229fce3710c1ae2b It may be possible for remote sources of analyzed packets to specifically craft traffic to trigger this behavior. Credit to OSS-Fuzz for discovery https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=22630 * Prevent IP fragment reassembly on packets without minimal IP header https://github.com/zeek/zeek/commit/a2f2f7a0dd8a2536a1688c63bc34431f17baae54 The IP fragment reassembly process assumes a packet contains at least the minimum IP header, but such a check did not previously occur, resulting in a heap buffer over-read. For example, a self-reported IPv4 IHL field with a value less than minimum IPv4 header length of 20 bytes. Such packets likely aren't routable on their own, but one can create an artifical pcap like that or possibly encapsulate it within another protocol to trigger this bug. Also fixed are the following bugs: * Limit rate of logging MaxMind DB diagnostic messages https://github.com/zeek/zeek/pull/963 * Fix wrong return value type for `topk_get_top()` BIF https://github.com/zeek/zeek/pull/996 * Fix opaque Broker types lacking a Type after (de)serialization https://github.com/zeek/zeek/pull/984 * Fix lack of descriptive printing for intervals converted from `double_to_interval()` https://github.com/zeek/zeek/pull/994/commits/e17487e79910fee1bb7ced30446dda0fdf4bcae4 * Fix some cases of known-services not being logged https://github.com/zeek/zeek/pull/965 https://github.com/zeek/zeek/commit/2f918ed9b24ae3aca2a7c36bf5b539e0a49b2f96 Reminder: Zeek 3.0.x is a Long-Term Support release, receiving bug fixes until at least October 2020.

zeek v3.1.4

2020-06-09T18:53:25+00:00

This release fixes the following security issues: * Fix potential stack overflow in NVT analyzer https://github.com/zeek/zeek/commit/3b51d72aa165fd7e4baac260b7cb8103a0a02822 The NVT_Analyzer (e.g. as instantiated to support the FTP analyzer) uses a recursive parsing function that may only advance one byte at a time and can easily cause a stack overflow as a result. Credit to OSS-Fuzz for discovery https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=22898 * Fix NVT analyzer memory leak from multiple telnet authn name options https://github.com/zeek/zeek/commit/e532335f9718bd902e2c600827c975824c0fd0a6 Credit to OSS-Fuzz for discovery https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=23069 * Fix multiple content-transfer-encoding headers causing a memory leak https://github.com/zeek/zeek/commit/0195880c4842cb2dbb8ea551078fb49f4f1e896f The MIME analyzer leaks memory if it sees many content-transfer-encoding headers or also if it see many multipart boundary parameters. Credit to OSS-Fuzz for discovery https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=22871 * Fix potential leak of Analyzers added to tree during Analyzer::Done https://github.com/zeek/zeek/commit/d2eb701b7e8a2444abcc5d8a229fce3710c1ae2b It may be possible for remote sources of analyzed packets to specifically craft traffic to trigger this behavior. Credit to OSS-Fuzz for discovery https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=22630 * Prevent IP fragment reassembly on packets without minimal IP header https://github.com/zeek/zeek/commit/a2f2f7a0dd8a2536a1688c63bc34431f17baae54 The IP fragment reassembly process assumes a packet contains at least the minimum IP header, but such a check did not previously occur, resulting in a heap buffer over-read. For example, a self-reported IPv4 IHL field with a value less than minimum IPv4 header length of 20 bytes. Such packets likely aren't routable on their own, but one can create an artifical pcap like that or possibly encapsulate it within another protocol to trigger this bug. Also fixed are the following bugs: * Fix compilation on Fedora 32 (GCC 10.0.1) https://github.com/zeek/zeek/commit/695457fe44c4adfbf2edab955fee0074ef365980 * Fix crash when using some deprecated environment variables https://github.com/zeek/zeek/commit/1c08be1c0f89e9a8437ee230fbdbcd306485847b * Fix use on CentOS 6 (Linux kernel < 3.8) https://github.com/mheily/libkqueue/commit/8707307ec5e09792782916205fb8de4f52ed24bb * Limit rate of logging MaxMind DB diagnostic messages https://github.com/zeek/zeek/pull/963 * Fix wrong return value type for `topk_get_top()` BIF https://github.com/zeek/zeek/pull/996 * Fix opaque Broker types lacking a Type after (de)serialization https://github.com/zeek/zeek/pull/984 * Fix lack of descriptive printing for intervals converted from `double_to_interval()` https://github.com/zeek/zeek/pull/994/commits/e17487e79910fee1bb7ced30446dda0fdf4bcae4 * Fix some cases of known-services not being logged https://github.com/zeek/zeek/pull/965 https://github.com/zeek/zeek/commit/2f918ed9b24ae3aca2a7c36bf5b539e0a49b2f96 Reminder: Zeek 3.0.x is the Long-Term Support release, receiving bug fixes until at least October 2020 while Zeek 3.1.x is the current feature release, receiving bug fixes until approximately July 2020 when the 3.2.x release series begins.

zeek v3.0.8

2020-07-27T21:57:55+00:00

This release fixes the following security issues: * Fix potential DNS analyzer stack overflow The Contents_DNS analyzer used a recursive message parsing function that determined the size of the next message from the input packet-data itself. A packet containing a sequence of many small messages could cause a stack overflow since a recursion happened after processing each message. This issue is remotely exploitable with Denial of Service potential due to crashing the Zeek process. Credit to OSS-Fuzz for discovery https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=24272 * Fix potential NetbiosSSN analyzer stack overflow The Contents_NetbiosSSN analyzer used a recursive message parsing function that determined the size of the next message from the input packet-data itself. A packet containing a sequence of many small messages could cause a stack overflow since a recursion happened after processing each message. This issue is remotely exploitable with Denial of Service potential due to crashing the Zeek process. This release fixes the following bugs: * Fix DHCP Client ID Option misformat for Hardware Type 0 https://github.com/zeek/zeek/pull/1003 * Fix/allow copying/cloning of `opaque of Broker::Store` https://github.com/zeek/zeek/pull/1028 * Fix ConnPolling memory over-use https://github.com/zeek/zeek/pull/1035 * Fix compress_path not normalizing some paths correctly https://github.com/zeek/zeek/issues/1041 https://github.com/zeek/zeek/pull/1050 https://github.com/zeek/zeek/commit/38cd56a3dba076c625045c2b433b5f956ed118c9 * Fix integer conversion error for Tag subtypes/enums https://github.com/zeek/zeek/issues/1062 https://github.com/zeek/zeek/pull/1064 * Fix `bro_prng()` results not staying within modulus https://github.com/zeek/zeek/issues/1076 https://github.com/zeek/zeek/commit/0f4eb9af0265a256a567ba4203950269b4b52d28 * Prevent providing a `0` seed to `bro_prng()` since the LCG parameters don't allow that https://github.com/zeek/zeek/issues/1076 https://github.com/zeek/zeek/commit/887b53b7f330ad93b090d6be0c4733690a58ff89 Reminder: Zeek 3.0.x is a Long-Term Support release, receiving bug fixes until at least October 2020.

zeek v3.1.5

2020-07-27T21:58:01+00:00

This release fixes the following security issues: * Fix potential DNS analyzer stack overflow The Contents_DNS analyzer used a recursive message parsing function that determined the size of the next message from the input packet-data itself. A packet containing a sequence of many small messages could cause a stack overflow since a recursion happened after processing each message. This issue is remotely exploitable with Denial of Service potential due to crashing the Zeek process. Credit to OSS-Fuzz for discovery https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=24272 * Fix potential NetbiosSSN analyzer stack overflow The Contents_NetbiosSSN analyzer used a recursive message parsing function that determined the size of the next message from the input packet-data itself. A packet containing a sequence of many small messages could cause a stack overflow since a recursion happened after processing each message. This issue is remotely exploitable with Denial of Service potential due to crashing the Zeek process. This release fixes the following bugs: * Fix DHCP Client ID Option misformat for Hardware Type 0 https://github.com/zeek/zeek/pull/1003 * Fix/allow copying/cloning of `opaque of Broker::Store` https://github.com/zeek/zeek/pull/1028 * Fix ConnPolling memory over-use https://github.com/zeek/zeek/pull/1035 * Fix compress_path not normalizing some paths correctly https://github.com/zeek/zeek/issues/1041 https://github.com/zeek/zeek/pull/1050 * Fix integer conversion error for Tag subtypes/enums https://github.com/zeek/zeek/issues/1062 https://github.com/zeek/zeek/pull/1064 * Fix `bro_prng()` results not staying within modulus https://github.com/zeek/zeek/issues/1076 https://github.com/zeek/zeek/commit/0f4eb9af0265a256a567ba4203950269b4b52d28 * Prevent providing a `0` seed to `bro_prng()` since the LCG parameters don't allow that https://github.com/zeek/zeek/issues/1076 https://github.com/zeek/zeek/commit/887b53b7f330ad93b090d6be0c4733690a58ff89 * Fix mishandling of `getrandom()` to seed RNG (caused unrandom/deterministic RNG -- opposite of what's desired/intended) https://github.com/zeek/zeek/issues/1076 https://github.com/zeek/zeek/commit/dba764386b38a5b4ac974eb74f142c6ae107acb0 Reminder: Zeek 3.0.x is the Long-Term Support release, receiving bug fixes until at least October 2020 while Zeek 3.1.x is the current feature release, receiving bug fixes until approximately July 2020 when the 3.2.x release series begins.

zeek v3.2.0

2020-07-27T21:58:08+00:00

Reminder: Zeek 3.0.x is the Long-Term Support release, receiving bug fixes until at least November 2020 while Zeek 3.2.x is the current feature release, receiving bug fixes until approximately November 2020 when the next 4.0.x LTS release series is expected to begin. New Functionality ----------------- - X509 Certificate caching: Zeek now caches certificates if they have (by default) been encountered more than 10 times in 62 seconds. Information for cached certificates is retained; if the certificate is encountered again it does not have to be re-parsed and already existing information is used to raise the events. This should especially help with performance in environments where the same certificates are seen very often. Certificate caching is very configureable; it is possible to disable the feature, change the time intervals or even suppress X509 events. For details see ``scripts/base/files/x509/main.zeek``. - Add parsing support for Remote Desktop Protocol UDP Transport Extension (RDPEUDP versions 1 and 2). This primarily only adds "rdpeudp" to connection record service fields when an RDPEUDP session handhake is detected, but also provides a few other events related to the RDPEUDP connection establishment. - Add the ``udp_content_ports`` configuration option. Any port added to this set will cause the ``udp_contents`` event to be raised. - Add the ``udp_content_delivery_ports_use_resp`` option which can be used to specify how the destination port for the ``udp_content_delivery_ports_orig`` and ``udp_content_delivery_ports_orig`` options is determined. The current value keeps behavior as it was in previous versions of Zeek. - Add a file signature to identify ISO9660 disk images (application/x-iso9660-image) - Add file signature to identify Python bytecode (application/x-python-bytecode) - Events and hooks are now allowed to have multiple, alternate prototype declarations. This allows for extending event/hook parameters in a way that won't break an existing user's handlers and also allows users to define their own custom event/hook prototypes that consume a subset of the parameters (convenience of typing/memory/etc). This feature is documented in detail here: https://docs.zeek.org/en/current/script-reference/types.html#type-event - Add ``flags`` parameters to ``rdp_connect_request``, ``rdp_negotiation_response``, and ``rdp_negotiation_failure`` events. - ``Reporter::conn_weird`` now correctly handles weirds for expired connections, for which no connection state information is available in the core anymore. These cases will raise the new ``expired_conn_weird`` event. - Broker Store table synchronization (experimental). Zeek now supports synchronizing tables/sets across clusters using a backing Broker store. The same feature also allows persistent storage of data in tables/sets over Zeek restarts. This feature is implemented using the new ``&backend`` attribute. To synchronize a table over a cluster, you can, e.g., use: global t: table[string] of count &backend=Broker::MEMORY; This feature is documented in detail here: https://docs.zeek.org/en/current/frameworks/broker.html#broker-store-backed-zeek-tables-for-data-synchronization-and-persistence Note: this feature is experimental and the syntax/specifics can change in the future. Changed Functionality --------------------- - Several C++ functions have been changed to pass smart pointers (``class IntrusivePtr<>``) instead of raw pointers. This makes the code more robust. External plugins may need to be updated to this API change. - BIFs that use `@ARG@`, `@ARGS@`, or `@ARGC@` may break since their type has changed: BIF arguments are now passed as a ``std::vector>`` rather than a ``val_list`` (i.e. ``List``). - RocksDB support for Broker data stores is now opt-in instead of automatically detected and used at configuration-time. Use the ``--enable-rocksdb`` and ``--with-rocksdb=`` flags to opt-in. - At the C++ level, a large number of functions had their return values and/or arguments changed to use ``bool`` types instead of ``int``. This includes some virtual methods, which may cause build failures in plugins that were overriding those methods. Those plugins will need to be updated to match these API changes. To make sure to catch changes you need to make to your plugin, double-check that all virtual method implementations use "override". - Many C++ classes were marked "final" which also has some performance benefits due to devirtualization optimizations. - Data members of many C++ classes/structs were reordered to achieve better packing and smaller memory footprint. - "using namespace std" was removed from the Zeek header files; Zeek now always explicitly specifies std when using STL functionality in headers. This may necessitate small changes in external plugins, if they relied on the using statement in Zeek headers. - The ``connection_external`` event was removed. This functionality that could raise this event (injecting connections via broccoli) was removed a while ago; the event handler served no purpose anymore. - Reorganize the file signatures to break them out into more groups. This may break scripts that had been explicitly loading any signature files that moved. - The DCE/RPC operation string of "NetrLogonSamLogonWithFlags" has been corrected from "NetrLogonSameLogonWithFlags". - ``AsRecord()`` and ``AsNonConstRecord()`` have changed to return ``std::vector>*``. - ``AsVector()`` has changed to return ``std::vector>*``. - Moved a large number of classes from the global namespace into either the ``zeek`` or ``zeek::detail`` namespace. See https://github.com/zeek/zeek/issues/266 for the rationale behind these changes. Most types that were moved and functions that used them have been marked as deprecated and will generate compiler warnings if used (a few exceptions will not generate compiler warnings, but the Deprecated Functionality section below will mention those ones specifically). This includes a number of renames to classes, removing 'Bro' from their names. Deprecation warnings should notify plugin developers of these name changes. Removed Functionality --------------------- - The fmt() function which takes a va_list argument is replaced, use the new vfmt() function for equivalent functionality. The former is deprecated because overloading it with the variadic fmt() function can cause the unintended overload to be chosen depending on how the platform implements va_list. Deprecated Functionality ------------------------ - The ``plugin::Plugin::HookCallFunction()`` method is deprecated. Note that compilers will not emit a deprecation warning, but the replacement method to now use is called ``HookFunctionCall`` and uses ``IntrusivePtr`` arguments and return value. - The ``Func::Call(val_list*, ...)`` method is now deprecated. Use ``Invoke()`` instead which takes a ``zeek::Args`` (``std::vector>``). There's also a variadic template for ``Invoke()`` that forwards all arguments into a ``zeek::Args`` for you. - The ``EventMgr::QueueEvent()`` and ``EventMgr::QueueEventFast()`` methods are now deprecated, use ``EventMgr::Enqueue()`` instead. - The ``Connection::ConnectionEvent()``, ``Connection::Event()``, and ``Connection::ConnectionEventFast()`` methods are now deprecated, use ``Connection::EnqueueEvent()`` instead. - The ``file_analysis::File::FileEvent()`` methods taking ``val_list`` arguments are now deprecated, use the overload that takes a ``zeek::Args`` instead. - The ``analyzer::Analyzer::ConnectionEvent()``, ``analyzer::Analyzer::Event``, and ``analyzer::Analyzer::ConectionEventFast()`` methods are deprecated, use ``analyzer::Analyzer::EnqueueConnEvent()`` instead. - All ``val_mgr`` methods starting with "Get" are deprecated, use the new ``val_mgr`` methods that return ``IntrusivePtr``. - ``Connection::BuildConnVal()`` is deprecated, use ``Connection::ConnVal()``. - ``Analyzer::BuildConnVal()`` is deprecated, use ``Analyzer::ConnVal()``. - ``BifEvent::generate_`` functions are deprecated, use ``zeek::BifEvent::enqueue_``. - ``binpac::bytestring_to_val()`` is deprecated, use ``binpac::to_stringval()``. - ``binpac::string_to_val()`` is deprecated, use ``StringVal`` constructor. - Returning ``Val*`` from BIFs is deprecated, return ``IntrusivePtr`` instead. - Various methods of converting protocol structures, like IP or packet headers, to associated ``Val`` type are now deprecated, the deprecation warning message will advise what new method to use instead. - Various methods of ``Tag`` classes are deprecated with the warning message advising what new method to use instead. - The ``utf16_bytestring_to_utf8_val()`` function is deprecated, use ``utf16_to_utf8_val()`` instead. - ``RecordType::FieldType()`` is deprecated, use ``RecordType::GetFieldType()`` - ``BroType::HasField()`` and ``BroType::FieldType()`` are deprecated, use the methods of ``RecordType`` directly. - ``BroType::YieldType()`` is deprecated, use ``BroType::Yield()``. - ``ID::AsType()`` is deprecated, use ``ID::IsType()`` and ``ID::GetType()``. - ``ID::Type()`` is deprecated, use ``ID::GetType()``. - ``ID::ID_Val()`` is deprecated, use ``ID::GetVal()``. - ``internal_type()`` is deprecated, use ``zeek::id::find_type()``. - ``internal_val()`` and ``internal_const_val()`` are deprecated, use ``zeek::id::find_val()`` or ``zeek::id::find_const()``. - ``internal_func()`` is deprecated, use ``zeek::id::find_func()``. - ``opt_internal_val()`` is deprecated, use ``lookup_ID()`` or ``zeek::id::find_val()``. - ``Val::Type()`` is deprecated, use ``Val::GetType``. - Most global type/value pointers in NetVar.h are deprecated, but one can still always perform the lookup themselves. - ``Scope::Lookup()`` is deprecated, use ``Scope::Find()``. - All generated ``BroType*`` names in the ``BifType::`` namespaces are deprecated, but there's an equivalent name in ``zeek::BifType::`` of ``IntrusivePtr`` type to use instead. - All generated ``BifConst::`` names are deprecated, but there's an equivalent name now in ``zeek::BifCont::``, and changed to ``IntrusivePtr`` if the old name was some ``Val*`` type. - Constructors for ``Val`` types that take a ``BroType*`` are all generally deprecated, with alternatives that instead take an ``IntrusivePtr`` argument. - ``FuncType::Args()`` is deprecated, use ``FuncType::Params()``. - ``FuncType::ArgTypes()`` is deprecated, use ``FuncType::ParamList()``. - ``RecordVal::Assign(int, Val*)`` is deprecated, use the overload taking ``IntrusivePtr``. - ``RecordVal::Lookup(int)`` is deprecated, use ``RecordVal::GetField(int)``. - ``RecordVal::LookupWithDefault(int)`` is deprecated, use ``RecordVal::GetFieldOrDefault(int)``. - ``RecordVal::Lookup(const char*, bool)`` is deprecated, use either ``RecordVal::GetField()`` or ``RecordVal::GetFieldOrDefault()``. - ``TableVal::Assign`` methods taking raw ``Val*`` are deprecated, use the overloads taking ``IntrusivePtr``. - ``TableVal::Lookup()`` is deprecated, use ``TableVal::Find()`` or ``TableVal::FindOrDefault()``. - ``VectorVal::Assign`` and ``Insert`` methods taking raw ``Val*`` are deprecated, use the methods that take ``IntrusivePtr``. - ``VectorVal::Lookup()`` is deprecated, use ``VectorVal::At()``. - The file analysis/analyzer API has deprecated methods taking raw ``RecordVal*`` for analyzer arguments and replaced those with methods taking ``IntrusivePtr``. - The ``Val(double, TypeTag)`` constructor is deprecated, use either ``IntervalVal()``, ``TimeVal()`` or ``DoubleVal()`` constructors. - The "BroString.h" file is deprecated, use "ZeekString.h" - The str_split() BIF is deprecated, use str_split_indices(). Note that the former returns a vector with indices starting at 1 while the later returns a vector with indices starting at 0. - The ``icmp_conn`` parameter of ICMP events is deprecated, there's an alternate version with an ``icmp_info`` parameter to use instead. The ``icmp_conn`` record passed to ICMP events has always been re-used amongst all events within an ICMP "connection", so the ``itype``, ``icode``, ``len``, and ``hlim`` fields as inspected in handlers never appears to change even if the underlying packet data has different values for those fields. However, it's not known if anyone relied on that behavior, so the new ``icmp_info`` record is introduced with the more-expected behavior of being created and populated for each new event. It also removes the orig_h/resp_h fields since those are redundant with what's already available in the connection parameter. - External plugins should include Zeek header files like ``#include `` instead of ``#include ``. The later style is considered deprecated. Reliance on ``zeek-config --include_dir`` to contain ``$prefix/include/zeek`` is also deprecated: its replacement output is expected to be just ``$prefix/include``, with it currently outputting both paths, delimited by a colon, during the deprecation period.

zeek v3.2.0-rc1

2020-07-27T21:58:08+00:00

zeek v3.0.10

2020-09-09T20:10:20+00:00

This release fixes the following security issues: * The AYIYA and GTPv1 parsing/decapsulation logic may leak memory if the inner packet uses the same connection tuple as the outer packet while also having another level of encapsulation within the inner packet using the same tunneling protocol, respectively (AYIYA or GTPv1). These leaks have potential for remote exploitation to cause Denial of Service via resource exhaustion. Credit to OSS-Fuzz for discovery https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=25256 This release fixes the following bugs: * Fix Input Framework 'change' events for 'set' destinations https://github.com/zeek/zeek/issues/1083 https://github.com/zeek/zeek/pull/1087 * Fix reported body-length of HTTP messages w/ sub-entities https://github.com/zeek/zeek/pull/1107 Note: the above changes were also part of v3.0.9, but a compilation failure related to the use of C++17 nested namespaces was discovered and fixed for the v3.0.10 official release announcement with v3.0.9 never receiving an official announcement. Reminder: Zeek 3.0.x is a Long-Term Support release, receiving bug fixes until at least November 2020.

zeek v3.2.1

2020-09-09T20:10:51+00:00

This release fixes the following security issues: * The AYIYA and GTPv1 parsing/decapsulation logic may leak memory if the inner packet uses the same connection tuple as the outer packet while also having another level of encapsulation within the inner packet using the same tunneling protocol, respectively (AYIYA or GTPv1). These leaks have potential for remote exploitation to cause Denial of Service via resource exhaustion. Credit to OSS-Fuzz for discovery https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=25256 This release fixes the following bugs: * Exclude installing "zeek -> ." include-dir symlink https://github.com/zeek/zeek/commit/bc3df067376ea80232f57393e69c84a6e045212d * Fix build for PowerPC architecture https://github.com/zeek/zeek/issues/1150 https://github.com/zeek/zeek/commit/e8efab541b49e8202c3896f9219334788e45d40e https://github.com/zeek/zeek/commit/05f7e3fa4353a30a8b105c551f5cacb4f7ff49aa * Fix ftp data-channel minimization function not returning a value https://github.com/zeek/zeek/issues/1120 * Fix `zeek -NN` not printing canonical file analyzer names https://github.com/zeek/zeek/pull/1136 * Fix closing timestamp of rotated log files in supervised-cluster mode https://github.com/zeek/zeek/commit/99d9a3a48c9c2469d6bc8f58add43901ab901901 Reminder: Zeek 3.0.x is the Long-Term Support release, receiving bug fixes until at least November 2020 while Zeek 3.2.x is the current feature release, receiving bug fixes until approximately November 2020 when the next 4.0.x LTS release series is expected to begin.

zeek v3.0.11

2020-10-07T19:00:40+00:00

This release fixes the following security issues: * Fix multipart MIME leak of sub-part found after closing-boundary After detecting a closing-boundary for a given multipart MIME entity, it enters into an "end of data" state, however any subsequent boundary delimiter could still cause the allocation of a sub-entity object that is never released due to cleanup logic being bypassed upon finding the "end of data" state already reached. This change prevents allocation/processing of sub-entities after the "end of data" state is reached (e.g. from detecting a multipart closing-boundary). This new behavior still aligns with RFC 2046 expectations: "There appears to be room for additional information prior to the first boundary delimiter line and following the final boundary delimiter line. These areas should generally be left blank, and implementations must ignore anything that appears before the first boundary delimiter line or after the last one." This leak has potential for remote exploitation and cause for Denial of Service via resource exhaustion. Credit to OSS-Fuzz for discovery https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=26027 (Link to details becomes public 30 days after patch release) This release fixes the following bugs: * Fix incorrect `RSTOS0` `conn_state` determinations https://github.com/zeek/zeek/issues/1164 https://github.com/zeek/zeek/pull/1166 Reminder: Zeek 3.0.x is a Long-Term Support release, receiving bug fixes until at least December 2020.

zeek v3.2.2

2020-10-07T19:00:46+00:00

This release fixes the following security issues: * Fix multipart MIME leak of sub-part found after closing-boundary After detecting a closing-boundary for a given multipart MIME entity, it enters into an "end of data" state, however any subsequent boundary delimiter could still cause the allocation of a sub-entity object that is never released due to cleanup logic being bypassed upon finding the "end of data" state already reached. This change prevents allocation/processing of sub-entities after the "end of data" state is reached (e.g. from detecting a multipart closing-boundary). This new behavior still aligns with RFC 2046 expectations: "There appears to be room for additional information prior to the first boundary delimiter line and following the final boundary delimiter line. These areas should generally be left blank, and implementations must ignore anything that appears before the first boundary delimiter line or after the last one." This leak has potential for remote exploitation and cause for Denial of Service via resource exhaustion. Credit to OSS-Fuzz for discovery https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=26027 (Link to details becomes public 30 days after patch release) This release fixes the following bugs: * Fix incorrect `RSTOS0` `conn_state` determinations https://github.com/zeek/zeek/issues/1164 https://github.com/zeek/zeek/pull/1166 Reminder: Zeek 3.0.x is the Long-Term Support release, receiving bug fixes until at least December 2020 while Zeek 3.2.x is the current feature release, receiving bug fixes until approximately December 2020 when the next 4.0.x LTS release series is expected to begin.

zeek v3.0.12

2020-12-15T17:34:34+00:00

This release fixes the following bugs: * Incorrect ICMP Neighbor Discovery Option length calculation https://github.com/zeek/zeek/issues/1225 https://github.com/zeek/zeek/pull/1228 * Fix SMB2 response status parsing https://github.com/zeek/zeek/pull/1311 https://github.com/zeek/zeek/commit/0b8535b879f1028d556b415ddccded27762e47c2 https://github.com/zeek/zeek/commit/07c4662dc4552ecd4d5b237de331c7b9ab369080 * Fix excessive `connection_status_update` events for ICMP connections https://github.com/zeek/zeek/pull/1322 Reminder: Zeek 3.0.x is a Long-Term Support (LTS) release, receiving bug fixes until at least February 2020 (estimate of 2 months after 4.0.0 release).

zeek v3.2.3

2020-12-15T17:34:39+00:00

This release fixes the following security issues: * In the parsing of IPv6 addresses within EDNS ECS options, data was written to a stack-buffer using as many bytes as supplied in the option even if it was in excess of the desired address prefix or maximum IPv6 address size. This could result in an overflow of that stack-buffer. This may be remotely exploitable by anyone creating such a DNS message. Credit to OSS-Fuzz for discovery https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=28336 (Link to details becomes public 30 days after patch release) This release fixes the following bugs: * The parsing of IPv4 addresses within EDNS ECS options would overwrite the storage used for that address as many times as there were bytes in the option in excess of the desired address prefix length or maximum IPv4 address size. This could cause the resulting IPv4 address to be derived from the incorrect data. * In parsing EDNS ECS options, upon encountering unexpected/excessive option-length or source-prefix parameters, the data pointer used for parsing was also not always advanced to the start of the next alleged option's data. Assuming all other parsing code correctly guards against invalid input, there's no further harm from that other than the subsequent parsing being more likely to encounter unexpected values and emitting more Weirds. * Incorrect ICMP Neighbor Discovery Option length calculation https://github.com/zeek/zeek/issues/1225 https://github.com/zeek/zeek/pull/1228 * Fix memory leak in deprecated Analyzer::ConnectionEvent() https://github.com/zeek/zeek/pull/1294 * Fix SMB2 response status parsing https://github.com/zeek/zeek/pull/1311 https://github.com/zeek/zeek/commit/0b8535b879f1028d556b415ddccded27762e47c2 https://github.com/zeek/zeek/commit/07c4662dc4552ecd4d5b237de331c7b9ab369080 * Fix excessive `connection_status_update` events for ICMP connections https://github.com/zeek/zeek/pull/1322 Reminder: Zeek 3.0.x is the Long-Term Support (LTS) release, receiving bug fixes until at least February 2020 while Zeek 3.2.x is the current feature release, receiving bug fixes until approximately February 2020 when the next 4.0.x LTS release series is expected to begin.

zeek v4.0.0-rc3

2020-12-15T17:34:44+00:00

New Functionality ----------------- - Added support for EDNS0 Cookie and Keep-Alive options. - Added new Packet Analysis plugin architecture for parsing packet headers at layers below the existing Session analysis plugins. This allows writing plugins to parse the various parts of a packet header separately, chaining down into other plugins as needed. - Add ``dce_rpc_request_stub`` and ``dce_rpc_response_stub`` events for accessing the contents of DCE-RPC request/response stub data. - Add support for log filter policy hooks, which supersede the current log predicates. The hook signature is as follows: hook(rec: any, id: Log::ID, filter: Log::Filter); The logging manager invokes hooks on each log record. Hooks can veto log records via a break, and modify them if necessary. Log filters inherit the stream-level hook, but can override or remove the hook as needed. The distribution's existing log streams now come with pre-defined hooks that users can add handlers to. The existing predicates are deprecated for removal in 4.1 but continue to work. See https://docs.zeek.org/en/master/frameworks/logging.html#filter-log-records for more details. - Added a ``udp-state`` signature condition to enforce matching against either "originator" or "responder" flow direction of UDP packets. - Improvements to capture-loss.zeek: - A new option, ``CaptureLoss::initial_watch_interval``. When restarting a Zeek cluster, one usually wants some immediate feedback as to the health of the monitoring via capture loss. However, you previously needed to wait a full ``CaptureLoss::watch_interval``, which defaults to 15 minutes. The new option specifies the interval for the first-time report. So the new default behavior provides stats after 1 minute and then after 15 minutes afterward. - A new notice type, ``CaptureLoss::Too_Little_Traffic``. If a Zeek process sees less than ``CaptureLoss::minimum_acks`` ACKs in a given interval, this notice gets raised. This can be a useful diagnostic if, for whatever reason, a Zeek process stops seeing traffic, but capture-loss.zeek would have previously only reported that "0 gaps and 0 ACKs is 0% loss". - A new ``zeek_script_args`` variable contains a list of arguments passed to a script. E.g. either when explicitly executing Zeek like ``zeek -- myscript.zeek -arg1 -arg2``, or when using Zeek to interpret executable scripts that contain a hashbang line at the top like:: #!/usr/local/zeek/bin/zeek -- - Added a new ``generate_all_events`` bif, which can be used to always raise events, even when they are not used by scripts. This can be used by the ``dump-events.zeek`` script to log all events that happen; the script got a new option to enable this behavior. - Added new unknown_protocols.log that will log analyzer and protocol pairs via the packet analysis framework for packet protocols that aren't supported by Zeek. It can be enabled by loading the ``policy/misc/unknown-protocols`` script. The script adds a new ``unknown_protocol`` event. - Added support for DNS resource records LOC, SSHFP, NSEC3PARAM, and custom BIND9 signaling. The associated events are: - dns_LOC - dns_SSHFP - dns_NSEC3PARAM - dns_BINDS - Zeek now supports SSH clients/servers that advertise SSH version 1.99, which is a special version indicating that the server/client supports both SSH2 and SSH1. - Added ``count_to_double()`` and ``int_to_double()`` type-conversion BIFs. - Added these string-processing BIFs: - count_substr - find_str - rfind_str - starts_with - ends_with - is_num - is_alpha - is_alnum - ljust - rjust - swap_case - to_title - zfill - remove_prefix - remove_suffix - Added a new ``Weird::sampling_global_list`` option to configure global rate-limiting of certain weirds instead of per connection/flow. - Added a ``Pcap::findalldevs()`` for obtaining available network devices. - Added ``enum_names()`` BIF to return names of an enum type's values - Added ``type_aliases`` BIF for introspecting type-names of types/values - Added composite-index support for ``&backend`` (Broker-backed tables). An example of a set with composite index is ``set[string, count, count]``. - Sumstats now allows manual epochs. If an ``epoch`` interval of 0 is specified, epochs will have to be manually ended by callis ``SumStats::next_epoch``. This can be convenient because epochs can be synced to other events. - The Zeek distribution now includes Zeek's package manager, zkg. Its code, configuration, and state reside in Zeek's installation tree, as follows: - The toplevel script, ``zkg``, installs alongside ``zeek`` in the distribution's ``$prefix/bin`` folder. - The config file installs into ``$prefix/etc/zkg/config``. The distribution's zkg command uses it by default, but you can switch to a different one via the ``ZKG_CONFIG_FILE`` environment variable or the ``--configfile`` command-line flag. - zkg's package state resides in ``$prefix/var/lib/zkg``. This implies that parallel Zeek installations now automatically separate their package installations. These folders have the same ownership and access permissions as the rest of the installation, , meaning that in order to manage zkg packages you need to run zkg as a user with corresponding access. Apart from these location overrides, the bundled zkg installation behaves as usual. local.zeek now contains a (commented out) ``@load`` statement you can use to source zkg's package state automatically. zkg's own Python module resides in ``zeek/python/zeekpkg`, in the installation tree's library folder. See below for additional changes around the library folder. zkg has external Python module dependencies. The Zeek configuration does not verify whether these dependencies are met. A new warning message at zkg launch flags missing packages and how to install them (e.g. via pip). Configuring with ``--disable-zkg`` disables the zkg inclusion. You can continue to install and use zkg independently. You're also free to use the config file in ``$prefix/etc/zkg/config`` with other zkg installations. The zkg source tree resides in ``auxil/package-manager`` as an additional Git submodule. Changed Functionality --------------------- - ``NetControl::DROP`` had 3 conflicting definitions that could potentially be used incorrectly without any warnings or type-checking errors. Such enum redefinition conflicts are now caught and treated as errors, so the ``NetControl::DROP`` enums had to be renamed: - The use as enum of type ``Log::ID`` is renamed to ``NetControl::DROP_LOG`` - The use as enum of type ``NetControl::CatchReleaseInfo`` is renamed to ``NetControl::DROP_REQUESTED`` - The use as enum of type ``NetControl::RuleType`` is unchanged and still named ``NetControl::DROP`` - The extract_email_addrs_vec() BIF now returns all occurrences of emails, including duplicates, with preserved order of occurrence. This seems like the original/documented intent of the function, but the previous implementation did not preserve ordering or duplicates. - The Dictionary implementation is replaced (no API changes). The new version uses clustered hashing, a variation of Robinhood / Open Addressing hashing. This implementation generally performs better and utilizes less memory than the previous one. A detailed explanation of the implementation is here: https://jasonlue.github.io/algo/2019/08/20/clustered-hashing.html - The ``p`` fields of ``Cluster::Node`` records now use a ``&default=0/unknown`` attribute with ``0/unknown`` meaning that the node is not pre-configured to listen for incoming connections from other cluster nodes. - The ``|x|`` operator, where ``x`` is an expression with an integral result, no longer performs an implicit coercion of that result into a signed ``int`` type. This was actually the behavior before Zeek 3.0 as well, but the attempt to prevent mistakes that easily result from integer literals in Zeek being unsigned like ``|5 - 9|`` causing an overflow/wraparound and yielding a very large number is not generally consistent since overflows are still generally able to happen in other ways and also in other contexts besides just the absolute-value operator. So the preference was to revert to a behavior that favors consistency. For reference, see https://github.com/zeek/zeek/pull/251#issuecomment-713956976 - The Zeek installation tree is now more consistent in using a ``lib64/`` (rather than ``lib/``) subdirectory for platforms where that's the common convention. If the old hardcoded ``lib/`` path exists while installing Zeek 4.0 and the new subdirectory differs, the old ``lib/`` remains untouched. This clutters the installation but is safe: the new installation does not require the old location, and any files you might require still in the old tree (e.g. ZeekControl plugins) remain available. Due to Zeek 4's reorganization of the installation tree we recommend a clean-slate install when possible. - Python modules installed with the Zeek distribution now reside in a common ``zeek/python`` directory below the library path (such as ``lib64/zeek/python``) and no longer assume ZeekControl. The ``zeek/python/zeekctl`` folder now contains only ZeekControl's own functionality, ``zeek/python/zeekpkg`` contains zkg's Python module, and Broker's Python bindings live in ``zeek/python/broker``. ``zeek-config --python_dir`` now reports this new ``zeek/python`` folder. Several new configure options allow you to customize the Python folder location, depending on your needs. As with the new libdir, no cleanup of the existing Python tree occurs. - Continued renaming/namespacing of many classes into either ``zeek`` or ``zeek::detail`` namespaces as already explained in Zeek 3.2's release notes. Deprecation warnings should generally help notify plugin developers of these changes. - Changed HTTP DPD signatures to trigger analyzer independent of peer state. This is to avoid missing large sessions where a single side exceeds the DPD buffer size. It comes with the trade-off that now the analyzer can be triggered by anybody controlling one of the endpoints (instead of both). For discussion, see https://github.com/zeek/zeek/issues/343. Removed Functionality --------------------- - The counter type was removed. This type was never fully functional/used anywhere. - Removed the PRI_PTR_COMPAT_INT, PRI_PTR_COMPAT_UINT, and PRI_SOURCE_ID macros. There are no deprecation warnings for these because they were C macros. Use the PRIdPTR and PRIuPTR macros from the standard library instead. - The ``successful_connection_remove`` and ``connection_successful`` events as well as ``connection$successful`` field that were added in Zeek v3.1.0 are now removed. They were found to be unintuitive along with having unresolved corner cases. The original goal/intent to avoid the overhead and scalability issues with every new protocol-analysis adding a new ``connection_state_remove`` handler can now be resolved with a less-confusing approach: see the ``Conn::register_removal_hook`` function. - Python 2 is no longer supported. Python 3.5 is the new minimum requirement. - CMake versions less than 3.5 are no longer supported. - CAF version 0.18 is now required and, by default, that is bundled with the Zeek distribution and will get built unless overridden with the ``--with-caf=`` configuration option. Deprecated Functionality ------------------------ - Marked the Continuation.h and PacketDumper.h files as deprecated. The code contained within them is unused by Zeek. - ``Type::GetAliases()`` and ``Type::AddAlias()`` are deprecated, use ``Type::Aliases()`` and ``Type::RegisterAlias()``. - The ``ssh1_server_host_key`` event's modulus and exponent parameters, *e* and *p*, were named in misleading way (*e* is the modulus) and now deprecated in favor of the new *modulus* and *exponent* parameters.

zeek v4.0.0

2020-12-15T17:34:44+00:00

zeek v4.0.0-rc1

2020-12-15T17:34:44+00:00

zeek v4.0.0-rc2

2020-12-15T17:34:44+00:00

zeek v3.0.13

2021-02-22T19:49:39+00:00

This release fixes the following security issues: * Fix ASCII Input reader's treatment of input files containing null-bytes https://github.com/zeek/zeek/issues/1398 https://github.com/zeek/zeek/commit/a636f8edbd903d3710ec4d7e06b8bb0d68d8aab9 This is mostly only significant for deployments that utilize input data feeds whose content is controlled by external sources: an input file containing null-bytes could lead to a buffer-over-read, crash Zeek, and be exploited to cause Denial of Service. This release fixes the following bugs: * MIME sub-entities overwrote top-level header values cause misleading SMTP log https://github.com/zeek/zeek/issues/1352 https://github.com/zeek/zeek/pull/1365 * Fix incorrect `major_subsys_version` field in `pe_optional_header` event https://github.com/zeek/zeek/pull/1401 Reminder: Zeek 3.0.x is a Long-Term Support (LTS) release, receiving bug fixes until at least May 2021 (estimate of 2 months after 4.0.0 release).

zeek v3.2.4

2021-02-22T19:50:01+00:00

This release fixes the following security issues: * Fix ASCII Input reader's treatment of input files containing null-bytes https://github.com/zeek/zeek/issues/1398 https://github.com/zeek/zeek/commit/a636f8edbd903d3710ec4d7e06b8bb0d68d8aab9 This is mostly only significant for deployments that utilize input data feeds whose content is controlled by external sources: an input file containing null-bytes could lead to a buffer-over-read, crash Zeek, and be exploited to cause Denial of Service. This release fixes the following bugs: * MIME sub-entities overwrote top-level header values cause misleading SMTP log https://github.com/zeek/zeek/issues/1352 https://github.com/zeek/zeek/pull/1365 * Fix incorrect `major_subsys_version` field in `pe_optional_header` event https://github.com/zeek/zeek/pull/1401 Reminder: Zeek 3.0.x is the Long-Term Support (LTS) release, receiving bug fixes until at least May 2021 (estimate of 2 months after 4.0.0 release) while Zeek 3.2.x is the current feature release, receiving bug fixes until approximately March 2021 when the next 4.0.x LTS release series is expected to begin.

zeek v3.0.14

2021-04-21T19:07:00+00:00

zeek v4.0.1

2021-04-21T19:10:34+00:00

This release fixes the following security issue: - Fix null-pointer dereference when encountering an invalid `enum` name in a config/input file that tries to read it into a `set[enum]`. For those that have such an input feed whose contents may come from external/remote sources, this is a potential DoS vulnerability. https://github.com/zeek/zeek/issues/1487 https://github.com/zeek/zeek/pull/1488 This release fixes the following bugs: - Fix mime type detection bug in IRC/FTP `file_transferred` event for file data containing null-bytes https://github.com/zeek/zeek/pull/1430 - Fix potential for missing timestamps in SMB logs https://github.com/zeek/zeek/pull/1436 - Remove use of LeakSanitizer API on FreeBSD where it's unsupported https://github.com/zeek/zeek/pull/1440 - Fix incorrect parsing of ERSPAN Type I https://github.com/zeek/zeek/pull/1445 https://github.com/zeek/zeek/commit/f53fb9a22e1ad5aa1ebdf726f241c9cad0ceeb32 - Fix incorrect/overflowed `n` value for `SSL_Heartbeat_Many_Requests` notices where number of server heartbeats is greater than number of client heartbeats. https://github.com/zeek/zeek/issues/1454 https://github.com/zeek/zeek/pull/1459 https://github.com/zeek/zeek/commit/c23e3ca1056c03be347da6163a8e447670ce06b1 - Fix missing `user_agent` existence check in `smtp/software.zeek` (causes `reporter.log` error noise, but no functional difference) https://github.com/zeek/zeek/pull/1455 https://github.com/zeek/zeek/commit/83d5b44462722c5571b33eb6bcb0491af90cccc7 - Fix include order of bundled headers to avoid conflicts with pre-existing/system-wide installs https://github.com/zeek/zeek/pull/1465 - Fix musl build (e.g. Void, Alpine, etc.) https://github.com/zeek/zeek/pull/1469 https://github.com/zeek/zeek/commit/2ad482535ee36c99d7f9ea3bcd3077cb9c81bc78 - Fix build with `-DENABLE_MOBILE_IPV6 ` / `./configure --enable-mobile-ipv6` https://github.com/zeek/zeek/issues/1493 https://github.com/zeek/zeek/pull/1495 - Add check for null packet data in pcap IOSource, which is an observed state in Myricom libpcap that crashes Zeek via null-pointer dereference https://github.com/zeek/zeek/pull/1498 - Allow CRLF line-endings in Zeek scripts and signature files https://github.com/zeek/zeek/issues/1497 https://github.com/zeek/zeek/pull/1499 - Fix armv7 build https://github.com/zeek/zeek/issues/1496 https://github.com/zeek/zeek/issues/1502 - Fix unserialization of `set[function]`, generally now used by `connection` record removal hooks, and specifically breaking `intel.log` of Zeek clusters https://github.com/zeek/zeek/issues/1506 https://github.com/zeek/zeek/pull/1513 - Fix indexing of set/table types with a vector https://github.com/zeek/zeek/pull/1514 - Fix precision loss in ASCII logging/printing of large double, time, or interval values https://github.com/zeek/zeek/issues/1450 https://github.com/zeek/zeek/pull/1494 - Improve handling of invalid SIP data before requests https://github.com/zeek/zeek/issues/1507 https://github.com/zeek/zeek/pull/1511 - Fix `copy()`/cloning vectors that have holes (indices w/ null values) https://github.com/zeek/zeek/commit/180ab3181f7743cf57b74a2840bd1d701679647b Reminder: Zeek 4.0.x is a Long-Term Support (LTS) release, receiving bug fixes until at least May 2022 (estimate of 2 months after 5.0.0 release).

zeek v4.0.2

2021-06-02T20:05:56+00:00

This release fixes the following security issues: - Fix potential Undefined Behavior in decode_netbios_name() and decode_netbios_name_type() BIFs. The latter has a possibility of a remote heap-buffer-overread, making this a potential DoS vulnerability. https://github.com/zeek/zeek/pull/1533 https://github.com/zeek/zeek/pull/1563 - Add some extra length checking when parsing mobile ipv6 packets. Due to the possibility of reading invalid headers from remote sources, this is a potential DoS vulnerability. https://github.com/zeek/zeek/commit/c5533f3e6a6772ce33607a048adfeffa95247bc1 This release fixes the following bugs: - Fix heap-use-after-free after clear_table() on a table that uses expiration attributes. https://github.com/zeek/zeek/commit/d51bd4bc4675d4d7234da0d11e31dbede5efe17c - Add fatal error for if table/Dictionary state ever becomes invalid since the behavior becomes unexpected/unclear at that point (e.g. when table bucket positions become large enough to overflow their 16-bit storage due to aggressive expiration-check settings preventing the re-positioning items) https://github.com/zeek/zeek/commit/292e3e18a3691d835b2d52ab8462a90ae47b0ae7 - Add missing "zeek/" to header includes, which can prevent external plugins from compiling against Zeek source-tree (e.g. via ./configure --zeek-dist=) https://github.com/zeek/zeek/issues/1547 https://github.com/zeek/zeek/pull/1549 - Fix reading empty set[enum] values and any vector of enum values from config files https://github.com/zeek/zeek/issues/1555 https://github.com/zeek/zeek/issues/1558 https://github.com/zeek/zeek/pull/1559 - Fix type-checks related to list-type equality https://github.com/zeek/zeek/issues/1296 https://github.com/zeek/zeek/pull/1358

zeek v4.0.3

2021-07-06T20:52:03+00:00

This release fixes the following bugs: - The input framework's handling of unset fields (commonly expressed as "-") in ingested data is now safer and more consistent. When reading data into records, Zeek now accepts unset fields in the input data only when the corresponding record field is ``&optional``. Unset fields for non-optional fields cause the input line to be skipped. Reading data into tables with complex index types (such as ``table[count, count] of string``) now also skips lines with unset fields, since such indexes require fields to be present. Note that this may change the behavior of existing scripts if you have unset fields in your input data. - The version field in ssh.log is now optional and will not be set if we cannot determine the version that was negotiated by the client and server. https://github.com/zeek/zeek/pull/1605 - Zeekctl could crash at startup on certain compilers and platforms due to a memory corruption issue in the Broker python bindings. https://github.com/zeek/broker/issues/187 - The highwayhash submodule was updated to fix a build failure on FreeBSD for PowerPC. https://github.com/zeek/zeek/issues/1591 This release deprecates the following functionality: - The stepping-stone analyzer is marked as deprecated. It was partially marked as deprecated in 2.0, and will be fully removed in v4.1. https://github.com/zeek/zeek/issues/1573 https://github.com/zeek/zeek/pull/1604

zeek v4.1.0-beta

2021-07-09T21:24:34+00:00

NOTE: This release is currently in beta. It is feature complete, but may undergo bug fixes before being fully released. New Functionality ----------------- - Lambda functions can now use capture-list to help specify exactly which local variables from outer scopes need to made available while evaluating the lambda and also the method by which they're made available: deep vs. shallow copy. For examples, see: https://docs.zeek.org/en/master/script-reference/types.html#type-function - Support for VN-Tag protocol headers: the new VN-Tag packet analyzer simply skips past the VN-Tag header, allowing for further analysis of subsequent packet contents. - Support for decapsulating Geneve packets to process the inner payload, similar in operation to the existing VXLAN support. - Support for Zeek script "Reaching Definitions" (RD) analysis: tracking the extent to which a given variable definition (assignment) can be visible elsewhere in the Zeek script. The analysis works on local variables in function/event/hook bodies, but not across them. The code tracks two forms of RDs, "minimal" (what's guaranteed to reach a given point in a function body) and "maximal" (what possibly could reach). Upcoming script optimization will use the latter, but the former currently allows identification of places for which a value is used where it does not appear that it will necessarily be defined. Specifying the ``zeek -u`` option will generate warnings for instances where this holds for local variables. Specifying ``zeek -uu`` turns on additional (expensive) analysis to report instances where record fields might be used without having previously been set. The ``zeek -u`` option can also identify assigned-to variables that aren't subsequently used (i.e. "dead code") and issues a warning. A new ``is_used`` attribute can be used situationally to suppress such warnings. The base scripts have some places where the static analysis lacks sufficient power to tell that values are being used safely (guaranteed to have been identified). In order to enable users to employ ``zeek -u`` on their own scripts without being distracted by these instances, this change also includes a new attribute, ``&is_assigned``, which can be associated with a variable or a record field to inform Zeek's analysis that the script writer asserts the value will be set, suppressing the associated warnings. - A Telemetry API was added to assist in gathering arbitrary runtime metrics and allows export to Prometheus. This is still work-in-progress, preliminary documentation for current, low-level API lives at https://github.com/zeek/zeek/wiki/Telemetry for now. - Experimental support for translating Zeek scripts to equivalent C++. The generated C++ can then be compiled directly into the ``zeek`` binary, replacing use of the interpreter and producing better runtime performance. See ``src/script_opt/CPP/README.md`` for a guide on how to use this feature. - Support for more generic session management. The NetSessions class has been renamed to SessionMgr (with the old name marked deprecated). The new class allows plugins to take advantage of session management similar to how Connection objects were handled previously, but without the need to be based on IP-based protocols. - The logging framework now provides a global policy hook, ``Log::log_stream_policy``. Like the existing filter-level hooks, handlers for the new hook can provide additional processing and veto the log write. The new hook runs once per write, prior to any filter-level policy hooks. Even when it vetoes, filter-level policy hooks still run, but cannot "un-veto" the write. - The ASCII writer gained a new option LogAscii::logdir, which can be used to change the logging output directory. - Added a ``--include-plugins`` argument to ``configure``. This argument takes a semicolon separated list of paths containing plugins that will be statically built into Zeek. - The X509 analyzer now can check if a specific hostname is valid for a certificate. Two new BIFs were added for this, ``x509_check_hostname`` and ``x509_check_cert_hostname``. A new field ``sni_matches_cert`` that tracks this information was added to ``ssl.log``. - Added a ``--plugindir`` argument to ``configure`` to set the installation path for plugins. - Added new functions to dynamically enable/disable file analyzers: - ``global enable_analyzer: function(tag: Files::Tag): bool;`` - ``global disable_analyzer: function(tag: Files::Tag): bool;`` - ``global analyzer_enabled: function(tag: Files::Tag): bool;`` - The Supervisor now defaults to starting with a minimal set of Zeek scripts controlled by a new init file, ``base/init-supervisor.zeek``. One may still run it with a larger configuration by loading additional scripts, including ``init-default.zeek``, as always. (Bare mode continues to work as usual, reducing the configuration to a minimum.) The ``NodeConfig`` record has two new members, providing additional control over launched nodes. The ``env`` member allows setting environment variables in the launched process. The ``bare_mode`` member, an optional boolean, provides control over the bare-mode state of the new node. When not provided, the node inherits the bare-mode status of the Supervisor, and setting the variable enables/disables bare mode. - Zeek now includes an incomplete, preliminary version of the future cluster controller framework. Loading ``policy/frameworks/cluster/agent`` and/or ``policy/frameworks/cluster/agent`` in a Zeek running with the Supervisor will launch the corresponding cluster management node(s). An experimental management client, `zeek-client`, connects to the controller and lets you issue commands. It requires configuration with ``--enable-zeek-client``. This does not yet provide a functional substitute for ``zeekctl``, which users should continue to use for now. Changed Functionality --------------------- - The default IP-based transport protocols (UDP, TCP, and ICMP) have been moved to the packet analysis framework. This change allows us to move other analyzers in the future that better align with the packet analysis framework than they do with session analysis. - The input framework's handling of unset fields (commonly expressed as "-") in ingested data is now safer and more consistent. When reading data into records, Zeek now accepts unset fields in the input data only when the corresponding record field is ``&optional``. Unset fields for non-optional fields cause the input line to be skipped. Reading data into tables with complex index types (such as ``table[count, count] of string``) now also skips lines with unset fields, since such indexes require fields to be present. Note that this may change the behavior of existing scripts if you have unset fields in your input data. - The version field in ssh.log is now optional and will not be set if we cannot determine the version that was negotiated by the client and server. - Add a new field ``email_dest`` to NOTICEs, which defines where to send email to. The email-related NOTICE actions fill this now, and then emails will be sent to all recorded addresses at the end of NOTICE processing. This makes email generation more consistent and extensible. - Add page and email administrator to mails processed by hostnames extension. - SSL and X509 handling was significantly overhauled with the goal to make the data that is logged by Zeek more helpful and compact. This change means that there are significant changes to the default log files, as well as changes to functionality: - ``x509.log`` is now indexed by the sha256 of the certificate, with deduplication being automatically performed. By default, the same certificate is only logged once per day. This also means that the file ID is no longer present in X509 log. Similarly, ``ssl.log`` now contains hashes for X509 certificates. The hash function that is used for indexing the certificates is changeable by changing the ``X509::hash_function`` option. The time period after which a certificate is logged again can be configured by changing ``X509::relog_known_certificates_after``. By default deduplication of certificates is done across the entire cluster using broker. If this is not desired due to the higher communication overhead, this behavior can be disabled using ``X509::known_log_certs_use_broker``. - X509 certificates are, by default, no longer logged into files.log. This behavior is configurable and the previous default can be restored by changing the ``X509::log_x509_in_files_log`` option. - ``x509.log`` now tracks if a certificate was encountered as a end-host certificate or as a client certificate. - OCSP logging is now enabled by default. - ``ssl.log`` now no longer includes information about the certificate issuer and subject. This information is still available in X509.log. If you need this information in ``ssl.log``, the old behavior can be restored by changing the ``SSL::log_include_server_certificate_subject_issuer`` and ``SSL::log_include_client_certificate_subject_issuer`` configuration options. - ``ssl.log`` now contains a ``ssl_history`` field, which tracks which protocol messages were seen in an SSL/TLS connection. - We added a policy script ``ssl-log-ext.zeek`` which greatly extends the amount of protocol information logged to SSL.log. The script is not loaded by default. - We added a ``disable-certificate-events-known-certs.zeek`` policy script. This script will completely disable X509 events for known certificates over SSL/TLS connections. For Zeek installations in settings where you encounter a lot of certificates, this could improve the performance of your installation. Before enabling this script, make sure that you do not use any third-party scripts that depend on the X509 events. The script is not loaded by default. - The ICSI SSL Notary script was deprecated. This functionality is superseeded by newer approaches, like SCT validation (which is supported by Zeek). - ``extract-certs-pem.zeek`` was deprecated - it never really worked in cluster modes. A new policy script, ``log-certs-base64.zeek`` that can be used to log raw certificates was added instead. - The CT logs listed in ``ct-list.zeek`` are now derived from the list of CT logs that are accepted by Google Chrome. In the past, we allowed the list of all known CT logs. This no longer makes sense since nowadays logs exist that contain, e.g., only outdated or invalid certificates. If the old behavior is desired, you can re-add Logs to ``SSL::ct_logs``. - The Mozilla CA list was updated to the state of NSS 3.67. - SQLite was updated to 3.36.0. Removed Functionality --------------------- - Support for the RocksDB Broker data store was previously broken and unusable, so all code/options related to it are now removed. - Support for the ``ENABLE_MOBILE_IPV6`` compiler variable has been removed. Mobile IPv6 is now enabled by default. The ``--enable-mobile-ipv6`` option for ``configure`` now returns a warning that it will be removed in v5.1 and no longer has any effect. Deprecated Functionality ------------------------ - Lambda/closure support: automatic capturing of references to variables outside a lambda's scope is now deprecated. An explicit capture list which also specifies the desired copy-semantics is now required when writing lambda functions that refer to local variables of an outer scope. For examples, see: https://docs.zeek.org/en/master/script-reference/types.html#type-function - The ``IterCookie`` version of iteration over ``Dictionary`` and ``PDict`` objects was marked as deprecated. It was replaced by standard-library compatible iterators. This enables the use of standard constructs such as ranged-for loops to iterate over those objects. - The ``zeek::util::zeekenv()`` function is deprecated since use of all environment variables prefixed by ``BRO_`` is now removed and calling ``getenv()`` directly with ``ZEEK_`` environment variables can be done. - ``supervisor_rotation_format_func`` is renamed to ``archiver_rotation_format_func`` - The ``MemoryAllocation()`` function implemented by a number of interfaces is now deprecated. In testing we found that the values returned were mostly incorrect and weren't useful. The ``val_size`` and ``global_sizes`` BIF methods have also both been marked deprecated.

zeek v4.1.0-rc2

2021-07-09T21:24:34+00:00

NOTE: This release is currently a release candidate. It is feature complete, but may undergo bug fixes before being fully released. New Functionality ----------------- - Lambda functions can now use capture-list to help specify exactly which local variables from outer scopes need to made available while evaluating the lambda and also the method by which they're made available: deep vs. shallow copy. For examples, see: https://docs.zeek.org/en/master/script-reference/types.html#type-function - Support for VN-Tag protocol headers: the new VN-Tag packet analyzer simply skips past the VN-Tag header, allowing for further analysis of subsequent packet contents. - Support for decapsulating Geneve packets to process the inner payload, similar in operation to the existing VXLAN support. - Support for Zeek script "Reaching Definitions" (RD) analysis: tracking the extent to which a given variable definition (assignment) can be visible elsewhere in the Zeek script. The analysis works on local variables in function/event/hook bodies, but not across them. The code tracks two forms of RDs, "minimal" (what's guaranteed to reach a given point in a function body) and "maximal" (what possibly could reach). Upcoming script optimization will use the latter, but the former currently allows identification of places for which a value is used where it does not appear that it will necessarily be defined. Specifying the ``zeek -u`` option will generate warnings for instances where this holds for local variables. Specifying ``zeek -uu`` turns on additional (expensive) analysis to report instances where record fields might be used without having previously been set. The ``zeek -u`` option can also identify assigned-to variables that aren't subsequently used (i.e. "dead code") and issues a warning. A new ``is_used`` attribute can be used situationally to suppress such warnings. The base scripts have some places where the static analysis lacks sufficient power to tell that values are being used safely (guaranteed to have been identified). In order to enable users to employ ``zeek -u`` on their own scripts without being distracted by these instances, this change also includes a new attribute, ``&is_assigned``, which can be associated with a variable or a record field to inform Zeek's analysis that the script writer asserts the value will be set, suppressing the associated warnings. - A Telemetry API was added to assist in gathering arbitrary runtime metrics and allows export to Prometheus. This is still work-in-progress, preliminary documentation for current, low-level API lives at https://github.com/zeek/zeek/wiki/Telemetry for now. - Experimental support for translating Zeek scripts to equivalent C++. The generated C++ can then be compiled directly into the ``zeek`` binary, replacing use of the interpreter and producing better runtime performance. See ``src/script_opt/CPP/README.md`` for a guide on how to use this feature. - Support for more generic session management. The NetSessions class has been renamed to SessionMgr (with the old name marked deprecated). The new class allows plugins to take advantage of session management similar to how Connection objects were handled previously, but without the need to be based on IP-based protocols. - The logging framework now provides a global policy hook, ``Log::log_stream_policy``. Like the existing filter-level hooks, handlers for the new hook can provide additional processing and veto the log write. The new hook runs once per write, prior to any filter-level policy hooks. Even when it vetoes, filter-level policy hooks still run, but cannot "un-veto" the write. - The ASCII writer gained a new option LogAscii::logdir, which can be used to change the logging output directory. - Added a ``--include-plugins`` argument to ``configure``. This argument takes a semicolon separated list of paths containing plugins that will be statically built into Zeek. - Added a ``--plugindir`` argument to ``configure`` to set the installation path for plugins. - The X509 analyzer now can check if a specific hostname is valid for a certificate. Two new BIFs were added for this, ``x509_check_hostname`` and ``x509_check_cert_hostname``. A new field ``sni_matches_cert`` that tracks this information was added to ``ssl.log``. - Added new functions to dynamically enable/disable file analyzers: - ``global enable_analyzer: function(tag: Files::Tag): bool;`` - ``global disable_analyzer: function(tag: Files::Tag): bool;`` - ``global analyzer_enabled: function(tag: Files::Tag): bool;`` - Zeek now includes its own BTest tooling in the distribution, enabling other tests (e.g. in Zeek packages) to use it. The ``$PREFIX/share/btest folder``, reported via ``zeek-config --btest_tools_dir``, includes: - ``scripts/`` for ``btest-diff`` canonifiers - ``data/`` for data files, including ``random.seed`` - ``data/pcaps`` for the test pcaps Configuring with ``--disable-btest-pcaps`` suppresses installation of the test pcaps. - The Supervisor now defaults to starting with a minimal set of Zeek scripts controlled by a new init file, ``base/init-supervisor.zeek``. One may still run it with a larger configuration by loading additional scripts, including ``init-default.zeek``, as always. (Bare mode continues to work as usual, reducing the configuration to a minimum.) The ``NodeConfig`` record has two new members, providing additional control over launched nodes. The ``env`` member allows setting environment variables in the launched process. The ``bare_mode`` member, an optional boolean, provides control over the bare-mode state of the new node. When not provided, the node inherits the bare-mode status of the Supervisor, and setting the variable enables/disables bare mode. - Zeek now includes an incomplete, preliminary version of the future cluster controller framework. Loading ``policy/frameworks/cluster/agent`` and/or ``policy/frameworks/cluster/agent`` in a Zeek running with the Supervisor will launch the corresponding cluster management node(s). An experimental management client, `zeek-client`, connects to the controller and lets you issue commands. It requires configuration with ``--enable-zeek-client``. This does not yet provide a functional substitute for ``zeekctl``, which users should continue to use for now. Changed Functionality --------------------- - The default IP-based transport protocols (UDP, TCP, and ICMP) have been moved to the packet analysis framework. This change allows us to move other analyzers in the future that better align with the packet analysis framework than they do with session analysis. - The input framework's handling of unset fields (commonly expressed as "-") in ingested data is now safer and more consistent. When reading data into records, Zeek now accepts unset fields in the input data only when the corresponding record field is ``&optional``. Unset fields for non-optional fields cause the input line to be skipped. Reading data into tables with complex index types (such as ``table[count, count] of string``) now also skips lines with unset fields, since such indexes require fields to be present. Note that this may change the behavior of existing scripts if you have unset fields in your input data. - The version field in ssh.log is now optional and will not be set if we cannot determine the version that was negotiated by the client and server. - Add a new field ``email_dest`` to NOTICEs, which defines where to send email to. The email-related NOTICE actions fill this now, and then emails will be sent to all recorded addresses at the end of NOTICE processing. This makes email generation more consistent and extensible. - Add page and email administrator to mails processed by hostnames extension. - SSL and X509 handling was significantly overhauled with the goal to make the data that is logged by Zeek more helpful and compact. This change means that there are significant changes to the default log files, as well as changes to functionality: - ``x509.log`` is now indexed by the sha256 of the certificate, with deduplication being automatically performed. By default, the same certificate is only logged once per day. This also means that the file ID is no longer present in X509 log. Similarly, ``ssl.log`` now contains hashes for X509 certificates. The hash function that is used for indexing the certificates is changeable by changing the ``X509::hash_function`` option. The time period after which a certificate is logged again can be configured by changing ``X509::relog_known_certificates_after``. By default deduplication of certificates is done across the entire cluster using broker. If this is not desired due to the higher communication overhead, this behavior can be disabled using ``X509::known_log_certs_use_broker``. - X509 certificates are, by default, no longer logged into files.log. This behavior is configurable and the previous default can be restored by changing the ``X509::log_x509_in_files_log`` option. - ``x509.log`` now tracks if a certificate was encountered as a end-host certificate or as a client certificate. - OCSP logging is now enabled by default. - ``ssl.log`` now no longer includes information about the certificate issuer and subject. This information is still available in X509.log. If you need this information in ``ssl.log``, the old behavior can be restored by changing the ``SSL::log_include_server_certificate_subject_issuer`` and ``SSL::log_include_client_certificate_subject_issuer`` configuration options. - ``ssl.log`` now contains a ``ssl_history`` field, which tracks which protocol messages were seen in an SSL/TLS connection. - We added a policy script ``ssl-log-ext.zeek`` which greatly extends the amount of protocol information logged to SSL.log. The script is not loaded by default. - We added a ``disable-certificate-events-known-certs.zeek`` policy script. This script will completely disable X509 events for known certificates over SSL/TLS connections. For Zeek installations in settings where you encounter a lot of certificates, this could improve the performance of your installation. Before enabling this script, make sure that you do not use any third-party scripts that depend on the X509 events. The script is not loaded by default. - The ICSI SSL Notary script was deprecated. This functionality is superseeded by newer approaches, like SCT validation (which is supported by Zeek). - ``extract-certs-pem.zeek`` was deprecated - it never really worked in cluster modes. A new policy script, ``log-certs-base64.zeek`` that can be used to log raw certificates was added instead. - The CT logs listed in ``ct-list.zeek`` are now derived from the list of CT logs that are accepted by Google Chrome. In the past, we allowed the list of all known CT logs. This no longer makes sense since nowadays logs exist that contain, e.g., only outdated or invalid certificates. If the old behavior is desired, you can re-add Logs to ``SSL::ct_logs``. - The Mozilla CA list was updated to the state of NSS 3.67. - SQLite was updated to 3.36.0. - The list of subnets that are considered private addresses was updated to mostly match IANA's list unroutable addresses. This brings it in line with Chrome, the W3C spec, and Python's ipaddress module. Removed Functionality --------------------- - Support for the RocksDB Broker data store was previously broken and unusable, so all code/options related to it are now removed. - Support for the ``ENABLE_MOBILE_IPV6`` compiler variable has been removed. Mobile IPv6 is now enabled by default. The ``--enable-mobile-ipv6`` option for ``configure`` now returns a warning that it will be removed in v5.1 and no longer has any effect. Deprecated Functionality ------------------------ - Lambda/closure support: automatic capturing of references to variables outside a lambda's scope is now deprecated. An explicit capture list which also specifies the desired copy-semantics is now required when writing lambda functions that refer to local variables of an outer scope. For examples, see: https://docs.zeek.org/en/master/script-reference/types.html#type-function - The ``IterCookie`` version of iteration over ``Dictionary`` and ``PDict`` objects was marked as deprecated. It was replaced by standard-library compatible iterators. This enables the use of standard constructs such as ranged-for loops to iterate over those objects. - The ``zeek::util::zeekenv()`` function is deprecated since use of all environment variables prefixed by ``BRO_`` is now removed and calling ``getenv()`` directly with ``ZEEK_`` environment variables can be done. - ``supervisor_rotation_format_func`` is renamed to ``archiver_rotation_format_func`` - The ``MemoryAllocation()`` function implemented by a number of interfaces is now deprecated. In testing we found that the values returned were mostly incorrect and weren't useful. The ``val_size`` and ``global_sizes`` BIF methods have also both been marked deprecated.

zeek v4.1.0-rc1

2021-07-09T21:24:34+00:00

zeek v4.1.0

2021-08-10T22:33:09+00:00

New Functionality ----------------- - Lambda functions can now use capture-list to help specify exactly which local variables from outer scopes need to made available while evaluating the lambda and also the method by which they're made available: deep vs. shallow copy. For examples, see: https://docs.zeek.org/en/master/script-reference/types.html#type-function - Support for VN-Tag protocol headers: the new VN-Tag packet analyzer simply skips past the VN-Tag header, allowing for further analysis of subsequent packet contents. - Support for decapsulating Geneve packets to process the inner payload, similar in operation to the existing VXLAN support. - Support for Zeek script "Reaching Definitions" (RD) analysis: tracking the extent to which a given variable definition (assignment) can be visible elsewhere in the Zeek script. The analysis works on local variables in function/event/hook bodies, but not across them. The code tracks two forms of RDs, "minimal" (what's guaranteed to reach a given point in a function body) and "maximal" (what possibly could reach). Upcoming script optimization will use the latter, but the former currently allows identification of places for which a value is used where it does not appear that it will necessarily be defined. Specifying the ``zeek -u`` option will generate warnings for instances where this holds for local variables. Specifying ``zeek -uu`` turns on additional (expensive) analysis to report instances where record fields might be used without having previously been set. The ``zeek -u`` option can also identify assigned-to variables that aren't subsequently used (i.e. "dead code") and issues a warning. A new ``is_used`` attribute can be used situationally to suppress such warnings. The base scripts have some places where the static analysis lacks sufficient power to tell that values are being used safely (guaranteed to have been identified). In order to enable users to employ ``zeek -u`` on their own scripts without being distracted by these instances, this change also includes a new attribute, ``&is_assigned``, which can be associated with a variable or a record field to inform Zeek's analysis that the script writer asserts the value will be set, suppressing the associated warnings. - A Telemetry API was added to assist in gathering arbitrary runtime metrics and allows export to Prometheus. This is still work-in-progress, preliminary documentation for current, low-level API lives at https://github.com/zeek/zeek/wiki/Telemetry for now. - Experimental support for translating Zeek scripts to equivalent C++. The generated C++ can then be compiled directly into the ``zeek`` binary, replacing use of the interpreter and producing better runtime performance. See ``src/script_opt/CPP/README.md`` for a guide on how to use this feature. - Support for more generic session management. The NetSessions class has been renamed to SessionMgr (with the old name marked deprecated). The new class allows plugins to take advantage of session management similar to how Connection objects were handled previously, but without the need to be based on IP-based protocols. - The logging framework now provides a global policy hook, ``Log::log_stream_policy``. Like the existing filter-level hooks, handlers for the new hook can provide additional processing and veto the log write. The new hook runs once per write, prior to any filter-level policy hooks. Even when it vetoes, filter-level policy hooks still run, but cannot "un-veto" the write. - The ASCII writer gained a new option LogAscii::logdir, which can be used to change the logging output directory. - Added a ``--include-plugins`` argument to ``configure``. This argument takes a semicolon separated list of paths containing plugins that will be statically built into Zeek. - Added a ``--plugindir`` argument to ``configure`` to set the installation path for plugins. - The X509 analyzer now can check if a specific hostname is valid for a certificate. Two new BIFs were added for this, ``x509_check_hostname`` and ``x509_check_cert_hostname``. A new field ``sni_matches_cert`` that tracks this information was added to ``ssl.log``. - Added new functions to dynamically enable/disable file analyzers: - ``global enable_analyzer: function(tag: Files::Tag): bool;`` - ``global disable_analyzer: function(tag: Files::Tag): bool;`` - ``global analyzer_enabled: function(tag: Files::Tag): bool;`` - Zeek now includes its own BTest tooling in the distribution, enabling other tests (e.g. in Zeek packages) to use it. The ``$PREFIX/share/btest folder``, reported via ``zeek-config --btest_tools_dir``, includes: - ``scripts/`` for ``btest-diff`` canonifiers - ``data/`` for data files, including ``random.seed`` - ``data/pcaps`` for the test pcaps Configuring with ``--disable-btest-pcaps`` suppresses installation of the test pcaps. - The Supervisor now defaults to starting with a minimal set of Zeek scripts controlled by a new init file, ``base/init-supervisor.zeek``. One may still run it with a larger configuration by loading additional scripts, including ``init-default.zeek``, as always. (Bare mode continues to work as usual, reducing the configuration to a minimum.) The ``NodeConfig`` record has two new members, providing additional control over launched nodes. The ``env`` member allows setting environment variables in the launched process. The ``bare_mode`` member, an optional boolean, provides control over the bare-mode state of the new node. When not provided, the node inherits the bare-mode status of the Supervisor, and setting the variable enables/disables bare mode. - Zeek now includes an incomplete, preliminary version of the future cluster controller framework. Loading ``policy/frameworks/cluster/agent`` and/or ``policy/frameworks/cluster/agent`` in a Zeek running with the Supervisor will launch the corresponding cluster management node(s). An experimental management client, `zeek-client`, connects to the controller and lets you issue commands. It requires configuration with ``--enable-zeek-client``. This does not yet provide a functional substitute for ``zeekctl``, which users should continue to use for now. Changed Functionality --------------------- - The default IP-based transport protocols (UDP, TCP, and ICMP) have been moved to the packet analysis framework. This change allows us to move other analyzers in the future that better align with the packet analysis framework than they do with session analysis. - The input framework's handling of unset fields (commonly expressed as "-") in ingested data is now safer and more consistent. When reading data into records, Zeek now accepts unset fields in the input data only when the corresponding record field is ``&optional``. Unset fields for non-optional fields cause the input line to be skipped. Reading data into tables with complex index types (such as ``table[count, count] of string``) now also skips lines with unset fields, since such indexes require fields to be present. Note that this may change the behavior of existing scripts if you have unset fields in your input data. - The version field in ssh.log is now optional and will not be set if we cannot determine the version that was negotiated by the client and server. - Add a new field ``email_dest`` to NOTICEs, which defines where to send email to. The email-related NOTICE actions fill this now, and then emails will be sent to all recorded addresses at the end of NOTICE processing. This makes email generation more consistent and extensible. - Add page and email administrator to mails processed by hostnames extension. - SSL and X509 handling was significantly overhauled with the goal to make the data that is logged by Zeek more helpful and compact. This change means that there are significant changes to the default log files, as well as changes to functionality: - ``x509.log`` is now indexed by the sha256 of the certificate, with deduplication being automatically performed. By default, the same certificate is only logged once per day. This also means that the file ID is no longer present in X509 log. Similarly, ``ssl.log`` now contains hashes for X509 certificates. The hash function that is used for indexing the certificates is changeable by changing the ``X509::hash_function`` option. The time period after which a certificate is logged again can be configured by changing ``X509::relog_known_certificates_after``. By default deduplication of certificates is done across the entire cluster using broker. If this is not desired due to the higher communication overhead, this behavior can be disabled using ``X509::known_log_certs_use_broker``. - X509 certificates are, by default, no longer logged into files.log. This behavior is configurable and the previous default can be restored by changing the ``X509::log_x509_in_files_log`` option. - ``x509.log`` now tracks if a certificate was encountered as a end-host certificate or as a client certificate. - OCSP logging is now enabled by default. - ``ssl.log`` now no longer includes information about the certificate issuer and subject. This information is still available in X509.log. If you need this information in ``ssl.log``, the old behavior can be restored by changing the ``SSL::log_include_server_certificate_subject_issuer`` and ``SSL::log_include_client_certificate_subject_issuer`` configuration options. - ``ssl.log`` now contains a ``ssl_history`` field, which tracks which protocol messages were seen in an SSL/TLS connection. - We added a policy script ``ssl-log-ext.zeek`` which greatly extends the amount of protocol information logged to SSL.log. The script is not loaded by default. - We added a ``disable-certificate-events-known-certs.zeek`` policy script. This script will completely disable X509 events for known certificates over SSL/TLS connections. For Zeek installations in settings where you encounter a lot of certificates, this could improve the performance of your installation. Before enabling this script, make sure that you do not use any third-party scripts that depend on the X509 events. The script is not loaded by default. - The ICSI SSL Notary script was deprecated. This functionality is superseeded by newer approaches, like SCT validation (which is supported by Zeek). - ``extract-certs-pem.zeek`` was deprecated - it never really worked in cluster modes. A new policy script, ``log-certs-base64.zeek`` that can be used to log raw certificates was added instead. - The CT logs listed in ``ct-list.zeek`` are now derived from the list of CT logs that are accepted by Google Chrome. In the past, we allowed the list of all known CT logs. This no longer makes sense since nowadays logs exist that contain, e.g., only outdated or invalid certificates. If the old behavior is desired, you can re-add Logs to ``SSL::ct_logs``. - The Mozilla CA list was updated to the state of NSS 3.67. - SQLite was updated to 3.36.0. - The list of subnets that are considered private addresses was updated to mostly match IANA's list unroutable addresses. This brings it in line with Chrome, the W3C spec, and Python's ipaddress module. Removed Functionality --------------------- - Support for the RocksDB Broker data store was previously broken and unusable, so all code/options related to it are now removed. - Support for the ``ENABLE_MOBILE_IPV6`` compiler variable has been removed. Mobile IPv6 is now enabled by default. The ``--enable-mobile-ipv6`` option for ``configure`` now returns a warning that it will be removed in v5.1 and no longer has any effect. Deprecated Functionality ------------------------ - Lambda/closure support: automatic capturing of references to variables outside a lambda's scope is now deprecated. An explicit capture list which also specifies the desired copy-semantics is now required when writing lambda functions that refer to local variables of an outer scope. For examples, see: https://docs.zeek.org/en/master/script-reference/types.html#type-function - The ``IterCookie`` version of iteration over ``Dictionary`` and ``PDict`` objects was marked as deprecated. It was replaced by standard-library compatible iterators. This enables the use of standard constructs such as ranged-for loops to iterate over those objects. - The ``zeek::util::zeekenv()`` function is deprecated since use of all environment variables prefixed by ``BRO_`` is now removed and calling ``getenv()`` directly with ``ZEEK_`` environment variables can be done. - ``supervisor_rotation_format_func`` is renamed to ``archiver_rotation_format_func`` - The ``MemoryAllocation()`` function implemented by a number of interfaces is now deprecated. In testing we found that the values returned were mostly incorrect and weren't useful. The ``val_size`` and ``global_sizes`` BIF methods have also both been marked deprecated.

zeek v4.2.0

2022-01-05T18:29:51+00:00

Breaking Changes ---------------- - The existing ``Tag`` types in C++ (``zeek::Analyzer::Tag``, etc) have been merged into a single type called ``zeek::Tag``. This is a breaking change, and may result in plugins failing to build where they were relying on those types being different for function overloading and such. We attempted to include deprecated versions of the old types, but were unable to do so because of changes to return types from a number of methods. With this change, any uses of the `zeek::*::Tag` types will need to be replaced by `zeek::Tag`. New Functionality ----------------- - We now provide minimal official Docker images for the Zeek project via two repositories on the Docker hub, ``zeekurity/zeek`` and ``zeekurity/zeek-dev``. The former receives all Zeek release versions, with tag ``zeek:latest`` being the most recent. An image corresponding to our latest merge into the master branch is tagged at ``zeek-dev:latest``. The images run Debian and provide a full install of the Zeek distribution into ``/usr/local/zeek``. They do not set Zeek-specific entrypoints or provide any particular configuration for operationally running Zeek. To keep the images lightweight they also do not contain a development toolchain as needed for example to build a Zeek plugin. You can add any required system packages in a derived image, or install them directly in the running container. - Zeek now supports formatting the C++ code using clang-format. Also provided is a configuration for ``pre-commit`` to run clang-format when add new commits via ``git``. More details can be found at https://github.com/zeek/zeek/wiki/Coding-Style-and-Conventions#clang-format. - Experimental support for speeding up Zeek script execution by compiling scripts to a low-level form called "ZAM". You activate this feature by specifying ``-O ZAM`` on the command line. See ``src/script_opt/ZAM/README.md`` for more information. - Improvements for compiling scripts to C++ (an experimental optimization feature introduced in 4.1). The generated C++ now compiles much faster than previously, though it can still take quite a while when using C++ optimization on large sets of scripts. You can incrementally compile additional scripts using ``-O add-C++``. See ``src/script_opt/CPP/README.md`` for details. - The new flags --optimize-files=/pat/ and --optimize-funcs=/pat/ apply to both ZAM and compile-to-C++ script optimization. The first instructs Zeek to optimize any functions/hooks/event handlers residing in files matching the given pattern (unanchored). The second does the same but based on the function name, and with the pattern anchored (so for example --optimize-funcs=foo will optimize any functions named "foo" but not those named "foobar", or "MYSCOPE::foo"). The flags can be combined and can also be used multiple times to specify a set of patterns. If neither flag is used then optimization is applied to all loaded scripts; if used, then only to those that match. - The ``-uu`` flag for analyzing potentially unused record fields has been removed because, due to other changes in script optimization, keeping it would now require about 1,800 lines of code not otherwise needed. - The DNS analyzer has initial support for the SVCB and HTTPS types. The new events are ``dns_SVCB`` and ``dns_HTTPS``. - The ``find_str`` and ``rfind_str`` bifs now support case-insensitive searches. - Added a new plugin hook for capturing packets that made it through analysis without being processed called ``Plugin::HookUnprocessedPacket``. Currently ARP packets or packets with a valid IP-based transport header are marked as processed. This also adds an event called ``packet_not_processed`` that reports the same packets. - A new command-line option ``-c`` or ``--capture-unprocessed`` will dump any packets not marked as being processed, similar to the new hook and event above. - In Zeek plugins, the new cmake function ``zeek_plugin_scripts()`` should be used alongside ``zeek_plugin_cc()`` and related functions to establish dependency tracking between Zeek scripts shipped with the plugin and plugin rebuilds. Previously, updates to included Zeek scripts didn't reliably trigger a rebuild. - Added PacketAnalyzer::register_for_port(s) functions to the packet analyzer framework in script-land. This allows a packet analyzer to register a port mapping with a parent analyzer just like any other numeric identifier, while also adding that port to the now-global Analyzer::ports table used by BPF filtering. - Added AllAnalyzers::Tag enum type that combines the existing Analyzer::Tag, PacketAnalyzer::Tag, and Files::Tags into a single enum. The existing types still exist, but the new type can be used as an argument for functions/hooks/events that need to handle any of the analyzer types. - Added protocol detection functionality to the packet analyzer framework. Packet analyzers can register for protocol detection using the ``PacketAnalyzer::register_protocol_detection`` script function and implement the ``PacketAnalyzer::DetectProtocol`` method in C++. This allows packet analyzer plugins to detect a protocol via byte matching or other heuristics instead of relying solely on a numeric identifier for forwarding. - The JSON logger's new LogAscii::json_include_unset_fields flag provides control over how to handle unset "&optional" fields. By default it continues to skip such fields entirely. When redef'ing the flag to T it includes such fields, with a "null" value. This simplifies data import use cases that require fields to be present at all times, regardless of their value. - A new external testsuite, https://github.com/zeek/zeek-testing-cluster, focuses on testing the emerging controller framework. It leverages the new official Zeek Docker image for building docker-compose test setups, driven via btest. The Github CI setup now includes a workflow that deploys and runs this testsuite. - The GRE analyzer now supports the Aruba WLAN protocol type. Changed Functionality --------------------- - Zeek now treats non-atomic index types for sets and tables more consistently. Indexing now generally works with any types Zeek's hashing subsystem knows how to serialize, meaning that you can now also index with sets, vectors, patterns, and even tables. - The traditional TSV Zeek logs are now valid UTF8 by default. It's possible to revert to the previous behavior by setting ``LogAscii::enable_utf_8`` to false. - The ``SYN_packet`` record now records TCP timestamps (TSval/TSecr) when available. - The ``init-plugin`` script now focuses purely on dynamic Zeek plugins. It no longer generates Zeek packages. To instantiate new Zeek packages, use the ``zkg create`` command instead. - The ``ignore_checksums`` options and the ``-C`` command-line option now additionally cause Zeek to accept IPv4 packets that provide a length of zero in the total-length IPv4 header field. When the length is set to zero, the capture length of the packet is used instead. This can be used to replay traces, or analyze traffic when TCP sequence offloading is enabled on the local NIC - which typically causes the total-length of affected packets to be set to zero. - The existing tunnel analyzers for AYIYA, Geneve, GTPv1, Teredo, and VXLAN are now packet analyzers. - C++ unit tests are now compiled in by default and can be disabled by configuring the build with --disable-cpp-tests. We removed the former --enable-cpp-tests configure flag. Unit tests now also work in (static and dynamic) Zeek plugins. - This release expands the emerging cluster controller framework. Most changes concern internals of the framework. Agent/controller connectivity management has become more flexible: configuration updates pushed by the client can now convey the agent topology, removing the need to hardwire/redef settings in the controller. The new ClusterController::API::notify_agents_ready event declares the management infrastructure ready for use. zeek-client's CLI has expanded to support the new functionality. The framework is still experimental and provides only a small subset of ZeekControl's functionality. ZeekControl remains the recommended tool for maintaining your cluster. - Entries in ``http.log`` now record the original ``HOST`` headers. Previously, they would skip any port specification a header might include. Deprecated Functionality ------------------------ - The ``protocol_confirmation`` and ``protocol_violation`` events along with the corresponding ``Analyzer::ProtocolConfirmation`` and ``Analyzer::ProtocolViolation`` C++ methods are marked as deprecated. They are replaced by ``analyzer_confirmation`` and ``analyzer_violation`` which can also now be implemented in packet analyzers. - Declaring a local variable in an inner scope and then accessing it in an outer scope is now deprecated. For example, ``` if ( foo() ) { local a = 5; ... } print a; ``` is deprecated. You can address the issue by hoisting the declaration to the outer scope, such as: ``` local a: count; if ( foo() ) { a = 5; ... } print a; ```

zeek v4.0.5

2022-01-25T20:35:06+00:00

This release fixes the following bugs: - The highwayhash module was updated to fix a build failure on FreeBSD. https://github.com/zeek/zeek/pull/1809 - A number of fixes for various problems on the CI infrastructure. https://github.com/zeek/zeek/pull/1809 https://github.com/zeek/zeek/pull/1579 https://github.com/zeek/zeek/commit/a84d1bffe9ff596b8d98dc6e9461ed8877141dbb https://github.com/zeek/zeek/commit/33a4e1cb6657d0a01e69addf2af26dcdd9da1ab0 - Fixed an issue with log rotation on Linux systems using shadow files. https://github.com/zeek/zeek/pull/1817 - Writers were not being cleaned up correctly when recreating log streams with the same ID as an existing stream. This could lead to a crash. https://github.com/zeek/zeek/pull/1877 - IP packets with bad/incorrect IP header lengths were not reporting weirds as they should be. https://github.com/zeek/zeek/pull/1569

zeek v4.0.6

2022-04-21T22:27:30+00:00

zeek v4.2.1

2022-04-21T22:27:36+00:00

This release fixes the following security issues: - Fix potential unbounded state growth in the FTP analyzer when receiving a specially-crafted stream of commands. This may lead to a buffer overflow and cause Zeek to crash. Due to the possibility of this happening with packets received from the network, this is a potential DoS vulnerabilty. Thank you to Jason Ish at OISF for reporting this vulnerability. This release fixes the following bugs: - Ensure both protocol and analyzer confirmation and violation events can be called. https://github.com/zeek/zeek/pull/2042 - Empty table constructors with &default attributes may cause a crash. https://github.com/zeek/zeek/commit/18fe9d84f602c90724dcec0e349789932ab9f931 - Fix a bug in ZAM when a function containing a loop is inlined. https://github.com/zeek/zeek/pull/2024 - Reduce the interpreter frames generated by ZAM when inlining function bodies. https://github.com/zeek/zeek/pull/2025 - Restore providing location information to LoadFile hooks. https://github.com/zeek/zeek/pull/2050 - Allow analyzer violations to explicitly set tag. This makes it consistent with analyzer confirmations. https://github.com/zeek/zeek/pull/2043 - Fix a number of bugs with robust dictionary iteration. https://github.com/zeek/zeek/pull/2040 - Fix compile error on GCC 12/Ubuntu 22.04 https://github.com/zeek/gen-zam/pull/1 - Fix missing "Reporter" entries when reporting hooks via `zeek -NN`. https://github.com/zeek/zeek/issues/2052

zeek v4.0.7

2022-06-03T16:58:25+00:00

This release fixes the following security issues: - Fix potential hang in the DNS analyzer when receiving a specially-crafted packet. Due to the possibility of this happening with packets received from the network, this is a potential DoS vulnerability. Thank you to Google's OSS-Fuzz project for reporting this vulernability. This release fixes the following bugs: - Fix issue with broken libpcaps that return repeat packets, most notably the version provided with Myricom hardware. https://github.com/zeek/zeek/issues/724

zeek v4.2.2

2022-06-03T17:01:19+00:00

zeek v5.0.0

2022-06-03T20:19:51+00:00

Breaking Changes ---------------- - Zeek now requires at least CMake version 3.15.0. - If Zeek is configured with support for included Spicy (the default) we now require at least Flex version 2.6 and its development headers, at least Bison version 3.3, and GCC version 8.3 or Clang version 9.0 or higher. - The script-land ``union`` and ``timer`` types have been removed. They haven't had any actual semantics backing them for some time and shouldn't have functioned in any useable way. We opted to skip the deprecation cycle for these types for that reason. - Broker now uses a new network backend with a custom network protocol that is incompatible with the pre-5.0 backend. In practice, this means Zeek 4.x will not be able to exchange events with Zeek 5.x. Going forward, this new backend will allow us to keep the Broker protocol more stable and add new capabilities in a backwards compatible way. New Functionality ----------------- - The Zeek cluster management framework now supports operational use of single-instance clusters, meaning setups in which all Zeek cluster nodes (manager, workers, etc) reside on a single machine. The framework builds upon Zeek's Supervisor (``-j`` is a requirement) and includes three components: (1) A cluster controller, loaded via ``policy/frameworks/management/controller`. The controller is the central management entity for a Zeek cluster. It interacts with one or more agents to enact instructions delivered by a management client. The controller is stateful and can persist cluster state to disk. (2) One or more cluster agents, loaded via ``policy/frameworks/management/agent`. Each agent assists the controller in managing the local instance (typically a machine or container) of Zeek cluster nodes. Agents interact with the local Supervisor to control nodes and peer directly with running Zeek nodes via Broker to carry out node-local tasks. (3) zeek-client, the cluster management client. A standalone client running outside of Zeek, it's now installed by default, alongside the other executables. The client supports uploading cluster configurations to the controller, deploying them, retrieving them, checking cluster node status, restarting individual cluster nodes, and retrieving the current value(s) of global identifiers in one or more Zeek nodes. Controller and agent come pre-configured for single-instance deployments of Zeek clusters, with automated log rotation and archival via Zeek's ``zeek-archiver`` tool. For further details on the framework, please refer to the Zeek documentation. zeekctl remains included and offers more knobs and features than the management framework. It remains the recommended solution for multi-machine clusters and those needing rich management capabilities. - Zeek now automatically warns about unused functions, hooks, and event handlers. The intent behind these checks is to catch instances where the script writer has introduced typos in names, or has forgotten to remove code that's no longer needed. For functions and hooks, "unused" means the function/hook is not exported or in the global scope (nor deprecated), and no "live" (i.e., not "unused") function/hook/event handler calls it. For event handlers, "unused" means that the event engine does not generate the event, nor do any "live" function/hook/event handler generates (and the event handler is not deprecated). The warnings can occasionally be or inappropriate or incorrect, such as due to the use of conditional code (which Zeek does not try to incorporate into its analysis), or events sent via Broker. You can mark such instances using the ``&is_used`` attribute to suppress the associated warning. You can also suppress all such warnings using the ``--no-unused-warnings`` command-line option. - A new feature, ``--profile-scripts[=file]`` instructs Zeek to write (upon termination) a profile of every function body to the given file (default: stdout), along with some aggregate profiles. A function body is associated with a function or a BiF (they can only have one), or a hook or event handler (they can have multiple). The profile is in TSV, quasi-Zeek-log format. It includes (1) the name of the script body, (2) its location, (3) its type (e.g., "BiF" or "function"), (4) how many times it was called, (5) the total CPU time that accumulated during its execution, (6) how much of that time was due to execution of its "children" (other functions it called), (7) the total memory allocated during its execution (approximately), and (8) how much of that memory was due to its children. Note that profiling is expensive and may not be suitable for execution on live traffic. - Zeek now includes support for building Spicy and spicy-plugin as part of Zeek. This feature is enabled by default, and can be turned off by passing the ``--disable-spicy`` flag to ``./configure``. - Zeek now supports generation and replay of event traces via the new ``--event-trace`` / ``-E`` command-line options. For details, see: https://docs.zeek.org/en/master/quickstart.html#tracing-events - Zeek now features limited TLS decryption capabilities. This feature is experimental and only works for TLS 1.2 connections that use the TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 ciphersuite. Furthermore Zeek requires access to the pre-master secret of each TLS connection. Typically this functionality will be most useful when analyzing trace-files where the TLS client recorded the key material. For more details and examples how to use this functionality, see the TLS Decryption documentation at https://docs.zeek.org/en/master/frameworks/tls-decryption.html - Zeek now provides WebSocket support for exchanging events with external clients. To enable it, call `Broker::listen_websocket()`. Zeek will then start listening on port 9997 for incoming WebSocket connections. Zeek/Broker communicates via JSON messages. See the Broker documentation for a description of the message format expected over these WebSocket connections at https://docs.zeek.org/projects/broker/en/current/web-socket.html. - Compiling scripts to C++ (via ``-O gen-C++``) is now "feature complete", i.e., the compiler should be able to process any script - other than those potentially affected by ``@if`` conditionals - regardless of which Zeek language constructs it uses. That said, such compilation remains experimental and only lightly tested. - ZAM script optimization (``-O ZAM``) now includes some major performance improvements. - The new --with-gen-zam configure flag and its corresponding GEN_ZAM_EXE_PATH cmake variable allow reuse of a previously built Gen-ZAM code generator. This aids cross-compilation: the Zeek build process normally compiles Gen-ZAM on the fly, but when cross-compiling will do so for the target platform, breaking its use on the host platform. Gen-ZAM is similar to binpac and bifcl in this regard. Like binpac and bifcl, it's now also available as a standalone git repository and hooked into the Zeek distribution as a submodule. - Zeek now uses the c-ares (https://c-ares.org) library for performing DNS requests, replacing an old custom implementation of a DNS resolver. Switching to this library simplifies the DNS code, adds support for IPv6 lookups, and adds the ability to support more DNS request types in the future. - Two new BiFs, val_footprint() and global_container_footprints(), offer substitutes for the deprecated val_size() and global_sizes() BiFs. A value's "footprint" is the number of elements it includes either directly or indirectly. The number is not meant to be precise, but rather comparable: larger footprint correlates with more memory consumption. - The Supervisor features two new events, ``Supervisor::node_status`` and ``SupervisorControl::node_status``, to notify recipients of the fact that the stem process has (re-)started a node. The events indicate the node's name and its new PID, reflecting the information the stem already shares with the Supervisor core. - The new ``configure --statedir`` option lets you adjust the installation's persistent state directory. It defaults to ``var/lib`` under your Zeek installation's root directory. - The ``base/misc/installation.zeek`` script provides your Zeek installation's key directories. Changed Functionality --------------------- - The behavior of the ``=``, ``+=``, and ``-=`` operators has been expanded and unified. It now covers ``{ ... }`` initializer lists, supports cross-product initialization, enables ``+=`` for table, set, vector and pattern values, similarly allows ``-=`` for table and set values, and supports listing multiple sets for ``+=`` initialization. For details, see: https://docs.zeek.org/en/master/script-reference/operators.html#assignment-operators - The is_num(), is_alpha(), and is_alnum() BiFs now return F for the empty string. - Type mismatch error messages now print the easier to understand type names, instead of the fully expanded raw type. - Errors caused by setting a filter at start-up are now more informative about what actually caused the error, including with the default ``ip or no ip`` filter. - Log messages about errors in input files are now more informative about where errors occured. - The ``--enable-zeek-client`` configure flag has been removed and is now the default. The new ``--disable-zeek-client`` flag allows users to skip installation of the client. Deprecated Functionality ------------------------ - For developers of asynchronous BiFs, or other uses of ``when`` statement triggers, the versions of ``Trigger::Cache`` and ``Trigger::Lookup`` that take ``const CallExpr*`` parameters have been deprecated. The replacements take ``const void*`` parameters. Usually all you'll need to do is replace previous uses of ``Frame::GetCall`` with ``Frame::GetTriggerAssoc``. - The ``--caf-root`` option of ``zeek-config`` was made obsolete by the changes to how we include CAF and has been marked as deprecated.

zeek v4.0.8

2022-08-26T20:23:36+00:00

zeek v5.0.1

2022-08-26T20:25:46+00:00

This release fixes the following security issues: - Fix a possible overflow and crash in the ARP analyzer when receiving a specially crafted packet. Due to the possibility of this happening with packets received from the network, this is a potential DoS vulnerability. - Fix a possible overflow and crash in the Modbus analyzer when receiving a specially crafted packet. Due to the possibility of this happening with packets received from the network, this is a potential DoS vulnerability. - Fix two possible crashes when converting IP headers for output via the ``raw_packet`` event. Due to the possibility of this happening with packets received from the network, this is a potential DoS vulnerability. Note that the ``raw_packet`` event is not enabled by default so these are likely low-severity issues. - Fix an abort related to an error related to the ordering of record fields when processing DNS EDNS headers via events. Due to the possibility of this happening with packets received from the network, this is a potential DoS vulnerability. Note that the ``dns_EDNS`` events are not implemented by default so this is likely a low-severity issue. Thank you to Google's OSS-Fuzz project for reporting all of the above issues. This release fixes the following bugs: - Fix a number of typos in the weak-keys.zeek script in the SSL framework. https://github.com/zeek/zeek/issues/2229 - Fix build of internal Spicy when using the --disable-cpp-tests configure flag. https://github.com/zeek/zeek/pull/2256 - Avoid calling ``safe_realloc`` unnecessarily from ``ODesc::Grow()``, providing a peformance improvement in some cases. https://github.com/zeek/zeek/pull/2244 - Remove use of fallible ``get_conn_transport_proto()`` in ``analyzer_violation`` event handlers. https://github.com/zeek/zeek/pull/2288 - Remove a warning when receiving packets with invalid or unknown IP protocol types, preventing it from spamming reporter.log. https://github.com/zeek/zeek/commit/40b1452905c0eed4d96300ce1aaf87a08166e396 - Fix workers failing to peer with proxies if they take too long to start https://github.com/zeek/zeek/issues/2334 - Fix Zeek build failures when building against an external version of Spicy https://github.com/zeek/zeek/pull/2364 - Update Spicy to version 1.5.1 and spicy-plugin to 1.3.17. https://github.com/zeek/zeek/pull/2368

zeek v4.0.9

2022-09-19T20:14:07+00:00

This release fixes the following security issues: - Fix a possible overflow and crash in the ICMP analyzer when receiving a specially crafted packet. Due to the possibility of this happening with packets received from the network, this is a potential DoS vulnerability. - Fix a possible overflow and crash in the IRC analyzer when receiving a specially crafted packet. Due to the possibility of this happening with packets received from the network, this is a potential DoS vulnerability. Note that this requires the ``irc_whois_channel_line`` to be handled. This event is not handled by default, so this is likely a low-severity issue. - Fix a possible overflow and crash in the SMB analyzer when receiving a specially crafted packet. Due to the possibility of this happening with packets received from the network, this is a potential DoS vulnerability. Note that this requires the ``smb1_transaction_secondary_request`` to be handled. This event is not handled by default, so this is likely a low-severity issue. - Fix two possible crashes when converting IP headers for output via the ``raw_packet`` event. Due to the possibility of this happening with packets received from the network, this is a potential DoS vulnerability. Note that the ``raw_packet`` event is not enabled by default so these are likely low-severity issues.

zeek v5.0.2

2022-09-19T20:17:54+00:00

zeek v5.1.0

2022-09-20T19:56:28+00:00

Breaking Changes ---------------- - The ``Packet::{l2,l3}_checksummed`` variables were reworked to correctly match the network layers that they apply to. A new ``Packet::l4_checksummed`` variable was added to cover the transport layer. See this GitHub issue for more detail: https://github.com/zeek/zeek/issues/2183. - The STREAM mode of the ASCII reader now behaves like `tail -F`: when file is removed/replaced, it will start tracking the new file. See https://github.com/zeek/zeek/pull/2097 for more detail - The Dictionary and PDict classes are now C++ templates. This may cause plugin/package builds to fail due to needing to modify uses of them to match. - By default, ``files.log`` does not have the fields ``tx_hosts``, ``rx_hosts`` and ``conn_uids`` anymore. These have been replaced with the more commonly used ``uid`` and ``id`` fields. They can be re-instantiated by loading the following policy script through ``local.zeek``: @load frameworks/files/deprecated-txhosts-rxhosts-connuids Note, however, that this script will be removed with Zeek 6.1. Consumers of ``files.log`` should convert to using the singular ``uid`` and ``id`` fields instead. - The ``files.log`` is now unrolled consistently. That is, when Zeek associates multiple connections with a single file, each of these connections will result in individual ``files.log`` entries with unique connection uids, all sharing the same file uid. This unrolling behavior always existed in a Zeek cluster when the network connections involved in a file transfer are load-balanced to different workers. Due to this affecting only a marginal ratio of files on real-world networks, unrolling the log was chosen as the more efficient approach over making the current logic cluster aware. The ``seen_bytes`` and ``missing_bytes`` fields of a ``File::Info`` record continue to represent the total number across all connections seen by the current instance of Zeek. - The barnyard2 policy scripts have been removed. The integration with the Barnyard2 project used the pre-Broker Broccoli library, which got removed in Zeek 3.0. - The unified2 analyzer and accompanying scripts have been removed without deprecation. - The return value of ``packet_analysis::IP::ParsePacket`` has changed to return enum values. This makes it easier to look at the result and immediately know what it means. Unfortunately, because we can't overload a method on the return value alone, we aren't able to deprecate the original version of the method. This may cause build of packages to fail if they were using this method. - Conditional directives (``@if``, ``@ifdef``, ``@ifndef``, ``@else`` and ``@endif``) can not be placed directly following ``if``, ``for`` or ``while`` statements anymore. This was interpreted non-intuitively and could lead to subtle bugs. The statement following the directive was placed outside of its intended block. Placing braces after ``if``, ``for`` or ``while`` should result in the intended behavior. - The ``bro`` symlink to the ``zeek`` binary has finally been removed. New Functionality ----------------- - Added support for the /s regular expression modifier. Using this modifier in patterns in Zeek scripts will cause the '.' character to also match newline characters. - Added a new telemetry framework for providing high-level access to Zeek's metric subsystem. This framework allows script writers to use different metric types (counters, gauges and histograms) for tracking metrics without using lower-level BiFs from ``telemetry.bif``. Additionally, metrics can now be accessed from script land using ``Telemetry::collect_metrics()`` and ``Telemetry::collect_histogram_metrics()``. The framework is located in ``base/frameworks/telemetry``. In addition to the Prometheus endpoint for metrics export that has existed since Zeek 4.1, two new log streams, ``telemetry.log`` and ``telemetry_histogram.log``, can be enabled by loading ``policy/frameworks/telemetry/log``. This policy script is included in ``local.zeek`` by default. For further details on the framework and examples, please refer to the Zeek documentation. - Allow redef'ing the ``&log`` attribute of record fields: redef Notice::Info$email_dest -= { &log }; While the syntax allows for any attribute, only ``&log`` is supported. The semantics for other record field attributes are not easy to grasp and there were no obvious use-cases identified. - Introduced a global ``disabling_analyzer()`` hook to allow vetoing calls to ``disable_analyzer()``. The contract is simple: Any script can veto a ``disable_analyzer()`` call by breaking from this hook. The decision is local to the script taking into account any state attached to the connection or state stored elsewhere. A script breaking from the hook takes over responsibility to call ``disable_analyzer()`` at a later point when it finds the condition due to which it vetoed fulfilled (which may be never). - Add support for iterating over indices and values of a vector using the same syntax as used for iterating over key-value pairs of tables, where ``value`` will be set to ``vec[idx]``. local vec = vector("zero", "one", "two"); for ( idx, value in vec ) print idx, value; - The Supervisor framework now allows better control over where to place additional scripts in the load sequence of new nodes. It previously always loaded such scripts after any other user scripts, which could create pitfalls when users expected their scripts to run last. Scripts placed in ``NodeConfig``'s new ``addl_base_scripts`` and ``addl_user_scripts`` fields will be loaded after the base scripts (and thus before any user scripts) and after any user scripts, respectively. The old ``NodeConfig$scripts` field still adds to the very end and is deprecated. - Added a new script-level option ``max_changes_per_connection`` to limit the number of ``tunnel_changed`` events that can be sent for a connection. This helps prevent log spam from connections that regularly swap. The option defaults to 5, and can be set to zero do disable the limiting. - Added a new BIF ``bytestring_to_float`` for converting 4-byte bytestrings to float values. - Added a new BIF ``pow``. - Added new bit-shift operators ``<<`` and ``>>`` for use in scripts. - Added a new BIF ``table_keys`` which returns a ``set`` of keys from a table. - Added a new BIF ``table_values`` which returns a ``vector`` of keys from a table. - Added new fields to the Modbus log for the Modbus PDU type, the transaction ID, and the unit ID. See https://github.com/zeek/zeek/pull/2281 for more information. - Added support for parsing TCP option 27, and fixed validation of lengths for TCP options 28, 29, and 34. - Added new packet-analzyer to handle the DLT_LINUX_SLL2 PCAP link type. Changed Functionality --------------------- - The SSL analyzer now determines the direction of the SSL/TLS session by examining the packets, and no longer assumes that the connection originator is the client. Due to this, the ``is_orig`` field in all SSL/TLS events was renamed to ``is_client``. Furthermore, the ``ssl_history`` now can indicate that the connection was flipped (meaning that it is not in the normal order of the originator is the client) using the ``^`` character. A new ``ssl_connection_flipped`` is raised when the connection is flipped. Furthermore, a ``SSL_unclear_connection_direction`` weird is raised when we cannot determine the connection direction, because both sides of the connection send packets that are associated with being a client/server. - The default logging directory is now set globally across all log writers through ``Log::default_logdir``. - Calling `Option::set()` when Zeek is terminating is now a noop and returns `F`. This prevents callbacks into script-land through change handlers when parts of the environment have already been torn down. - When running in cluster mode, the manager by default now imports metrics from all other cluster nodes and opens port 9911/tcp for Prometheus metrics exposition. - The ``smb2_file_delete`` event will now be raised for SMB2 ``CREATE`` requests marked with the ``FILE_DELETE_ON_CLOSE`` option. - Fixed ``bytestring_to_count`` to handle 3-, 5-, 6-, and 7-byte strings. - Updated the list of DNS type strings to reflect the correct mappings. Note that the following mappings where changed: - type 30 is now NXT instead of EID - type 31 is now EID instead of NIMLOC - type 32 is now NIMLOC instead of NB> NB was originally defined in RFC 1002, but was later made obsolete and replaced by NIMLOC. Similarly, type 33 was originally defined as NBSTAT, but was replaced by SRV (Zeek had this one correct already). Removed Functionality --------------------- - The barnyard2 policy scripts have been removed. - The unified2 analyzer and accompanying scripts have been removed. Deprecated Functionality ------------------------ - The ``PDict`` class is now an alias to ``Dictionary`` and has been deprecated. Use ``Dictionary`` directly, passing a pointer type to the template. - ``LogAscii::logdir`` and per-writer log directories have been deprecated in favor of the new ``Log::default_logdir``. - The ``HOOK_BRO_OBJ_DTOR`` hook and associated methods have been deprecated. They are replaced by the ``HOOK_OBJ_DTOR`` hook and methods. - The ``bro_int_t`` and ``bro_uint_t`` types have been deprecated and replaced by ``zeek_int_t`` and ``zeek_uint_t``. - The ``bro_inet_ntop.h`` and ``bro_inet_ntop.c`` files have been deprecated and replaced by ``zeek_*`` files. - The ``BRO_PLUGIN_API_VERSION`` has been deprecated and replaced by ``zeek::PLUGIN_API_VERSION``. - The ``misc/scan.zeek`` script has been marked for removal in Zeek 6.1. Use github.com/ncsa/bro-simple-scan instead. - The Supervisor framework's ``NodeConfig$scripts`` field has been deprecated and marked for removal in Zeek 6.1. Use ``NodeConfig$addl_user_scripts`` instead.

zeek v5.0.3

2022-11-08T23:12:54+00:00

Zeek 5.0.3 ========== This release fixes the following security issues: - Fix an issue where a specially-crafted FTP packet can cause Zeek to spend large amounts of time attempting to search for valid commands in the data stream. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. - Fix a possible overflow in the Zeek dictionary code that may lead to a memory leak. Due to the possibility of this happening with packets received from the network, this is a potential DoS vulnerability. - Fix an issue where a specially-crafted packet can cause Zeek to spend large amounts of time reporting analyzer violations. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. - Fix a possible assert and crash in the HTTP analyzer when receiving a specially-crafted packet. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. - Fix an issue where a specially-crafted HTTP or SMTP packet can cause Zeek to spend a large amount of time attempting to search for filenames within the packet data. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. - Fix two separate possible crashes when converting processed IP headers for logging via the `raw_packet` event handlers. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. This event handler is not enabled by default, so this can be considered low-priority. This release fixes the following bugs: - Fix a possible crash with ``when`` statements where lambda captures of local variables sometimes overflowed the frame counter. - Reduced the amount of ``analyzer_confirmation`` events that are raised for packets that contain tunnels. - Fix a long-standing bug where TCP reassembly would not function correctly for some analyzers if ``dpd_reassemble_first_packets`` was set to false. - Fix a performance bug in the Zeek dictionary code in certain cases, such as copying a large number of entries from one dictionary into another. - Fix a performance issue when inserting large numbers of elements into a Broker store when ``Broker::scheduler_policy`` is set to ``stealing``. - Fix a Broker performance issue when distributing large amounts of data from the input framework to proxies/workers at startup. - Fix an issue with messaging between proxies and workers that resulted in error messages being reported. - Updated the list of DNS type strings to reflect the correct mappings. Note that the following mappings where changed: - type 30 is now NXT instead of EID - type 31 is now EID instead of NIMLOC - type 32 is now NIMLOC instead of NB> NB was originally defined in RFC 1002, but was later made obsolete and replaced by NIMLOC. Similarly, type 33 was originally defined as NBSTAT, but was replaced by SRV (Zeek had this one correct already).

zeek v5.0.4

2022-11-22T19:00:44+00:00

This release fixes the following security issues: - A specially-crafted series of HTTP 0.9 packets can cause Zeek to spend large amounts of time processing the packets. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. The fix included is to report a weird and an analyzer violation for streams that include such malformed packets. - A specially-crafted FTP packet can cause Zeek to spend large amounts of time processing the command. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. The fix included is to cap FTP commands at 100 bytes and report a weird for violations. - A specially-crafted IPv6 packet can cause Zeek to overflow memory and potentially crash. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. The fix included is better length checking and reporting a weird for violations. This release fixes the following bugs: - Fix a potential stall in Broker’s internal data pipeline. This bug was manifesting itself as logging completely stopping on certain combinations of system configuration (number of workers) and the amount of data being received from the network.

zeek v5.1.1

2022-11-22T19:04:11+00:00

zeek v5.0.5

2023-01-19T18:01:13+00:00

This release fixes the following bugs: - Update broker to version 2.3.6. This broker release fixes some failures when building against Python 3.11 and above.

zeek v5.1.2

2023-02-01T17:15:37+00:00

This release fixes the following security issues: - A missing field in the SMB FSControl script-land record could cause a heap buffer overflow when receiving packets containing those header types. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. The fix included is to add the missing field to the record type. - Receiving a series of packets that start with HTTP/1.0 and then switch to HTTP/0.9 could cause Zeek to spend a large amount of time processing the packets. Due to the possiblity of receiving these packets from remote hosts, this is a DoS risk. The fix included is to ensure that the HTTP analyzer switches versions correctly. - Receiving large numbers of FTP commands sequentially from the network with bad data in them could cause Zeek to spend a large amount of time processing the packets, and generate a large amount of events. Due to the possiblity of receiving these packets from remote hosts, this is a DoS risk. The fix included adds better validation for FTP command strings and request codes. This release fixes the following bugs: - Zeek could throw a scripting error when receiving SMB1 packets containing connect_andx_response messages prior to receiving an associated request. A new weird ``smb_tree_connect_andx_response_without_tree`` will now be raised in these cases and the packet will be ignored. - A performance regression from 4.2 to 5.0 when reading pcap files related to Broker's internal clock was fixed. - Notices created for files transferred over multiple connections will now be associated with one of the connections rather than none. - A new ``file_over_new_connection`` event was added to the Intel framework, for use when receiving files over established connections (for example, HTTP). This ensures that the filename in the file info record is populated correctly. - The error message returned when trying use invalid enums in scripts now correctly includes the script location. - Analyzer confirmation events are no longer raised per-packet for tunnels. This greatly reduces the amount of events/log messages on networks containing a lot of tunnel traffic. - The Teredo packet analyzer is now less greedy when matching packets. This should greatly reduce the amount of log messages from that analyzer, as well as reduce the number of analyzer violations reported by it.

zeek v5.0.6

2023-02-01T17:18:04+00:00

zeek v5.2.0

2023-02-03T17:51:59+00:00

Breaking Changes ---------------- - Zeekctl now assigns network ports to workers starting at port 27760. This fixes an issue where workers were starting up with ports within Linux's ephemeral port range, and were potentially failing to startup due the ports already being in use. This change may require changes in firewall/routing configurations between hosts in a Zeek cluster. This should not affect clusters running on FreeBSD, as that OS uses a different range for ephemeral ports. - Zeekctl support for the AF_PACKET plugin specific options (af_packet_*) has been integrated into zeekctl directly. Upgrading to Zeek 5.2 with a builtin AF_PACKET packet source (default on Linux) requires an upgrade of zeekctl to the version bundled with Zeek to continue using these options. - The blank identifier ``_`` cannot be used in expressions and options anymore. Outside of obfuscation exercises, this should have little real-world impact. - A new ``mysql_eof`` event has been introduced and the ``mysql_ok`` event is not raised in its place or artificially anymore. The base scripts were adapted accordingly. Users of ``mysql_ok()`` likely need to switch to ``mysql_eof()``. - Zeek will now exit at startup if an external plugin (e.g. from a package) is discovered to have the same name as a built-in plugin. See below for the change regarding the AF_PACKET plugin now being built-in for an example of this potentially being triggered. - DNS query type strings were updated to match the current standardized list of strings. This changes the string reported for a small subset of query types: 30: Changed from "EID" to "NXT" 31: Changed from "NIMLOC" to "EID" 32: Changed from "NB" to "NIMLOC" - The ``--with-caf`` option for the ``configure`` script was removed. Broker now requires specific versions of CAF per Zeek release, and passing an externally-built version of CAF often lead to build failures. New Functionality ----------------- - Experimental support added for building and running Zeek on Microsoft Windows environments. This is considered experimental due to the fact that our standard testing setup (btest) doesn't run properly on Windows. This will be fixed in the future. In the meantime we have done some basic testing against builds done with Visual Studio 2019. Information on how to build on Windows is available in the Zeek documentation. Note also that Spicy is currently unsupported and will be fixed in the future. The feature as checked into the repository is not considered production-ready. There are many bugs to squash and features to improve, and we will be steadily fixing things over the next few months. The Zeek team wants to give a huge thank you to the team at Microsoft for all of their effort in completing this port. - Zeek container images are now being published to zeek/zeek and zeek/zeek-dev rather than zeekurity/zeek and zeekurity/zeek-dev on Docker Hub (and continue to be published to public.ecr.aws) Further, container images for amd64 and arm64 platforms are now available. Main driver for the latter was to allow usage of the official container images on Apple's M1 systems. - Zeekctl support for using ``af_packet`` as ``lb_method`` has been added. - New ``analyzer_confirmation_info`` and ``analyzer_violation_info`` events with accompanying record types ``AnalyzerConfirmationInfo`` and ``AnalyzerViolationInfo`` have been added. These supersede ``analyzer_confirmation`` and ``analyzer_violation``, which have been deprecated. - Added helpers to determine protocol, packet or file analyzer based on ``AllAnalyzers::Tag`` values named ``is_protocol_analyzer()``, ``is_packet_analyzer()`` and ``is_file_analyzer()``. - File analyzers can now raise analyzer violations to the script-layer via the new ``AnalyzerViolation()`` method. - Packet and file analyzers can now be disabled and enabled at runtime using the ``Analyzer::enable_analyzer()`` and ``Analyzer::disable_analyzer()`` wrappers. While initially for protocol analyzers only, these have been extended to work for packet and file analyzers. This now allows to leverage ``Analyzer::disabled_analyzers`` for these kinds of analyzers. - The blank identifier ``_`` can now be used to ignore loop variables of different types without type clash errors. This allows to do the following within the same scope: local vec = vector("a", "b", "c"); for ( _, v in vec ) print v; for ( i, _ in vec ) print v; Iterating over only the values of a table can be done by ignoring the full index with a single blank identifier. Due to the internal structure of Zeek tables, this can result in a performance improvement. local tab = table(["a", 1, T] = "a1T", ["b", 2, F] = "b2f"); for ( _, v in tab ) print v; It's also possible ignore individual indices of different types with the blank identifier ``_`` as follows: for ( [_, i, _], v in tab ) print i, v; As noted under breaking changes, the blank identifier ``_`` cannot be referenced in expression anymore. - It is now possible to put trailing commas within table, vector, set and record construction. For example, the following code is now valid, which can make for more uniform style and smaller diffs. local vec = vector( "1", "2", ); local tab: table[string] of count = [ ["a"] = 1, ["b"] = 2, ]; Function calls and record constructors can have a trailing comma after the last argument. Analyzer::schedule_analyzer( chan$orig_h, chan$resp_h, chan$resp_p, Analyzer::ANALYZER_FTP_DATA, 5mins, ); - Re-introduce event groups. Allow the ``&group`` attribute on event and hook handlers for annotating them with one or more event groups. These groups can be disabled and enable during runtime. Disabling an event group implies disabling all event and hook handlers that are part of it. The main difference to a previous implementation in (very) old Zeek versions is its granularity: It is now possible to toggle individual event handlers (event handler bodies). The original implementation worked at the level of events, disabling or enabling all event handlers for a given event at once. Additionally, support for hooks was added as these are structurally similar to events. The BIFs ``disable_event_group()`` and ``enable_event_group()`` are re-instantiated and allow controlling event groups based on the group attribute. Additionally, event and hook handlers are implicitly placed into event module groups based on the module they are implemented in. All events implemented in a given module can be toggled with ``disable_module_events()`` and ``enable_module_events()``. - Extend the ``Logging::Stream`` record with an ``event_groups`` field and toggle these during ``Log::disable_stream`` and ``Log::enable_stream`` invocations. This allows for explicit/manual opt-in performance optimizations by turning off event handlers at runtime that are only needed for log generation. - On Linux, the AF_PACKET packet source plugin (https://github.com/zeek/zeek-af_packet-plugin) is included as a builtin plugin by default. To select this packet source, prefix the interface name with ``af_packet``. zeek -i af_packet::eth0 - Usage of ``break`` and ``next`` statements is now validated. It was previously possible to place these outside of ``for``, ``while`` or ``switch`` statements without any error indication. - Add two BIFs ``get_identifier_declaring_script()`` and ``get_record_field_declaring_script()`` to query the declaring scripts for identifiers and record fields from Zeek scripts. - Extend the SSH analyzer to produce new events (``ssh2_ecc_init``, ``ssh2_gh_gex_init``, ``ssh2_gss_init``, ssh2_rsa_secret``) to detect when SSH client and server roles are reversed. - Analyzers found in the new ``Analyzer::requested_analyzers`` set will be enabled at ``zeek_init()`` time. The set can be populated via :zeek:see:`redef`. This change only has an effect in settings where ``Analyzer::disable_all`` is changed to ``T``. By default, all analyzers continue to be enabled. - A new ``analyzer.log`` was added to log all analyzer violations and optionally analyzer confirmations. This log can be useful during development of new analyzers as well as for collecting operational data in production environments. - Expose configurability of for SQLite's synchronous and journal_mode PRAGMAs for SQLite backed Broker data stores. Setting these to synchronous=normal and journal_mode=wal can significantly improve throughput at the cost of some durability in the presence of power loss or OS crash. In the context of Zeek, this is likely more than acceptable. Additionally, add integrity_check and failure_mode options to support detecting and deleting corrupted SQLite database at store initialization. - A new ``join_string_set`` BIF was added, replacing the existing script-level version from utils/strings.zeek. - A new ``&ordered`` attribute for tables and sets was added. This attribute causes iteration over a table/set to return elements in the order of their insertion. - A new ``-D`` argument was added to the ``configure`` script to allow passing parameters directly to the underlying CMake call. - Added parsing for the challenge and response fields to the NTLM analyzer. - A new ``FTP::max_command_length`` value was added to script-land, defaulting to 100. This value is used by the FTP analyzer to limit the size of commands accepted by the analyzer. A ``FTP_max_command_length_exceeded`` weird is raised for any violations of that length. - The MySQL analyzer has been extended to detect when client and server negotiate to use a SSL encrypted session. This allows analysis of the subsequent SSL handshake. The service field for encrypted MySQL connections in the conn.log will have entries for both, mysql and ssl. Changed Functionality --------------------- - Violations for packet analyzers that have sessions attached with them will be raised once only. Further, analyzer confirmations are not raised after a violation. - The parameter given to ``enum_names()`` can now be a string naming the enum type, rather than the type itself. - The ``type_name`` of enum types produced by ``record_fields()`` now includes the actual type name rather than just ``"enum"``. - Passing non-string ``sep`` and ``def`` arguments to ``cat_sep()`` isn't a fatal error anymore. More descriptive error messages are produced, too. - The number of analyzer violation events that can be raised by protocol analyzer instances is now capped by the const ``max_analyzer_violation_events``. - The number of analyzer violation events that can be raised by protocol and file analyzer instances is now capped by the const ``max_analyzer_violation_events``. Its default is 1000 and the main purpose is to prevent analyzers from scheduling too many ``analyzer_violation_info`` events before the DPD ``max_violations`` script-level logic has a chance to run and disable the problematic analyzer. - The TCP analyzer now continues processing payload for some connections missing initial packets where it would previously have stopped. This fixes a few cases where we already had the logic to continue in place, but we still ended up considering them partial. - Count underflows via ``--c`` or subtract from statements (``c = c - 1``) are now consistently warned about. Previously, underflows through ``--c`` were treated as runtime errors, while "subtract from" underflows were silently accepted. The following (surprising behavior) now causes a warning, too: $ zeek -e 'print 1 - 2' expression warning in , line 1: count underflow (1 - 2) 18446744073709551615 - The MQTT scripts registering the analyzer and DPD signatures have been moved from the policy folder to base and are loaded by default. - Notices created for files transferred over multiple connections will now be associated with one of the connections rather than none. - The MySQL analyzer has been switched to parse in little endian. This avoids analyzer violations due to out of bound errors for length encoded strings. - Non-fatal errors when setting up BPF filtering will no longer cause Zeek to exit, but instead will log the error in reporter.log and continue processing. - The languages reported for the ``keyboard_layout`` field in rdp.log were updated to match the current standardized set of languages. Unknown layout values now attempt to fallback to a "parent" layout if one is available. - In the cluster management framework, the controller now supports Broker's WebSocket data transport for communication with clients. It listens on TCP port 2149 for this purpose. zeek-client now likewise uses the WebSocket transport, removing its runtime dependency on the Broker library and enabling standalone installation. The client still bundles with Zeek by default but is now also available on PyPI and installable via ``pip install zeek-client``. The documentation provides additional details. Deprecated Functionality ------------------------ - The global ``disabling_analyzer()`` hook has been deprecated and replaced with ``Analyzer::disabling_analyzer()`` that has the same semantics. - The ``analyzer_confirmation`` and ``analyzer_violation`` events have been deprecated in favor of the more generic ``analyzer_confirmation_info`` and ``analyzer_violation_info`` events. - The const values for toggling individual tunnel packet analyzers have been deprecated in favor of using ``Analyzer::disable_analyzer()`` directly. This affects: Tunnel::enable_ip Tunnel::enable_ayiya Tunnel::enable_teredo Tunnel::enable_gtpv1 Tunnel::enable_gre Setting these explicitly to F can be achieved by leveraging ``Analyzers::disabled_analyzers``, for example: redef Analyzer::disabled_analyzers += { PacketAnalyzer::ANALYZER_GRE }; - The ``zeek::merge_type_list()`` function has been deprecated. Please consider the partially compatible and saner ``zeek::maximal_type()`` instead. See GH-2604 for context. - The pre-authentication data field (pa_data) available in certain Kerberos events now exposes the (encrypted) PA-ENC-TIMESTAMP field (padata-type=2). - The ``SupressWeirds()`` method in the ContentLine analyzer was deprecated in favor of the correctly-spelled ``SuppressWeirds()`` method. - The `bro` symlink has finally been removed.

zeek v5.1.3

2023-02-21T20:35:58+00:00

This release fixes the following security issues: - Receiving DNS responses from async DNS requests (via the lookup_addr, etc BIF methods) with the TTL set to zero could cause the DNS manager to eventually stop being able to make new requests. This would lead to a memory leak that causes Zeek to crash. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. The fix included is to more carefully handle these types of responses to allow DNS resolution to continue. - Specially-crafted FTP packets with excessively long usernames, passwords, or other fields could cause log writes to use large amounts of disk space. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. The fix included adds new script-level options for limiting the possible input length for these fields, reporting weirds when those limits are exceeded. - The find_all and find_all_ordered BIF methods could take extremely large amounts of time to process incoming data depending on the size of the input. This has been encountered with packet input to multiple analyzers, but most notably the SMTP analyzer. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. The fix included adds a new argument to those BIF methods allowing callers to a set a maximum size for an input string to be processed. If the limit is exceeded, a weird is reported.

zeek v5.0.7

2023-02-21T20:37:22+00:00

zeek v5.0.8

2023-04-11T19:54:11+00:00

This release fixes the following security issues: - A specially-crafted stream of FTP packets containing a command reply with many intermediate lines can cause Zeek to spend a large amount of time processing data. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. The fix included is to ignore intermediate lines of multiline replies, as they do not contain status data and can’t generally be matched to a corresponding command. - A specially-crafted set of packets containing extremely large file offsets cause cause the reassembler code to allocate large amounts of memory. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. This was seen originally with the SMB analyzer, but could be reproduced with the HTTP analyzer as well. The fix included adds some additional length checking to avoid integer overflows in the calculations involving those offsets. - The DNS manager does not correctly expire responses that don’t contain any data, such those containing NXDOMAIN or NODATA status codes. This can lead to Zeek allocating large amounts of memory for these responses and never deallocating them. This is a DoS risk as it is possible to cause Zeek to repeatedly make DNS lookups to hostnames that will always return these responses. The fix included removes the check for whether empty responses should be expired, instead always allowing them to expire once they have hit their TTL. - A specially-crafted stream of RDP packets can cause Zeek to spend large amounts of time processing data. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. The fix included here is two parts. The first adds some additional length checking to the RDP analyzer for protocol validation. The second is to remove the use of the PIA analyzer when processing internal packets, instead sending all encrypted packet data directly to the SSL analyzer. - A specially-crafted stream of SMTP packets can cause Zeek to spend large amounts of time processing data. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. The fix included adds some validation of the order that SMTP commands are received, ensuring that they are received in the proper order. This release fixes the following bugs: - Data stores used by the known-{hosts,certs,services} policies now default to using local stores instead of Broker stores. This change was made to avoid some performance issues with those stores on high-traffic networks. - The VXLAN and Geneve report analyzer confirmations once their protocols have been fully parsed, but before attempting to forward to the tunneled packets to other analyzers. This should make the confirmation of these protocols more consistent. - New wierds were added to the AYIYA, Geneve, and VXLAN analyzers (``ayiya_empty_packet``, ``geneve_empty_packet``, and ``vxlan_empty_packet``). These are reported when packets containing these protocols do not have any further data after the tunnel header. - A new script-level option ``Pcap::non_fd_timeout`` was added to allow fine-tuning the amount of time to sleep on each IO loop when using a packet source that doesn't provide a file descriptor (e.g. Myricom). The option defaults to 20 microseconds. Tuning this option can greatly reduce the amount of CPU time used by Zeek on low-traffic networks, but may also increase the number of dropped packets if set to too high of a value. - Avoid attempting to retrieve packets during every loop for a packet source, instead switching to a predictive approach that keeps track of whether or not that packet source has previously seen traffic. This improves performance somewhat, especially on traffic links and on certain packet sources that may block when traffic isn't available.

zeek v5.2.1

2023-04-11T19:54:49+00:00

This release fixes the following security issues: - A specially-crafted stream of FTP packets containing a command reply with many intermediate lines can cause Zeek to spend a large amount of time processing data. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. The fix included is to ignore intermediate lines of multiline replies, as they do not contain status data and can’t generally be matched to a corresponding command. - A specially-crafted set of packets containing extremely large file offsets cause cause the reassembler code to allocate large amounts of memory. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. This was seen originally with the SMB analyzer, but could be reproduced with the HTTP analyzer as well. The fix included adds some additional length checking to avoid integer overflows in the calculations involving those offsets. - The DNS manager does not correctly expire responses that don’t contain any data, such those containing NXDOMAIN or NODATA status codes. This can lead to Zeek allocating large amounts of memory for these responses and never deallocating them. This is a DoS risk as it is possible to cause Zeek to repeatedly make DNS lookups to hostnames that will always return these responses. The fix included removes the check for whether empty responses should be expired, instead always allowing them to expire once they have hit their TTL. - A specially-crafted stream of RDP packets can cause Zeek to spend large amounts of time processing data. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. The fix included here is two parts. The first adds some additional length checking to the RDP analyzer for protocol validation. The second is to remove the use of the PIA analyzer when processing internal packets, instead sending all encrypted packet data directly to the SSL analyzer. - A specially-crafted stream of SMTP packets can cause Zeek to spend large amounts of time processing data. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. The fix included adds some validation of the order that SMTP commands are received, ensuring that they are received in the proper order. This release fixes the following bugs: - Data stores used by the known-{hosts,certs,services} policies now default to using local stores instead of Broker stores. This change was made to avoid some performance issues with those stores on high-traffic networks. - Zeekctl now assigns network ports to workers starting at port 27760. This fixes an issue where workers were starting up with ports within Linux's ephemeral port range, and were potentially failing to startup due the ports already being in use. This change may require changes in firewall/routing configurations between hosts in a Zeek cluster. This should not affect clusters running on FreeBSD, as that OS uses a different range for ephemeral ports. - The languages reported for the ``keyboard_layout`` field in rdp.log were updated to match the current standardized set of languages. Unknown layout values now attempt to fallback to a "parent" layout if one is available. - Replies to HTTP/0.9 requests are now handled in a separate fashion from HTTP/1.x in order to avoid analyzer violations. - The VXLAN analyzer no longer reports analzyer violations if the tunneled packets were not able to be parsed. - The VXLAN and Geneve report analyzer confirmations once their protocols have been fully parsed, but before attempting to forward to the tunneled packets to other analyzers. This should make the confirmation of these protocols more consistent. - New wierds were added to the AYIYA, Geneve, and VXLAN analyzers (``ayiya_empty_packet``, ``geneve_empty_packet``, and ``vxlan_empty_packet``). These are reported when packets containing these protocols do not have any further data after the tunnel header. - A new script-level option ``Pcap::non_fd_timeout`` was added to allow fine-tuning the amount of time to sleep on each IO loop when using a packet source that doesn't provide a file descriptor (e.g. Myricom). The option defaults to 20 microseconds. Tuning this option can greatly reduce the amount of CPU time used by Zeek on low-traffic networks, but may also increase the number of dropped packets if set to too high of a value.

zeek v5.0.9

2023-05-19T16:18:13+00:00

This release fixes the following security issues: - A specially-crafted series of FTP packets with a CMD command with a large path followed by a very large number of replies could cause Zeek to spend a long time processing the data. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. The fix included prevents Zeek from reusing the CMD command if it was already consumed by path-traversal logic. - A specially-crafted with a truncated header can cause Zeek to overflow memory and potentially crash. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. This overflow requires implementing the raw_packet event handler which isn’t implemented by default, which makes the risk of this issue low. The fix included adds additional length checking during handling of raw_packet events. - A specially-crafted series of SMTP packets can cause Zeek to generate a very large number of events and take a long time to process them. Zeek correctly disables the SMTP analyzer while processing these packets but continues to feed packets to it, generating more events. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. The fix included prevents an analyzer from calling another analyzer that has already been disabled for a connection. - A specially-crafted series of POP3 packets containing MIME data can cause Zeek to spend a long time dealing with each individual file ID. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. The fix included attempts to reuse an existing file ID for a connection instead of recreating it each pass through the MIME analyzer. This release fixes the following bugs: - This release includes a fixes to Zeek and updates to the Broker and Spicy submodules to support building against GCC 13.

zeek v5.2.2

2023-05-19T16:21:50+00:00

This release fixes the following security issues: - A specially-crafted series of FTP packets with a CMD command with a large path followed by a very large number of replies could cause Zeek to spend a long time processing the data. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. The fix included prevents Zeek from reusing the CMD command if it was already consumed by path-traversal logic. - A specially-crafted with a truncated header can cause Zeek to overflow memory and potentially crash. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. This overflow requires implementing the raw_packet event handler which isn’t implemented by default, which makes the risk of this issue low. The fix included adds additional length checking during handling of raw_packet events. - A specially-crafted series of SMTP packets can cause Zeek to generate a very large number of events and take a long time to process them. Zeek correctly disables the SMTP analyzer while processing these packets but continues to feed packets to it, generating more events. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. The fix included prevents an analyzer from calling another analyzer that has already been disabled for a connection. - A specially-crafted series of POP3 packets containing MIME data can cause Zeek to spend a long time dealing with each individual file ID. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. The fix included attempts to reuse an existing file ID for a connection instead of recreating it each pass through the MIME analyzer. This release fixes the following bugs: - The config parser implements handling of commas at the end of input files in a safer way now, avoiding some crashes on Linux systems during parsing. - The AF_Packet plugin wasn't properly masking the tp_vlan_tci values received from the kernel, and so could return invalid values for the VLAN ID reported to Zeek. The value is now correctly masked. - The AF_Packet plugin now checks whether the interface is up during setup, ensuring that a more useful error message is reported.

zeek v6.0.0

2023-05-31T11:56:33+00:00

Breaking Changes ---------------- - Zeek now treats private address space (i.e., non-routable IP address ranges) as local by default, matching the intuition of many users that e.g. a 192.168/16 IP address should show up as local in the logs. To do this, Zeek automatically adds ``Site::private_address_space`` to ``Site::local_nets`` at startup. Subsequent runtime updates to ``Site::private_address_space`` propagate to ``Site::local_nets``, while updates to the latter don't affect the former. You're free to define ``Site::local_nets`` as before and do not need to update your configurations. If you added standard private address space to ``Site::local_nets`` in the past, you no longer need to do so. This also applies to zeekctl's ``networks.cfg`` file. The new global Boolean ``Site::private_address_space_is_local``, true by default, controls the behavior. A redef to false brings back Zeek's prior behavior of considering private address space an unrelated concept, which will come in handy for example when working with tests that compare results against log baselines that have not yet been updated. - Telemetry centralization and Prometheus exposition is not enabled by default anymore. Previously, the manager node would open port 9911/tcp by default and import all metrics from other nodes. For large clusters, the current implementation introduces significant processing overhead on the manager even if the Prometheus functionality is not used. While inconvenient, disable this functionality (assumed to be used by few as of now) by default to preserve resources. The script to enable centralization and the Prometheus endpoint is now located in the ``policy/`` folder. Re-enable the old functionality with: @load frameworks/telemetry/prometheus You may experiment with increasing ``Broker::metrics_export_interval`` (default 1s) to reduce the extra overhead and communication at the expense of stale metrics. - Custom source tarballs require a ``repo-info.json`` file. Note, should you be using official Zeek release tarballs only, or build Zeek solely from git checkouts, this does not affect you. However, if you're building your own Zeek source tarballs, it is now required that a ``repo-info.json`` file exists at the top-level. The ``dist`` target was extended to add this file and official Zeek release source tarballs will contain it going forward. The following command can be used to produce ``repo-info.json``: python3 ./ci/collect-repo-info.py --only-git > ../path/to/tarballdir/repo-info.json This is required to support the new ``-V`` / ``--build-info`` option that provides information about git submodules and included plugins used during the build. The ``ci/collect-repo-info.py`` tool runs at ``./configure`` time and either collects the required information from a git clone (when git is installed), or otherwise uses the content of a file named ``repo-info.json``. If you see opportunities to extend ``repo-info.json`` with further information, please get in touch. - Plugin authors should raise the minimum required CMake version to 3.15 to ensure compatibility with new CMake scaffolding included in this release. Older versions will trigger a warning at configuration time and, depending on the functionality included in the plugin, may trigger subsequent errors during configuration or build. - Zeek container images are not pushed to the zeekurity organization anymore. Please switch to using the ``zeek/zeek`` image on DockerHub, or the images published to ``public.ecr.aws/zeek/zeek``. - The IRC_Data analyzer declaration has been moved to protocols/irc/IRC.h. - The error message returned when using ``bro_init``, ``bro_done``, and ``bro_script_loaded`` events is now removed. removed. Usage of these events has returned that error during script parsing for a few years, and time has come to finally remove it. New Functionality ----------------- - Zeek now features experimental JavaScript support: /* hello.js */ zeek.on('zeek_init', () => { console.log('Hello, Zeek!'); }); $ zeek ./hello.js Hello, Zeek! When building Zeek on a system that features a recent (16.13+) version of the libnode package with development headers, Zeek automatically includes the externally-maintained ZeekJS plugin (https://github.com/corelight/zeekjs) as a builtin plugin. This allows Zeek to load and execute JavaScript code located in ``.js`` or ``.cjs`` files. When no such files are passed to Zeek, the JavaScript engine and Node.js environment aren't initialized and there is no runtime impact. The Linux distributions Fedora 37 & 38, Ubuntu 22.10, and the upcoming Debian 12 release provide suitable packages. On other platforms, Node.js can be built from source with the ``--shared`` option. To disable this functionality, pass ``--disable-javascript`` to configure. - Zeek now comes with Spicy support built in, meaning it can now leverage any analyzers written in Spicy out of the box. While the interface layer connecting Zeek and Spicy used to be implemented through an external Zeek plugin, that code has now moved into the Zeek code base itself. We also added infrastructure to Zeek that enables its built-in standard analyzers to use Spicy instead of Binpac. As initial (simple) examples, Zeek's Syslog and Finger analyzers are now implemented in Spicy. While their legacy versions remain available as fallbacks for now in case Spicy gets explicitly disabled at build time, their use is deprecated and their code won't be maintained any further. (Some of these Spicy updates were part of Zeek 5.2 already, but hadn't been included in its NEWS section.) - Zeek events now hold network timestamps. For scheduled events, the timestamp represents the network time for which the event was scheduled for, otherwise it is the network time at event creation. A new bif ``current_event_time()`` allows to retrieve the current event's network timestamp within the script-layer. When Zeek sends events via Broker to other nodes in a cluster, an event's network timestamp is attached to the Broker messages. On a receiving Zeek node executing a handler for a remote event, ``current_event_time()`` returns the network time of the sending node at the time the event was created. The Broker level implementation allows to exchange arbitrary event metadata, but Zeek's script and C++ APIs currently only expose network timestamp functionality. - A new bif ``from_json()`` can be used to parse JSON strings into records. type A: record { a: addr; }; local p = from_json({\"a\": \"192.168.0.1\"}", A); if ( p$valid ) print (p$v as A) Implicit conversion from JSON to Zeek types is implemented for bool, int, count, real, interval (number as seconds) and time (number as unix timestamp), port (strings in "80/tcp" notation), patterns, addr, subnet, enum, sets, vectors and records similar to the rules of the input framework. Optional or default record fields are allowed to be missing or null in the input. - Zeek now provides native "Community ID" support with a new bif called ``community_id_v1()``. Two policy scripts ``protocols/conn/community-id-logging`` and ``frameworks/notice/community-id`` extend the respective logs with a ``community_id`` field the same way as the external zeek-community-id plugin provides. A main difference to the external ``hash_conn()`` bif is that the ``community_id_v1()`` takes a ``conn_id`` record instead of a ``connection``. Loading the new policy scripts and using the external zeek-community-id plugin at the same time is unsupported. - ZeekControl is now multi-logger aware. When multiple logger nodes are configured in ZeekControl's node.cfg, by default the log archival logic adds a logger's name as suffix to the rotated file name: stats.11:18:57-11:19:00-logger-1.log.gz stats.11:18:57-11:19:00-logger-2.log.gz Previously, in a multi-logger setup, individual logger processes would overwrite each other's log files during rotation, causing data loss. For setups with a single logger, there's no change in behavior. The naming of the final logs can be customized by providing an alternative ``make-archive-name`` script and using the new ``ZEEK_ARG_LOG_SUFFIX`` environment variable. - A supervisor controlled Zeek cluster is now multi-logger aware. This avoids loggers overwriting each other's log files within a single log-queue directory. By default, a logger's name is appended to the rotated logs by zeek-archiver. - Introduce a new command-line option ``-V`` / ``--build-info``. It produces verbose output in JSON format about the repository state and any included plugins. - The X.509 certificate parser now exposes the signature type that is given inside the signed portion of the certificate. - The SSL parser now parses the CertificateRequest handshake message. There is a new ``ssl_certificate_request`` event and a new ``parse_distinguished_name`` function. We also added the ``protocols/ssl/certificate-request-info`` policy script, that adds some additional information to ``ssl.log``. - Add logging metrics for streams (``zeek-log-stream-writes``) and writers (``zeek-log-writer-writes-total``). - Add networking metrics via the telemetry framework. These are enabled when the ``misc/stats`` script is loaded. zeek-net-dropped-packets zeek-net-link-packets zeek-net-received-bytes zeek-net-packet-lag-seconds zeek-net-received-packets-total Except for lag, metrics originate from the ``get_net_stats()`` bif and are updated through the ``Telemetry::sync()`` hook every 15 seconds by default. - The DNS analyzer now parses RFC 2535's AD ("authentic data") and CD ("checking disabled") flags from DNS requests and responses, making them available in the ``dns_msg`` record provided by many of the ``dns_*`` events. The existing ``Z`` field remains unchanged and continues to subsume the two flags, for backward compatibility. - The supervisor framework can now start worker nodes that read from a trace file. - Zeek can be prevented from updating ``network_time()`` to the current time by setting ``allow_network_time_forward=F``. Together with ``set_network_time()`` or a custom plugin, this allows control of ``network_time()`` without Zeek interfering. - The setting ``Pcap::non_fd_timeout`` can be used to configure the timeout used by non-selectable packet sources in the idle case (default 20usec). This value has previously been hard-coded, but increasing it can significantly reduce idle CPU usage in low packet rate deployments. - Zeek now supports a new ``@pragma`` directive. It currently allows suppressing deprecation warnings in Zeek scripts by opening with ``@pragma push ignore-deprecations`` and closing with ``@pragma pop``. This particularly helps in situations where use of the Zeek base scripts, for example to populate a deprecated field for API compatibility, would otherwise trigger deprecation warnings. - The ``Reporter`` class was extended by a ``Deprecation()`` method to use for logging deprecations rather than using ad-hoc ``Warning()`` calls. - The network statistics record type features a new ``pkts_filtered`` field for reporting the number of packets that the interface filtered before hand-off to Zeek. Packet source implementations are free to fill this field as feasible. The default pcap packet source does not provide this information because its availability depends on the libpcap version. - Packet statistics (packets received, packets dropped, bytes received, packets seen on link, and packets filtered) are now reported to the Telemetry framework, under the ``zeek_net`` prefix. - Zeek's cluster framework provides the new ``get_node_count(node_type: NodeType)`` function to obtain the number of nodes for a given node type as defined in the cluster layout. Furthermore, ``broadcast_topics`` was added as a collection of broker topics that can be used to reach all nodes in a cluster. - The new ``Cluster::Experimental`` namespace has been introduced to Zeek's cluster framework to provide experimental features. Based on practical experiences and the adoption of an experimental feature, it may become a regular feature or be removed in future releases. Experimental features are loaded via: ``@load policy/frameworks/cluster/experimental`` - Zeek's cluster framework provides two new experimental events: - ``cluster_started``: This event will be broadcasted from the manager once all cluster-level connections have been established based on the given cluster layout. If any node restarts (including the manager itself), the event will neither be rebroadcasted nor raised locally for the restarted node. - ``node_fully_connected``: This event will be sent to the manager and raised locally once a cluster node has successfully conducted cluster-level handshakes for all its outgoing connections to other cluster nodes based on the given cluster layout. Note: There is no tracking of cluster node connectivity. Thus, there is no guarantee that all peerings still exist at the time of these events being raised. - The IEEE 802.11 packet analyzer gains the ability to parse encapsulated A-MSDU packets, instead of just dropping them. It also gains the ability to properly recognize CCMP-encrypted packets. These encrypted packets are currently dropped to Zeek's inability to do anything with them. - Add packet analzyers for LLC, SNAP, and Novell 802.3, called from the Ethernet and VLAN analyzers by default. - Environment variables for the execution of log rotation postprocessors can be set via ``Log::default_rotation_postprocessor_cmd_env``. - The ``record_field`` record was extended by ``optional`` and ``record_fields()`` can now be used to determine the optionality of record fields. - The ``ip4_hdr`` record was extended by ``DF``, ``MF``, ``offset`` and ``sum`` to aid packet-level analysis use-cases. - Zeek now supports parsing the recently standardized DTLS 1.3. Besides the protocol messages being correctly parsed and raising the typical SSL/TLS events, the biggest visible change is the newly added ``ssl_extension_connection_id`` event. - The NTP analyzer now recognizes when client and server mode messages disagree with the notion of "originator" and "responder" and flips the connection. This can happen in packet loss or packet re-ordering scenarios. Such connections will have a ``^`` added to their history. - New bifs for ``ceil()`` and ``log2()`` have been added. - Seeds for deterministic processing can now also be set through a new environment variable called ``ZEEK_SEED_VALUES``. The format is expected to contain 21 positive numbers separated by spaces. Changed Functionality --------------------- - The base distribution of the Zeek container images has been upgraded to Debian 12 "bookworm" and JavaScript support was enabled. - When ``get_file_handle()`` is invoked for an analyzer that did not register an appropriate callback function, log a warning and return a generic handle value based on the analyzer and connection information. - The ``&on_change`` attribute of set and tables is propagated through ``copy()``. - Revert back to old method of preallocating ``PortVal`` objects for all valid port numbers, as it was implemented prior to the Windows port. Not preallocating these objects saves a minor amount of memory for short runs of Zeek, but comes at a performance cost for having to allocate the objects every time a new port is seen plus do map lookups for each port. This memory savings is mostly lost for long runs of Zeek, since all of the ports will likely end up allocated in time. If the version from the Windows port is desired, a new configure option ``--disable-port-prealloc`` will disable the preallocation and enable the map lookup version. - The main-loop has been changed to process all ready IO sources with a zero timeout in the same loop iteration. Previously, two zero-timeout sources would require two main-loop iterations. Further, when the main-loop is polling IO sources with file descriptors, zero timeout IO sources are added to the list of sources to be processed as well. The intervals to decide when Zeek checks FD-based IO sources for readiness have been made configurable through ``io_poll_interval_default`` and ``io_poll_interval_live`` for ease of testing, development and debugging of the main-loop. - Zeek does not arbitrarily update ``network_time()`` to current time anymore. When a packet source is providing a constant stream of packets, packets drive network time. Previously, Zeek updated network time to current time in various situations, disregarding timestamps of network packets. Zeek will now update ``network_time()`` only when a packet source has been inactive/idle for an interval of ``packet_source_inactivity_timeout`` (default 100msec). When a worker process suddenly observes no packets, timer expiration may initially be delayed by ``packet_source_inactivity_timeout``. - Calling ``suspend_processing()`` when reading traces does not update network time to the current time anymore. Instead, Zeek keeps ``network_time()`` according to the trace file. This causes scheduled events to not fire once ``suspend_processing()`` is called, which seems more reasonable than arbitrarily setting ``network_time()`` to current time. Processing can still be continued from broker events or input readers. - Previously, Zeek would process and dispatch events for the very first packet in a trace file in order to initialize time, even if ``suspend_processing()`` was called in a ``zeek_init()`` handler. This has been changed such that the first packet will only be processed once ``continue_processing()`` has been invoked again. Some background around the previous behavior can be found in GH-938. Given that the ``network_time_init()`` event explicitly signals initialization of network time, this behavior seems more reasonable. - If an event is scheduled with a 0.0sec timeout from a ``zeek_init()`` handler that also invokes ``suspend_processing()``, the scheduled event will fire immediately with ``network_time()`` still yielding ``0.0``. Previously, ``network_time()`` was set to the current time. The new behavior provides more deterministic operation and aligns with timers stopping during a ``suspend_processing()``. - Broker no longer initializes network time to current time when processing input. Particularly in combination with pcap processing this was not desirable behavior. - The IO loop's poll interval is now correctly reduced from 100 to 10 for live packet sources. This should lower CPU usage for deployments with non-selectable packet sources. - Zeek's CMake scaffolding has received an overhaul for modernizing the build system and to make it easier to maintain going forward. Plugins can now use a declarative interface for adding all sources, BIFs, etc. in one block instead of using the previous begin/end functions. While the old plugin functions still exist for backward compatibility, the underlying codebase requires newer CMake features. Plugin authors should raise their minimum required CMake version to 3.15, to match Zeek's. - The IRC data analyzer does not extract DCC acknowledgements to files anymore. Instead, ``irc_dcc_send_ack`` is raised with the bytes acknowledged by the recipient. - The IRC base script now use ``file_sniff()`` instead of ``file_new()`` for DCC file transfers to capture ``fuid`` and inferred MIME type in irc.log. - The ``ignore_checksums`` script variable now reflects the correct value when using the ``-C`` command-line flag. - Support for ARUBA GRE tunnels now covers all of the known protocol type values for those tunnels. - The vlan field reported by the AF_PACKET packet source is now properly masked to exclude PCP and DEI bits. Previously, these bits were included and could cause invalid vlan values > 4095 to be reported. - Libpcap based packet source now avoids the 32bit wraparound of link and dropped packet counters as reported by users. - The `ssl_history` field in ssl.log indicates that the letter `j` is reserved for hello retry requests. However, this logging was never fully implemented; instead, hello retry requests were logged like as a server hello (with the letter `s`). This oversight was fixed, and hello retry requests are now correctly logged. - When per-connection SMB parser state (read offsets, tree ids, ...) exceeds ``SMB::max_pending_messages`` (default 1000), Zeek discards such per-connection state and raises a new ``smb2_discarded_messages_state()`` event. This event is used to reset script-layer SMB state. This change provides protection against unbounded state growth due to partial or one-sided SMB connections. Setting ``SMB::max_pending_messages`` to 0 can be used to switch back to the previous behavior of not discarding state. Setting ``SMB::enable_state_clear`` to ``F`` skips the script-layer state clearing logic. - Fix ``disable_analyzer()`` builtin function crashing when attempting to disable connection's root analyzers. - Zeek script vectors now support negative indices. local v = vector(1, 2, 3); print v[-1]; # prints 3 - Function parameters are rendered by Zeekygen as ``:param x`` rather than just ``:x:``. This allows to group parameters Zeek's documentation. Removed Functionality --------------------- - Mixing vector and scalar operands for binary expressions, like addition, multiplication, etc., is now an error. - Using deprecated ``when`` semantics without capturing variables is now an error. - Referencing local variables in a more outer scope than where they were declared is now an error Deprecated Functionality ------------------------ - The cluster framework's ``worker_count`` has been deprecated in favor of the new function ``get_active_node_count(node_type: NodeType)`` that can be used to obtain the number of nodes of a given type the calling node is currently connected to.

zeek v5.0.10

2023-09-12T20:45:34+00:00

zeek v6.0.1

2023-09-12T20:46:31+00:00

This release fixes the following security issues: - File extraction limits were not correctly enforced for files containing large amounts of missing bytes. Crafting files with large amounts of missing bytes in them could cause Zeek to spend a long time processing data, allocate a lot of main memory, and write a lot of data to disk. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. The fix included makes Zeek correctly enforce file size limits. We also added a new option (default_limit_includes_missing) which allows to customize the behavior when encountering large amounts of missed bytes in file. This issue was discovered by Luca Cigarini. - Sessions are sometimes not cleaned up completely within Zeek during shutdown, potentially causing a crash when using the -B dpd flag for debug logging. This is low priority because it only happens at shutdown and only when using that flag. The fix included is to reorder shutdown to cleanup all the sessions prior to tearing down the analyzers. - A specially-crafted HTTP packet can cause Zeek’s filename extraction code to take a long time to process the data. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. The fix included adjusts the regular expression used in the extract_filename_from_content_disposition() script function to more-correctly match the data. - A specially-crafted series of FTP packets made up of a CWD request followed by a large amount of ERPT requests may cause Zeek to spend a long time logging the commands. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. The fix included is to prevent logging of pending commands for FTP packets. - A specially-crafted VLAN packet can cause Zeek to overflow memory and potentially crash. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. The fix included is to add some additional length checking to the VLAN analyzer. This release fixes the following bugs: - Fixed a base64 decoding issue with the authorization field of HTTP request headers that was sometimes causing Zeek to output error messages. Thank you to GitHub user @progmboy for reporting and providing a fix for this issue. - Ensure that Zeek builds use the internal version of Spicy instead of external installations, unless specifically configured for that mode. - Support was added for ``switch`` fields when exporting Spicy types to Zeek. - A number of fixes were added to protect against potential unbounded state growth with the SMB and DCE-RPC analyzers. SMB close requests will properly tear down an related DCE-RPC analyzers. A new ``SMB::max_dce_rpc_analyzers`` script variable was added that allows finer control of how many DCE-RPC analyzers are allowed to be created per SMB connection. Thanks to Zeek Slack user Takomi Sugawara for reporting this issue. - Fixed a regression in the UDP and TCP analyzers that was causing more data than necessary to be forwarded to the next analyzer in the chain. Thanks to Zeek Slack user Hiep Long Tan for reporting this issue. - A connection's value is now updated in-place when its directionality is flipped due to Zeek's heuristics (for example, SYN/SYN-ACK reversal or protocol specific approaches). Previously, a connection's value was discarded when flipped, including any values set in a ``new_connection()`` handler. A new ``connection_flipped()`` event is added to allow updating custom state in script-land. - Fixed undefined symbols being reported from Spicy when building some of the binary packages for Zeek. - Loading ``policy/frameworks/notice/community-id.zeek`` now also automatically community ID logging. In the past, loading the script had no effect unless ``policy/protocols/conn/community-id-logging.zeek`` was loaded before. This was fairly unusual and hard to debug behavior. - Spicy no longer registers an extra port for every port registered in a plugin's .evt file. - Timeouts in DNS resolution no longer cause uncontrolled memory growth. - Fix check to skip DNS hostname lookups for notices that are not delivered via email in ``policy/frameworks/notice/extend-email/hostnames``. Due to that policy script being loaded in the Zeek's default ``local.zeek``, this previously caused unneeded DNS lookups for every generated notice instead of just those delivered via email.

zeek v6.1.0

2023-10-13T18:57:19+00:00

Breaking Changes ---------------- - ``assert`` is now a reserved keyword for the new ``assert`` statement. - The ``__bro_plugin__`` file that gets generated as part of plugin builds was renamed to ``__zeek_plugin__``. This will affect the ability for older versions of ``zkg`` to use the ``zkg unload`` and ``zkg load`` commands. This should only cause breakage for people using a version of ``zkg` that doesn't come bundled with Zeek (which we generally don't recommend doing). - Zeek does not traverse into dot directories to find plugins or hlto files anymore. Any dot directories found below the directories specified in ZEEK_PLUGIN_PATH or ZEEK_SPICY_MODULE_PATH are now skipped. Dot directories explicitly listed in ZEEK_PLUGIN_PATH or ZEEK_SPICY_MODULE_PATH are not skipped. - External plugins will fail to configure if their minimum required CMake version is below 3.15. This was a warning with Zeek 6.0, but has caused user confusion due to unhelpful error messages around the IN_LIST operator policy. - The FindBISON, FindOpenSSL, FindPackageHandleStandardArgs, FindPackageMessage, and SelectLibraryConfigurations cmake files were removed from our cmake repository in favor of the versions that come with CMake. This should not cause any breakage, but it is possible in the case that someone was using these in a plugin. New Functionality ----------------- - Zeek now includes the LDAP protocol analyzer from the zeek/spicy-ldap project (https://github.com/zeek/spicy-ldap). This analyzer is enabled by default. The analyzer's events and its ``ldap.log`` and ``ldap_search.log`` should be considered preliminary and experimental until the arrival of Zeek's next long-term-stable release (7.0). If you observe unusually high CPU consumption or other issues due to this analyzer being enabled by default, the easiest way to disable it is via the ``Analyzer::disabled_analyzers`` const as follows: redef Analyzer::disabled_analyzers += { Analyzer::ANALYZER_LDAP_UDP, Analyzer::ANALYZER_LDAP_TCP, }; Please do report issues to us including diagnostic information in case this is necessary in your environment. We're also open to general feedback about the structure of the new logs. - Zeek now includes the QUIC protocol analyzer from the zeek/spicy-quic project (https://github.com/zeek/spicy-quic). This project is a fork of Fox-IT's initial implementation (https://github.com/fox-ds/spicy-quic). As for the LDAP analyzer, the analyzer's events and the new ``quic.log`` should be considered preliminary and experimental until the arrival of Zeek's next long-term-stable release (7.0). As above, any feedback and contributions to this analyzer and the new log are welcome. The analyzer's functionality is limited to decryption of the INITIAL packets of QUIC version 1. If decryption of these packets is successful, the handshake data is forwarded to Zeek's SSL analyzer. An ``ssl.log`` entry will appear in ``ssl.log`` for QUIC connections. The entry in the ``conn.log`` will contain ``quic`` and ``ssl`` in the service field. To disable the analyzer in case of issues, use the following snippet: redef Analyzer::disabled_analyzers += { Analyzer::ANALYZER_QUIC, }; - Added a new ``assert`` statement for assertion based testing and asserting runtime state. assert [, ]; This statement comes with two hooks. First, ``assertion_failure()`` that is invoked for every failing assert statement. Second, ``assertion_result()`` which is invoked for every assert statement and its outcome. The latter allows to construct a summary of failing and passing assert statements. Both hooks receive the location and call stack for the ``assert`` statement via a ``Backtrace`` vector. A failing assert will abort execution of the current event handler similar to scripting errors. By default, a reporter error message is logged. Using the break statement within ``assertion_failure()`` or ``assertion_result()`` allows to suppress the default message. - Add a new ``&default_insert`` attribute for tables. This behaves as ``&default`` with the addition that the default value is inserted into the table upon a failed lookup. Particularly for tables with nested container values, the ``&default`` behavior of not inserting the value can be of little use. - The ``from_json()`` function now takes an optional key_func argument to normalize JSON object key names. This can be useful if the keys in a JSON object are not valid Zeek identifiers or reserved keywords. - Module names are now included in ``global_ids()``. Their key in the returned table is prefixed with "module " and their value will have the ``type_name`` field set to "module". - Identifiers in the global scope can now be referenced and defined from within modules by prefixing their names with ``::``. Previously, these required an explicit ``GLOBAL::`` prefix to be used. Using ``GLOBAL::`` has been deprecated. - The ``as`` keyword now supports casting between ``set`` and ``vector`` values with the same element type. Converting ``set`` values with multiple index values is not supported. We plan to extend the use of the ``as`` keyword to support more type conversions in the future. - Added new packet analyzer to handle PCAP files DLT_PPP link type. - Fixed appending of ``any`` to ``vector of any``. - The ModBus analyzer's function support was expanded, with new handling of the Encapsulation Interface Transport (function 28) And Diagnostics (function 8) functions. This adds new ``modbus_encap_interface_transport_{request,response}`` and ``modbus_diagnostics_{request,response}`` events. - The ModBus file record read and write events now provide the full data from the request and response messages as part of the event data. - The full PDU length was added to the ``ModBusHeader`` record type passed with all of the ModBus events. Changed Functionality --------------------- - A connection's value is now updated in-place when its directionality is flipped due to Zeek's heuristics (for example, SYN/SYN-ACK reversal or protocol specific approaches). Previously, a connection's value was discarded when flipped, including any values set in a ``new_connection()`` handler. A new ``connection_flipped()`` event is added to allow updating custom state in script-land. - Loading ``policy/frameworks/notice/community-id.zeek`` now also automatically community ID logging. In the past, loading the script had no effect unless ``policy/protocols/conn/community-id-logging.zeek`` was loaded before. This was fairly unusual and hard to debug behavior. - Connections to broadcast addresses are not flipped based on ``likely_server_ports`` anymore. Previously, broadcast packets originating from a likely server port resulted in 255.255.255.255 being the originator in ``conn.log``. - When too many HTTP requests are pending, Zeek will now log them at once and reset request/response correlation instead of running into unbounded state growth. This behavior is configurable via a new option ``HTTP::max_pending_requests``. The default is ``100``. - Fix deferred initialization of nested records containing non-const &default attributes. - Parameter lists for functions, events and hooks now use commas instead of semicolons in error messages or when printing such functions. - The IO buffer size used for PCAP file reading is now always 128kb. This new default can be changed via ``Pcap::bufsize_offline_bytes``. - The input framework now provides better information in error messages when encountering missing non-optional field while loading data. - The SSL analyzer will now parse a configurable maximum of 10 SSL Alerts per SSL message. For TLS 1.3, the maximum is implicitly 1 as defined by RFC 8446. If there are more alerts, a new weird "SSL_excessive_alerts_in_record" is raised. For non-TLS 1.3, the maximum can be redefined via ``SSL::max_alerts_per_record``. - The ``ssl_history`` field in the ssl.log is now capped at a configurable limit of 100 characters prevent unbounded growth. The limit can be changed via the option ``SSL::max_ssl_history_length``. When reached, a new weird named "SSL_max_ssl_history_length_reached" is raised. Deprecated Functionality ------------------------ - Accessing globals with ``GLOBAL::name`` has been deprecated and will be removed with Zeek 7.1. Use ``::name`` instead. - The original ``trigger::Trigger`` constructor has been deprecated and will be removed with Zeek 7.1. Use the new alternative constructor (per ``src/Trigger.h``) instead, including replacing any use of ``new ...`` with ``make_intrusive<...>``. The new constructor differs only in the placement of the ``timeout`` parameter, and in that - unlike the original - it always returns a valid pointer, which must be Unref()'d after construction, either explicitly (if using ``new``) or implicitly (if using ``make_intrusive<...>``).

zeek v6.0.2

2023-10-27T19:51:06+00:00

This release fixes the following security issues: - A specially-crafted SSL packet could cause Zeek to leak memory and potentially crash. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. The fix included adds additional memory cleanup to the x509 file analyzer. - A specially-crafted series of FTP packets could cause Zeek to log entries for requests that have already been completed, using resources unnecessarily and potentially causing Zeek to lose other traffic. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. The fix included changes the way that we track the pending FTP commands, avoiding possibly reusing the same value for subsequent commands. - A specially-crafted series of SSL packets could cause Zeek to output a very large number of unnecessary alerts for the same record. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. The fix included adds a new option SSL::max_alerts_per_record that caps the number of alerts that can be generated for an SSL record. For TLS 1.3 this is capped at 1 as defined in RFC 8446. For non-TLS 1.3 it is a configurable value. A SSL_excessive_alerts_in_record weird will be raised if the cap is exceeded. - A specially-crafted series of SSL packets could cause Zeek to generate very long ssl_history fields in the ssl.log, potentially using a large amount of memory due to unbounded state growth. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. The fix included adds a new option SSL::max_ssl_history_length that caps this to 100 characters by default. A SSL_max_ssl_history_length_reached weird will be raised if the cap is exceeded. - A specially-crafted IEEE802.11 packet could cause Zeek to overflow memory and potentially crash. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. The fix included adds additional bounds checking to the IEEE802.11 packet analyzer. This release fixes the following bugs: - Fixed Spicy type names from causing collisions with existing Zeek types. - On some systems with low values for the maximum number of file descriptors, it was possible to run into crashes when doing DNS lookups if all of the file descriptors were used. This is now avoided with better checking for the number of available file descriptors before trying a lookup. Thank you to Zeek Slack user h-mikami for reporting this issue. - Tables backed by a Broker backend now correctly support deletion if they have complex index types. Zeek previously reported an error when trying to delete elements from these tables. - A significant performance issue with Zeek's supervisor code was fixed, revolving around the re-initialization of the Event Manager object used to track events. Thank you to Jan Grashoefer for reporting this issue. - The MaxMind DB code now cleans up after itself, resolving a memory leak with the loaded database files. - The ZeekJS submodule was updated to version 0.9.6, bringing fixes for zeek.invoke and zeek.event crashes, garbage collection, and an issue where Zeek may stop executing events from ZeekJS.