http://open-source-security-software.net/releases.atom Recent releases 2022-09-30T01:10:59.905196+00:00 python-feedgen reko version-0.11.1 reko version-0.11.1 2022-04-07T22:17:52+00:00 This maintenance release provides minor enhancements and bugfixes, including: * More ARM32 rewriters. * More uses of generic `IntrinsicProcedures`. * Replace recursive `SccFinder` with an non-recursive implementation. * Initial support for Terse Executable format. * Don't try tracing into nonexecutable code. * Ctrl+0 resets the zoom level of the Graph Viewer * Many more PowerPC instructions supported * C parser issues reported by @smx-smx * Make SSA analysis use bit-accurate analysis for stack variables. * Multithreaded robustness. It also has some refactorings and new classes, setting the stage for a future refactoring of the Scanner: * New `RtlSwitch` subclass of `RtlInstruction` * Support for platform-specific patterns for procedure entries. 2022-04-07T22:17:52+00:00 PeekabooAV v2.1rc3 PeekabooAV v2.1rc3 2022-04-13T10:41:50+00:00 Install using venv/bin/pip install peekabooav==2.1rc3. See [CHANGELOG.md](https://github.com/scVENUS/PeekabooAV/blob/v2.1rc3/CHANGELOG.md) for changes since last release. 2022-04-13T10:41:50+00:00 OpenTAXII 0.2.4 OpenTAXII 0.2.4 2022-04-13T11:19:28+00:00 Changelog ========= 0.2.4 (2022-04-13) ------------------ * Make sure werkzeug <2.1 and >=2.1 work correctly with auth system 0.2.3 (2021-12-22) ------------------ * Fix bug in multithreaded use of sqlite (`#210 <https://github.com/eclecticiq/OpenTAXII/issues/114>`_ thanks `@rohits144 <https://github.com/rohits144>`_ for the report) 0.2.2 (2021-11-05) ------------------ * Fix readthedocs build 0.2.1 (2021-11-03) ------------------ * Add tests for python 3.6, 3.7, 3.8, 3.9, pypy * Add tests for sqlite, mysql, mariadb, postgresql * Fix bug that broke ``delete_content_blocks`` when using mysql on sqlalchemy 1.3 * Docs: Add db schema diagram * Docs: Clarify how to get default data in a default (development) docker instance * Fix implicit routing in TAXII 1.1 Inboxes * Update jwt usage to pyjwt >= 2.0 (thanks `@SanyaKapoor <https://github.com/SanyaKapoor>`_) 0.2.0 (2020-06-30) ------------------ * Enforce UTC usage in datetime fields in SQL DB Persistence API. * `Fix for #114 <https://github.com/eclecticiq/OpenTAXII/issues/114>`_: reintroduce ``opentaxii-create-account`` CLI command. * `Fix for #153 <https://github.com/eclecticiq/OpenTAXII/issues/152>`_: check if user can modify a collection before advertising it over inbox service. * Multiple coding style fixes. * Various documentation updates. 0.1.12 (2019-03-06) ------------------- * Remove unnecessary print statements. 0.1.11 (2019-02-13) ------------------- * Make JSON logging consistent when the application is run via Gunicorn. * Set ``acceptable_destination`` key in status details instead of extended headers * Allow passing ``engine_parameters`` to ``SQLDatabaseAPI`` for those who want to customize SQLAlchemy engine parameters. * Require recent version of ``lxml`` for security reasons. * Various test and Docker infrastructure improvements. 0.1.10 (2018-06-03) ------------------- * Replace separate service/collection/account creation process with single ``opentaxii-sync-data`` CLI command. * Persistence and Auth APIs extended with missing CRUD methods, that are used by ``opentaxii-sync-data``. * Read/modify collection level ACL added. * DB models for default implementation of Persistence API and Auth API were changed. No automatic migration code is provided (sorry!), so upgrading might require manual DB migration. * Drop python2.7 from testing scope. * Various bug fixes and improvements. 0.1.9 (2017-06-19) ------------------ * `libtaxii <https://github.com/TAXIIProject/libtaxii>`_ dependency upgraded to 1.1.111. * Various bug fixes and improvements (thanks to `@bjigmp <https://github.com/bjigmp>`_, `@chorsley <https://github.com/chorsley>`_, `@rjprins <https://github.com/rjprins>`_). 0.1.8 (2017-02-21) ------------------ * Ability to enable/disable "huge trees" support in XML parser. Configuration property ``xml_parser_supports_huge_tree`` set to ``yes`` or ``true`` will disable security restrictions and force XML parser to support very deep trees and very long text content. * Adjust SQL Persistence API implemetation so it works smoothly with MySQL backend. * Use Python 3.5 instead of Python 3.4 for tests. 0.1.7 (2016-10-18) ------------------ * Minor fixes. * Dependencies were changed from hard-pinned to more flexible. * Example of production DB configuration added to docs. 0.1.6 (2016-06-01) ------------------ * Python 3.4 compatibility of the codebase. Tox configuration extended with python 3.4 environment run. * Flake8 full style compatibility. Flake8 check added to Tox configuration. * SQLAlchemy session scope issue fixed (related to `#38 <https://github.com/EclecticIQ/OpenTAXII/issues/38>`_). * `opentaxii-delete-blocks` CLI command added (related to `#45 <https://github.com/EclecticIQ/OpenTAXII/issues/45>`_). * `delete_content_blocks` method `added <https://github.com/EclecticIQ/OpenTAXII/commit/dc6fddc27a98e8450c7e05e583b2bfb741f6e197#diff-6814849ac352b2b74132f8fa52e0ec4eR213>`_ to Persistence API. * Collection's name is `required <https://github.com/EclecticIQ/OpenTAXII/commit/dc6fddc27a98e8450c7e05e583b2bfb741f6e197#diff-ce3f7b939e5c540480ac655aef32c513R116>`_ to be unique in default SQL DB Auth API implementation. 0.1.5 (2016-03-15) ------------------ * Fix for the issue with persistence backend returning ``None`` instead of ``InboxMessage`` object 0.1.4 (2016-02-25) ------------------ * Hard-coded dependencies in ``setup.py`` removed. 0.1.3 (2016-02-25) ------------------ * Versions of dependencies are pinned. * Code adjusted for a new version of `anyconfig <https://pypi.python.org/pypi/anyconfig>`_ API. * Test for configuration loading added. 0.1.2 (2015-07-24) ------------------ * Docker configuration added. * Health check endpoint added. * Basic authentication support added. * Temporary workaround for `Issue #191 <https://github.com/TAXIIProject/libtaxii/issues/191>`_. * Method ``get_domain`` in Persistence API returns domain value configured for ``service_id``. If nothing returned, value set in configuration file will be used. * Performance optimisations. * Bug fixes and style improvements. 0.1.1 (2015-04-08) ------------------ * Alias for Root Logger added to logging configuration. * Context object in a request scope that holds account and token added. * Support for OPTIONS HTTP request to enable auto version negotiation added. * Documentation improved. 0.1.0 (2015-03-31) ------------------ * Initial release 2022-04-13T11:19:28+00:00 OpenTAXII 0.3.0a4 OpenTAXII 0.3.0a4 2022-04-13T12:20:59+00:00 Changelog ========= 0.3.0a4 (2022-04-13) -------------------- * Merge changes from 0.2.4 maintenance release 0.3.0a3 (2022-01-21) -------------------- * Fix bug that prevented booting with only taxii1 config (`#217 <https://github.com/eclecticiq/OpenTAXII/issues/217>`_ thanks `@azurekid <https://github.com/azurekid>`_ for the report) 0.3.0a2 (2021-12-27) -------------------- * Merge changes from 0.2.3 maintenance release 0.3.0a1 ------- * Add python 3.10 support 0.3.0a0 ------- * Enablement for future taxii2 implementation * Fix documentation build issues 0.2.4 (2022-04-13) ------------------ * Make sure werkzeug <2.1 and >=2.1 work correctly with auth system 0.2.3 (2021-12-22) ------------------ * Fix bug in multithreaded use of sqlite (`#210 <https://github.com/eclecticiq/OpenTAXII/issues/210>`_ thanks `@rohits144 <https://github.com/rohits144>`_ for the report) 0.2.2 (2021-11-05) ------------------ * Fix readthedocs build 0.2.1 (2021-11-03) ------------------ * Add tests for python 3.6, 3.7, 3.8, 3.9, pypy * Add tests for sqlite, mysql, mariadb, postgresql * Fix bug that broke ``delete_content_blocks`` when using mysql on sqlalchemy 1.3 * Docs: Add db schema diagram * Docs: Clarify how to get default data in a default (development) docker instance * Fix implicit routing in TAXII 1.1 Inboxes * Update jwt usage to pyjwt >= 2.0 (thanks `@SanyaKapoor <https://github.com/SanyaKapoor>`_) 0.2.0 (2020-06-30) ------------------ * Enforce UTC usage in datetime fields in SQL DB Persistence API. * `Fix for #114 <https://github.com/eclecticiq/OpenTAXII/issues/114>`_: reintroduce ``opentaxii-create-account`` CLI command. * `Fix for #153 <https://github.com/eclecticiq/OpenTAXII/issues/152>`_: check if user can modify a collection before advertising it over inbox service. * Multiple coding style fixes. * Various documentation updates. 0.1.12 (2019-03-06) ------------------- * Remove unnecessary print statements. 0.1.11 (2019-02-13) ------------------- * Make JSON logging consistent when the application is run via Gunicorn. * Set ``acceptable_destination`` key in status details instead of extended headers * Allow passing ``engine_parameters`` to ``SQLDatabaseAPI`` for those who want to customize SQLAlchemy engine parameters. * Require recent version of ``lxml`` for security reasons. * Various test and Docker infrastructure improvements. 0.1.10 (2018-06-03) ------------------- * Replace separate service/collection/account creation process with single ``opentaxii-sync-data`` CLI command. * Persistence and Auth APIs extended with missing CRUD methods, that are used by ``opentaxii-sync-data``. * Read/modify collection level ACL added. * DB models for default implementation of Persistence API and Auth API were changed. No automatic migration code is provided (sorry!), so upgrading might require manual DB migration. * Drop python2.7 from testing scope. * Various bug fixes and improvements. 0.1.9 (2017-06-19) ------------------ * `libtaxii <https://github.com/TAXIIProject/libtaxii>`_ dependency upgraded to 1.1.111. * Various bug fixes and improvements (thanks to `@bjigmp <https://github.com/bjigmp>`_, `@chorsley <https://github.com/chorsley>`_, `@rjprins <https://github.com/rjprins>`_). 0.1.8 (2017-02-21) ------------------ * Ability to enable/disable "huge trees" support in XML parser. Configuration property ``xml_parser_supports_huge_tree`` set to ``yes`` or ``true`` will disable security restrictions and force XML parser to support very deep trees and very long text content. * Adjust SQL Persistence API implemetation so it works smoothly with MySQL backend. * Use Python 3.5 instead of Python 3.4 for tests. 0.1.7 (2016-10-18) ------------------ * Minor fixes. * Dependencies were changed from hard-pinned to more flexible. * Example of production DB configuration added to docs. 0.1.6 (2016-06-01) ------------------ * Python 3.4 compatibility of the codebase. Tox configuration extended with python 3.4 environment run. * Flake8 full style compatibility. Flake8 check added to Tox configuration. * SQLAlchemy session scope issue fixed (related to `#38 <https://github.com/EclecticIQ/OpenTAXII/issues/38>`_). * `opentaxii-delete-blocks` CLI command added (related to `#45 <https://github.com/EclecticIQ/OpenTAXII/issues/45>`_). * `delete_content_blocks` method `added <https://github.com/EclecticIQ/OpenTAXII/commit/dc6fddc27a98e8450c7e05e583b2bfb741f6e197#diff-6814849ac352b2b74132f8fa52e0ec4eR213>`_ to Persistence API. * Collection's name is `required <https://github.com/EclecticIQ/OpenTAXII/commit/dc6fddc27a98e8450c7e05e583b2bfb741f6e197#diff-ce3f7b939e5c540480ac655aef32c513R116>`_ to be unique in default SQL DB Auth API implementation. 0.1.5 (2016-03-15) ------------------ * Fix for the issue with persistence backend returning ``None`` instead of ``InboxMessage`` object 0.1.4 (2016-02-25) ------------------ * Hard-coded dependencies in ``setup.py`` removed. 0.1.3 (2016-02-25) ------------------ * Versions of dependencies are pinned. * Code adjusted for a new version of `anyconfig <https://pypi.python.org/pypi/anyconfig>`_ API. * Test for configuration loading added. 0.1.2 (2015-07-24) ------------------ * Docker configuration added. * Health check endpoint added. * Basic authentication support added. * Temporary workaround for `Issue #191 <https://github.com/TAXIIProject/libtaxii/issues/191>`_. * Method ``get_domain`` in Persistence API returns domain value configured for ``service_id``. If nothing returned, value set in configuration file will be used. * Performance optimisations. * Bug fixes and style improvements. 0.1.1 (2015-04-08) ------------------ * Alias for Root Logger added to logging configuration. * Context object in a request scope that holds account and token added. * Support for OPTIONS HTTP request to enable auto version negotiation added. * Documentation improved. 0.1.0 (2015-03-31) ------------------ * Initial release 2022-04-13T12:20:59+00:00 OpenTAXII 0.3.0 OpenTAXII 0.3.0 2022-04-13T13:25:40+00:00 Changelog ========= 0.3.0 (2022-04-13) ------------------ * Implement taxii2.1 support 0.3.0a4 (2022-04-13) -------------------- * Merge changes from 0.2.4 maintenance release 0.3.0a3 (2022-01-21) -------------------- * Fix bug that prevented booting with only taxii1 config (`#217 <https://github.com/eclecticiq/OpenTAXII/issues/217>`_ thanks `@azurekid <https://github.com/azurekid>`_ for the report) 0.3.0a2 (2021-12-27) -------------------- * Merge changes from 0.2.3 maintenance release 0.3.0a1 ------- * Add python 3.10 support 0.3.0a0 ------- * Enablement for future taxii2 implementation * Fix documentation build issues 0.2.4 (2022-04-13) ------------------ * Make sure werkzeug <2.1 and >=2.1 work correctly with auth system 0.2.3 (2021-12-22) ------------------ * Fix bug in multithreaded use of sqlite (`#210 <https://github.com/eclecticiq/OpenTAXII/issues/210>`_ thanks `@rohits144 <https://github.com/rohits144>`_ for the report) 0.2.2 (2021-11-05) ------------------ * Fix readthedocs build 0.2.1 (2021-11-03) ------------------ * Add tests for python 3.6, 3.7, 3.8, 3.9, pypy * Add tests for sqlite, mysql, mariadb, postgresql * Fix bug that broke ``delete_content_blocks`` when using mysql on sqlalchemy 1.3 * Docs: Add db schema diagram * Docs: Clarify how to get default data in a default (development) docker instance * Fix implicit routing in TAXII 1.1 Inboxes * Update jwt usage to pyjwt >= 2.0 (thanks `@SanyaKapoor <https://github.com/SanyaKapoor>`_) 0.2.0 (2020-06-30) ------------------ * Enforce UTC usage in datetime fields in SQL DB Persistence API. * `Fix for #114 <https://github.com/eclecticiq/OpenTAXII/issues/114>`_: reintroduce ``opentaxii-create-account`` CLI command. * `Fix for #153 <https://github.com/eclecticiq/OpenTAXII/issues/152>`_: check if user can modify a collection before advertising it over inbox service. * Multiple coding style fixes. * Various documentation updates. 0.1.12 (2019-03-06) ------------------- * Remove unnecessary print statements. 0.1.11 (2019-02-13) ------------------- * Make JSON logging consistent when the application is run via Gunicorn. * Set ``acceptable_destination`` key in status details instead of extended headers * Allow passing ``engine_parameters`` to ``SQLDatabaseAPI`` for those who want to customize SQLAlchemy engine parameters. * Require recent version of ``lxml`` for security reasons. * Various test and Docker infrastructure improvements. 0.1.10 (2018-06-03) ------------------- * Replace separate service/collection/account creation process with single ``opentaxii-sync-data`` CLI command. * Persistence and Auth APIs extended with missing CRUD methods, that are used by ``opentaxii-sync-data``. * Read/modify collection level ACL added. * DB models for default implementation of Persistence API and Auth API were changed. No automatic migration code is provided (sorry!), so upgrading might require manual DB migration. * Drop python2.7 from testing scope. * Various bug fixes and improvements. 0.1.9 (2017-06-19) ------------------ * `libtaxii <https://github.com/TAXIIProject/libtaxii>`_ dependency upgraded to 1.1.111. * Various bug fixes and improvements (thanks to `@bjigmp <https://github.com/bjigmp>`_, `@chorsley <https://github.com/chorsley>`_, `@rjprins <https://github.com/rjprins>`_). 0.1.8 (2017-02-21) ------------------ * Ability to enable/disable "huge trees" support in XML parser. Configuration property ``xml_parser_supports_huge_tree`` set to ``yes`` or ``true`` will disable security restrictions and force XML parser to support very deep trees and very long text content. * Adjust SQL Persistence API implemetation so it works smoothly with MySQL backend. * Use Python 3.5 instead of Python 3.4 for tests. 0.1.7 (2016-10-18) ------------------ * Minor fixes. * Dependencies were changed from hard-pinned to more flexible. * Example of production DB configuration added to docs. 0.1.6 (2016-06-01) ------------------ * Python 3.4 compatibility of the codebase. Tox configuration extended with python 3.4 environment run. * Flake8 full style compatibility. Flake8 check added to Tox configuration. * SQLAlchemy session scope issue fixed (related to `#38 <https://github.com/EclecticIQ/OpenTAXII/issues/38>`_). * `opentaxii-delete-blocks` CLI command added (related to `#45 <https://github.com/EclecticIQ/OpenTAXII/issues/45>`_). * `delete_content_blocks` method `added <https://github.com/EclecticIQ/OpenTAXII/commit/dc6fddc27a98e8450c7e05e583b2bfb741f6e197#diff-6814849ac352b2b74132f8fa52e0ec4eR213>`_ to Persistence API. * Collection's name is `required <https://github.com/EclecticIQ/OpenTAXII/commit/dc6fddc27a98e8450c7e05e583b2bfb741f6e197#diff-ce3f7b939e5c540480ac655aef32c513R116>`_ to be unique in default SQL DB Auth API implementation. 0.1.5 (2016-03-15) ------------------ * Fix for the issue with persistence backend returning ``None`` instead of ``InboxMessage`` object 0.1.4 (2016-02-25) ------------------ * Hard-coded dependencies in ``setup.py`` removed. 0.1.3 (2016-02-25) ------------------ * Versions of dependencies are pinned. * Code adjusted for a new version of `anyconfig <https://pypi.python.org/pypi/anyconfig>`_ API. * Test for configuration loading added. 0.1.2 (2015-07-24) ------------------ * Docker configuration added. * Health check endpoint added. * Basic authentication support added. * Temporary workaround for `Issue #191 <https://github.com/TAXIIProject/libtaxii/issues/191>`_. * Method ``get_domain`` in Persistence API returns domain value configured for ``service_id``. If nothing returned, value set in configuration file will be used. * Performance optimisations. * Bug fixes and style improvements. 0.1.1 (2015-04-08) ------------------ * Alias for Root Logger added to logging configuration. * Context object in a request scope that holds account and token added. * Support for OPTIONS HTTP request to enable auto version negotiation added. * Documentation improved. 0.1.0 (2015-03-31) ------------------ * Initial release 2022-04-13T13:25:40+00:00 caddy v2.5.0-rc.1 caddy v2.5.0-rc.1 2022-04-13T18:56:51+00:00 Caddy 2.5 introduces new features you'll love as well as a huge number of bug fixes and enhancements. Thank you to everyone who contributed. **:warning: This is a release candidate version. We think it's stable enough to use in production, but we want to be extra sure, so please try it out! It needs to be used and tested for regressions. Let us know if there are any issues.** Documentation on the website has mostly been updated (JSON docs will be last). Feel free to ask on the [forum](https://caddy.community) if you have any questions or feedback! ## Highlights - **Reverse proxy:** [:sparkles: _Dynamic upstreams_](https://github.com/caddyserver/caddy/pull/4470), which is the ability to get the list of upstreams at every request (more specifically, every iteration in the proxy loop of every request) rather than just once at config-load time. Dynamic upstream modules can be plugged in to provide Caddy with the latest list of backends in real-time. Two standard modules have been implemented which can get upstreams from SRV and A/AAAA record lookups. - :warning: This deprecates the `lookup_srv` JSON field for upstreams (and `srv+` scheme prefix in the Caddyfile), which will be removed in the future. - **Automatic HTTPS:** Caddy will automatically try to get relevant certificates from the local [Tailscale](https://tailscale.com) instance (if running with permission to access the Tailscale socket). This makes services running on a Tailscale network [automatically available](https://github.com/caddyserver/caddy/pull/4541) over trusted HTTPS with Caddy. - **Tracing:** New [OpenTelemetry](https://opentelemetry.io/) integration with the [`tracing` handler module and associated `tracing` directive](https://github.com/caddyserver/caddy/pull/4361). - **Reverse proxy:** When using the response handlers, a new handler `copy_response` is available to copy the proxy's response back to the client, and `copy_response_headers` may be used to selectively copy header values from the proxy's response. - **API:** Added new endpoints `/pki/ca/<id>` and `/pki/ca/<id>/certificates` for getting information about Caddy's managed CAs, including the chain of root and intermediate certificates. ## Notable - **Reverse proxy:** The `X-Forwarded-Host` header will now be automatically set, along with `X-Forwarded-For` and `X-Forwarded-Proto`. - :warning: **Reverse proxy:** Incoming `X-Forwarded-*` headers will no longer be automatically trusted, to prevent spoofing. Now, `trusted_proxies` must be configured to specify a list of downstream proxies which are trusted to have sent good values. You only need to configure trusted proxies if Caddy is not the first server being connected to. For example, if you have Cloudflare in front of Caddy, then you should configure this with Cloudflare's [list of IP ranges](https://www.cloudflare.com/en-ca/ips/). - **Automatic HTTPS:** Revoked certificates will be automatically [replaced more reliably](https://github.com/caddyserver/certmagic/pull/166). - **Automatic HTTPS:** [Can now get certificates from _Managers_.](https://github.com/caddyserver/caddy/pull/4541) As opposed to _Issuers_ (such as the default ACME issuers) which give Caddy certificates to manage from a CSR, Managers give Caddy certificates to serve (rather than manage) during TLS handshakes. - **Automatic HTTPS:** A DNS challenge [domain override](https://github.com/caddyserver/caddy/issues/4071) can be configured to delegate the solving of the challenge to a different domain. - **Reverse proxy:** The default dial timeout for the HTTP transport [has been adjusted](https://github.com/caddyserver/caddy/pull/4436) down to `3s` (was `10s`), which should allow for more easily configuring load balancing retries. - **Logging:** HTTP access logs will now [render empty values](https://github.com/caddyserver/caddy/commit/7d5047c1f190421528695e1cc3a4ad71c97eb022) for often-sensitive HTTP headers such as Cookie, Authorization, and Proxy-Authorization. Logging such credentials is now [opt-in](https://github.com/caddyserver/caddy/commit/5bf0adad8748e96e10529d5fc5777afc9236a7b5) with the `log_credentials` global option in the Caddyfile, or the server's `logs > should_log_credentials` field in JSON. - **Logging:** Logs can now be filtered by [query string parameters](https://github.com/caddyserver/caddy/commit/bcac2beee7e419f8cdab2ed16f388d1af282a46b), [cookie values](https://github.com/caddyserver/caddy/commit/8887adb027982e844965b4707b8595cee5845d54), and [regular expressions](https://github.com/caddyserver/caddy/commit/789efa5deef53071b57479d37e4022bf372c4eef); and log values can be [hashed](https://github.com/caddyserver/caddy/commit/a1b417c832b4ab3dab9eaa9690e1d07672a949b8). These features are useful for redacting sensitive information. - **Logging:** Errors during request handling [will now be logged at `DEBUG` level](https://github.com/caddyserver/caddy/pull/4429) if the error was [handled via `errors` routes](https://github.com/caddyserver/caddy/pull/4584) (`handle_errors` in Caddyfile). - :warning: **Logging:** Removed the [deprecated](https://github.com/caddyserver/caddy/issues/4148) `common_log` field from HTTP access logs, and the `single_field` encoder. If you relied on this, you may use the [transform encoder plugin](https://github.com/caddyserver/transform-encoder) to encode logs in Common Log format. - :warning: **Logging:** The `remote_addr` field [has been replaced](https://github.com/caddyserver/caddy/commit/f55b123d63132e290789bcd07077375c76b6e1dd) by `remote_ip` and `remote_port` fields in HTTP access logs, which split up the two parts of the remote address. This improves ease of use for some tooling which only expect an IP address, without a port. - **HTTP server:** The [`vars` matcher](https://github.com/caddyserver/caddy/commit/ecac03cdcb6cceae743aac16faca7f32e5da1607) can now match on multiple possible values. - **HTTP server:** Requests [can now be assigned](https://github.com/caddyserver/caddy/commit/180ae0cc4843ecc3c7ddcb6e978ebfd474ed07f9) a random and unique UUID from the new `{http.request.uuid}` placeholder. - **HTTP server:** [New `http_redirect` listener wrapper](https://github.com/caddyserver/caddy/pull/4585) which can be used to redirect HTTP requests that come in on a server listening for HTTPS requests to be redirected to `https://`. - **Caddyfile:** [New `default_bind` global option](https://github.com/caddyserver/caddy/pull/4531) lets you specify the default interface all sockets should bind to. - **Caddyfile:** [New `pki` global option](https://github.com/caddyserver/caddy/pull/4450) lets you configure the properties of the internal CAs managed by Caddy. - **Caddyfile:** [New `method` directive](https://github.com/caddyserver/caddy/pull/4528) allows rewriting the request method via Caddyfile. - :warning: **Caddyfile:** The `reverse_proxy` directive's `handle_response` subdirective has had its status replacement functionality [moved to a new `replace_status`](https://github.com/caddyserver/caddy/pull/4300) subdirective. This makes sure that the functionality of `handle_response` is not overloaded, and usage is clearer. - :warning: **Admin:** [Renamed](https://github.com/caddyserver/caddy/commit/bc447e307f195b80eeec0f6157e0d8e641af9155) experimental property `load_interval` :arrow_right: `load_delay` for clarification, and improved dynamic config loading. --- :shield: Thanks to [David Leadbeater](https://github.com/dgl) for reporting a security vulnerability related to HTTP methods and metrics cardinality, which was fixed in this release. ## New Contributors * @adamburgess made their first contribution in https://github.com/caddyserver/caddy/pull/4460 * @12f23eddde made their first contribution in https://github.com/caddyserver/caddy/pull/4444 * @rayjlinden made their first contribution in https://github.com/caddyserver/caddy/pull/4023 * @GallopingKylin made their first contribution in https://github.com/caddyserver/caddy/pull/4522 * @ForestJohnson made their first contribution in https://github.com/caddyserver/caddy/pull/4534 * @VojtechVitek made their first contribution in https://github.com/caddyserver/caddy/pull/4535 * @Ikke made their first contribution in https://github.com/caddyserver/caddy/pull/4544 * @YourTechBud made their first contribution in https://github.com/caddyserver/caddy/pull/4603 * @BitWuehler made their first contribution in https://github.com/caddyserver/caddy/pull/4597 * @ttys3 made their first contribution in https://github.com/caddyserver/caddy/pull/4572 * @crccw made their first contribution in https://github.com/caddyserver/caddy/pull/4596 * @andriikushch made their first contribution in https://github.com/caddyserver/caddy/pull/4361 * @renbou made their first contribution in https://github.com/caddyserver/caddy/pull/4654 * @cuishuang made their first contribution in https://github.com/caddyserver/caddy/pull/4702 ## Changelog * 2e46c2ac admin, reverseproxy: Stop timers if canceled to avoid goroutine leak (#4482) * 40b54434 admin: Enforce and refactor origin checking * b4bfa29b admin: Require identity for remote (fix #4478) * 32aad909 admin: Write proper status on invalid requests (#4569) (fix #4561) * ff137d17 caddyconfig: Support placeholders in HTTP loader * b47af6ef caddyfile: Copy input before parsing (fix #4422) * e90d7517 caddyfile: impove fmt warning message (#4444) * 5e5af50e caddyfile: make renew_interval option configurable (#4451) * ddbb234d caddyhttp: Always log handled errors at debug level (#4584) * 6b385a36 caddyhttp: Don't attempt to manage Tailscale certs * ecac03cd caddyhttp: Enhance vars matcher (#4433) * 6e6ce2be caddyhttp: Fix HTTP->HTTPS redir not preferring HTTPS port if ambiguous (#4530) * 3fe2c73d caddyhttp: Fix `MatchPath` sanitizing (#4499) * 44e5e9e4 caddyhttp: Fix test when /tmp/etc already exists (#4544) * 2bb8550a caddyhttp: Honor wildcard hosts in log SkipHosts (#4606) * 180ae0cc caddyhttp: Implement http.request.uuid placeholder (#4285) * 7d5047c1 caddyhttp: Log empty value for typical password headers * eead3373 caddyhttp: Log non-500 handler errors at debug level (#4429) * 5bf0adad caddyhttp: Make logging of credential headers opt-in (#4438) * 186fdba9 caddyhttp: Move HTTP redirect listener to an optional module (#4585) * 80d7a356 caddyhttp: Redirect HTTP requests on the HTTPS port to https:// (#4313) * bf380d00 caddyhttp: Reject absurd methods (#4538) * 850e1605 caddyhttp: Return HTTP 421 for mismatched Host header (#4023) * f55b123d caddyhttp: Split up logged remote address into IP and port (#4403) * ac14b64e caddyhttp: Support zone identifiers in remote_ip matcher (#4597) * a1c41210 caddypki: Minor tweak, don't use context pointer * 78e381b2 caddypki: Refactor /pki/ admin endpoints * c634bbe9 caddypki: Return error if no PEM data found * 9b7cdfa2 caddypki: Try to fix lint warnings * a79b4055 caddytls: Add internal Caddyfile `lifetime`, `sign_with_root` opts (#4513) * 66de438a caddytls: Fix `MatchRemoteIP` provisoning with multiple CIDR ranges (#4522) * 57a708d1 caddytls: Support external certificate Managers (like Tailscale) (#4541) * d9b1d463 caddytls: dns_challenge_override_domain for challenge delegation (#4596) * 1a7a78a1 cmd: Print error if fmt overwrite fails (fix #4524) * bc447e30 core: Config LoadInterval -> LoadDelay for clarity * 7ea5b2a8 core: Config load interval only reloads if changed (#4603) * 7f364c77 core: Load config at interval instead of just once * a72acd21 core: Retry dynamic config load if config unchanged * ceef70db core: Retry dynamic config load if error or no-op (#4603) * acbee947 core: Revert 7f364c7; simplify dynamic config load * 64a3218f core: Simplify shared listeners, fix deadline bug * 8e5aafa5 fastcgi: Fix a TODO, prevent zap using reflection for logging env (#4437) * c8f2834b fastcgi: Protect against requests with null bytes in the path (#4614) * de490c7c fastcgi: Set SERVER_PORT to 80 or 443 depending on scheme (#4572) * 09ba9e99 fileserver: Add `pass_thru` Caddyfile option (#4613) * 15c95e9d fileserver: Canonical redir when whole path is stripped (#4549) * c8b5a816 fileserver: Fix handling of symlink sizes in directory listings (#4415) * e81369e2 fileserver: Move default browse template into a separate file (#4417) * 1e10f6f7 fileserver: browse: do not encode the paths in breadcrumbs and page title (#4410) * 78b5356f fileserver: do not double-escape paths (#4447) * 0de51593 go.mod: Revert version bump of CEL (#4587) * 6f9b6ad7 go.mod: Update smallstep/certificates, no longer need replace (#4475) * 4906b935 go.mod: Update smallstep/truststore, fix build on FreeBSD (#4473) * c1331534 go.mod: Update to latest smallstep/truststore, support FreeBSD (#4453) * ff74a0aa go.mod: Upgrade dependencies * e9dde230 headers: Fix `+` in Caddyfile to properly append rather than set (#4506) * 1b7ff5d7 httpcaddyfile: Add `default_bind` global option (#4531) * 5a071568 httpcaddyfile: Add pki app `root` and `intermediate` cert/key config (#4514) * 26d633ba httpcaddyfile: Disabling OCSP stapling for both managed and unmanaged (#4589) * 93a7a45e httpcaddyfile: Fix incorrect handling of IPv6 bind addresses (#4532) * 81ee34e9 httpcaddyfile: Fix sorting edgecase for nested `handle_path` (#4477) * 4b9849c7 httpcaddyfile: Support configuring `pki` app names via global options (#4450) * 5bd96a6a httpcaddyfile: Support explicitly turning off `strict_sni_host` (#4592) * c921e082 logging: Add `roll_local_time` Caddyfile option (#4583) * 0eb0b60f logging: Remove common_log field and single_field encoder (#4149) (#4282) * 249adc1c logging: Support turning off roll compression via Caddyfile (#4505) * 8887adb0 logging: add a filter for cookies (#4425) * bcac2bee logging: add a filter for query parameters (#4424) * 789efa5d logging: add a regexp filter (#4426) * a1b417c8 logging: add support for hashing data (#4434) * eb891d46 metrics: Enforce smaller set of method labels * c04d24ca pki: Avoid provisioning the `local` CA when not necessary (#4463) * bbad6931 pki: Implement API endpoints for certs and `caddy trust` (#4443) * 9ee68c1b reverseproxy: Adjust defaults, document defaults (#4436) * 7557d1d9 reverseproxy: Avoid returning a `nil` error during GetClientCertificate (#4550) * ab045592 reverseproxy: Dynamic upstreams (with SRV and A/AAAA support) (#4470) * 5333c352 reverseproxy: Fix incorrect `health_headers` Caddyfile parsing (#4485) * c50094fc reverseproxy: Implement trusted proxies for `X-Forwarded-*` headers (#4507) * f5e10494 reverseproxy: Make shallow-ish clone of the request (#4551) * 87a1f228 reverseproxy: Move status replacement intercept to `replace_status` (#4300) * d058dee1 reverseproxy: Refactor dial address parsing, augment command parsing (#4616) * c7d6c4cb reverseproxy: copy_response and copy_response_headers for handle_response routes (#4391) * bcb7a19c rewrite: Add `method` Caddyfile directive (#4528) * 1feb6595 rewrite: Fix a double-encode issue when using the `{uri}` placeholder (#4516) * 6cadb60f templates: Document .OriginalReq * 1d0425b2 templates: Elaborate on what's supported by the markdown function (#4564) * a6199cf8 templates: Fix docs for .Args * ec14ccdd templates: fix inconsistent nested includes (#4452) * d0b608af tracing: New OpenTelemetry module (#4361) * 134b8056 caddyfile: Prevent bad block opening tokens (#4655) * c5fffb4a caddyfile: Support for raw token values; improve `map`, `expression` (#4643) * b82e22b4 caddyhttp: retain all values of vars matcher when specified multiple times (#4629) * bc15b4b0 caddypki: Load intermediate for signing on-the-fly (#4669) * 6512832f cmd: Add `--diff` option for `caddy fmt` (#4695) * 30b6d1f4 cmd: Enhance .env (dotenv) file parsing * 22d8edb9 cmd: Fix defaulting admin address if empty in config, fixes `reload` (#4674) * c2327161 cmd: Set Origin header properly on API requests * d06d0e79 go.mod: Upgrade CertMagic to v0.16.0 * bf6a1b75 go.mod: Upgrade some dependencies * 79cbe7bf httpcaddyfile: Add 'vars' directive * a58f240d httpcaddyfile: Fix #4640 (auto-HTTPS edgecase) (#4661) * 7d229665 logging: Caddyfile support for `duration_format` (#4684) * 55b4c12e map: Evaluate placeholders in output vals (#4650) * 93c99f67 map: Support numeric and bool types with Caddyfile * 3d616e8c requestbody: Return HTTP 413 (fix #4558) * 3e3bb002 reverseproxy: Add `_ms` placeholders for proxy durations (#4666) * 2196c92c reverseproxy: Don't clear name in SRV upstreams * dc4d1473 reverseproxy: Expand SRV/A addrs for cache key * b8dbecb8 reverseproxy: Include port in A upstreams cache * e4ce40f8 reverseproxy: Sync up `handleUpgradeResponse` with stdlib (#4664) * afca2421 staticfiles: Expand placeholder for index files (#4679) * 00234c8a templates: Switch to `BurntSushi/toml` (#4700) **Full Changelog**: https://github.com/caddyserver/caddy/compare/v2.4.6...v2.5.0-rc.1 2022-04-13T18:56:51+00:00 rocksdb v7.1.1 rocksdb v7.1.1 2022-04-13T21:20:40+00:00 ## 7.1.1 (04/07/2022) ### Bug Fixes * Fix segfault in FilePrefetchBuffer with async_io as it doesn't wait for pending jobs to complete on destruction. ## 7.1.0 (03/23/2022) ### New Features * Allow WriteBatchWithIndex to index a WriteBatch that includes keys with user-defined timestamps. The index itself does not have timestamp. * Add support for user-defined timestamps to write-committed transaction without API change. The `TransactionDB` layer APIs do not allow timestamps because we require that all user-defined-timestamps-aware operations go through the `Transaction` APIs. * Added BlobDB options to `ldb` * `BlockBasedTableOptions::detect_filter_construct_corruption` can now be dynamically configured using `DB::SetOptions`. * Automatically recover from retryable read IO errors during backgorund flush/compaction. * Experimental support for preserving file Temperatures through backup and restore, and for updating DB metadata for outside changes to file Temperature (`UpdateManifestForFilesState` or `ldb update_manifest --update_temperatures`). * Experimental support for async_io in ReadOptions which is used by FilePrefetchBuffer to prefetch some of the data asynchronously, if reads are sequential and auto readahead is enabled by rocksdb internally. ### Bug Fixes * Fixed a major performance bug in which Bloom filters generated by pre-7.0 releases are not read by early 7.0.x releases (and vice-versa) due to changes to FilterPolicy::Name() in #9590. This can severely impact read performance and read I/O on upgrade or downgrade with existing DB, but not data correctness. * Fixed a data race on `versions_` between `DBImpl::ResumeImpl()` and threads waiting for recovery to complete (#9496) * Fixed a bug caused by race among flush, incoming writes and taking snapshots. Queries to snapshots created with these race condition can return incorrect result, e.g. resurfacing deleted data. * Fixed a bug that DB flush uses `options.compression` even `options.compression_per_level` is set. * Fixed a bug that DisableManualCompaction may assert when disable an unscheduled manual compaction. * Fix a race condition when cancel manual compaction with `DisableManualCompaction`. Also DB close can cancel the manual compaction thread. * Fixed a potential timer crash when open close DB concurrently. * Fixed a race condition for `alive_log_files_` in non-two-write-queues mode. The race is between the write_thread_ in WriteToWAL() and another thread executing `FindObsoleteFiles()`. The race condition will be caught if `__glibcxx_requires_nonempty` is enabled. * Fixed a bug that `Iterator::Refresh()` reads stale keys after DeleteRange() performed. * Fixed a race condition when disable and re-enable manual compaction. * Fixed automatic error recovery failure in atomic flush. * Fixed a race condition when mmaping a WritableFile on POSIX. ### Public API changes * Added pure virtual FilterPolicy::CompatibilityName(), which is needed for fixing major performance bug involving FilterPolicy naming in SST metadata without affecting Customizable aspect of FilterPolicy. This change only affects those with their own custom or wrapper FilterPolicy classes. * `options.compression_per_level` is dynamically changeable with `SetOptions()`. * Added `WriteOptions::rate_limiter_priority`. When set to something other than `Env::IO_TOTAL`, the internal rate limiter (`DBOptions::rate_limiter`) will be charged at the specified priority for writes associated with the API to which the `WriteOptions` was provided. Currently the support covers automatic WAL flushes, which happen during live updates (`Put()`, `Write()`, `Delete()`, etc.) when `WriteOptions::disableWAL == false` and `DBOptions::manual_wal_flush == false`. * Add DB::OpenAndTrimHistory API. This API will open DB and trim data to the timestamp specified by trim_ts (The data with timestamp larger than specified trim bound will be removed). This API should only be used at a timestamp-enabled column families recovery. If the column family doesn't have timestamp enabled, this API won't trim any data on that column family. This API is not compatible with avoid_flush_during_recovery option. * Remove BlockBasedTableOptions.hash_index_allow_collision which already takes no effect. 2022-04-13T21:20:40+00:00 syncthing v1.20.0-rc.1 syncthing v1.20.0-rc.1 2022-04-15T04:50:34+00:00 Bugfixes: - #7855: Watching for changes using too many system resources on BSDs (kqueue) - #8020: Dangling symlink prevents filesystem watcher on Solaris - #8232: Missing include file endlessly spams log - #8258: The latest version does not support go1.18 compilation!! Enhancements: - #8202: Indicate folders / devices where the remote end has not accepted sharing. Other issues: - #8279: Simplify tests using `T.TempDir` 2022-04-15T04:50:34+00:00 rocksdb v7.1.2 rocksdb v7.1.2 2022-04-20T01:48:15+00:00 ## 7.1.2 (04/19/2022) ### Bug Fixes * Fixed bug which caused rocksdb failure in the situation when rocksdb was accessible using UNC path * Fixed a race condition when 2PC is disabled and WAL tracking in the MANIFEST is enabled. The race condition is between two background flush threads trying to install flush results, causing a WAL deletion not tracked in the MANIFEST. A future DB open may fail. * Fixed a heap use-after-free race with DropColumnFamily. * Fixed a bug that `rocksdb.read.block.compaction.micros` cannot track compaction stats (#9722). * Fixed `file_type`, `relative_filename` and `directory` fields returned by `GetLiveFilesMetaData()`, which were added in inheriting from `FileStorageInfo`. * Fixed a bug affecting `track_and_verify_wals_in_manifest`. Without the fix, application may see "open error: Corruption: Missing WAL with log number" while trying to open the db. The corruption is a false alarm but prevents DB open (#9766). 2022-04-20T01:48:15+00:00 MISP v2.4.158 MISP v2.4.158 2022-04-20T07:43:37+00:00 We are pleased to announce the immediate availability of MISP v2.4.158. This release includes a series of security fixes and as such **we highly encourage everyone to update to this version as soon as possible**. Thanks to Dawid Czarnecki of Zigrin Security for the in-depth penetration test and its findings and thanks to the Luxembourg Army for financing the penetration test. This is the follow up to the Cerebrate penetration test also conducted by Zigrin Security on behalf of the Luxembourg Army, as described [here](https://www.cerebrate-project.org/2022/01/27/Cerebate-version-1.4-released.html). # Security fixes Several security issues have been resolved, head over to [the security page](https://www.misp-project.org/security/) for a detailed break-down of the advisories including the associated CVEs. Whilst most of the vulnerabilities listed are mitigated by requiring compromised high privilege accounts, we nevertheless advise all users to update their instances as soon as possible. - Phar deserialisation - [Global fix](https://github.com/MISP/MISP/commit/0108f1bde2117ac5c1e28d124128f60c8bb09a8e) - [Individual additional mitigations](https://github.com/MISP/MISP/commit/93821c0de6a7dd32262ce62212773f43136ca66e) - [XSS in LinOTP login](https://github.com/MISP/MISP/commit/9623de2f5cca011afc581d55cfa5ce87682894fd) - [XSS in Galaxy clusters](https://github.com/MISP/MISP/commit/107e271d78c255d658ce998285fe6f6c4f291b41) - [XSS in organisation fetchSGOrgRow](https://github.com/MISP/MISP/commit/ce6bc88e330f5ef50666b149d86c0d94f545f24e) - [XSS in Event graph via tags](https://github.com/MISP/MISP/commit/bb3b7a7e91862742cae228c43b3091bad476dcc0) - [XSS in Cerebrate view](https://github.com/MISP/MISP/commit/60c85b80e3ab05c3ef015bca5630e95eddbb1436) - [Password confirmation bypass](https://github.com/MISP/MISP/commit/01120163a6b4d905029d416e7305575df31df8af) ## Announcement of a silent fix of phar deserialisation RCE in a previous release (v2.4.156) As of the previous security release (v2.4.156), based on the pentest conducted by Ianis BERNARD of the NATO Cyber Security Centre, a high criticality vulnerability was also identiefied. We have opted for a silent fix to the critical vulnerability whilst upgrading the announced criticality of the other security fixes included in the release. This is an extreme measure that we take whenever we want to ensure that the community is both aware that they do need to update as soon as possible whilst not drawing attention to the actual critical vulnerability. If you have followed our guidance over the past month to update you are already safe - if you are running a MISP instance below 2.4.156 **we highly encourage you to update to the latest version as soon as possible**. - [Phar deserialisation silent fix](https://github.com/MISP/MISP/commit/8eff854fce1fea1521f33fffc2440df5b7e5c410) # Custom email templates Added the ability to override some of the standard e-mail templates with custom ones, just drop the templates mirroring the naming convention of the existing ones in `/var/www/MISP/app/View/Email/text` and `/var/www/MISP/app/View/Email/html` into `/var/www/MISP/app/View/Email/text/Custom/` and `/var/www/MISP/app/View/Email/html/Custom/`. Currently supported templates: alert, password_reset. # RestSearch improvements Fixing a baffling oversight on our side, thanks to Tom King we can now search by sharing groups besides just distribution levels. # A long list of refactors and bugfixes Massive thanks to Jakub Onderka for the continuous refactoring, simplifying and cleaning up of the code-base. For a full list of all the improvements that are part of this herculean effort, refer to the [changelog](https://www.misp-project.org/Changelog.txt) # Acknowledgement We would like to thank all the [contributors](https://www.misp-project.org/contributors), reporters and users who have helped us in the past months to improve MISP and information sharing at large. This release includes multiple updates in [misp-objects](https://www.misp-project.org/objects.html), [misp-taxonomies](https://www.misp-project.org/taxonomies.html) and [misp-galaxy](https://www.misp-project.org/galaxy.html). As always, a detailed and [complete changelog is available](https://www.misp-project.org/Changelog.txt) with all the fixes, changes and improvements in MISP core. 2022-04-20T07:43:37+00:00 syncthing v1.20.0-rc.2 syncthing v1.20.0-rc.2 2022-04-21T13:57:00+00:00 Bugfixes: - #7855: Watching for changes using too many system resources on BSDs (kqueue) - #8020: Dangling symlink prevents filesystem watcher on Solaris - #8232: Missing include file endlessly spams log - #8258: The latest version does not support go1.18 compilation!! Enhancements: - #8202: Indicate folders / devices where the remote end has not accepted sharing. - #8282: Change default log format to include date Other issues: - #8279: Simplify tests using `T.TempDir` 2022-04-21T13:57:00+00:00 caddy v2.5.0 caddy v2.5.0 2022-04-25T17:34:35+00:00 Caddy 2.5 introduces new features you'll love as well as a huge number of bug fixes and enhancements. Thank you to everyone who contributed! Feel free to ask on the [forum](https://caddy.community) if you have any questions or feedback. ## Highlights - **Reverse proxy:** [:sparkles: _Dynamic upstreams_](https://github.com/caddyserver/caddy/pull/4470), which is the ability to get the list of upstreams at every request (more specifically, every iteration in the proxy loop of every request) rather than just once at config-load time. Dynamic upstream modules can be plugged in to provide Caddy with the latest list of backends in real-time. Two standard modules have been implemented which can get upstreams from SRV and A/AAAA record lookups. - :warning: This deprecates the `lookup_srv` JSON field for upstreams (and `srv+` scheme prefix in the Caddyfile), which will be removed in the future. - **Automatic HTTPS:** Caddy will automatically try to get relevant certificates from the local [Tailscale](https://tailscale.com) instance (if running with permission to access the Tailscale socket). This makes services running on a Tailscale network [automatically available](https://github.com/caddyserver/caddy/pull/4541) over trusted HTTPS with Caddy. - **Tracing:** New [OpenTelemetry](https://opentelemetry.io/) integration with the [`tracing` handler module and associated `tracing` directive](https://github.com/caddyserver/caddy/pull/4361). - **Reverse proxy:** When using the response handlers, a new handler `copy_response` is available to copy the proxy's response back to the client, and `copy_response_headers` may be used to selectively copy header values from the proxy's response. - **API:** Added new endpoints `/pki/ca/<id>` and `/pki/ca/<id>/certificates` for getting information about Caddy's managed CAs, including the chain of root and intermediate certificates. ## Notable - **Reverse proxy:** The `X-Forwarded-Host` header will now be automatically set, along with `X-Forwarded-For` and `X-Forwarded-Proto`. - :warning: **Reverse proxy:** Incoming `X-Forwarded-*` headers will no longer be automatically trusted, to prevent spoofing. Now, `trusted_proxies` must be configured to specify a list of downstream proxies which are trusted to have sent good values. You only need to configure trusted proxies if Caddy is not the first server being connected to. For example, if you have Cloudflare in front of Caddy, then you should configure this with Cloudflare's [list of IP ranges](https://www.cloudflare.com/en-ca/ips/). - **Automatic HTTPS:** Revoked certificates will be automatically [replaced more reliably](https://github.com/caddyserver/certmagic/pull/166). - **Automatic HTTPS:** [Can now get certificates from _Managers_.](https://github.com/caddyserver/caddy/pull/4541) As opposed to _Issuers_ (such as the default ACME issuers) which give Caddy certificates to manage from a CSR, Managers give Caddy certificates to serve (rather than manage) during TLS handshakes. - **Automatic HTTPS:** A DNS challenge [domain override](https://github.com/caddyserver/caddy/issues/4071) can be configured to delegate the solving of the challenge to a different domain. - **Automatic HTTPS:** The DNS challenge propagation checks can now be delayed or disabled by setting `propagation_delay` or `propagation_timeout` to -1, respectively. - **Reverse proxy:** The default dial timeout for the HTTP transport [has been adjusted](https://github.com/caddyserver/caddy/pull/4436) down to `3s` (was `10s`), which should allow for more easily configuring load balancing retries. - **Logging:** HTTP access logs will now [render empty values](https://github.com/caddyserver/caddy/commit/7d5047c1f190421528695e1cc3a4ad71c97eb022) for often-sensitive HTTP headers such as Cookie, Authorization, and Proxy-Authorization. Logging such credentials is now [opt-in](https://github.com/caddyserver/caddy/commit/5bf0adad8748e96e10529d5fc5777afc9236a7b5) with the `log_credentials` global option in the Caddyfile, or the server's `logs > should_log_credentials` field in JSON. - **Logging:** Logs can now be filtered by [query string parameters](https://github.com/caddyserver/caddy/commit/bcac2beee7e419f8cdab2ed16f388d1af282a46b), [cookie values](https://github.com/caddyserver/caddy/commit/8887adb027982e844965b4707b8595cee5845d54), and [regular expressions](https://github.com/caddyserver/caddy/commit/789efa5deef53071b57479d37e4022bf372c4eef); and log values can be [hashed](https://github.com/caddyserver/caddy/commit/a1b417c832b4ab3dab9eaa9690e1d07672a949b8). These features are useful for redacting sensitive information. - **Logging:** Errors during request handling [will now be logged at `DEBUG` level](https://github.com/caddyserver/caddy/pull/4429) if the error was [handled via `errors` routes](https://github.com/caddyserver/caddy/pull/4584) (`handle_errors` in Caddyfile). - :warning: **Logging:** Removed the [deprecated](https://github.com/caddyserver/caddy/issues/4148) `common_log` field from HTTP access logs, and the `single_field` encoder. If you relied on this, you may use the [transform encoder plugin](https://github.com/caddyserver/transform-encoder) to encode logs in Common Log format. - :warning: **Logging:** The `remote_addr` field [has been replaced](https://github.com/caddyserver/caddy/commit/f55b123d63132e290789bcd07077375c76b6e1dd) by `remote_ip` and `remote_port` fields in HTTP access logs, which split up the two parts of the remote address. This improves ease of use for some tooling which only expect an IP address, without a port. - **HTTP server:** The [`vars` matcher](https://github.com/caddyserver/caddy/commit/ecac03cdcb6cceae743aac16faca7f32e5da1607) can now match on multiple possible values. - **HTTP server:** Requests [can now be assigned](https://github.com/caddyserver/caddy/commit/180ae0cc4843ecc3c7ddcb6e978ebfd474ed07f9) a random and unique UUID from the new `{http.request.uuid}` placeholder. - **HTTP server:** [New `http_redirect` listener wrapper](https://github.com/caddyserver/caddy/pull/4585) which can be used to redirect HTTP requests that come in on a server listening for HTTPS requests to be redirected to `https://`. - :warning: **Caddyfile:** [Deprecated paths in site addresses.](https://github.com/caddyserver/caddy/pull/4728) Prefer using path matchers within your site block instead. - **Caddyfile:** [New `default_bind` global option](https://github.com/caddyserver/caddy/pull/4531) lets you specify the default interface all sockets should bind to. - **Caddyfile:** [New `pki` global option](https://github.com/caddyserver/caddy/pull/4450) lets you configure the properties of the internal CAs managed by Caddy. - **Caddyfile:** [New `method` directive](https://github.com/caddyserver/caddy/pull/4528) allows rewriting the request method via Caddyfile. - :warning: **Caddyfile:** The `reverse_proxy` directive's `handle_response` subdirective has had its status replacement functionality [moved to a new `replace_status`](https://github.com/caddyserver/caddy/pull/4300) subdirective. This makes sure that the functionality of `handle_response` is not overloaded, and usage is clearer. - **Caddyfile**: The `map` directive [now casts outputs](https://github.com/caddyserver/caddy/commit/93c99f67342504efe9f6b58a734aaec3929fe785) to the appropriate scalar type if possible (int, float, bool). If you need to force a string, you may use double quotes or backticks https://github.com/caddyserver/caddy/pull/4643. - **Caddyfile**: [New `vars` directive](https://github.com/caddyserver/caddy/commit/79cbe7bfd06565d0e7ab0717119f78960ed54c08) allows setting some variables during request handling for later use in another handler or matcher. - **Caddyfile**: The Caddyfile adapter [is now stricter](https://github.com/caddyserver/caddy/pull/4655) about curly braces for block openers to try to prevent parsing ambiguities. - **Caddyfile**: The `caddy fmt` CLI command now has a [`--diff` option](https://github.com/caddyserver/caddy/pull/4695) which lets you visually see the formatting differences. - :warning: **Admin:** [Renamed](https://github.com/caddyserver/caddy/commit/bc447e307f195b80eeec0f6157e0d8e641af9155) experimental property `load_interval` :arrow_right: `load_delay` for clarification, and improved dynamic config loading. --- :shield: Thanks to [David Leadbeater](https://github.com/dgl) for reporting a security vulnerability related to HTTP methods and metrics cardinality, which was fixed in this release. ## New Contributors * @adamburgess made their first contribution in https://github.com/caddyserver/caddy/pull/4460 * @12f23eddde made their first contribution in https://github.com/caddyserver/caddy/pull/4444 * @rayjlinden made their first contribution in https://github.com/caddyserver/caddy/pull/4023 * @GallopingKylin made their first contribution in https://github.com/caddyserver/caddy/pull/4522 * @ForestJohnson made their first contribution in https://github.com/caddyserver/caddy/pull/4534 * @VojtechVitek made their first contribution in https://github.com/caddyserver/caddy/pull/4535 * @Ikke made their first contribution in https://github.com/caddyserver/caddy/pull/4544 * @YourTechBud made their first contribution in https://github.com/caddyserver/caddy/pull/4603 * @BitWuehler made their first contribution in https://github.com/caddyserver/caddy/pull/4597 * @ttys3 made their first contribution in https://github.com/caddyserver/caddy/pull/4572 * @crccw made their first contribution in https://github.com/caddyserver/caddy/pull/4596 * @andriikushch made their first contribution in https://github.com/caddyserver/caddy/pull/4361 * @renbou made their first contribution in https://github.com/caddyserver/caddy/pull/4654 * @cuishuang made their first contribution in https://github.com/caddyserver/caddy/pull/4702 ## Changelog * 2e46c2ac admin, reverseproxy: Stop timers if canceled to avoid goroutine leak (#4482) * 40b54434 admin: Enforce and refactor origin checking * b4bfa29b admin: Require identity for remote (fix #4478) * 32aad909 admin: Write proper status on invalid requests (#4569) (fix #4561) * ff137d17 caddyconfig: Support placeholders in HTTP loader * b47af6ef caddyfile: Copy input before parsing (fix #4422) * e90d7517 caddyfile: impove fmt warning message (#4444) * 5e5af50e caddyfile: make renew_interval option configurable (#4451) * ddbb234d caddyhttp: Always log handled errors at debug level (#4584) * 6b385a36 caddyhttp: Don't attempt to manage Tailscale certs * ecac03cd caddyhttp: Enhance vars matcher (#4433) * 6e6ce2be caddyhttp: Fix HTTP->HTTPS redir not preferring HTTPS port if ambiguous (#4530) * 3fe2c73d caddyhttp: Fix `MatchPath` sanitizing (#4499) * 44e5e9e4 caddyhttp: Fix test when /tmp/etc already exists (#4544) * 2bb8550a caddyhttp: Honor wildcard hosts in log SkipHosts (#4606) * 180ae0cc caddyhttp: Implement http.request.uuid placeholder (#4285) * 7d5047c1 caddyhttp: Log empty value for typical password headers * eead3373 caddyhttp: Log non-500 handler errors at debug level (#4429) * 5bf0adad caddyhttp: Make logging of credential headers opt-in (#4438) * 186fdba9 caddyhttp: Move HTTP redirect listener to an optional module (#4585) * 80d7a356 caddyhttp: Redirect HTTP requests on the HTTPS port to https:// (#4313) * bf380d00 caddyhttp: Reject absurd methods (#4538) * 850e1605 caddyhttp: Return HTTP 421 for mismatched Host header (#4023) * f55b123d caddyhttp: Split up logged remote address into IP and port (#4403) * ac14b64e caddyhttp: Support zone identifiers in remote_ip matcher (#4597) * a1c41210 caddypki: Minor tweak, don't use context pointer * 78e381b2 caddypki: Refactor /pki/ admin endpoints * c634bbe9 caddypki: Return error if no PEM data found * 9b7cdfa2 caddypki: Try to fix lint warnings * a79b4055 caddytls: Add internal Caddyfile `lifetime`, `sign_with_root` opts (#4513) * 77a77c02 caddytls: Add `propagation_delay`, support `propagation_timeout -1` (#4723) * 66de438a caddytls: Fix `MatchRemoteIP` provisoning with multiple CIDR ranges (#4522) * 57a708d1 caddytls: Support external certificate Managers (like Tailscale) (#4541) * d9b1d463 caddytls: dns_challenge_override_domain for challenge delegation (#4596) * 1a7a78a1 cmd: Print error if fmt overwrite fails (fix #4524) * bc447e30 core: Config LoadInterval -> LoadDelay for clarity * 7ea5b2a8 core: Config load interval only reloads if changed (#4603) * 7f364c77 core: Load config at interval instead of just once * a72acd21 core: Retry dynamic config load if config unchanged * ceef70db core: Retry dynamic config load if error or no-op (#4603) * acbee947 core: Revert 7f364c7; simplify dynamic config load * 64a3218f core: Simplify shared listeners, fix deadline bug * 8e5aafa5 fastcgi: Fix a TODO, prevent zap using reflection for logging env (#4437) * c8f2834b fastcgi: Protect against requests with null bytes in the path (#4614) * de490c7c fastcgi: Set SERVER_PORT to 80 or 443 depending on scheme (#4572) * 09ba9e99 fileserver: Add `pass_thru` Caddyfile option (#4613) * 15c95e9d fileserver: Canonical redir when whole path is stripped (#4549) * c8b5a816 fileserver: Fix handling of symlink sizes in directory listings (#4415) * e81369e2 fileserver: Move default browse template into a separate file (#4417) * 1e10f6f7 fileserver: browse: do not encode the paths in breadcrumbs and page title (#4410) * 78b5356f fileserver: do not double-escape paths (#4447) * 0de51593 go.mod: Revert version bump of CEL (#4587) * 6f9b6ad7 go.mod: Update smallstep/certificates, no longer need replace (#4475) * 4906b935 go.mod: Update smallstep/truststore, fix build on FreeBSD (#4473) * c1331534 go.mod: Update to latest smallstep/truststore, support FreeBSD (#4453) * ff74a0aa go.mod: Upgrade dependencies * e9dde230 headers: Fix `+` in Caddyfile to properly append rather than set (#4506) * 1b7ff5d7 httpcaddyfile: Add `default_bind` global option (#4531) * a8bb4a66 httpcaddyfile: Add `{vars.*}` placeholder shortcut, reverse `vars` sort order (#4726) * 5a071568 httpcaddyfile: Add pki app `root` and `intermediate` cert/key config (#4514) * 3a1e0dbf httpcaddyfile: Deprecate paths in site addresses; use zap logs (#4728) * 26d633ba httpcaddyfile: Disabling OCSP stapling for both managed and unmanaged (#4589) * 93a7a45e httpcaddyfile: Fix incorrect handling of IPv6 bind addresses (#4532) * 81ee34e9 httpcaddyfile: Fix sorting edgecase for nested `handle_path` (#4477) * 4b9849c7 httpcaddyfile: Support configuring `pki` app names via global options (#4450) * 5bd96a6a httpcaddyfile: Support explicitly turning off `strict_sni_host` (#4592) * c921e082 logging: Add `roll_local_time` Caddyfile option (#4583) * 0eb0b60f logging: Remove common_log field and single_field encoder (#4149) (#4282) * 249adc1c logging: Support turning off roll compression via Caddyfile (#4505) * 8887adb0 logging: add a filter for cookies (#4425) * bcac2bee logging: add a filter for query parameters (#4424) * 789efa5d logging: add a regexp filter (#4426) * a1b417c8 logging: add support for hashing data (#4434) * eb891d46 metrics: Enforce smaller set of method labels * c04d24ca pki: Avoid provisioning the `local` CA when not necessary (#4463) * bbad6931 pki: Implement API endpoints for certs and `caddy trust` (#4443) * 9ee68c1b reverseproxy: Adjust defaults, document defaults (#4436) * 7557d1d9 reverseproxy: Avoid returning a `nil` error during GetClientCertificate (#4550) * ab045592 reverseproxy: Dynamic upstreams (with SRV and A/AAAA support) (#4470) * 5333c352 reverseproxy: Fix incorrect `health_headers` Caddyfile parsing (#4485) * c50094fc reverseproxy: Implement trusted proxies for `X-Forwarded-*` headers (#4507) * f5e10494 reverseproxy: Make shallow-ish clone of the request (#4551) * 87a1f228 reverseproxy: Move status replacement intercept to `replace_status` (#4300) * d058dee1 reverseproxy: Refactor dial address parsing, augment command parsing (#4616) * c7d6c4cb reverseproxy: copy_response and copy_response_headers for handle_response routes (#4391) * bcb7a19c rewrite: Add `method` Caddyfile directive (#4528) * 1feb6595 rewrite: Fix a double-encode issue when using the `{uri}` placeholder (#4516) * 6cadb60f templates: Document .OriginalReq * 1d0425b2 templates: Elaborate on what's supported by the markdown function (#4564) * a6199cf8 templates: Fix docs for .Args * ec14ccdd templates: fix inconsistent nested includes (#4452) * d0b608af tracing: New OpenTelemetry module (#4361) * 134b8056 caddyfile: Prevent bad block opening tokens (#4655) * c5fffb4a caddyfile: Support for raw token values; improve `map`, `expression` (#4643) * b82e22b4 caddyhttp: retain all values of vars matcher when specified multiple times (#4629) * bc15b4b0 caddypki: Load intermediate for signing on-the-fly (#4669) * 6512832f cmd: Add `--diff` option for `caddy fmt` (#4695) * 30b6d1f4 cmd: Enhance .env (dotenv) file parsing * 22d8edb9 cmd: Fix defaulting admin address if empty in config, fixes `reload` (#4674) * c2327161 cmd: Set Origin header properly on API requests * d06d0e79 go.mod: Upgrade CertMagic to v0.16.0 * bf6a1b75 go.mod: Upgrade some dependencies * 79cbe7bf httpcaddyfile: Add 'vars' directive * a58f240d httpcaddyfile: Fix #4640 (auto-HTTPS edgecase) (#4661) * 7d229665 logging: Caddyfile support for `duration_format` (#4684) * 55b4c12e map: Evaluate placeholders in output vals (#4650) * 93c99f67 map: Support numeric and bool types with Caddyfile * 3d616e8c requestbody: Return HTTP 413 (fix #4558) * 3e3bb002 reverseproxy: Add `_ms` placeholders for proxy durations (#4666) * 2196c92c reverseproxy: Don't clear name in SRV upstreams * dc4d1473 reverseproxy: Expand SRV/A addrs for cache key * b8dbecb8 reverseproxy: Include port in A upstreams cache * e4ce40f8 reverseproxy: Sync up `handleUpgradeResponse` with stdlib (#4664) * afca2421 staticfiles: Expand placeholder for index files (#4679) * 00234c8a templates: Switch to `BurntSushi/toml` (#4700) **Full Changelog**: https://github.com/caddyserver/caddy/compare/v2.4.6...v2.5.0 2022-04-25T17:34:35+00:00 PeekabooAV v2.1 PeekabooAV v2.1 2022-04-26T08:20:37+00:00 - Peekaboo now provides a REST API. The old UNIX domain socket is gone and there's no longer a long-lived client connection providing a summary report on multiple samples. Samples are now submitted individually, yielding a job ID for subsequent attempts at retrieving a report. Both inputs and outputs of the API are JSON. The AMaViS plugin and peekaboo-util are updated to match. - Embedded Cuckoo mode and python2 support are removed. - Breaking change: Equality operators in expressions using regexes do now need to match the whole string up to the end. - New database schema version 9. Removes tables PeekabooMetadata and AnalysisJournal, and adds field analysis\_time as well as state to SampleInfo. - Generic rules can now make use of the new analyser `knownreport` - Introduce cortexreport toolbox analyser to connect to Cortex by TheHive. There already are a few sub analysers that can be used. - Reduce amount of data copied from Cuckoo reports for memory efficiency and security reasons. Reduces the amount of information available in Peekaboo processing failure dumps as well. URL to access original report via Cuckoo API is provided instead. - The CortexAnalyser or more precisely every CortexAnalyser can now access domain, hash, and ip artifacts from within the Generic rules. - FileInfoAnalyzerReport has new attibutes md5sum, sha256sum, and ssdeepsum (now don't get to excited, ssdeep hashes can only be used as strings) - Input validation of reports adds a new pip requirement: schema - Availability of external resources, particularly Cuckoo and Cortex APIs is no longer checked at startup. Lack of availability is reported as individual job failure. - PID file is no longer created by default (but can be re-enabled by specifying a path). 2022-04-26T08:20:37+00:00 syncthing v1.20.0-rc.3 syncthing v1.20.0-rc.3 2022-04-26T11:55:37+00:00 Bugfixes: - #7855: Watching for changes using too many system resources on BSDs (kqueue) - #8020: Dangling symlink prevents filesystem watcher on Solaris - #8232: Missing include file endlessly spams log - #8258: The latest version does not support go1.18 compilation!! - #8261: TypeError: Cannot read properties of null (reading 'status') in the GUI after killing Syncthing Enhancements: - #8202: Indicate folders / devices where the remote end has not accepted sharing. - #8282: Change default log format to include date Other issues: - #8279: Simplify tests using `T.TempDir` 2022-04-26T11:55:37+00:00 dalton v3.2.2 dalton v3.2.2 2022-04-28T14:34:15+00:00 - Added explicit requirement for `Jinja2==3.0.3` 2022-04-28T14:34:15+00:00 maltrail 0.45 maltrail 0.45 2022-04-30T22:11:05+00:00 Start-of-month release 2022-04-30T22:11:05+00:00 syncthing v1.20.0 syncthing v1.20.0 2022-05-03T15:24:19+00:00 Bugfixes: - #7855: Watching for changes using too many system resources on BSDs (kqueue) - #8020: Dangling symlink prevents filesystem watcher on Solaris - #8232: Missing include file endlessly spams log - #8258: The latest version does not support go1.18 compilation!! - #8261: TypeError: Cannot read properties of null (reading 'status') in the GUI after killing Syncthing Enhancements: - #8202: Indicate folders / devices where the remote end has not accepted sharing. - #8282: Change default log format to include date Other issues: - #8279: Simplify tests using `T.TempDir` 2022-05-03T15:24:19+00:00 syncthing v1.20.1 syncthing v1.20.1 2022-05-04T16:32:31+00:00 Bugfixes: - #8320: Spurious failure of new connections ("unknown (newer?) version of the protocol" etc) 2022-05-04T16:32:31+00:00 rocksdb v7.2.2 rocksdb v7.2.2 2022-05-05T22:33:28+00:00 ## 7.2.2 (2022-04-28) ### Bug Fixes * Fixed a bug in async_io path where incorrect length of data is read by FilePrefetchBuffer if data is consumed from two populated buffers and request for more data is sent. ## 7.2.1 (2022-04-26) ### Bug Fixes * Fixed a bug where RocksDB could corrupt DBs with `avoid_flush_during_recovery == true` by removing valid WALs, leading to `Status::Corruption` with message like "SST file is ahead of WALs" when attempting to reopen. * RocksDB calls FileSystem::Poll API during FilePrefetchBuffer destruction which impacts performance as it waits for read requets completion which is not needed anymore. Calling FileSystem::AbortIO to abort those requests instead fixes that performance issue. ## 7.2.0 (2022-04-15) ### Bug Fixes * Fixed bug which caused rocksdb failure in the situation when rocksdb was accessible using UNC path * Fixed a race condition when 2PC is disabled and WAL tracking in the MANIFEST is enabled. The race condition is between two background flush threads trying to install flush results, causing a WAL deletion not tracked in the MANIFEST. A future DB open may fail. * Fixed a heap use-after-free race with DropColumnFamily. * Fixed a bug that `rocksdb.read.block.compaction.micros` cannot track compaction stats (#9722). * Fixed `file_type`, `relative_filename` and `directory` fields returned by `GetLiveFilesMetaData()`, which were added in inheriting from `FileStorageInfo`. * Fixed a bug affecting `track_and_verify_wals_in_manifest`. Without the fix, application may see "open error: Corruption: Missing WAL with log number" while trying to open the db. The corruption is a false alarm but prevents DB open (#9766). * Fix segfault in FilePrefetchBuffer with async_io as it doesn't wait for pending jobs to complete on destruction. * Fix ERROR_HANDLER_AUTORESUME_RETRY_COUNT stat whose value was set wrong in portal.h * Fixed a bug for non-TransactionDB with avoid_flush_during_recovery = true and TransactionDB where in case of crash, min_log_number_to_keep may not change on recovery and persisting a new MANIFEST with advanced log_numbers for some column families, results in "column family inconsistency" error on second recovery. As a solution the corrupted WALs whose numbers are larger than the corrupted wal and smaller than the new WAL will be moved to archive folder. * Fixed a bug in RocksDB DB::Open() which may creates and writes to two new MANIFEST files even before recovery succeeds. Now writes to MANIFEST are persisted only after recovery is successful. ### New Features * For db_bench when --seed=0 or --seed is not set then it uses the current time as the seed value. Previously it used the value 1000. * For db_bench when --benchmark lists multiple tests and each test uses a seed for a RNG then the seeds across tests will no longer be repeated. * Added an option to dynamically charge an updating estimated memory usage of block-based table reader to block cache if block cache available. To enable this feature, set `BlockBasedTableOptions::reserve_table_reader_memory = true`. * Add new stat ASYNC_READ_BYTES that calculates number of bytes read during async read call and users can check if async code path is being called by RocksDB internal automatic prefetching for sequential reads. * Enable async prefetching if ReadOptions.readahead_size is set along with ReadOptions.async_io in FilePrefetchBuffer. * Add event listener support on remote compaction compactor side. * Added a dedicated integer DB property `rocksdb.live-blob-file-garbage-size` that exposes the total amount of garbage in the blob files in the current version. * RocksDB does internal auto prefetching if it notices sequential reads. It starts with readahead size `initial_auto_readahead_size` which now can be configured through BlockBasedTableOptions. * Add a merge operator that allows users to register specific aggregation function so that they can does aggregation using different aggregation types for different keys. See comments in include/rocksdb/utilities/agg_merge.h for actual usage. The feature is experimental and the format is subject to change and we won't provide a migration tool. * Meta-internal / Experimental: Improve CPU performance by replacing many uses of std::unordered_map with folly::F14FastMap when RocksDB is compiled together with Folly. * Experimental: Add CompressedSecondaryCache, a concrete implementation of rocksdb::SecondaryCache, that integrates with compression libraries (e.g. LZ4) to hold compressed blocks. ### Behavior changes * Disallow usage of commit-time-write-batch for write-prepared/write-unprepared transactions if TransactionOptions::use_only_the_last_commit_time_batch_for_recovery is false to prevent two (or more) uncommitted versions of the same key in the database. Otherwise, bottommost compaction may violate the internal key uniqueness invariant of SSTs if the sequence numbers of both internal keys are zeroed out (#9794). * Make DB::GetUpdatesSince() return NotSupported early for write-prepared/write-unprepared transactions, as the API contract indicates. ### Public API changes * Exposed APIs to examine results of block cache stats collections in a structured way. In particular, users of `GetMapProperty()` with property `kBlockCacheEntryStats` can now use the functions in `BlockCacheEntryStatsMapKeys` to find stats in the map. * Add `fail_if_not_bottommost_level` to IngestExternalFileOptions so that ingestion will fail if the file(s) cannot be ingested to the bottommost level. * Add output parameter `is_in_sec_cache` to `SecondaryCache::Lookup()`. It is to indicate whether the handle is possibly erased from the secondary cache after the Lookup. 2022-05-05T22:33:28+00:00 caddy v2.5.1 caddy v2.5.1 2022-05-06T17:23:16+00:00 This is a minor patch release that fixes some bugs and also enhances `reverse_proxy` with capabilities that weren't ready in time for v2.5.0. ### Highlights - Fixed regression in Unix socket admin endpoints. - Fixed regression in `caddy trust` commands. - Hash-based load balancing policies (ip_hash, uri_hash, header, and cookie) use an improved highest-random-weight (HRW) algorithm for increased consistency. The new rendezvous hash will ensure a client or request is _consistently_ mapped to a particular upstream even if the list of upstreams changes. - The reverse proxy is now able to rewrite the method and URI on its internal copy of the request that goes to the upstream. Combined with new `handle_response` capabilities, this enables the reverse proxy to fire off ["pre-check requests"](https://github.com/caddyserver/caddy/pull/4739) (for lack of a better term) to make routing decisions based on the results of that call. This enables a commonly-emerging pattern called _forward authentication_ wherein a backend is queried to assess a client's authorization to be proxied. The [full, verbose config for this](https://github.com/caddyserver/caddy/pull/4739#issuecomment-1113901951) is very flexible but tedious, so we made a new wrapper directive called `forward_auth` that eliminates the boilerplate (very similar to the [`php_fastcgi` directive](https://caddyserver.com/docs/caddyfile/directives/php_fastcgi#expanded-form)): ``` forward_auth authelia:9091 { uri /api/verify?rd=https://auth.example.com copy_headers Remote-User Remote-Groups Remote-Name Remote-Email } ``` This works with authentication providers like Authelia, and more. ## What's Changed * caddypki: Fix `caddy trust` command to use the correct API endpoint by @francislavoie in https://github.com/caddyserver/caddy/pull/4730 * reverseproxy: Improve hashing LB policies with HRW by @mholt in https://github.com/caddyserver/caddy/pull/4724 * Add missing backticks by @mahgoh in https://github.com/caddyserver/caddy/pull/4737 * caddyhttp: Improve listen addr error message for IPv6 by @francislavoie in https://github.com/caddyserver/caddy/pull/4740 * cmd: Fix unix socket addresses for admin API requests by @francislavoie in https://github.com/caddyserver/caddy/pull/4742 * logging: Use `RedirectStdLog` by @francislavoie in https://github.com/caddyserver/caddy/pull/4732 * logging: Implement rename filter, changes field key names by @francislavoie in https://github.com/caddyserver/caddy/pull/4745 * httpcaddyfile: Fix duplicate access log when debug is on by @francislavoie in https://github.com/caddyserver/caddy/pull/4746 * reverseproxy: Fix Caddyfile support for `replace_status` by @francislavoie in https://github.com/caddyserver/caddy/pull/4754 * templates: Add custom template function registration by @kroppt in https://github.com/caddyserver/caddy/pull/4757 * reverseproxy: Permit resolver addresses to not specify a port by @francislavoie in https://github.com/caddyserver/caddy/pull/4760 * caddyfile: Shortcut for `remote_ip` for private IP ranges by @francislavoie in https://github.com/caddyserver/caddy/pull/4753 * reverseproxy: Support performing pre-check requests by @francislavoie in https://github.com/caddyserver/caddy/pull/4739 * map: Prevent output destinations overlap with Caddyfile shorthands by @francislavoie in https://github.com/caddyserver/caddy/pull/4657 ## New Contributors * @mahgoh made their first contribution in https://github.com/caddyserver/caddy/pull/4737 ## Changelog * ec86a2f7 caddyfile: Shortcut for `remote_ip` for private IP ranges (#4753) * dcc98da4 caddyhttp: Improve listen addr error message for IPv6 (#4740) * d543ad1f caddypki: Fix `caddy trust` command to use the correct API endpoint (#4730) * 2e4c0915 cmd: Fix unix socket addresses for admin API requests (#4742) * af732151 httpcaddyfile: Fix duplicate access log when debug is on (#4746) * 0be3d995 logging: Implement rename filter, changes field key names (#4745) * 3017b245 logging: Use `RedirectStdLog` to capture more stdlib logs (#4732) * f7be0ee1 map: Prevent output destinations overlap with Caddyfile shorthands (#4657) * 4a223f52 reverseproxy: Fix Caddyfile support for `replace_status` (#4754) * 40b193fb reverseproxy: Improve hashing LB policies with HRW (#4724) * e7fbee8c reverseproxy: Permit resolver addresses to not specify a port (#4760) * f6900fcf reverseproxy: Support performing pre-check requests (#4739) * e84e19a0 templates: Add custom template function registration (#4757) * 3ab64838 templates: Add missing backticks in docs (#4737) **Full Changelog**: https://github.com/caddyserver/caddy/compare/v2.5.0...v2.5.1 2022-05-06T17:23:16+00:00 syncthing v1.20.2-rc.1 syncthing v1.20.2-rc.1 2022-05-10T06:35:01+00:00 Bugfixes: - #7289: TCP port 0 is announced in the LAN beacon - #8314: https://relays.syncthing.net/endpoint contains non-valid URLs with spaces Enhancements: - #8264: Include default ignore patterns in the GUI's advanced configuration editor 2022-05-10T06:35:01+00:00 TheHive 4.1.20 TheHive 4.1.20 2022-05-16T12:06:25+00:00 ## [4.1.20](https://github.com/TheHive-Project/TheHive/milestone/90) (2022-05-16) **Implemented enhancements:** - [Enhancement] Improve artifact creation from responder operation [\#2383](https://github.com/TheHive-Project/TheHive/issues/2383) - [Enhancement] Accept operations in analyzer reports [\#2384](https://github.com/TheHive-Project/TheHive/issues/2384) - [Enhancement] Add queries to filter observables based on the type of object it belongs to [\#2385](https://github.com/TheHive-Project/TheHive/issues/2385) **Fixed bugs:** - [Bug] Unable to use AWS S3 as storage backend [\#2316](https://github.com/TheHive-Project/TheHive/issues/2316) - [Bug] Typo on migration elasticsearch http config [\#2374](https://github.com/TheHive-Project/TheHive/issues/2374) 2022-05-16T12:06:25+00:00 lynis 3.0.8 lynis 3.0.8 2022-05-17T13:10:32+00:00 ### Added - MALW-3274 - Detect McAfee VirusScan Command Line Scanner - PKGS-7346 Check Alpine Package Keeper (apk) - PKGS-7395 Check Alpine upgradeable packages - EOL for Alpine Linux 3.14 and 3.15 ### Changed - AUTH-9408 - Check for pam_faillock as well (replacement for pam_tally2) - FILE-7524 - Test enhanced to support symlinks - HTTP-6643 - Support ModSecurity version 2 and 3 - KRNL-5788 - Only run relevant tests and improved logging - KRNL-5820 - Additional path for security/limits.conf - KRNL-5830 - Check for /var/run/needs_restarting (Slackware) - KRNL-5830 - Add a presence check for /boot/vmlinuz - PRNT-2308 - Bugfix that prevented test from storing values correctly - Extended location of PAM files for AARCH64 - Some messages in log improved 2022-05-17T13:10:32+00:00 OpenTAXII 0.4.0 OpenTAXII 0.4.0 2022-05-20T11:55:43+00:00 Changelog ========= 0.4.0 (2022-05-20) ------------------ * Move next_param handling into `OpenTAXII2PersistenceAPI` 0.3.0 (2022-04-13) ------------------ * Implement taxii2.1 support 0.3.0a4 (2022-04-13) -------------------- * Merge changes from 0.2.4 maintenance release 0.3.0a3 (2022-01-21) -------------------- * Fix bug that prevented booting with only taxii1 config (`#217 <https://github.com/eclecticiq/OpenTAXII/issues/217>`_ thanks `@azurekid <https://github.com/azurekid>`_ for the report) 0.3.0a2 (2021-12-27) -------------------- * Merge changes from 0.2.3 maintenance release 0.3.0a1 ------- * Add python 3.10 support 0.3.0a0 ------- * Enablement for future taxii2 implementation * Fix documentation build issues 0.2.4 (2022-04-13) ------------------ * Make sure werkzeug <2.1 and >=2.1 work correctly with auth system 0.2.3 (2021-12-22) ------------------ * Fix bug in multithreaded use of sqlite (`#210 <https://github.com/eclecticiq/OpenTAXII/issues/210>`_ thanks `@rohits144 <https://github.com/rohits144>`_ for the report) 0.2.2 (2021-11-05) ------------------ * Fix readthedocs build 0.2.1 (2021-11-03) ------------------ * Add tests for python 3.6, 3.7, 3.8, 3.9, pypy * Add tests for sqlite, mysql, mariadb, postgresql * Fix bug that broke ``delete_content_blocks`` when using mysql on sqlalchemy 1.3 * Docs: Add db schema diagram * Docs: Clarify how to get default data in a default (development) docker instance * Fix implicit routing in TAXII 1.1 Inboxes * Update jwt usage to pyjwt >= 2.0 (thanks `@SanyaKapoor <https://github.com/SanyaKapoor>`_) 0.2.0 (2020-06-30) ------------------ * Enforce UTC usage in datetime fields in SQL DB Persistence API. * `Fix for #114 <https://github.com/eclecticiq/OpenTAXII/issues/114>`_: reintroduce ``opentaxii-create-account`` CLI command. * `Fix for #153 <https://github.com/eclecticiq/OpenTAXII/issues/152>`_: check if user can modify a collection before advertising it over inbox service. * Multiple coding style fixes. * Various documentation updates. 0.1.12 (2019-03-06) ------------------- * Remove unnecessary print statements. 0.1.11 (2019-02-13) ------------------- * Make JSON logging consistent when the application is run via Gunicorn. * Set ``acceptable_destination`` key in status details instead of extended headers * Allow passing ``engine_parameters`` to ``SQLDatabaseAPI`` for those who want to customize SQLAlchemy engine parameters. * Require recent version of ``lxml`` for security reasons. * Various test and Docker infrastructure improvements. 0.1.10 (2018-06-03) ------------------- * Replace separate service/collection/account creation process with single ``opentaxii-sync-data`` CLI command. * Persistence and Auth APIs extended with missing CRUD methods, that are used by ``opentaxii-sync-data``. * Read/modify collection level ACL added. * DB models for default implementation of Persistence API and Auth API were changed. No automatic migration code is provided (sorry!), so upgrading might require manual DB migration. * Drop python2.7 from testing scope. * Various bug fixes and improvements. 0.1.9 (2017-06-19) ------------------ * `libtaxii <https://github.com/TAXIIProject/libtaxii>`_ dependency upgraded to 1.1.111. * Various bug fixes and improvements (thanks to `@bjigmp <https://github.com/bjigmp>`_, `@chorsley <https://github.com/chorsley>`_, `@rjprins <https://github.com/rjprins>`_). 0.1.8 (2017-02-21) ------------------ * Ability to enable/disable "huge trees" support in XML parser. Configuration property ``xml_parser_supports_huge_tree`` set to ``yes`` or ``true`` will disable security restrictions and force XML parser to support very deep trees and very long text content. * Adjust SQL Persistence API implemetation so it works smoothly with MySQL backend. * Use Python 3.5 instead of Python 3.4 for tests. 0.1.7 (2016-10-18) ------------------ * Minor fixes. * Dependencies were changed from hard-pinned to more flexible. * Example of production DB configuration added to docs. 0.1.6 (2016-06-01) ------------------ * Python 3.4 compatibility of the codebase. Tox configuration extended with python 3.4 environment run. * Flake8 full style compatibility. Flake8 check added to Tox configuration. * SQLAlchemy session scope issue fixed (related to `#38 <https://github.com/EclecticIQ/OpenTAXII/issues/38>`_). * `opentaxii-delete-blocks` CLI command added (related to `#45 <https://github.com/EclecticIQ/OpenTAXII/issues/45>`_). * `delete_content_blocks` method `added <https://github.com/EclecticIQ/OpenTAXII/commit/dc6fddc27a98e8450c7e05e583b2bfb741f6e197#diff-6814849ac352b2b74132f8fa52e0ec4eR213>`_ to Persistence API. * Collection's name is `required <https://github.com/EclecticIQ/OpenTAXII/commit/dc6fddc27a98e8450c7e05e583b2bfb741f6e197#diff-ce3f7b939e5c540480ac655aef32c513R116>`_ to be unique in default SQL DB Auth API implementation. 0.1.5 (2016-03-15) ------------------ * Fix for the issue with persistence backend returning ``None`` instead of ``InboxMessage`` object 0.1.4 (2016-02-25) ------------------ * Hard-coded dependencies in ``setup.py`` removed. 0.1.3 (2016-02-25) ------------------ * Versions of dependencies are pinned. * Code adjusted for a new version of `anyconfig <https://pypi.python.org/pypi/anyconfig>`_ API. * Test for configuration loading added. 0.1.2 (2015-07-24) ------------------ * Docker configuration added. * Health check endpoint added. * Basic authentication support added. * Temporary workaround for `Issue #191 <https://github.com/TAXIIProject/libtaxii/issues/191>`_. * Method ``get_domain`` in Persistence API returns domain value configured for ``service_id``. If nothing returned, value set in configuration file will be used. * Performance optimisations. * Bug fixes and style improvements. 0.1.1 (2015-04-08) ------------------ * Alias for Root Logger added to logging configuration. * Context object in a request scope that holds account and token added. * Support for OPTIONS HTTP request to enable auto version negotiation added. * Documentation improved. 0.1.0 (2015-03-31) ------------------ * Initial release 2022-05-20T11:55:43+00:00 syncthing v1.20.2-rc.2 syncthing v1.20.2-rc.2 2022-05-24T08:16:12+00:00 Bugfixes: - #7289: TCP port 0 is announced in the LAN beacon - #8314: https://relays.syncthing.net/endpoint contains non-valid URLs with spaces - #8355: Upgrading from v1.19.2 to v1.20.x now requires chmod for syncing files Enhancements: - #8264: Include default ignore patterns in the GUI's advanced configuration editor - #8310: Indicate folders / devices where the remote end is paused. 2022-05-24T08:16:12+00:00 OpenTAXII 0.5.0 OpenTAXII 0.5.0 2022-05-24T11:54:11+00:00 Changelog ========= 0.5.0 (2022-05-24) ------------------ * Add support for publicly readable taxii 2 collections 0.4.0 (2022-05-20) ------------------ * Move next_param handling into `OpenTAXII2PersistenceAPI` 0.3.0 (2022-04-13) ------------------ * Implement taxii2.1 support 0.3.0a4 (2022-04-13) -------------------- * Merge changes from 0.2.4 maintenance release 0.3.0a3 (2022-01-21) -------------------- * Fix bug that prevented booting with only taxii1 config (`#217 <https://github.com/eclecticiq/OpenTAXII/issues/217>`_ thanks `@azurekid <https://github.com/azurekid>`_ for the report) 0.3.0a2 (2021-12-27) -------------------- * Merge changes from 0.2.3 maintenance release 0.3.0a1 ------- * Add python 3.10 support 0.3.0a0 ------- * Enablement for future taxii2 implementation * Fix documentation build issues 0.2.4 (2022-04-13) ------------------ * Make sure werkzeug <2.1 and >=2.1 work correctly with auth system 0.2.3 (2021-12-22) ------------------ * Fix bug in multithreaded use of sqlite (`#210 <https://github.com/eclecticiq/OpenTAXII/issues/210>`_ thanks `@rohits144 <https://github.com/rohits144>`_ for the report) 0.2.2 (2021-11-05) ------------------ * Fix readthedocs build 0.2.1 (2021-11-03) ------------------ * Add tests for python 3.6, 3.7, 3.8, 3.9, pypy * Add tests for sqlite, mysql, mariadb, postgresql * Fix bug that broke ``delete_content_blocks`` when using mysql on sqlalchemy 1.3 * Docs: Add db schema diagram * Docs: Clarify how to get default data in a default (development) docker instance * Fix implicit routing in TAXII 1.1 Inboxes * Update jwt usage to pyjwt >= 2.0 (thanks `@SanyaKapoor <https://github.com/SanyaKapoor>`_) 0.2.0 (2020-06-30) ------------------ * Enforce UTC usage in datetime fields in SQL DB Persistence API. * `Fix for #114 <https://github.com/eclecticiq/OpenTAXII/issues/114>`_: reintroduce ``opentaxii-create-account`` CLI command. * `Fix for #153 <https://github.com/eclecticiq/OpenTAXII/issues/152>`_: check if user can modify a collection before advertising it over inbox service. * Multiple coding style fixes. * Various documentation updates. 0.1.12 (2019-03-06) ------------------- * Remove unnecessary print statements. 0.1.11 (2019-02-13) ------------------- * Make JSON logging consistent when the application is run via Gunicorn. * Set ``acceptable_destination`` key in status details instead of extended headers * Allow passing ``engine_parameters`` to ``SQLDatabaseAPI`` for those who want to customize SQLAlchemy engine parameters. * Require recent version of ``lxml`` for security reasons. * Various test and Docker infrastructure improvements. 0.1.10 (2018-06-03) ------------------- * Replace separate service/collection/account creation process with single ``opentaxii-sync-data`` CLI command. * Persistence and Auth APIs extended with missing CRUD methods, that are used by ``opentaxii-sync-data``. * Read/modify collection level ACL added. * DB models for default implementation of Persistence API and Auth API were changed. No automatic migration code is provided (sorry!), so upgrading might require manual DB migration. * Drop python2.7 from testing scope. * Various bug fixes and improvements. 0.1.9 (2017-06-19) ------------------ * `libtaxii <https://github.com/TAXIIProject/libtaxii>`_ dependency upgraded to 1.1.111. * Various bug fixes and improvements (thanks to `@bjigmp <https://github.com/bjigmp>`_, `@chorsley <https://github.com/chorsley>`_, `@rjprins <https://github.com/rjprins>`_). 0.1.8 (2017-02-21) ------------------ * Ability to enable/disable "huge trees" support in XML parser. Configuration property ``xml_parser_supports_huge_tree`` set to ``yes`` or ``true`` will disable security restrictions and force XML parser to support very deep trees and very long text content. * Adjust SQL Persistence API implemetation so it works smoothly with MySQL backend. * Use Python 3.5 instead of Python 3.4 for tests. 0.1.7 (2016-10-18) ------------------ * Minor fixes. * Dependencies were changed from hard-pinned to more flexible. * Example of production DB configuration added to docs. 0.1.6 (2016-06-01) ------------------ * Python 3.4 compatibility of the codebase. Tox configuration extended with python 3.4 environment run. * Flake8 full style compatibility. Flake8 check added to Tox configuration. * SQLAlchemy session scope issue fixed (related to `#38 <https://github.com/EclecticIQ/OpenTAXII/issues/38>`_). * `opentaxii-delete-blocks` CLI command added (related to `#45 <https://github.com/EclecticIQ/OpenTAXII/issues/45>`_). * `delete_content_blocks` method `added <https://github.com/EclecticIQ/OpenTAXII/commit/dc6fddc27a98e8450c7e05e583b2bfb741f6e197#diff-6814849ac352b2b74132f8fa52e0ec4eR213>`_ to Persistence API. * Collection's name is `required <https://github.com/EclecticIQ/OpenTAXII/commit/dc6fddc27a98e8450c7e05e583b2bfb741f6e197#diff-ce3f7b939e5c540480ac655aef32c513R116>`_ to be unique in default SQL DB Auth API implementation. 0.1.5 (2016-03-15) ------------------ * Fix for the issue with persistence backend returning ``None`` instead of ``InboxMessage`` object 0.1.4 (2016-02-25) ------------------ * Hard-coded dependencies in ``setup.py`` removed. 0.1.3 (2016-02-25) ------------------ * Versions of dependencies are pinned. * Code adjusted for a new version of `anyconfig <https://pypi.python.org/pypi/anyconfig>`_ API. * Test for configuration loading added. 0.1.2 (2015-07-24) ------------------ * Docker configuration added. * Health check endpoint added. * Basic authentication support added. * Temporary workaround for `Issue #191 <https://github.com/TAXIIProject/libtaxii/issues/191>`_. * Method ``get_domain`` in Persistence API returns domain value configured for ``service_id``. If nothing returned, value set in configuration file will be used. * Performance optimisations. * Bug fixes and style improvements. 0.1.1 (2015-04-08) ------------------ * Alias for Root Logger added to logging configuration. * Context object in a request scope that holds account and token added. * Support for OPTIONS HTTP request to enable auto version negotiation added. * Documentation improved. 0.1.0 (2015-03-31) ------------------ * Initial release 2022-05-24T11:54:11+00:00 Lookyloo v1.12.0 Lookyloo v1.12.0 2022-05-24T13:33:18+00:00 # New Features ## Playwright The captures are now made via [Playwright](https://playwright.dev/) instead of [Splash](https://github.com/scrapinghub/splash). It is a major improvement as Playwright uses actual up-to-date browsers, in headless mode (instead of qt-webkit from ~2016). You can read more about the research that lead to this change [in the discussion](https://github.com/Lookyloo/lookyloo/discussions/243). The main other advantages of using playwright are the following: * Easier to install: it doesn't requires Docker in order to use Splash * Much better control of what happen in the browser while capturing: Playwright makes it extremely simple to instrument everything in the browsers. The capturing module already tries to solve reCaptcha if it detects it on the page. The capture is made by a [standalone](https://github.com/Lookyloo/PlaywrightCapture) python module that you can use in your own tools if you wish to. ## De-duplication If the exact same capture is triggered multiple times within 5 min, it is skipped and the requestor is redirected to the capture done before. # Fixes * Avoid discarding a capture on network error: when a redirect is broken down the line, we keep the chain up to that point * Issue when the MISP was submitted as un-published * [Docker] Properly handle archiving * [Docker] Init SRI hashes # Changes * Improve subsequent capture template on long URLs * Improve view of the capture page on small-ish screens * General maintenance and code cleanup * Improvement in the tree generation on edge cases * Bump JS/CSS libraries * Update bundled-in User-Agent file * Use pydeep2, comes with a bundled-in libfuzzy, easier to install. 2022-05-24T13:33:18+00:00 osquery 5.3.0 osquery 5.3.0 2022-05-24T20:33:25+00:00 <a name="5.3.0"></a> ## [5.3.0](https://github.com/osquery/osquery/releases/tag/5.3.0) [Git Commits](https://github.com/osquery/osquery/compare/5.2.3...5.3.0) osquery 5.3.0 brings several table improvements and bugfixes. Worth mentioning also the deprecation of the `smart_drive_info` table and the new warning added when incorrectly configuring a CLI only flag via the config file. In the next release CLI only flags will not be configurable through the config file or refresh anymore. This release represents commits from 15 contributors! Thank you all. ### Deprecation Notices - Deprecate unmaintainable legacy table, `smart_drive_info` [#7464](https://github.com/osquery/osquery/issues/7464) ### New Features - Add the option `tls_disable_status_log` to prevent status logs from being sent via TLS [#7550](https://github.com/osquery/osquery/pull/7550) - Add SQLite function `in_cidr_block` to check if IPv4/v6 addresses are within the supplied CIDR block [#7563](https://github.com/osquery/osquery/pull/7563) ### Table Changes - Add the `admindir` column to the `deb_packages` table to parse package databases on different paths [#7549](https://github.com/osquery/osquery/pull/7549) - Implement and fix `wifi_networks` on macOS Big Sur and newer [#7503](https://github.com/osquery/osquery/pull/7503) - Add windows/darwin support to `npm_packages` [#7536](https://github.com/osquery/osquery/pull/7536) - Move `apt_sources` and `yum_sources` tables to linux only [#7537](https://github.com/osquery/osquery/pull/7537) - Add homebrew paths to the `python_packages` table [#7535](https://github.com/osquery/osquery/pull/7535) - Mark `wall_time` column in `osquery_schedule` as hidden [#7501](https://github.com/osquery/osquery/pull/7501) - Add new metrics and improve description of existing ones in `osquery_schedule` [#7438](https://github.com/osquery/osquery/pull/7438) - Add the `mirrorlist` column in the table `yum_sources` [#7479](https://github.com/osquery/osquery/pull/7479) - Implement `output_size` for `osquery_schedule` [#7436](https://github.com/osquery/osquery/pull/7436) - `deb_packages` table: Use additional instead of index for the `admindir` column [#7573](https://github.com/osquery/osquery/pull/7573) - `certificates` table: Add Linux support [#7570](https://github.com/osquery/osquery/pull/7570) - Add `translated` column to `processes` table to indicate whether the process is running under Apple Rosetta [#7507](https://github.com/osquery/osquery/pull/7507) - Add the "internet password" type to the macOS `keychain_items` table [#7576](https://github.com/osquery/osquery/pull/7576) - Add `original filename` column to `file` table on Windows [#7156](https://github.com/osquery/osquery/pull/7156) ### Bug Fixes - Fix watchdog not killing unhealthy worker/extension fast enough [#7474](https://github.com/osquery/osquery/pull/7474) - Fix the `test_http_server.py` `--persist` option [#7497](https://github.com/osquery/osquery/pull/7497) - Update`profile.py --leaks` for python3 [#7534](https://github.com/osquery/osquery/pull/7534) - Fixes osquery tls connections to aws kinesis when tls_server_certs is set [#7450](https://github.com/osquery/osquery/pull/7450) - Fix parsing issue when a backslash as the last character on sudoers file line [#7440](https://github.com/osquery/osquery/pull/7440) - Change the JSON of the results coming from an event scheduled query to an array [#7434](https://github.com/osquery/osquery/pull/7434) - Fix globToRegex truncating UTF16 characters [#7430](https://github.com/osquery/osquery/pull/7430) - Prevent hanging when the WMI server does not respond [#7429](https://github.com/osquery/osquery/pull/7429) - Fix `python_packages` table so that it lists python packages from any user Python installations [#7414](https://github.com/osquery/osquery/pull/7414) - Set string size limit on thrift protocol factory to prevent a crash [#7484](https://github.com/osquery/osquery/pull/7484) - Fix driver image path in `drivers` table [#7444](https://github.com/osquery/osquery/pull/7444) - Do not remove nonblocking flag when reading "special" files, to prevent hangs [#7530](https://github.com/osquery/osquery/pull/7530) - Fix crash due to interaction between distributed and config plugin [#7504](https://github.com/osquery/osquery/pull/7504) - bpf: Disable the BPF publisher in case of error [#7500](https://github.com/osquery/osquery/pull/7500) - Warn about setting CLI_FLAGs in the config [#7583](https://github.com/osquery/osquery/pull/7583) - Explicitly set context for the tables reading utmpx databases [#7578](https://github.com/osquery/osquery/pull/7578) - bpf: Improve socket event handling [#7446](https://github.com/osquery/osquery/pull/7446) - certificates: Refactor the OpenSSL utilities [#7581](https://github.com/osquery/osquery/pull/7581) - Fix shared_resources accessing uninitialized variables [#7600](https://github.com/osquery/osquery/pull/7600) ### Under the Hood improvements - Implement a performant cache for users and groups on Windows [#7516](https://github.com/osquery/osquery/pull/7516) - Replace WmiRequest constructor with static factory method to improve error handling and prevent crashes [#7489](https://github.com/osquery/osquery/pull/7489) - Remove redundant string conversion [#7603](https://github.com/osquery/osquery/pull/7603) ### Build - Fix DebPackages.test_sanity test when the `size` column is empty [#7569](https://github.com/osquery/osquery/pull/7569) - libs: Update libdpkg from version v1.19.0.5 to v1.21.7 [#7549](https://github.com/osquery/osquery/pull/7549) - CI: Restore some release checks [#7558](https://github.com/osquery/osquery/pull/7558) - Prevent ebpfpub linking against the system zlib [#7557](https://github.com/osquery/osquery/pull/7557) - Fix mdfind.test_sanity flaky behavior [#7533](https://github.com/osquery/osquery/pull/7533) - Enable fuzzing and Asan on Windows, enable Asan on macOS [#7470](https://github.com/osquery/osquery/pull/7470) - Update cppcheck to version 2.6.3 and skip analysis for third party code [#7455](https://github.com/osquery/osquery/pull/7455) - Change `cpu_info` test to expect *at least* one socket, not just one [#7490](https://github.com/osquery/osquery/pull/7490) - Fix third party libraries flags leaking to osquery targets [#7480](https://github.com/osquery/osquery/pull/7480) - Add third party libraries target [#7467](https://github.com/osquery/osquery/pull/7467) - Do not run clang-tidy on third party libraries [#7432](https://github.com/osquery/osquery/pull/7432) - CI: Create github workflow target to gate mergeability [#7427](https://github.com/osquery/osquery/pull/7427) - Fix some warnings about unrecognized special characters in the Windows event log test [#7478](https://github.com/osquery/osquery/pull/7478) - Change where the macOS Info.plist is generated [#7566](https://github.com/osquery/osquery/pull/7566) - Add OSQUERY_ENABLE_THREAD_SANITIZER to optionally enable TSan [#6997](https://github.com/osquery/osquery/pull/6997) - Add an option to specify a path to the openssl archive [#7559](https://github.com/osquery/osquery/pull/7559) - packs: Update reverse shell query pack to check for a valid remote_port [#7567](https://github.com/osquery/osquery/pull/7567) - Remove the test_daemon_sighup test [#7584](https://github.com/osquery/osquery/pull/7584) ### Documentation - docs: remove FreeBSD [#7508](https://github.com/osquery/osquery/pull/7508) - Pin Jinja2 ReadTheDocs dependency to 3.0.3 [#7533](https://github.com/osquery/osquery/pull/7533) - CHANGELOG 5.2.3 [#7571](https://github.com/osquery/osquery/pull/7571) - CHANGELOG 5.2.2 [#7447](https://github.com/osquery/osquery/pull/7447) - Bump mkdocs from 1.1.2 to 1.2.3 in /docs [#7457](https://github.com/osquery/osquery/pull/7457) - Replace OS X with macOS in table specs [#7587](https://github.com/osquery/osquery/pull/7587) - Update `osquery.example.conf` to omit the CLI only flags [#7595](https://github.com/osquery/osquery/pull/7595) 2022-05-24T20:33:25+00:00 OpenTAXII 0.6.0 OpenTAXII 0.6.0 2022-05-25T11:15:18+00:00 Changelog ========= 0.6.0 (2022-05-25 ------------------ * Add `public_discovery` option to taxii2 config * Add support for publicly readable taxii 2 api roots 0.5.0 (2022-05-24) ------------------ * Add support for publicly readable taxii 2 collections 0.4.0 (2022-05-20) ------------------ * Move next_param handling into `OpenTAXII2PersistenceAPI` 0.3.0 (2022-04-13) ------------------ * Implement taxii2.1 support 0.3.0a4 (2022-04-13) -------------------- * Merge changes from 0.2.4 maintenance release 0.3.0a3 (2022-01-21) -------------------- * Fix bug that prevented booting with only taxii1 config (`#217 <https://github.com/eclecticiq/OpenTAXII/issues/217>`_ thanks `@azurekid <https://github.com/azurekid>`_ for the report) 0.3.0a2 (2021-12-27) -------------------- * Merge changes from 0.2.3 maintenance release 0.3.0a1 ------- * Add python 3.10 support 0.3.0a0 ------- * Enablement for future taxii2 implementation * Fix documentation build issues 0.2.4 (2022-04-13) ------------------ * Make sure werkzeug <2.1 and >=2.1 work correctly with auth system 0.2.3 (2021-12-22) ------------------ * Fix bug in multithreaded use of sqlite (`#210 <https://github.com/eclecticiq/OpenTAXII/issues/210>`_ thanks `@rohits144 <https://github.com/rohits144>`_ for the report) 0.2.2 (2021-11-05) ------------------ * Fix readthedocs build 0.2.1 (2021-11-03) ------------------ * Add tests for python 3.6, 3.7, 3.8, 3.9, pypy * Add tests for sqlite, mysql, mariadb, postgresql * Fix bug that broke ``delete_content_blocks`` when using mysql on sqlalchemy 1.3 * Docs: Add db schema diagram * Docs: Clarify how to get default data in a default (development) docker instance * Fix implicit routing in TAXII 1.1 Inboxes * Update jwt usage to pyjwt >= 2.0 (thanks `@SanyaKapoor <https://github.com/SanyaKapoor>`_) 0.2.0 (2020-06-30) ------------------ * Enforce UTC usage in datetime fields in SQL DB Persistence API. * `Fix for #114 <https://github.com/eclecticiq/OpenTAXII/issues/114>`_: reintroduce ``opentaxii-create-account`` CLI command. * `Fix for #153 <https://github.com/eclecticiq/OpenTAXII/issues/152>`_: check if user can modify a collection before advertising it over inbox service. * Multiple coding style fixes. * Various documentation updates. 0.1.12 (2019-03-06) ------------------- * Remove unnecessary print statements. 0.1.11 (2019-02-13) ------------------- * Make JSON logging consistent when the application is run via Gunicorn. * Set ``acceptable_destination`` key in status details instead of extended headers * Allow passing ``engine_parameters`` to ``SQLDatabaseAPI`` for those who want to customize SQLAlchemy engine parameters. * Require recent version of ``lxml`` for security reasons. * Various test and Docker infrastructure improvements. 0.1.10 (2018-06-03) ------------------- * Replace separate service/collection/account creation process with single ``opentaxii-sync-data`` CLI command. * Persistence and Auth APIs extended with missing CRUD methods, that are used by ``opentaxii-sync-data``. * Read/modify collection level ACL added. * DB models for default implementation of Persistence API and Auth API were changed. No automatic migration code is provided (sorry!), so upgrading might require manual DB migration. * Drop python2.7 from testing scope. * Various bug fixes and improvements. 0.1.9 (2017-06-19) ------------------ * `libtaxii <https://github.com/TAXIIProject/libtaxii>`_ dependency upgraded to 1.1.111. * Various bug fixes and improvements (thanks to `@bjigmp <https://github.com/bjigmp>`_, `@chorsley <https://github.com/chorsley>`_, `@rjprins <https://github.com/rjprins>`_). 0.1.8 (2017-02-21) ------------------ * Ability to enable/disable "huge trees" support in XML parser. Configuration property ``xml_parser_supports_huge_tree`` set to ``yes`` or ``true`` will disable security restrictions and force XML parser to support very deep trees and very long text content. * Adjust SQL Persistence API implemetation so it works smoothly with MySQL backend. * Use Python 3.5 instead of Python 3.4 for tests. 0.1.7 (2016-10-18) ------------------ * Minor fixes. * Dependencies were changed from hard-pinned to more flexible. * Example of production DB configuration added to docs. 0.1.6 (2016-06-01) ------------------ * Python 3.4 compatibility of the codebase. Tox configuration extended with python 3.4 environment run. * Flake8 full style compatibility. Flake8 check added to Tox configuration. * SQLAlchemy session scope issue fixed (related to `#38 <https://github.com/EclecticIQ/OpenTAXII/issues/38>`_). * `opentaxii-delete-blocks` CLI command added (related to `#45 <https://github.com/EclecticIQ/OpenTAXII/issues/45>`_). * `delete_content_blocks` method `added <https://github.com/EclecticIQ/OpenTAXII/commit/dc6fddc27a98e8450c7e05e583b2bfb741f6e197#diff-6814849ac352b2b74132f8fa52e0ec4eR213>`_ to Persistence API. * Collection's name is `required <https://github.com/EclecticIQ/OpenTAXII/commit/dc6fddc27a98e8450c7e05e583b2bfb741f6e197#diff-ce3f7b939e5c540480ac655aef32c513R116>`_ to be unique in default SQL DB Auth API implementation. 0.1.5 (2016-03-15) ------------------ * Fix for the issue with persistence backend returning ``None`` instead of ``InboxMessage`` object 0.1.4 (2016-02-25) ------------------ * Hard-coded dependencies in ``setup.py`` removed. 0.1.3 (2016-02-25) ------------------ * Versions of dependencies are pinned. * Code adjusted for a new version of `anyconfig <https://pypi.python.org/pypi/anyconfig>`_ API. * Test for configuration loading added. 0.1.2 (2015-07-24) ------------------ * Docker configuration added. * Health check endpoint added. * Basic authentication support added. * Temporary workaround for `Issue #191 <https://github.com/TAXIIProject/libtaxii/issues/191>`_. * Method ``get_domain`` in Persistence API returns domain value configured for ``service_id``. If nothing returned, value set in configuration file will be used. * Performance optimisations. * Bug fixes and style improvements. 0.1.1 (2015-04-08) ------------------ * Alias for Root Logger added to logging configuration. * Context object in a request scope that holds account and token added. * Support for OPTIONS HTTP request to enable auto version negotiation added. * Documentation improved. 0.1.0 (2015-03-31) ------------------ * Initial release 2022-05-25T11:15:18+00:00 dalton v3.2.3 dalton v3.2.3 2022-05-26T14:51:26+00:00 - Added share link on job results page to recreate job automatically - Added PCAP drag and drop to UI when creating a new job 2022-05-26T14:51:26+00:00 OpenTAXII 0.7.0 OpenTAXII 0.7.0 2022-05-27T18:01:28+00:00 Changelog ========= 0.7.0 (2022-05-27) ------------------ * Nest taxii2 endpoints under `/taxii2/` 0.6.0 (2022-05-25) ------------------ * Add `public_discovery` option to taxii2 config * Add support for publicly readable taxii 2 api roots 0.5.0 (2022-05-24) ------------------ * Add support for publicly readable taxii 2 collections 0.4.0 (2022-05-20) ------------------ * Move next_param handling into `OpenTAXII2PersistenceAPI` 0.3.0 (2022-04-13) ------------------ * Implement taxii2.1 support 0.3.0a4 (2022-04-13) -------------------- * Merge changes from 0.2.4 maintenance release 0.3.0a3 (2022-01-21) -------------------- * Fix bug that prevented booting with only taxii1 config (`#217 <https://github.com/eclecticiq/OpenTAXII/issues/217>`_ thanks `@azurekid <https://github.com/azurekid>`_ for the report) 0.3.0a2 (2021-12-27) -------------------- * Merge changes from 0.2.3 maintenance release 0.3.0a1 ------- * Add python 3.10 support 0.3.0a0 ------- * Enablement for future taxii2 implementation * Fix documentation build issues 0.2.4 (2022-04-13) ------------------ * Make sure werkzeug <2.1 and >=2.1 work correctly with auth system 0.2.3 (2021-12-22) ------------------ * Fix bug in multithreaded use of sqlite (`#210 <https://github.com/eclecticiq/OpenTAXII/issues/210>`_ thanks `@rohits144 <https://github.com/rohits144>`_ for the report) 0.2.2 (2021-11-05) ------------------ * Fix readthedocs build 0.2.1 (2021-11-03) ------------------ * Add tests for python 3.6, 3.7, 3.8, 3.9, pypy * Add tests for sqlite, mysql, mariadb, postgresql * Fix bug that broke ``delete_content_blocks`` when using mysql on sqlalchemy 1.3 * Docs: Add db schema diagram * Docs: Clarify how to get default data in a default (development) docker instance * Fix implicit routing in TAXII 1.1 Inboxes * Update jwt usage to pyjwt >= 2.0 (thanks `@SanyaKapoor <https://github.com/SanyaKapoor>`_) 0.2.0 (2020-06-30) ------------------ * Enforce UTC usage in datetime fields in SQL DB Persistence API. * `Fix for #114 <https://github.com/eclecticiq/OpenTAXII/issues/114>`_: reintroduce ``opentaxii-create-account`` CLI command. * `Fix for #153 <https://github.com/eclecticiq/OpenTAXII/issues/152>`_: check if user can modify a collection before advertising it over inbox service. * Multiple coding style fixes. * Various documentation updates. 0.1.12 (2019-03-06) ------------------- * Remove unnecessary print statements. 0.1.11 (2019-02-13) ------------------- * Make JSON logging consistent when the application is run via Gunicorn. * Set ``acceptable_destination`` key in status details instead of extended headers * Allow passing ``engine_parameters`` to ``SQLDatabaseAPI`` for those who want to customize SQLAlchemy engine parameters. * Require recent version of ``lxml`` for security reasons. * Various test and Docker infrastructure improvements. 0.1.10 (2018-06-03) ------------------- * Replace separate service/collection/account creation process with single ``opentaxii-sync-data`` CLI command. * Persistence and Auth APIs extended with missing CRUD methods, that are used by ``opentaxii-sync-data``. * Read/modify collection level ACL added. * DB models for default implementation of Persistence API and Auth API were changed. No automatic migration code is provided (sorry!), so upgrading might require manual DB migration. * Drop python2.7 from testing scope. * Various bug fixes and improvements. 0.1.9 (2017-06-19) ------------------ * `libtaxii <https://github.com/TAXIIProject/libtaxii>`_ dependency upgraded to 1.1.111. * Various bug fixes and improvements (thanks to `@bjigmp <https://github.com/bjigmp>`_, `@chorsley <https://github.com/chorsley>`_, `@rjprins <https://github.com/rjprins>`_). 0.1.8 (2017-02-21) ------------------ * Ability to enable/disable "huge trees" support in XML parser. Configuration property ``xml_parser_supports_huge_tree`` set to ``yes`` or ``true`` will disable security restrictions and force XML parser to support very deep trees and very long text content. * Adjust SQL Persistence API implemetation so it works smoothly with MySQL backend. * Use Python 3.5 instead of Python 3.4 for tests. 0.1.7 (2016-10-18) ------------------ * Minor fixes. * Dependencies were changed from hard-pinned to more flexible. * Example of production DB configuration added to docs. 0.1.6 (2016-06-01) ------------------ * Python 3.4 compatibility of the codebase. Tox configuration extended with python 3.4 environment run. * Flake8 full style compatibility. Flake8 check added to Tox configuration. * SQLAlchemy session scope issue fixed (related to `#38 <https://github.com/EclecticIQ/OpenTAXII/issues/38>`_). * `opentaxii-delete-blocks` CLI command added (related to `#45 <https://github.com/EclecticIQ/OpenTAXII/issues/45>`_). * `delete_content_blocks` method `added <https://github.com/EclecticIQ/OpenTAXII/commit/dc6fddc27a98e8450c7e05e583b2bfb741f6e197#diff-6814849ac352b2b74132f8fa52e0ec4eR213>`_ to Persistence API. * Collection's name is `required <https://github.com/EclecticIQ/OpenTAXII/commit/dc6fddc27a98e8450c7e05e583b2bfb741f6e197#diff-ce3f7b939e5c540480ac655aef32c513R116>`_ to be unique in default SQL DB Auth API implementation. 0.1.5 (2016-03-15) ------------------ * Fix for the issue with persistence backend returning ``None`` instead of ``InboxMessage`` object 0.1.4 (2016-02-25) ------------------ * Hard-coded dependencies in ``setup.py`` removed. 0.1.3 (2016-02-25) ------------------ * Versions of dependencies are pinned. * Code adjusted for a new version of `anyconfig <https://pypi.python.org/pypi/anyconfig>`_ API. * Test for configuration loading added. 0.1.2 (2015-07-24) ------------------ * Docker configuration added. * Health check endpoint added. * Basic authentication support added. * Temporary workaround for `Issue #191 <https://github.com/TAXIIProject/libtaxii/issues/191>`_. * Method ``get_domain`` in Persistence API returns domain value configured for ``service_id``. If nothing returned, value set in configuration file will be used. * Performance optimisations. * Bug fixes and style improvements. 0.1.1 (2015-04-08) ------------------ * Alias for Root Logger added to logging configuration. * Context object in a request scope that holds account and token added. * Support for OPTIONS HTTP request to enable auto version negotiation added. * Documentation improved. 0.1.0 (2015-03-31) ------------------ * Initial release 2022-05-27T18:01:28+00:00 MISP v2.4.159 MISP v2.4.159 2022-05-30T17:05:38+00:00 We are pleased to announce the immediate availability of MISP v2.4.159. This releases includes many improvements, bugs fixing and improvement concerning performance on large dataset. ![](https://www.misp-project.org/img/blog/graph-syria.png) # Performance Improvements - [DB] Add MysqlExtended DboSource to support index query hints. - [Query] Add new setting to disable taxonomy checks when browsing data. - We discovered that some MISP users are still using slow file-based session handling in PHP. Now, we added in the diagnostic, if session is file based. We recommend everyone to use the Redis session. - Many additional speed-up and faster functions in the MISP internals. - Reduce memory usage when generating all correlations. # Improvements - [Feed] Allow option to disable correlations for all events coming from a feed. This can be useful when correlation requires to be disabled for an imported feed. - [UI] Allow to upload MISP event by pasting data to textarea in addition to the file upload. - An optional feature `clusters:attachMultipleClusters` is now available to allow the mirroring of attribute clusters to event. - [auditlog] Support for fetch event changes from specific time. - [UI] Allow to filter attributes from Related Events box. - [UI] Allow to filter attributes from warninglist box. - [UI] Many UI improvements to make the interface easier to read. - [UI] Disable correlation checkbox for non correlating types. - [STIX 2 import] Better Galaxies parsing by looking for the ATT&CK technique id. - [API] Enable sharing group filter for Event controller not just attribute. # Fixes - [STIX] Avoiding non RFC-4122 UUIDs to be imported (and therefore skipped) - [STIX 1 import] Save process network connections. - [STIX 1 import] Fixed galaxy tag_names fetching from TTP names. # Knowledge Bases ## MISP Taxonomies - [dga] First version of the DGA taxonomy based on https://www.usenix.org/system/files/conference/usenixsecurity16/sec16_paper_plohmann.pdf. - GrayZone of Active Defense, originaly published by Washington University, v2 created and updated by DCG420 - Various fixes to existing taxonomies. ## MISP Objects Template - A new PaloAlto Threat Event object template has been added. - A updated security playbook has been added. - A new ransom negotiation object has been added. - An improved Passive SSH template object. - Various fixes and improvements to different object templates such as email, virustotal-submissions and others. ## MISP Galaxy - Improved Cryptominers galaxy. - Improved backdoors galaxy. - Threat Actor galaxy updated and extended with new threat-actors. - MISP Galaxy updated for MITRE ATT&CK v11.2. # Acknowledgement We would like to thank all the [contributors](https://www.misp-project.org/contributors), reporters and users who have helped us in the past months to improve MISP and information sharing at large. This release includes multiple updates in [misp-objects](https://www.misp-project.org/objects.html), [misp-taxonomies](https://www.misp-project.org/taxonomies.html) and [misp-galaxy](https://www.misp-project.org/galaxy.html). As always, a detailed and [complete changelog is available](https://www.misp-project.org/Changelog.txt) with all the fixes, changes and improvements in MISP core. Additional changelogs are available for [misp galaxy](https://www.misp-project.org/Changelog-misp-galaxy.txt), [misp-taxonomies](https://www.misp-project.org/Changelog-misp-taxonomies.txt), [misp-objects](https://www.misp-project.org/Changelog-misp-objects.txt) and [misp-modules](https://www.misp-project.org/Changelog-misp-modules.txt) 2022-05-30T17:05:38+00:00 HyperDbg v0.1.0 HyperDbg v0.1.0 2022-05-31T14:08:32+00:00 # HyperDbg v0.1 is released! **If you’re enjoying HyperDbg, don’t forget to give a star 🌟 on GitHub!** Please visit [Build & Install](https://docs.hyperdbg.org/getting-started/build-and-install) to configure the environment for running **HyperDbg**. Check out the [Quick Start](https://docs.hyperdbg.org/getting-started/quick-start) and [Frequently Asked Questions (FAQs)](https://docs.hyperdbg.org/getting-started/faq) to learn more. You can use the examples of [using the debugger](https://docs.hyperdbg.org/using-hyperdbg/kernel-mode-debugging/examples) and the [script engine](https://docs.hyperdbg.org/commands/scripting-language/examples) to get started with **HyperDbg**. ## New Fearues * Advanced Hypervisor-based Kernel Mode Debugger [<a href="https://docs.hyperdbg.org/using-hyperdbg/kernel-mode-debugging" target="_blank">link</a>][<a href="https://docs.hyperdbg.org/getting-started/attach-to-hyperdbg/debug" target="_blank">link</a>][<a href="https://docs.hyperdbg.org/getting-started/attach-to-hyperdbg/local-debugging" target="_blank">link</a>] * Classic EPT Hook (Hidden Breakpoint) [<a href="https://docs.hyperdbg.org/commands/extension-commands/epthook" target="_blank">link</a>][<a href="https://docs.hyperdbg.org/design/features/vmm-module/design-of-epthook" target="_blank">link</a>][<a href="https://docs.hyperdbg.org/using-hyperdbg/kernel-mode-debugging/examples/events/hooking-any-function" target="_blank">link</a>] * Inline EPT Hook (Inline Hook) [<a href="https://docs.hyperdbg.org/commands/extension-commands/epthook2" target="_blank">link</a>][<a href="https://docs.hyperdbg.org/design/features/vmm-module/design-of-epthook2" target="_blank">link</a>] * Monitor Memory For R/W (Emulating Hardware Debug Registers Without Limitation) [<a href="https://docs.hyperdbg.org/commands/extension-commands/monitor" target="_blank">link</a>][<a href="https://docs.hyperdbg.org/design/features/vmm-module/design-of-monitor" target="_blank">link</a>][<a href="https://docs.hyperdbg.org/using-hyperdbg/kernel-mode-debugging/examples/events/monitoring-accesses-to-structures" target="_blank">link</a>] * SYSCALL Hook (Disable EFER & Handle #UD) [<a href="https://docs.hyperdbg.org/commands/extension-commands/syscall" target="_blank">link</a>][<a href="https://docs.hyperdbg.org/design/features/vmm-module/design-of-syscall-and-sysret" target="_blank">link</a>][<a href="https://docs.hyperdbg.org/using-hyperdbg/kernel-mode-debugging/examples/events/intercepting-all-syscalls" target="_blank">link</a>] * SYSRET Hook (Disable EFER & Handle #UD) [<a href="https://docs.hyperdbg.org/commands/extension-commands/sysret" target="_blank">link</a>][<a href="https://docs.hyperdbg.org/design/features/vmm-module/design-of-syscall-and-sysret" target="_blank">link</a>] * CPUID Hook & Monitor [<a href="https://docs.hyperdbg.org/commands/extension-commands/cpuid" target="_blank">link</a>][<a href="https://docs.hyperdbg.org/using-hyperdbg/kernel-mode-debugging/examples/events/triggering-special-instructions" target="_blank">link</a>] * RDMSR Hook & Monitor [<a href="https://docs.hyperdbg.org/commands/extension-commands/msrread" target="_blank">link</a>][<a href="https://docs.hyperdbg.org/using-hyperdbg/kernel-mode-debugging/examples/events/identifying-system-behavior" target="_blank">link</a>] * WRMSR Hook & Monitor [<a href="https://docs.hyperdbg.org/commands/extension-commands/msrwrite" target="_blank">link</a>][<a href="https://docs.hyperdbg.org/using-hyperdbg/kernel-mode-debugging/examples/events/identifying-system-behavior" target="_blank">link</a>] * RDTSC/RDTSCP Hook & Monitor [<a href="https://docs.hyperdbg.org/commands/extension-commands/tsc" target="_blank">link</a>] * RDPMC Hook & Monitor [<a href="https://docs.hyperdbg.org/commands/extension-commands/pmc" target="_blank">link</a>] * VMCALL Hook & Monitor [<a href="https://docs.hyperdbg.org/commands/extension-commands/vmcall" target="_blank">link</a>] * Debug Registers Hook & Monitor [<a href="https://docs.hyperdbg.org/commands/extension-commands/dr" target="_blank">link</a>] * I/O Port (In Instruction) Hook & Monitor [<a href="https://docs.hyperdbg.org/commands/extension-commands/ioin" target="_blank">link</a>][<a href="https://docs.hyperdbg.org/using-hyperdbg/kernel-mode-debugging/examples/events/triggering-special-instructions" target="_blank">link</a>] * I/O Port (Out Instruction) Hook & Monitor [<a href="https://docs.hyperdbg.org/commands/extension-commands/ioout" target="_blank">link</a>][<a href="https://docs.hyperdbg.org/using-hyperdbg/kernel-mode-debugging/examples/events/triggering-special-instructions" target="_blank">link</a>] * MMIO Monitor [<a href="https://docs.hyperdbg.org/commands/extension-commands/monitor" target="_blank">link</a>] * Exception (IDT < 32) Monitor [<a href="https://docs.hyperdbg.org/commands/extension-commands/exception" target="_blank">link</a>][<a href="https://docs.hyperdbg.org/design/features/vmm-module/design-of-exception-and-interrupt" target="_blank">link</a>][<a href="https://docs.hyperdbg.org/using-hyperdbg/kernel-mode-debugging/examples/events/identifying-system-behavior" target="_blank">link</a>] * External-Interrupt (IDT > 32) Monitor [<a href="https://docs.hyperdbg.org/commands/extension-commands/interrupt" target="_blank">link</a>][<a href="https://docs.hyperdbg.org/design/features/vmm-module/design-of-exception-and-interrupt" target="_blank">link</a>][<a href="https://docs.hyperdbg.org/using-hyperdbg/kernel-mode-debugging/examples/events/identifying-system-behavior" target="_blank">link</a>] * Running Automated Scripts [<a href="https://docs.hyperdbg.org/commands/scripting-language/hyperdbg-scripts" target="_blank">link</a>] * Transparent-mode (Anti-debugging and Anti-hypervisor Resistance) [<a href="https://docs.hyperdbg.org/tips-and-tricks/considerations/transparent-mode" target="_blank">link</a>][<a href="https://docs.hyperdbg.org/using-hyperdbg/kernel-mode-debugging/examples/misc/defeating-anti-debug-and-anti-hypervisor-methods" target="_blank">link</a>] * Running Custom Assembly In Both VMX-root, VMX non-root (Kernel & User) [<a href="https://docs.hyperdbg.org/using-hyperdbg/prerequisites/how-to-create-an-action" target="_blank">link</a>] * Checking For Custom Conditions [<a href="https://docs.hyperdbg.org/using-hyperdbg/prerequisites/how-to-create-a-condition" target="_blank">link</a>][<a href="https://docs.hyperdbg.org/design/debugger-internals/conditions" target="_blank">link</a>] * Process-specific & Thread-specific Debugging [<a href="https://docs.hyperdbg.org/commands/meta-commands/.process" target="_blank">link</a>][<a href="https://docs.hyperdbg.org/commands/meta-commands/.thread" target="_blank">link</a>][<a href="https://docs.hyperdbg.org/using-hyperdbg/user-mode-debugging/examples/basics/switching-to-a-specific-process-or-thread" target="_blank">link</a>] * VMX-root Compatible Message Tracing [<a href="https://docs.hyperdbg.org/design/features/vmm-module/vmx-root-mode-compatible-message-tracing" target="_blank">link</a>] * Powerful Kernel Side Scripting Engine [<a href="https://docs.hyperdbg.org/commands/scripting-language" target="_blank">link</a>][<a href="https://docs.hyperdbg.org/design/script-engine" target="_blank">link</a>] * Support To Symbols (Parsing PDB Files) [<a href="https://docs.hyperdbg.org/commands/meta-commands/.sympath" target="_blank">link</a>][<a href="https://docs.hyperdbg.org/commands/meta-commands/.sym" target="_blank">link</a>] * Event Forwarding (#DFIR) [<a href="https://docs.hyperdbg.org/tips-and-tricks/misc/event-forwarding" target="_blank">link</a>][<a href="https://docs.hyperdbg.org/commands/debugging-commands/output" target="_blank">link</a>] * Transparent Breakpoint Handler [<a href="https://docs.hyperdbg.org/commands/debugging-commands/bp" target="_blank">link</a>][<a href="https://docs.hyperdbg.org/using-hyperdbg/kernel-mode-debugging/examples/basics/setting-breakpoints-and-stepping-instructions" target="_blank">link</a>] * Various Custom Scripts [<a href="https://github.com/HyperDbg/scripts" target="_blank">link</a>] **Note**: community contributions are always welcomed and appreciated. If you plan to contribute a new feature, it's best to [discuss](https://github.com/HyperDbg/HyperDbg/discussions) it first. Bug fixes, tests, and documentation improvements are greatly appreciated. 2022-05-31T14:08:32+00:00 maltrail 0.46 maltrail 0.46 2022-05-31T22:11:06+00:00 Start-of-month release 2022-05-31T22:11:06+00:00 syncthing v1.20.2-rc.3 syncthing v1.20.2-rc.3 2022-06-01T05:34:59+00:00 Bugfixes: - #7289: TCP port 0 is announced in the LAN beacon - #8314: https://relays.syncthing.net/endpoint contains non-valid URLs with spaces - #8355: Upgrading from v1.19.2 to v1.20.x now requires chmod for syncing files Enhancements: - #8264: Include default ignore patterns in the GUI's advanced configuration editor - #8310: Indicate folders / devices where the remote end is paused. 2022-06-01T05:34:59+00:00 PyPCAPKit v0.16.0 PyPCAPKit v0.16.0 2022-06-01T05:35:59+00:00 ## What's Changed * revised entire project * type annotations * protocol redesign * linter compliance * added and revised tests (see #82, bugfix help wanted) * general bugfix in the project (#101 and many more) * reviewed and rearranged APIs, especially * added `pcapkit.corekit.multidict` module based on `Werkzeug` project * removed `validators` & some decorators from `pcapkit.utilities` * moved PCAP & aux protocols under `pcapkit.misc` * redesigned `Protocol` class to integrate parsing & construction at the same time, as well as better subclass protocol implementation experience * initialised PyPCAPKit Enhancement Proposals discussion channel (see #106) **Full Changelog**: https://github.com/JarryShaw/PyPCAPKit/compare/v0.15.5...v0.16.0 2022-06-01T05:35:59+00:00 OpenTAXII 0.8.0 OpenTAXII 0.8.0 2022-06-05T14:57:50+00:00 Changelog ========= 0.8.0 (2022-06-05) ------------------ * Nest details inside taxii2 job and allow counts without details 0.7.0 (2022-05-27) ------------------ * Nest taxii2 endpoints under `/taxii2/` 0.6.0 (2022-05-25) ------------------ * Add `public_discovery` option to taxii2 config * Add support for publicly readable taxii 2 api roots 0.5.0 (2022-05-24) ------------------ * Add support for publicly readable taxii 2 collections 0.4.0 (2022-05-20) ------------------ * Move next_param handling into `OpenTAXII2PersistenceAPI` 0.3.0 (2022-04-13) ------------------ * Implement taxii2.1 support 0.3.0a4 (2022-04-13) -------------------- * Merge changes from 0.2.4 maintenance release 0.3.0a3 (2022-01-21) -------------------- * Fix bug that prevented booting with only taxii1 config (`#217 <https://github.com/eclecticiq/OpenTAXII/issues/217>`_ thanks `@azurekid <https://github.com/azurekid>`_ for the report) 0.3.0a2 (2021-12-27) -------------------- * Merge changes from 0.2.3 maintenance release 0.3.0a1 ------- * Add python 3.10 support 0.3.0a0 ------- * Enablement for future taxii2 implementation * Fix documentation build issues 0.2.4 (2022-04-13) ------------------ * Make sure werkzeug <2.1 and >=2.1 work correctly with auth system 0.2.3 (2021-12-22) ------------------ * Fix bug in multithreaded use of sqlite (`#210 <https://github.com/eclecticiq/OpenTAXII/issues/210>`_ thanks `@rohits144 <https://github.com/rohits144>`_ for the report) 0.2.2 (2021-11-05) ------------------ * Fix readthedocs build 0.2.1 (2021-11-03) ------------------ * Add tests for python 3.6, 3.7, 3.8, 3.9, pypy * Add tests for sqlite, mysql, mariadb, postgresql * Fix bug that broke ``delete_content_blocks`` when using mysql on sqlalchemy 1.3 * Docs: Add db schema diagram * Docs: Clarify how to get default data in a default (development) docker instance * Fix implicit routing in TAXII 1.1 Inboxes * Update jwt usage to pyjwt >= 2.0 (thanks `@SanyaKapoor <https://github.com/SanyaKapoor>`_) 0.2.0 (2020-06-30) ------------------ * Enforce UTC usage in datetime fields in SQL DB Persistence API. * `Fix for #114 <https://github.com/eclecticiq/OpenTAXII/issues/114>`_: reintroduce ``opentaxii-create-account`` CLI command. * `Fix for #153 <https://github.com/eclecticiq/OpenTAXII/issues/152>`_: check if user can modify a collection before advertising it over inbox service. * Multiple coding style fixes. * Various documentation updates. 0.1.12 (2019-03-06) ------------------- * Remove unnecessary print statements. 0.1.11 (2019-02-13) ------------------- * Make JSON logging consistent when the application is run via Gunicorn. * Set ``acceptable_destination`` key in status details instead of extended headers * Allow passing ``engine_parameters`` to ``SQLDatabaseAPI`` for those who want to customize SQLAlchemy engine parameters. * Require recent version of ``lxml`` for security reasons. * Various test and Docker infrastructure improvements. 0.1.10 (2018-06-03) ------------------- * Replace separate service/collection/account creation process with single ``opentaxii-sync-data`` CLI command. * Persistence and Auth APIs extended with missing CRUD methods, that are used by ``opentaxii-sync-data``. * Read/modify collection level ACL added. * DB models for default implementation of Persistence API and Auth API were changed. No automatic migration code is provided (sorry!), so upgrading might require manual DB migration. * Drop python2.7 from testing scope. * Various bug fixes and improvements. 0.1.9 (2017-06-19) ------------------ * `libtaxii <https://github.com/TAXIIProject/libtaxii>`_ dependency upgraded to 1.1.111. * Various bug fixes and improvements (thanks to `@bjigmp <https://github.com/bjigmp>`_, `@chorsley <https://github.com/chorsley>`_, `@rjprins <https://github.com/rjprins>`_). 0.1.8 (2017-02-21) ------------------ * Ability to enable/disable "huge trees" support in XML parser. Configuration property ``xml_parser_supports_huge_tree`` set to ``yes`` or ``true`` will disable security restrictions and force XML parser to support very deep trees and very long text content. * Adjust SQL Persistence API implemetation so it works smoothly with MySQL backend. * Use Python 3.5 instead of Python 3.4 for tests. 0.1.7 (2016-10-18) ------------------ * Minor fixes. * Dependencies were changed from hard-pinned to more flexible. * Example of production DB configuration added to docs. 0.1.6 (2016-06-01) ------------------ * Python 3.4 compatibility of the codebase. Tox configuration extended with python 3.4 environment run. * Flake8 full style compatibility. Flake8 check added to Tox configuration. * SQLAlchemy session scope issue fixed (related to `#38 <https://github.com/EclecticIQ/OpenTAXII/issues/38>`_). * `opentaxii-delete-blocks` CLI command added (related to `#45 <https://github.com/EclecticIQ/OpenTAXII/issues/45>`_). * `delete_content_blocks` method `added <https://github.com/EclecticIQ/OpenTAXII/commit/dc6fddc27a98e8450c7e05e583b2bfb741f6e197#diff-6814849ac352b2b74132f8fa52e0ec4eR213>`_ to Persistence API. * Collection's name is `required <https://github.com/EclecticIQ/OpenTAXII/commit/dc6fddc27a98e8450c7e05e583b2bfb741f6e197#diff-ce3f7b939e5c540480ac655aef32c513R116>`_ to be unique in default SQL DB Auth API implementation. 0.1.5 (2016-03-15) ------------------ * Fix for the issue with persistence backend returning ``None`` instead of ``InboxMessage`` object 0.1.4 (2016-02-25) ------------------ * Hard-coded dependencies in ``setup.py`` removed. 0.1.3 (2016-02-25) ------------------ * Versions of dependencies are pinned. * Code adjusted for a new version of `anyconfig <https://pypi.python.org/pypi/anyconfig>`_ API. * Test for configuration loading added. 0.1.2 (2015-07-24) ------------------ * Docker configuration added. * Health check endpoint added. * Basic authentication support added. * Temporary workaround for `Issue #191 <https://github.com/TAXIIProject/libtaxii/issues/191>`_. * Method ``get_domain`` in Persistence API returns domain value configured for ``service_id``. If nothing returned, value set in configuration file will be used. * Performance optimisations. * Bug fixes and style improvements. 0.1.1 (2015-04-08) ------------------ * Alias for Root Logger added to logging configuration. * Context object in a request scope that holds account and token added. * Support for OPTIONS HTTP request to enable auto version negotiation added. * Documentation improved. 0.1.0 (2015-03-31) ------------------ * Initial release 2022-06-05T14:57:50+00:00 syncthing v1.20.2 syncthing v1.20.2 2022-06-07T10:13:46+00:00 Bugfixes: - #7289: TCP port 0 is announced in the LAN beacon - #8314: https://relays.syncthing.net/endpoint contains non-valid URLs with spaces - #8355: Upgrading from v1.19.2 to v1.20.x now requires chmod for syncing files Enhancements: - #8264: Include default ignore patterns in the GUI's advanced configuration editor - #8310: Indicate folders / devices where the remote end is paused. 2022-06-07T10:13:46+00:00 PyPCAPKit v0.16.1 PyPCAPKit v0.16.1 2022-06-08T19:59:55+00:00 ## What's Changed * changed `warnings.warn` calls to `pcapkit.utilities.warnings.warn` * revised missing optional dependency warnings (add for CLI & vendor) * updated version strings in repository * `make isort` **Full Changelog**: https://github.com/JarryShaw/PyPCAPKit/compare/v0.16.0...v0.16.1 2022-06-08T19:59:55+00:00 rocksdb v7.3.1 rocksdb v7.3.1 2022-06-10T23:08:05+00:00 ## 7.3.1 (06/08/2022) ### Bug Fixes * Fix a bug in WAL tracking. Before this PR (#10087), calling `SyncWAL()` on the only WAL file of the db will not log the event in MANIFEST, thus allowing a subsequent `DB::Open` even if the WAL file is missing or corrupted. * Fixed a bug for non-TransactionDB with avoid_flush_during_recovery = true and TransactionDB where in case of crash, min_log_number_to_keep may not change on recovery and persisting a new MANIFEST with advanced log_numbers for some column families, results in "column family inconsistency" error on second recovery. As a solution, RocksDB will persist the new MANIFEST after successfully syncing the new WAL. If a future recovery starts from the new MANIFEST, then it means the new WAL is successfully synced. Due to the sentinel empty write batch at the beginning, kPointInTimeRecovery of WAL is guaranteed to go after this point. If future recovery starts from the old MANIFEST, it means the writing the new MANIFEST failed. We won't have the "SST ahead of WAL" error. * Fixed a bug where RocksDB DB::Open() may creates and writes to two new MANIFEST files even before recovery succeeds. Now writes to MANIFEST are persisted only after recovery is successful. ## 7.3.0 (05/20/2022) ### Bug Fixes * Fixed a bug where manual flush would block forever even though flush options had wait=false. * Fixed a bug where RocksDB could corrupt DBs with `avoid_flush_during_recovery == true` by removing valid WALs, leading to `Status::Corruption` with message like "SST file is ahead of WALs" when attempting to reopen. * Fixed a bug in async_io path where incorrect length of data is read by FilePrefetchBuffer if data is consumed from two populated buffers and request for more data is sent. * Fixed a CompactionFilter bug. Compaction filter used to use `Delete` to remove keys, even if the keys should be removed with `SingleDelete`. Mixing `Delete` and `SingleDelete` may cause undefined behavior. * Fixed a bug in `WritableFileWriter::WriteDirect` and `WritableFileWriter::WriteDirectWithChecksum`. The rate_limiter_priority specified in ReadOptions was not passed to the RateLimiter when requesting a token. * Fixed a bug which might cause process crash when I/O error happens when reading an index block in MultiGet(). ### New Features * DB::GetLiveFilesStorageInfo is ready for production use. * Add new stats PREFETCHED_BYTES_DISCARDED which records number of prefetched bytes discarded by RocksDB FilePrefetchBuffer on destruction and POLL_WAIT_MICROS records wait time for FS::Poll API completion. * RemoteCompaction supports table_properties_collector_factories override on compaction worker. * Start tracking SST unique id in MANIFEST, which will be used to verify with SST properties during DB open to make sure the SST file is not overwritten or misplaced. A db option `verify_sst_unique_id_in_manifest` is introduced to enable/disable the verification, if enabled all SST files will be opened during DB-open to verify the unique id (default is false), so it's recommended to use it with `max_open_files = -1` to pre-open the files. * Added the ability to concurrently read data blocks from multiple files in a level in batched MultiGet. This can be enabled by setting the async_io option in ReadOptions. Using this feature requires a FileSystem that supports ReadAsync (PosixFileSystem is not supported yet for this), and for RocksDB to be compiled with folly and c++20. * Add FileSystem::ReadAsync API in io_tracing. ### Public API changes * Add rollback_deletion_type_callback to TransactionDBOptions so that write-prepared transactions know whether to issue a Delete or SingleDelete to cancel a previous key written during prior prepare phase. The PR aims to prevent mixing SingleDeletes and Deletes for the same key that can lead to undefined behaviors for write-prepared transactions. * EXPERIMENTAL: Add new API AbortIO in file_system to abort the read requests submitted asynchronously. * CompactionFilter::Decision has a new value: kRemoveWithSingleDelete. If CompactionFilter returns this decision, then CompactionIterator will use `SingleDelete` to mark a key as removed. * Renamed CompactionFilter::Decision::kRemoveWithSingleDelete to kPurge since the latter sounds more general and hides the implementation details of how compaction iterator handles keys. * Added ability to specify functions for Prepare and Validate to OptionsTypeInfo. Added methods to OptionTypeInfo to set the functions via an API. These methods are intended for RocksDB plugin developers for configuration management. * Added a new immutable db options, enforce_single_del_contracts. If set to false (default is true), compaction will NOT fail due to a single delete followed by a delete for the same key. The purpose of this temporay option is to help existing use cases migrate. * Introduce `BlockBasedTableOptions::cache_usage_options` and use that to replace `BlockBasedTableOptions::reserve_table_builder_memory` and `BlockBasedTableOptions::reserve_table_reader_memory`. * Changed `GetUniqueIdFromTableProperties` to return a 128-bit unique identifier, which will be the standard size now. The old functionality (192-bit) is available from `GetExtendedUniqueIdFromTableProperties`. Both functions are no longer "experimental" and are ready for production use. * In IOOptions, mark `prio` as deprecated for future removal. * In `file_system.h`, mark `IOPriority` as deprecated for future removal. * Add an option, `CompressionOptions::use_zstd_dict_trainer`, to indicate whether zstd dictionary trainer should be used for generating zstd compression dictionaries. The default value of this option is true for backward compatibility. When this option is set to false, zstd API `ZDICT_finalizeDictionary` is used to generate compression dictionaries. * Seek API which positions itself every LevelIterator on the correct data block in the correct SST file which can be parallelized if ReadOptions.async_io option is enabled. * Add new stat number_async_seek in PerfContext that indicates number of async calls made by seek to prefetch data. ### Bug Fixes * RocksDB calls FileSystem::Poll API during FilePrefetchBuffer destruction which impacts performance as it waits for read requets completion which is not needed anymore. Calling FileSystem::AbortIO to abort those requests instead fixes that performance issue. * Fixed unnecessary block cache contention when queries within a MultiGet batch and across parallel batches access the same data block, which previously could cause severely degraded performance in this unusual case. (In more typical MultiGet cases, this fix is expected to yield a small or negligible performance improvement.) ### Behavior changes * Enforce the existing contract of SingleDelete so that SingleDelete cannot be mixed with Delete because it leads to undefined behavior. Fix a number of unit tests that violate the contract but happen to pass. * ldb `--try_load_options` default to true if `--db` is specified and not creating a new DB, the user can still explicitly disable that by `--try_load_options=false` (or explicitly enable that by `--try_load_options`). * During Flush write or Compaction write/read, the WriteController is used to determine whether DB writes are stalled or slowed down. The priority (Env::IOPriority) can then be determined accordingly and be passed in IOOptions to the file system. 2022-06-10T23:08:05+00:00 cwe_checker v0.6 cwe_checker v0.6 2022-06-13T05:48:56+00:00 Version 0.6 contains improved abstract domains able to represent data more precise and more complete. Furthermore, the Pointer Inference analysis was reworked to be a bottom-up analysis and an additional function signature analysis step was added to the analysis pipeline. These improvements allow all analyses depending on the Pointer Inference to be both more precise and more complete. Other highlights include: - A new command line flag for analyzing bare-metal binaries. - The check for *CWE-78: Command line injections* was completely rewritten using abstract domains for strings. - The check for *CWE-119: Buffer Overflow* was completely rewritten and now emits additional data flow information in the JSON output to help with root cause analysis. - The check for *CWE-416: Use After Free* was completely rewritten and now emits additional data flow information in the JSON output to help with root cause analysis. See the CHANGES.md for more details. 2022-06-13T05:48:56+00:00 OpenTAXII 0.9.0 OpenTAXII 0.9.0 2022-06-13T14:44:11+00:00 Changelog ========= 0.9.0 (2022-06-13) ------------------ * Allow custom properties. This can be disabled by config option ``allow_custom_properties`` 0.8.0 (2022-06-05) ------------------ * Nest details inside taxii2 job and allow counts without details 0.7.0 (2022-05-27) ------------------ * Nest taxii2 endpoints under `/taxii2/` 0.6.0 (2022-05-25) ------------------ * Add `public_discovery` option to taxii2 config * Add support for publicly readable taxii 2 api roots 0.5.0 (2022-05-24) ------------------ * Add support for publicly readable taxii 2 collections 0.4.0 (2022-05-20) ------------------ * Move next_param handling into `OpenTAXII2PersistenceAPI` 0.3.0 (2022-04-13) ------------------ * Implement taxii2.1 support 0.3.0a4 (2022-04-13) -------------------- * Merge changes from 0.2.4 maintenance release 0.3.0a3 (2022-01-21) -------------------- * Fix bug that prevented booting with only taxii1 config (`#217 <https://github.com/eclecticiq/OpenTAXII/issues/217>`_ thanks `@azurekid <https://github.com/azurekid>`_ for the report) 0.3.0a2 (2021-12-27) -------------------- * Merge changes from 0.2.3 maintenance release 0.3.0a1 ------- * Add python 3.10 support 0.3.0a0 ------- * Enablement for future taxii2 implementation * Fix documentation build issues 0.2.4 (2022-04-13) ------------------ * Make sure werkzeug <2.1 and >=2.1 work correctly with auth system 0.2.3 (2021-12-22) ------------------ * Fix bug in multithreaded use of sqlite (`#210 <https://github.com/eclecticiq/OpenTAXII/issues/210>`_ thanks `@rohits144 <https://github.com/rohits144>`_ for the report) 0.2.2 (2021-11-05) ------------------ * Fix readthedocs build 0.2.1 (2021-11-03) ------------------ * Add tests for python 3.6, 3.7, 3.8, 3.9, pypy * Add tests for sqlite, mysql, mariadb, postgresql * Fix bug that broke ``delete_content_blocks`` when using mysql on sqlalchemy 1.3 * Docs: Add db schema diagram * Docs: Clarify how to get default data in a default (development) docker instance * Fix implicit routing in TAXII 1.1 Inboxes * Update jwt usage to pyjwt >= 2.0 (thanks `@SanyaKapoor <https://github.com/SanyaKapoor>`_) 0.2.0 (2020-06-30) ------------------ * Enforce UTC usage in datetime fields in SQL DB Persistence API. * `Fix for #114 <https://github.com/eclecticiq/OpenTAXII/issues/114>`_: reintroduce ``opentaxii-create-account`` CLI command. * `Fix for #153 <https://github.com/eclecticiq/OpenTAXII/issues/152>`_: check if user can modify a collection before advertising it over inbox service. * Multiple coding style fixes. * Various documentation updates. 0.1.12 (2019-03-06) ------------------- * Remove unnecessary print statements. 0.1.11 (2019-02-13) ------------------- * Make JSON logging consistent when the application is run via Gunicorn. * Set ``acceptable_destination`` key in status details instead of extended headers * Allow passing ``engine_parameters`` to ``SQLDatabaseAPI`` for those who want to customize SQLAlchemy engine parameters. * Require recent version of ``lxml`` for security reasons. * Various test and Docker infrastructure improvements. 0.1.10 (2018-06-03) ------------------- * Replace separate service/collection/account creation process with single ``opentaxii-sync-data`` CLI command. * Persistence and Auth APIs extended with missing CRUD methods, that are used by ``opentaxii-sync-data``. * Read/modify collection level ACL added. * DB models for default implementation of Persistence API and Auth API were changed. No automatic migration code is provided (sorry!), so upgrading might require manual DB migration. * Drop python2.7 from testing scope. * Various bug fixes and improvements. 0.1.9 (2017-06-19) ------------------ * `libtaxii <https://github.com/TAXIIProject/libtaxii>`_ dependency upgraded to 1.1.111. * Various bug fixes and improvements (thanks to `@bjigmp <https://github.com/bjigmp>`_, `@chorsley <https://github.com/chorsley>`_, `@rjprins <https://github.com/rjprins>`_). 0.1.8 (2017-02-21) ------------------ * Ability to enable/disable "huge trees" support in XML parser. Configuration property ``xml_parser_supports_huge_tree`` set to ``yes`` or ``true`` will disable security restrictions and force XML parser to support very deep trees and very long text content. * Adjust SQL Persistence API implemetation so it works smoothly with MySQL backend. * Use Python 3.5 instead of Python 3.4 for tests. 0.1.7 (2016-10-18) ------------------ * Minor fixes. * Dependencies were changed from hard-pinned to more flexible. * Example of production DB configuration added to docs. 0.1.6 (2016-06-01) ------------------ * Python 3.4 compatibility of the codebase. Tox configuration extended with python 3.4 environment run. * Flake8 full style compatibility. Flake8 check added to Tox configuration. * SQLAlchemy session scope issue fixed (related to `#38 <https://github.com/EclecticIQ/OpenTAXII/issues/38>`_). * `opentaxii-delete-blocks` CLI command added (related to `#45 <https://github.com/EclecticIQ/OpenTAXII/issues/45>`_). * `delete_content_blocks` method `added <https://github.com/EclecticIQ/OpenTAXII/commit/dc6fddc27a98e8450c7e05e583b2bfb741f6e197#diff-6814849ac352b2b74132f8fa52e0ec4eR213>`_ to Persistence API. * Collection's name is `required <https://github.com/EclecticIQ/OpenTAXII/commit/dc6fddc27a98e8450c7e05e583b2bfb741f6e197#diff-ce3f7b939e5c540480ac655aef32c513R116>`_ to be unique in default SQL DB Auth API implementation. 0.1.5 (2016-03-15) ------------------ * Fix for the issue with persistence backend returning ``None`` instead of ``InboxMessage`` object 0.1.4 (2016-02-25) ------------------ * Hard-coded dependencies in ``setup.py`` removed. 0.1.3 (2016-02-25) ------------------ * Versions of dependencies are pinned. * Code adjusted for a new version of `anyconfig <https://pypi.python.org/pypi/anyconfig>`_ API. * Test for configuration loading added. 0.1.2 (2015-07-24) ------------------ * Docker configuration added. * Health check endpoint added. * Basic authentication support added. * Temporary workaround for `Issue #191 <https://github.com/TAXIIProject/libtaxii/issues/191>`_. * Method ``get_domain`` in Persistence API returns domain value configured for ``service_id``. If nothing returned, value set in configuration file will be used. * Performance optimisations. * Bug fixes and style improvements. 0.1.1 (2015-04-08) ------------------ * Alias for Root Logger added to logging configuration. * Context object in a request scope that holds account and token added. * Support for OPTIONS HTTP request to enable auto version negotiation added. * Documentation improved. 0.1.0 (2015-03-31) ------------------ * Initial release 2022-06-13T14:44:11+00:00 syncthing v1.20.3-rc.1 syncthing v1.20.3-rc.1 2022-06-14T12:24:10+00:00 Bugfixes: - #8376: Chrome Autofill Breaks Authentication 2022-06-14T12:24:10+00:00 MONARC v2.11.1-p2 MONARC v2.11.1-p2 2022-06-15T06:56:04+00:00 2022-06-15T06:56:04+00:00 monarc-stats-service v0.5.0 monarc-stats-service v0.5.0 2022-06-20T12:27:47+00:00 - chg: [documentation] Updated documentation (9f38db11b382d8516fb71b60154aa0c7ba77004c); - chg: [API] CLIENT_REGISTRATION_OPEN is now set to True by default (b277436f81bbac1445822f0399dc348c5e283f70); - fix: [security] prevent the creation of new admin users (even by an admin) (257c16fed890bda8974594238a743f8afda0ff5a); - fix: replaced after_request by before_request for the API (88a276bf4d5f35e4e5da6ac065e1eb62f2892670); - small codebase fix for container image (82cdeaa714dbff14b9068c0b65c302ec4d02b3c4); - dockerfile and build pipeline (f8c663b61e1c8475d0f17060690d9920a1cc9e90); - updated Python dependencies. 2022-06-20T12:27:47+00:00 MONARC v2.12.0 MONARC v2.12.0 2022-06-20T12:44:13+00:00 ### New - [compliance scale](https://github.com/monarc-project/MonarcAppFO/discussions/439) - [metadata assets](https://github.com/monarc-project/MonarcAppFO/discussions/437) - [two-factor authentication](https://github.com/monarc-project/MonarcAppFO/discussions/442) - new build deployment is available and based on GitHub Actions ### Fix - [Stats provider] removed the leading slash in the URI ([e7dfba1](https://github.com/monarc-project/zm-client/commit/e7dfba1cf64322bc3e83630df6729b525d7d5c8d)) Details about upcoming related releases: https://github.com/orgs/monarc-project/projects/3 2022-06-20T12:44:13+00:00 monarc-stats-service v0.5.1 monarc-stats-service v0.5.1 2022-06-21T21:59:16+00:00 Changes ~~~~~~~ - [dependenvies] Updated Python dependencies. 2022-06-21T21:59:16+00:00 MONARC v2.12.1 MONARC v2.12.1 2022-06-22T07:39:21+00:00 2022-06-22T07:39:21+00:00 TheHive 4.1.21 TheHive 4.1.21 2022-06-22T11:21:10+00:00 ## [4.1.21](https://github.com/TheHive-Project/TheHive/milestone/91) (2022-06-22) **Fixed bugs:** - [Bug] S3 storage fails with old version of Minio [\#2388](https://github.com/TheHive-Project/TheHive/issues/2388) - [Bug ] Authentication Bypass Vulnerability [\#2391](https://github.com/TheHive-Project/TheHive/issues/2391) 2022-06-22T11:21:10+00:00 Lookyloo v1.13.0 Lookyloo v1.13.0 2022-06-26T16:06:50+00:00 # Maintenance and bug-fixes release All releases don't need to contain new features, sometimes, it is just some cleanup, and it is okay. * Properly handle exceptions in some edge cases (fixes in har2tree) * Properly display an error message if the capture fails * Use the same default User-Agent in when a capture is submitted via the API as via the web interface. * Cleanup some legacy code * Bump all dependencies (JS/CSS and Python) # Still, there is a new-ish thing We revamped the package generator, and it should be [more usable](https://github.com/Lookyloo/lookyloo/pkgs/container/lookyloo). If it is not, let us know! 2022-06-26T16:06:50+00:00 DC3-MWCP 3.7.0 DC3-MWCP 3.7.0 2022-06-28T13:39:42+00:00 2022-06-28T13:39:42+00:00 syncthing v1.20.3-rc.2 syncthing v1.20.3-rc.2 2022-06-29T06:28:24+00:00 Bugfixes: - #8369: Shared device names are missing from "Edit Folder -> Sharing" - #8376: Chrome Autofill Breaks Authentication - #8386: Ignore patterns with wildcard and non-ASCII characters don't work as expected Enhancements: - #8393: Warn if two devices are introducers to each other 2022-06-29T06:28:24+00:00 MONARC v2.12.2 MONARC v2.12.2 2022-06-29T07:20:25+00:00 2022-06-29T07:20:25+00:00 pandora v1.0.0 pandora v1.0.0 2022-06-29T16:30:21+00:00 This is the first official stable open source release of [Pandora](https://github.com/pandora-analysis/pandora). Pandora is an analysis framework to discover if a file is suspicious and conveniently show the results. The solution can be installed on-prem to avoid information leak in organisations. It has been tested relatively extensively over the last few months, but there might still be issues. If anything goes wrong, please open an issue and we will do our best to solve it. If you want to test Pandora without installing it, the online version is available at [pandora.circl.lu](https://pandora.circl.lu/). # Core functionalities * Easy to implement workers to analyze specific file formats, or connect to third party services * Admin interface * Session-based user management interface, and sharing * Generate a preview of the submitted document (if applicable) * Extract indicators/observables from submitted files * Extract content of archives * Extract attachments from email in EML and MSG format * Extract text content (if applicable) * Extract EXIF metadata * Pool service to fetch emails from an IMAP mailbox * Notify Administrator * MISP export and submission (admin only) * Statistics (admin only) * Role management (admin only) * Locally defined observables (legitimate/suspicious) (admin only) # Screenshots ## Submission interface ![pandora1](https://user-images.githubusercontent.com/248875/176490605-28763966-f924-4fad-855e-017f88f1954a.png) ## Result page ![pandora2](https://user-images.githubusercontent.com/248875/176490653-d0a1c658-1dc5-4b90-90e5-a565404f2447.png) --------------- ![pandora3](https://user-images.githubusercontent.com/248875/176492185-b4bb61cc-78c0-4146-8ff8-447fbf719a0c.png) 2022-06-29T16:30:21+00:00 maltrail 0.47 maltrail 0.47 2022-06-30T22:11:05+00:00 Start-of-month release 2022-06-30T22:11:05+00:00 TheHive 4.1.22 TheHive 4.1.22 2022-07-01T15:00:56+00:00 ## [4.1.22](https://github.com/TheHive-Project/TheHive/milestone/93) (2022-07-01) **Implemented enhancements:** - [Enhancement] Add check on user role [\#2401](https://github.com/TheHive-Project/TheHive/issues/2401) **Fixed bugs:** - [Bug] Use dedicated stream topic for stream dispatcher subscription [\#2400](https://github.com/TheHive-Project/TheHive/issues/2400) 2022-07-01T15:00:56+00:00 monarc-stats-service v0.5.2 monarc-stats-service v0.5.2 2022-07-04T08:41:37+00:00 ## Changes - [API] patch on client now expects again a model from Namespace (client_ns). - Updated Python dependencies. - [documentation] Updated links to documentation. - [documentation] Updated information about installation. - [deployment] added docker-compose.yml file. ## Fix - [API] enable patch method for enabling/disabling stats sharing. 2022-07-04T08:41:37+00:00 MONARC v2.12.2-p1 MONARC v2.12.2-p1 2022-07-04T08:56:31+00:00 2022-07-04T08:56:31+00:00 osquery 5.4.0 osquery 5.4.0 2022-07-06T21:20:21+00:00 2022-07-06T21:20:21+00:00 GDPRDPIAT v3.0.1 GDPRDPIAT v3.0.1 2022-07-07T10:35:43+00:00 ## What's Changed * Bugfixes/surveyjs cdn by @simonarnell in https://github.com/simonarnell/GDPRDPIAT/pull/6 **Full Changelog**: https://github.com/simonarnell/GDPRDPIAT/compare/v3.0.0...v3.0.1 2022-07-07T10:35:43+00:00 MONARC v2.12.2-p2 MONARC v2.12.2-p2 2022-07-07T10:44:13+00:00 Fixed an issue with sortable 1.15.0 https://github.com/monarc-project/ng-client/commit/4d8e61e56edf7b858db68381f2b99389b3921866 2022-07-07T10:44:13+00:00 MONARC v2.12.2-p3 MONARC v2.12.2-p3 2022-07-07T12:42:37+00:00 2022-07-07T12:42:37+00:00 pandora v1.0.1 pandora v1.0.1 2022-07-11T14:44:26+00:00 Quick release to improve handling of submissions with passwords. # New features * Support password on submit via API. # Maintenance * Bugfixes * Dependencies update. 2022-07-11T14:44:26+00:00 OpenTAXII 0.9.1 OpenTAXII 0.9.1 2022-07-11T20:28:28+00:00 Changelog ========= 0.9.1 (2022-07-11) ------------------ * Implement `raise_unauthorized` for taxii2, this was missing and lead to 500 errors. 0.9.0 (2022-06-13) ------------------ * Allow custom properties. This can be disabled by config option ``allow_custom_properties`` 0.8.0 (2022-06-05) ------------------ * Nest details inside taxii2 job and allow counts without details 0.7.0 (2022-05-27) ------------------ * Nest taxii2 endpoints under `/taxii2/` 0.6.0 (2022-05-25) ------------------ * Add `public_discovery` option to taxii2 config * Add support for publicly readable taxii 2 api roots 0.5.0 (2022-05-24) ------------------ * Add support for publicly readable taxii 2 collections 0.4.0 (2022-05-20) ------------------ * Move next_param handling into `OpenTAXII2PersistenceAPI` 0.3.0 (2022-04-13) ------------------ * Implement taxii2.1 support 0.3.0a4 (2022-04-13) -------------------- * Merge changes from 0.2.4 maintenance release 0.3.0a3 (2022-01-21) -------------------- * Fix bug that prevented booting with only taxii1 config (`#217 <https://github.com/eclecticiq/OpenTAXII/issues/217>`_ thanks `@azurekid <https://github.com/azurekid>`_ for the report) 0.3.0a2 (2021-12-27) -------------------- * Merge changes from 0.2.3 maintenance release 0.3.0a1 ------- * Add python 3.10 support 0.3.0a0 ------- * Enablement for future taxii2 implementation * Fix documentation build issues 0.2.4 (2022-04-13) ------------------ * Make sure werkzeug <2.1 and >=2.1 work correctly with auth system 0.2.3 (2021-12-22) ------------------ * Fix bug in multithreaded use of sqlite (`#210 <https://github.com/eclecticiq/OpenTAXII/issues/210>`_ thanks `@rohits144 <https://github.com/rohits144>`_ for the report) 0.2.2 (2021-11-05) ------------------ * Fix readthedocs build 0.2.1 (2021-11-03) ------------------ * Add tests for python 3.6, 3.7, 3.8, 3.9, pypy * Add tests for sqlite, mysql, mariadb, postgresql * Fix bug that broke ``delete_content_blocks`` when using mysql on sqlalchemy 1.3 * Docs: Add db schema diagram * Docs: Clarify how to get default data in a default (development) docker instance * Fix implicit routing in TAXII 1.1 Inboxes * Update jwt usage to pyjwt >= 2.0 (thanks `@SanyaKapoor <https://github.com/SanyaKapoor>`_) 0.2.0 (2020-06-30) ------------------ * Enforce UTC usage in datetime fields in SQL DB Persistence API. * `Fix for #114 <https://github.com/eclecticiq/OpenTAXII/issues/114>`_: reintroduce ``opentaxii-create-account`` CLI command. * `Fix for #153 <https://github.com/eclecticiq/OpenTAXII/issues/152>`_: check if user can modify a collection before advertising it over inbox service. * Multiple coding style fixes. * Various documentation updates. 0.1.12 (2019-03-06) ------------------- * Remove unnecessary print statements. 0.1.11 (2019-02-13) ------------------- * Make JSON logging consistent when the application is run via Gunicorn. * Set ``acceptable_destination`` key in status details instead of extended headers * Allow passing ``engine_parameters`` to ``SQLDatabaseAPI`` for those who want to customize SQLAlchemy engine parameters. * Require recent version of ``lxml`` for security reasons. * Various test and Docker infrastructure improvements. 0.1.10 (2018-06-03) ------------------- * Replace separate service/collection/account creation process with single ``opentaxii-sync-data`` CLI command. * Persistence and Auth APIs extended with missing CRUD methods, that are used by ``opentaxii-sync-data``. * Read/modify collection level ACL added. * DB models for default implementation of Persistence API and Auth API were changed. No automatic migration code is provided (sorry!), so upgrading might require manual DB migration. * Drop python2.7 from testing scope. * Various bug fixes and improvements. 0.1.9 (2017-06-19) ------------------ * `libtaxii <https://github.com/TAXIIProject/libtaxii>`_ dependency upgraded to 1.1.111. * Various bug fixes and improvements (thanks to `@bjigmp <https://github.com/bjigmp>`_, `@chorsley <https://github.com/chorsley>`_, `@rjprins <https://github.com/rjprins>`_). 0.1.8 (2017-02-21) ------------------ * Ability to enable/disable "huge trees" support in XML parser. Configuration property ``xml_parser_supports_huge_tree`` set to ``yes`` or ``true`` will disable security restrictions and force XML parser to support very deep trees and very long text content. * Adjust SQL Persistence API implemetation so it works smoothly with MySQL backend. * Use Python 3.5 instead of Python 3.4 for tests. 0.1.7 (2016-10-18) ------------------ * Minor fixes. * Dependencies were changed from hard-pinned to more flexible. * Example of production DB configuration added to docs. 0.1.6 (2016-06-01) ------------------ * Python 3.4 compatibility of the codebase. Tox configuration extended with python 3.4 environment run. * Flake8 full style compatibility. Flake8 check added to Tox configuration. * SQLAlchemy session scope issue fixed (related to `#38 <https://github.com/EclecticIQ/OpenTAXII/issues/38>`_). * `opentaxii-delete-blocks` CLI command added (related to `#45 <https://github.com/EclecticIQ/OpenTAXII/issues/45>`_). * `delete_content_blocks` method `added <https://github.com/EclecticIQ/OpenTAXII/commit/dc6fddc27a98e8450c7e05e583b2bfb741f6e197#diff-6814849ac352b2b74132f8fa52e0ec4eR213>`_ to Persistence API. * Collection's name is `required <https://github.com/EclecticIQ/OpenTAXII/commit/dc6fddc27a98e8450c7e05e583b2bfb741f6e197#diff-ce3f7b939e5c540480ac655aef32c513R116>`_ to be unique in default SQL DB Auth API implementation. 0.1.5 (2016-03-15) ------------------ * Fix for the issue with persistence backend returning ``None`` instead of ``InboxMessage`` object 0.1.4 (2016-02-25) ------------------ * Hard-coded dependencies in ``setup.py`` removed. 0.1.3 (2016-02-25) ------------------ * Versions of dependencies are pinned. * Code adjusted for a new version of `anyconfig <https://pypi.python.org/pypi/anyconfig>`_ API. * Test for configuration loading added. 0.1.2 (2015-07-24) ------------------ * Docker configuration added. * Health check endpoint added. * Basic authentication support added. * Temporary workaround for `Issue #191 <https://github.com/TAXIIProject/libtaxii/issues/191>`_. * Method ``get_domain`` in Persistence API returns domain value configured for ``service_id``. If nothing returned, value set in configuration file will be used. * Performance optimisations. * Bug fixes and style improvements. 0.1.1 (2015-04-08) ------------------ * Alias for Root Logger added to logging configuration. * Context object in a request scope that holds account and token added. * Support for OPTIONS HTTP request to enable auto version negotiation added. * Documentation improved. 0.1.0 (2015-03-31) ------------------ * Initial release 2022-07-11T20:28:28+00:00 caddy v2.5.2 caddy v2.5.2 2022-07-12T19:06:28+00:00 This patch release fixes bugs, adds some new features, and makes worthwhile enhancements. We recommend everyone test and upgrade! Many improvements have been made to the `reverse_proxy` module. Highlights: - **New [`/adapt` admin endpoint](https://caddyserver.com/docs/api#post-adapt):** Use your installed config adapters via API in addition to the existing `caddy adapt` CLI command. - **New `Etag`/`If-Match` support for config API:** Safely update your config concurrently and avoid collisions by using [our unique Etag implementation](https://caddyserver.com/docs/api#concurrent-config-changes). - **Rename copied headers from reverse_proxy:** If you're using `handle_response`, you can more easily map headers to a different name for clients. - **Many HTTP matchers have been added to CEL:** You can now use the logic of our HTTP request matchers in CEL expressions. - **Notable bug fixes:** EAB reuse, various QUIC & HTTP/3 fixes, more specific HTTP status codes, various reverse proxy fixes. ## Changelog * 660c59b6 admin: Implement /adapt endpoint (close #4465) (#4846) * ad3a83fb admin: expect quoted ETags (#4879) * f259ed52 admin: support ETag on config endpoints (#4579) * 1498132e caddyhttp: Log error from CEL evaluation (fix #4832) * 0a14f97e caddytls: Make peer certificate verification pluggable (#4389) * 412dcc07 caddytls: Reuse issuer between PreCheck and Issue (#4866) * 499ad6d1 core: Micro-optim in run() (#4810) * c0f76e9e fileserver: Use safe redirects in file browser * 58e05cab forwardauth: Fix case when `copy_headers` is omitted (#4856) * 0b6f7643 forwardauth: Support renaming copied headers, block support (#4783) * 8bac134f go.mod: Bump up quic-go to v0.28.0, fixes for BC breaks (#4867) * 3d18bc56 go.mod: Update go-yaml to v3 * 56013934 go.mod: Update some dependencies * 8e6bc360 go.mod: Upgrade some dependencies * 53c4d788 headers: Only replace known placeholders (#4880) * 0bcd02d5 headers: Support wildcards for delete ops (close #4830) (#4831) * 58970cae httpcaddyfile: Add `{err.*}` placeholder shortcut (#4798) * b687d7b9 httpcaddyfile: Support multiple values for `default_bind` (#4774) * a9267791 reverseproxy: Add --internal-certs CLI flag #3589 (#4817) * aaf6794b reverseproxy: Add renegotiation param in TLS client (#4784) * 54d1923c reverseproxy: Adjust new TLS Caddyfile directive names (#4872) * 7f9b1f43 reverseproxy: Correct the `tls_server_name` docs (#4827) * c82fe911 reverseproxy: Dynamic ServerName for TLS upstreams (#4836) * d6bc9e0b reverseproxy: Err 503 if all upstreams unavailable * 98468af8 reverseproxy: Fix double headers in response handlers (#4847) * 25f10511 reverseproxy: Fix panic when TLS is not configured (#4848) * 5e729c1e reverseproxy: HTTP 504 for upstream timeouts (#4824) * f9b42c37 reverseproxy: Make TLS renegotiation optional * b6e96fa3 reverseproxy: Skip TLS for certain configured ports (#4843) * 57d27c1b reverseproxy: Support http1.1>h2c (close #4777) (#4778) * 9864b138 reverseproxy: api: Remove misleading 'healthy' value * 693e9b52 rewrite: Handle fragment before query (fix #4775) * 6891f7f4 templates: Add `humanize` function (#4767) * 9e760e2e templates: Documentation consistency (#4796) ## New Contributors * @nekohasekai made their first contribution in https://github.com/caddyserver/caddy/pull/4782 * @davidbgk made their first contribution in https://github.com/caddyserver/caddy/pull/4796 * @git001 made their first contribution in https://github.com/caddyserver/caddy/pull/4767 * @varianone made their first contribution in https://github.com/caddyserver/caddy/pull/4817 * @Gr33nbl00d made their first contribution in https://github.com/caddyserver/caddy/pull/4389 * @yaslama made their first contribution in https://github.com/caddyserver/caddy/pull/4784 * @kresike made their first contribution in https://github.com/caddyserver/caddy/pull/4836 * @TristonianJones made their first contribution in https://github.com/caddyserver/caddy/pull/4715 * @jhwz made their first contribution in https://github.com/caddyserver/caddy/pull/4579 **Full Changelog**: https://github.com/caddyserver/caddy/compare/v2.5.1...v2.5.2 2022-07-12T19:06:28+00:00 AIL-framework v4.2 AIL-framework v4.2 2022-07-16T08:40:47+00:00 ## AIL Framework version 4.2 released including typo squatting tracker, improved AIL2AIL sync, zerobinz fetcher and many bugs fixes v4.2 (2022-06-24) AIL Framework version 4.2 has been released including: - A new tracker for tracking potential typo squatted domains. This feature relies on the new [ail-typo-squatting ](https://github.com/ail-project/ail-typo-squatting) library which can be also used outside of AIL framework. This contribution is from @DavidCruciani - Many improvement and bugs fixed for the AIL2AIL sync. A huge thanks to @aaronkaplan from EU Directorate-General for Informatics (DIGIT) for support and tests during the long debugging sessions. - A new module for zerobinz to create an immediate crawler request if a zerobinz link appears in an item. The module can be used for other services with ephemeral content. Thanks to @gallypette for the contribution and the improvement ideas. - A new hosts detection module has been introduced. - Multiple bugs were fixed. ### Detailed Changes * [Tracker} Tracker_Typo_Squatting. [David Cruciani] * [v4.2] add v4.2 update. [Terrtia] * [investigation] fix investigation by user + delete an obj from all investigation. [Terrtia] * [install vitualenv] remove travis env. [Terrtia] * [Retro Hunt] add logs. [Terrtia] * [Retro Hunt] add logs. [Terrtia] * [Retro Hunt] add logs. [Terrtia] * [AIL2AIL Sync] update exchange format. [Terrtia] * [AIL2AIL Sync] update exchange format. [Terrtia] * [add Hosts module] [Terrtia] * [sync module] debug. [Terrtia] * [sync client] debug. [Terrtia] * [websockets client] bind client ip. [Terrtia] * [websocket server] add host and port config. [Terrtia] * [telegram importer] add username correlation. [Terrtia] * [UI subtype objs] get obj by subtype + name. [Terrtia] * [misp export] add username. [Terrtia] ### Fix * [typosquatting] remove unused import. [Thirion Aurélien] * [tracker] clean import. [Thirion Aurélien] * [tracker term] fix typosquatting key. [Thirion Aurélien] * [Typo] tracker typo. [David Cruciani] * [tracker] UI for other than typosquat. [David Cruciani] * [typo] UI. [David Cruciani] * [Language] fix cld3 import. [Terrtia] * [launcher] kill AIL_2_AIL screen. [Terrtia] * [cld3] enable cld3. [Terrtia] * [cld3 python3.10] temp disable cld3. [Terrtia] * [launcher] remove Travis test. [Terrtia] * [Retro Hunt] item directory. [Terrtia] * [Retro Hunt] item directory. [Terrtia] * [Retro Hunt] fix item directory. [Terrtia] * [AIL exchange mime-type] [Terrtia] * [Hosts module] module + launcher. [Terrtia] * [abstract module] exception traceback #145. [Terrtia] * [ui tag selector] force custom tags. [Terrtia] * [installer] remove old tor install. [Terrtia] * [sync module] fix redis tag queue. [Terrtia] * [sync module] fix tags filter. [Terrtia] * [sync client] debug. [Terrtia] * [sync client] debug. [Terrtia] * [sync module] debug. [Terrtia] * [websockets client] fix client bind. [Terrtia] * [websockets] remove size limit. [Terrtia] * [UI subtype objs] fix form. [Terrtia] * [misp config] https. [Thirion Aurélien] ### Other * Merge pull request #147 from ail-project/typo. [Thirion Aurélien] Integration of the typo-squatting tracker * Fix; [set tracker] missing function. [Thirion Aurélien] * Merge branch 'master' into typo. [David Cruciani] * Add: [tracker] typo-squatting. [David Cruciani] * Merge branch 'master' of github.com:ail-project/ail-framework. [Terrtia] * Merge pull request #146 from gallypette/master. [Thirion Aurélien] add: [modules] zerobinz * Add: [modules] zerobinz. [huynenjl@gmail.com] * Merge branch 'master' of github.com:ail-project/ail-framework. [Terrtia] 2022-07-16T08:40:47+00:00 rocksdb v7.4.3 rocksdb v7.4.3 2022-07-18T15:18:20+00:00 ## 7.4.3 (07/13/2022) ### Behavior Changes * For track_and_verify_wals_in_manifest, revert to the original behavior before #10087: syncing of live WAL file is not tracked, and we track only the synced sizes of **closed** WALs. (PR #10330). ## 7.4.2 (06/30/2022) ### Bug Fixes * Fix a bug in Logger where if dbname and db_log_dir are on different filesystems, dbname creation would fail wrt to db_log_dir path returning an error and fails to open the DB. ## 7.4.1 (06/28/2022) ### Bug Fixes * Pass `rate_limiter_priority` through filter block reader functions to `FileSystem`. ## 7.4.0 (06/19/2022) ### Bug Fixes * Fixed a bug in calculating key-value integrity protection for users of in-place memtable updates. In particular, the affected users would be those who configure `protection_bytes_per_key > 0` on `WriteBatch` or `WriteOptions`, and configure `inplace_callback != nullptr`. * Fixed a bug where a snapshot taken during SST file ingestion would be unstable. * Fixed a bug for non-TransactionDB with avoid_flush_during_recovery = true and TransactionDB where in case of crash, min_log_number_to_keep may not change on recovery and persisting a new MANIFEST with advanced log_numbers for some column families, results in "column family inconsistency" error on second recovery. As a solution, RocksDB will persist the new MANIFEST after successfully syncing the new WAL. If a future recovery starts from the new MANIFEST, then it means the new WAL is successfully synced. Due to the sentinel empty write batch at the beginning, kPointInTimeRecovery of WAL is guaranteed to go after this point. If future recovery starts from the old MANIFEST, it means the writing the new MANIFEST failed. We won't have the "SST ahead of WAL" error. * Fixed a bug where RocksDB DB::Open() may creates and writes to two new MANIFEST files even before recovery succeeds. Now writes to MANIFEST are persisted only after recovery is successful. * Fix a race condition in WAL size tracking which is caused by an unsafe iterator access after container is changed. * Fix unprotected concurrent accesses to `WritableFileWriter::filesize_` by `DB::SyncWAL()` and `DB::Put()` in two write queue mode. * Fix a bug in WAL tracking. Before this PR (#10087), calling `SyncWAL()` on the only WAL file of the db will not log the event in MANIFEST, thus allowing a subsequent `DB::Open` even if the WAL file is missing or corrupted. * Fix a bug that could return wrong results with `index_type=kHashSearch` and using `SetOptions` to change the `prefix_extractor`. * Fixed a bug in WAL tracking with wal_compression. WAL compression writes a kSetCompressionType record which is not associated with any sequence number. As result, WalManager::GetSortedWalsOfType() will skip these WALs and not return them to caller, e.g. Checkpoint, Backup, causing the operations to fail. * Avoid a crash if the IDENTITY file is accidentally truncated to empty. A new DB ID will be written and generated on Open. * Fixed a possible corruption for users of `manual_wal_flush` and/or `FlushWAL(true /* sync */)`, together with `track_and_verify_wals_in_manifest == true`. For those users, losing unsynced data (e.g., due to power loss) could make future DB opens fail with a `Status::Corruption` complaining about missing WAL data. * Fixed a bug in `WriteBatchInternal::Append()` where WAL termination point in write batch was not considered and the function appends an incorrect number of checksums. * Fixed a crash bug introduced in 7.3.0 affecting users of MultiGet with `kDataBlockBinaryAndHash`. * Add some fixes in async_io which was doing extra prefetching in shorter scans. ### Public API changes * Add new API GetUnixTime in Snapshot class which returns the unix time at which Snapshot is taken. * Add transaction `get_pinned` and `multi_get` to C API. * Add two-phase commit support to C API. * Add `rocksdb_transaction_get_writebatch_wi` and `rocksdb_transaction_rebuild_from_writebatch` to C API. * Add `rocksdb_options_get_blob_file_starting_level` and `rocksdb_options_set_blob_file_starting_level` to C API. * Add `blobFileStartingLevel` and `setBlobFileStartingLevel` to Java API. * Add SingleDelete for DB in C API * Add User Defined Timestamp in C API. * `rocksdb_comparator_with_ts_create` to create timestamp aware comparator * Put, Get, Delete, SingleDelete, MultiGet APIs has corresponding timestamp aware APIs with suffix `with_ts` * And Add C API's for Transaction, SstFileWriter, Compaction as mentioned [here](https://github.com/facebook/rocksdb/wiki/User-defined-Timestamp-(Experimental)) * The contract for implementations of Comparator::IsSameLengthImmediateSuccessor has been updated to work around a design bug in `auto_prefix_mode`. * The API documentation for `auto_prefix_mode` now notes some corner cases in which it returns different results than `total_order_seek`, due to design bugs that are not easily fixed. Users using built-in comparators and keys at least the size of a fixed prefix length are not affected. * Obsoleted the NUM_DATA_BLOCKS_READ_PER_LEVEL stat and introduced the NUM_LEVEL_READ_PER_MULTIGET and MULTIGET_COROUTINE_COUNT stats * Introduced `WriteOptions::protection_bytes_per_key`, which can be used to enable key-value integrity protection for live updates. ### New Features * Add FileSystem::ReadAsync API in io_tracing * Add blob garbage collection parameters `blob_garbage_collection_policy` and `blob_garbage_collection_age_cutoff` to both force-enable and force-disable GC, as well as selectively override age cutoff when using CompactRange. * Add an extra sanity check in `GetSortedWalFiles()` (also used by `GetLiveFilesStorageInfo()`, `BackupEngine`, and `Checkpoint`) to reduce risk of successfully created backup or checkpoint failing to open because of missing WAL file. * Add a new column family option `blob_file_starting_level` to enable writing blob files during flushes and compactions starting from the specified LSM tree level. * Add support for timestamped snapshots (#9879) * Provide support for AbortIO in posix to cancel submitted asynchronous requests using io_uring. * Add support for rate-limiting batched `MultiGet()` APIs ### Behavior changes * DB::Open(), DB::OpenAsSecondary() will fail if a Logger cannot be created (#9984) * Removed support for reading Bloom filters using obsolete block-based filter format. (Support for writing such filters was dropped in 7.0.) For good read performance on old DBs using these filters, a full compaction is required. * Per KV checksum in write batch is verified before a write batch is written to WAL to detect any corruption to the write batch (#10114). ### Performance Improvements * When compiled with folly (Meta-internal integration; experimental in open source build), improve the locking performance (CPU efficiency) of LRUCache by using folly DistributedMutex in place of standard mutex. 2022-07-18T15:18:20+00:00 rocksdb v7.4.4 rocksdb v7.4.4 2022-07-28T18:34:26+00:00 ## 7.4.4 (07/19/2022) ### Public API changes * Removed Customizable support for RateLimiter and removed its CreateFromString() and Type() functions. ### Bug Fixes * Fix a bug where `GenericRateLimiter` could revert the bandwidth set dynamically using `SetBytesPerSecond()` when a user configures a structure enclosing it, e.g., using `GetOptionsFromString()` to configure an `Options` that references an existing `RateLimiter` object. 2022-07-28T18:34:26+00:00 rocksdb v7.4.5 rocksdb v7.4.5 2022-08-02T23:17:54+00:00 ## 7.4.5 (08/02/2022) ### Bug Fixes * Fix a bug starting in 7.4.0 in which some fsync operations might be skipped in a DB after any DropColumnFamily on that DB, until it is re-opened. This can lead to data loss on power loss. (For custom FileSystem implementations, this could lead to `FSDirectory::Fsync` or `FSDirectory::Close` after the first `FSDirectory::Close`; Also, valgrind could report call to `close()` with `fd=-1`.) 2022-08-02T23:17:54+00:00 whids v1.8.0-beta.7 whids v1.8.0-beta.7 2022-08-03T12:33:11+00:00 2022-08-03T12:33:11+00:00 PyPCAPKit v0.16.2 PyPCAPKit v0.16.2 2022-08-04T02:55:47+00:00 2022-08-04T02:55:47+00:00 maltrail 0.48 maltrail 0.48 2022-08-04T06:36:06+00:00 Start-of-month release 2022-08-04T06:36:06+00:00 pcileech v4.15 pcileech v4.15 2022-08-04T06:39:53+00:00 * Support for MemProcFS v5.0 2022-08-04T06:39:53+00:00 MISP v2.4.160 MISP v2.4.160 2022-08-08T12:32:32+00:00 We are pleased to announce the immediate availability of MISP v2.4.160. With the August summer-holiday season kicking into high gear, we have a very special release for you all, containing a long list of major new features, improvements and general quality of life improvements. Unlike we do normally, this time around we're preparing separate blog posts for some of those major features, so follow the links below to read up on in-depth descriptions of each. # Workflows Something that has been in the works for quite some time now is finally hitting a release version of MISP, as of 2.4.160, we have the first release of the built in workflow system released. This system allows you to use an easy to use, yet extremely powerful graphical interface to modify how MISP handles certain tasks such as event publishing, user enrollment, synchronisation, etc., by adding additional logical steps in their respective executions, utilising a module system similar to what was already common to MISP from enrichment subsystems, exports as well as imports. This is merely the first step (or leap rather) towards customising and sharing custom workflows, stay tuned for new features, improvements as well as triggers and modules in the near future. Head over to the [README](https://github.com/MISP/misp-workflow-blueprints/blob/main/README.md) as well as a nifty [slide deck](https://www.misp-project.org/misp-training/a.12-misp-workflows.pdf), to find out what this incredibly powerful can do for you and your community. # New correlation engine One of the biggest pain points as of recently has been our dated and rather bloated correlation engine, which could easily bring a long running MISP instance to its knees when certain highly correlated data sources were synchronised. As of 2.4.160, we now have 2 brand new correlation engines at your disposal, with the old engine being retired immediately. Please be aware that upgrading to the current version will regenerate your correlations using the new engine, something that can take quite a long time (on our largest instance it took a whopping 40 hours!). With that said, we can assure you it's well worth the wait and should resolve several long standing performance bottlenecks as well as heavily cut down on the space requirements for your data. For more information, on the new engines, their differences, the various new support tools as well as what benefits you should expect, head over to the [dedicated blog post](https://github.com/MISP/MISP/blob/2.4/docs/correlations.rework.md). # STIX 2 library reworks There has been a massive amount of work going into the STIX 2.x library rework, bringing us closer and closer to having a full mapping of everything expressable. We're collaborating with CISA and Mitre to ensure that MISP can both express and understand STIX to its fullest extent. For more information, head over to the [release notes](https://github.com/MISP/misp-stix/releases/tag/v2.4.160) over on the MISP STIX library's repo. # Mermaid support for Event reports added Writing custom reports has become more and more popular, but one annoyance has been the lack of a way to depict graphs and flow charts without relying on external tools to create those (and share them as images for example). Using Mermaid, you now have a nifty tool to build graphs out of simple markdown directly in the event report editor. # Various other improvements A long list of other improvements, affecting the performance and stability of the platform as well as improvements to existing features. Head over to the changelog for a detailed list of changes. # Acknowledgement We would like to thank all the [contributors](https://www.misp-project.org/contributors), reporters and users who have helped us in the past months to improve MISP and information sharing at large. This release includes multiple updates in [misp-objects](https://www.misp-project.org/objects.html), [misp-taxonomies](https://www.misp-project.org/taxonomies.html) and [misp-galaxy](https://www.misp-project.org/galaxy.html). As always, a detailed and [complete changelog is available](https://www.misp-project.org/Changelog.txt) with all the fixes, changes and improvements in MISP core. Additional changelogs are available for [misp galaxy](https://www.misp-project.org/Changelog-misp-galaxy.txt), [misp-taxonomies](https://www.misp-project.org/Changelog-misp-taxonomies.txt), [misp-objects](https://www.misp-project.org/Changelog-misp-objects.txt) and [misp-modules](https://www.misp-project.org/Changelog-misp-modules.txt) 2022-08-08T12:32:32+00:00 Lookyloo v1.14.0 Lookyloo v1.14.0 2022-08-08T13:53:34+00:00 # New features * Trigger a capture on a web enabled document provided as a file instead of a URL. Useful for HTML files attached to emails, or HTML body in email. ![Screenshot_20220808_131607](https://user-images.githubusercontent.com/248875/183406016-e02f99bd-fe8c-47ea-906e-39e9da2280b6.png) -------- * Compress (gzip) the HAR file in archived captures - saves a lot of disk space. * Support for RiskIQ Passive DNS (requires API key) * Display SSL/TLS information available in the HAR dump from Playwright ![Screenshot_20220808_132643](https://user-images.githubusercontent.com/248875/183407809-4475d6ce-2311-43fe-bb79-8a0697bae78c.png) -------- * Optional DoNotTrack HTTP header in capture ![Screenshot_20220808_132302](https://user-images.githubusercontent.com/248875/183407193-06aaf9a0-8377-49e7-bceb-5cb9dadad6fb.png) -------- * Display size of rendered page on hostnode popup. * [WiP] Download files when the URL captured to a downloadable file (PDF, Office doc, ...) (**Important note** the downloaded file is not exposed to the user yet) * [WiP] List all hashes available in the capture, sort them by frequency. Makes it easier to find phishing sides using the same resources. ![Screenshot_20220808_132149](https://user-images.githubusercontent.com/248875/183407015-daf83393-e605-4f3c-a1a2-0d3885023422.png) # Fixes * Major speed improvements when displaying the hostnode popups (only show the recent cached captures by default) * Improvements in the caching mechanism * Cleanup data showed by monitoring script * Avoid crashes when RiskIQ isn't reachable # Changes * Update dependencies (js, python) * Improve logging in archiver * Improve config file 2022-08-08T13:53:34+00:00 whids v1.8.0-beta.8 whids v1.8.0-beta.8 2022-08-08T16:38:56+00:00 2022-08-08T16:38:56+00:00 MISP v2.4.161 MISP v2.4.161 2022-08-11T15:30:58+00:00 We are pleased to announce the immediate availability of [MISP v2.4.161](https://github.com/MISP/MISP/releases/tag/v2.4.161). ![](https://www.misp-project.org//img/blog/workflow.png) # Small improvements - A new option added to log the last API request of an API key. (Thanks to Tom King for the contribution) - Overcorrelation features have some new improvements such as: - A new tool to generate occurrence counts (real numbers this time) - A hook to truncate the over-correlating value table on recorrelation - We no longer store the partial counts as occurrences when generating correlations - Performance improvements in event fetching - Various performance tuning in the new correlation engine including the full recorrelation # Bugs fixed - `tlp:amber+strict` and `tlp:clear` are now valid tags - [stix2 import] Better `external_references` parsing for attack patterns objects Thanks to all the contributors and users reporting bugs to make the software better. As always, a detailed and [complete changelog is available](https://www.misp-project.org/Changelog.txt) with all the fixes, changes and improvements in MISP core. 2022-08-11T15:30:58+00:00 osquery 5.5.0 osquery 5.5.0 2022-08-12T17:47:19+00:00 draft 2022-08-12T17:47:19+00:00 dnstwist 20220815 dnstwist 20220815 2022-08-15T15:39:49+00:00 2022-08-15T15:39:49+00:00 dalton v3.2.4 dalton v3.2.4 2022-08-16T18:51:57+00:00 - New URL for downloading Suricata source code 2022-08-16T18:51:57+00:00 osquery 5.5.1 osquery 5.5.1 2022-08-18T13:24:43+00:00 Draft! (think 5.5.0 plus sqlite) 2022-08-18T13:24:43+00:00 rocksdb v7.5.3 rocksdb v7.5.3 2022-08-24T22:39:19+00:00 ## 7.5.2 (08/02/2022) ### Bug Fixes * Fix a bug starting in 7.4.0 in which some fsync operations might be skipped in a DB after any DropColumnFamily on that DB, until it is re-opened. This can lead to data loss on power loss. (For custom FileSystem implementations, this could lead to `FSDirectory::Fsync` or `FSDirectory::Close` after the first `FSDirectory::Close`; Also, valgrind could report call to `close()` with `fd=-1`.) ## 7.5.1 (08/01/2022) ### Bug Fixes * Fix a bug where rate_limiter_parameter is not passed into `PartitionedFilterBlockReader::GetFilterPartitionBlock`. ## 7.5.0 (07/15/2022) ### New Features * Mempurge option flag `experimental_mempurge_threshold` is now a ColumnFamilyOptions and can now be dynamically configured using `SetOptions()`. * Support backward iteration when `ReadOptions::iter_start_ts` is set. * Provide support for ReadOptions.async_io with direct_io to improve Seek latency by using async IO to parallelize child iterator seek and doing asynchronous prefetching on sequential scans. * Added support for blob caching in order to cache frequently used blobs for BlobDB. * User can configure the new ColumnFamilyOptions `blob_cache` to enable/disable blob caching. * Either sharing the backend cache with the block cache or using a completely separate cache is supported. * A new abstraction interface called `BlobSource` for blob read logic gives all users access to blobs, whether they are in the blob cache, secondary cache, or (remote) storage. Blobs can be potentially read both while handling user reads (`Get`, `MultiGet`, or iterator) and during compaction (while dealing with compaction filters, Merges, or garbage collection) but eventually all blob reads go through `Version::GetBlob` or, for MultiGet, `Version::MultiGetBlob` (and then get dispatched to the interface -- `BlobSource`). * Add experimental tiered compaction feature `AdvancedColumnFamilyOptions::preclude_last_level_data_seconds`, which makes sure the new data inserted within preclude_last_level_data_seconds won't be placed on cold tier (the feature is not complete). ### Public API changes * Add metadata related structs and functions in C API, including * `rocksdb_get_column_family_metadata()` and `rocksdb_get_column_family_metadata_cf()` to obtain `rocksdb_column_family_metadata_t`. * `rocksdb_column_family_metadata_t` and its get functions & destroy function. * `rocksdb_level_metadata_t` and its and its get functions & destroy function. * `rocksdb_file_metadata_t` and its and get functions & destroy functions. * Add suggest_compact_range() and suggest_compact_range_cf() to C API. * When using block cache strict capacity limit (`LRUCache` with `strict_capacity_limit=true`), DB operations now fail with Status code `kAborted` subcode `kMemoryLimit` (`IsMemoryLimit()`) instead of `kIncomplete` (`IsIncomplete()`) when the capacity limit is reached, because Incomplete can mean other specific things for some operations. In more detail, `Cache::Insert()` now returns the updated Status code and this usually propagates through RocksDB to the user on failure. * NewClockCache calls temporarily return an LRUCache (with similar characteristics as the desired ClockCache). This is because ClockCache is being replaced by a new version (the old one had unknown bugs) but this is still under development. * Add two functions `int ReserveThreads(int threads_to_be_reserved)` and `int ReleaseThreads(threads_to_be_released)` into `Env` class. In the default implementation, both return 0. Newly added `xxxEnv` class that inherits `Env` should implement these two functions for thread reservation/releasing features. * Removed Customizable support for RateLimiter and removed its CreateFromString() and Type() functions. ### Bug Fixes * Fix a bug in which backup/checkpoint can include a WAL deleted by RocksDB. * Fix a bug where concurrent compactions might cause unnecessary further write stalling. In some cases, this might cause write rate to drop to minimum. * Fix a bug in Logger where if dbname and db_log_dir are on different filesystems, dbname creation would fail wrt to db_log_dir path returning an error and fails to open the DB. * Fix a CPU and memory efficiency issue introduce by https://github.com/facebook/rocksdb/pull/8336 which made InternalKeyComparator configurable as an unintended side effect * Fix a bug where `GenericRateLimiter` could revert the bandwidth set dynamically using `SetBytesPerSecond()` when a user configures a structure enclosing it, e.g., using `GetOptionsFromString()` to configure an `Options` that references an existing `RateLimiter` object. ## Behavior Change * In leveled compaction with dynamic levelling, level multiplier is not anymore adjusted due to oversized L0. Instead, compaction score is adjusted by increasing size level target by adding incoming bytes from upper levels. This would deprioritize compactions from upper levels if more data from L0 is coming. This is to fix some unnecessary full stalling due to drastic change of level targets, while not wasting write bandwidth for compaction while writes are overloaded. * For track_and_verify_wals_in_manifest, revert to the original behavior before #10087: syncing of live WAL file is not tracked, and we track only the synced sizes of **closed** WALs. (PR #10330). * WAL compression now computes/verifies checksum during compression/decompression. ### Performance Improvements * Rather than doing total sort against all files in a level, SortFileByOverlappingRatio() to only find the top 50 files based on score. This can improve write throughput for the use cases where data is loaded in increasing key order and there are a lot of files in one LSM-tree, where applying compaction results is the bottleneck. * In leveled compaction, L0->L1 trivial move will allow more than one file to be moved in one compaction. This would allow L0 files to be moved down faster when data is loaded in sequential order, making slowdown or stop condition harder to hit. Also seek L0->L1 trivial move when only some files qualify. * In leveled compaction, try to trivial move more than one files if possible, up to 4 files or max_compaction_bytes. This is to allow higher write throughput for some use cases where data is loaded in sequential order, where appying compaction results is the bottleneck. 2022-08-24T22:39:19+00:00 Lookyloo v1.15.0 Lookyloo v1.15.0 2022-08-25T12:43:29+00:00 # Breaking change * Lookyloo requires Redis 7.0 or more decent. The upgrade process is as follows: 1. Go to the Redis directory (should be in the same directory as where you cloned Lookyloo) 2. Run the following commands ``` git fetch git checkout 7.0 make distclean make -j4 make test ``` 3. You now have the new version of redis in place, you can update lookyloo as usual. # New features * Use pre-configured devices from Playwright (mobile only for now) ![Device select for mobile](https://user-images.githubusercontent.com/248875/186662401-b6486584-ea7f-4f83-8e6f-f0d67d191e77.png) * Download files when the URLs points to a downloadable content ![Download file and submit to pandora](https://user-images.githubusercontent.com/248875/186667605-a5c0c667-cdbf-4fc2-ac84-e0a7b51c405c.png) * Submit downloadable content to [Pandora](https://pandora.circl.lu/submit) (if available) * Automatically select the most appropriate browser engine based on the user-agent # Fixes * Make sure all the gunicorn instances displays all the recent captures * Other bugfixes and GUI improvements # Changes * Improve capture page with radio button to select which user-agent to submit * Bump dependencies 2022-08-25T12:43:29+00:00 OpenTAXII 0.9.2 OpenTAXII 0.9.2 2022-08-26T12:58:36+00:00 Changelog ========= 0.9.2 (2022-08-26) ------------------ * Improve readability and navigation of docs (`#238 <https://github.com/eclecticiq/OpenTAXII/pull/238>`_ thanks `@zed-eiq <https://github.com/zed-eiq>`_ for the improvement). 0.9.1 (2022-07-11) ------------------ * Implement `raise_unauthorized` for taxii2, this was missing and lead to 500 errors. 0.9.0 (2022-06-13) ------------------ * Allow custom properties. This can be disabled by config option ``allow_custom_properties`` 0.8.0 (2022-06-05) ------------------ * Nest details inside taxii2 job and allow counts without details 0.7.0 (2022-05-27) ------------------ * Nest taxii2 endpoints under `/taxii2/` 0.6.0 (2022-05-25) ------------------ * Add `public_discovery` option to taxii2 config * Add support for publicly readable taxii 2 api roots 0.5.0 (2022-05-24) ------------------ * Add support for publicly readable taxii 2 collections 0.4.0 (2022-05-20) ------------------ * Move next_param handling into `OpenTAXII2PersistenceAPI` 0.3.0 (2022-04-13) ------------------ * Implement taxii2.1 support 0.3.0a4 (2022-04-13) -------------------- * Merge changes from 0.2.4 maintenance release 0.3.0a3 (2022-01-21) -------------------- * Fix bug that prevented booting with only taxii1 config (`#217 <https://github.com/eclecticiq/OpenTAXII/issues/217>`_ thanks `@azurekid <https://github.com/azurekid>`_ for the report) 0.3.0a2 (2021-12-27) -------------------- * Merge changes from 0.2.3 maintenance release 0.3.0a1 ------- * Add python 3.10 support 0.3.0a0 ------- * Enablement for future taxii2 implementation * Fix documentation build issues 0.2.4 (2022-04-13) ------------------ * Make sure werkzeug <2.1 and >=2.1 work correctly with auth system 0.2.3 (2021-12-22) ------------------ * Fix bug in multithreaded use of sqlite (`#210 <https://github.com/eclecticiq/OpenTAXII/issues/210>`_ thanks `@rohits144 <https://github.com/rohits144>`_ for the report) 0.2.2 (2021-11-05) ------------------ * Fix readthedocs build 0.2.1 (2021-11-03) ------------------ * Add tests for python 3.6, 3.7, 3.8, 3.9, pypy * Add tests for sqlite, mysql, mariadb, postgresql * Fix bug that broke ``delete_content_blocks`` when using mysql on sqlalchemy 1.3 * Docs: Add db schema diagram * Docs: Clarify how to get default data in a default (development) docker instance * Fix implicit routing in TAXII 1.1 Inboxes * Update jwt usage to pyjwt >= 2.0 (thanks `@SanyaKapoor <https://github.com/SanyaKapoor>`_) 0.2.0 (2020-06-30) ------------------ * Enforce UTC usage in datetime fields in SQL DB Persistence API. * `Fix for #114 <https://github.com/eclecticiq/OpenTAXII/issues/114>`_: reintroduce ``opentaxii-create-account`` CLI command. * `Fix for #153 <https://github.com/eclecticiq/OpenTAXII/issues/152>`_: check if user can modify a collection before advertising it over inbox service. * Multiple coding style fixes. * Various documentation updates. 0.1.12 (2019-03-06) ------------------- * Remove unnecessary print statements. 0.1.11 (2019-02-13) ------------------- * Make JSON logging consistent when the application is run via Gunicorn. * Set ``acceptable_destination`` key in status details instead of extended headers * Allow passing ``engine_parameters`` to ``SQLDatabaseAPI`` for those who want to customize SQLAlchemy engine parameters. * Require recent version of ``lxml`` for security reasons. * Various test and Docker infrastructure improvements. 0.1.10 (2018-06-03) ------------------- * Replace separate service/collection/account creation process with single ``opentaxii-sync-data`` CLI command. * Persistence and Auth APIs extended with missing CRUD methods, that are used by ``opentaxii-sync-data``. * Read/modify collection level ACL added. * DB models for default implementation of Persistence API and Auth API were changed. No automatic migration code is provided (sorry!), so upgrading might require manual DB migration. * Drop python2.7 from testing scope. * Various bug fixes and improvements. 0.1.9 (2017-06-19) ------------------ * `libtaxii <https://github.com/TAXIIProject/libtaxii>`_ dependency upgraded to 1.1.111. * Various bug fixes and improvements (thanks to `@bjigmp <https://github.com/bjigmp>`_, `@chorsley <https://github.com/chorsley>`_, `@rjprins <https://github.com/rjprins>`_). 0.1.8 (2017-02-21) ------------------ * Ability to enable/disable "huge trees" support in XML parser. Configuration property ``xml_parser_supports_huge_tree`` set to ``yes`` or ``true`` will disable security restrictions and force XML parser to support very deep trees and very long text content. * Adjust SQL Persistence API implemetation so it works smoothly with MySQL backend. * Use Python 3.5 instead of Python 3.4 for tests. 0.1.7 (2016-10-18) ------------------ * Minor fixes. * Dependencies were changed from hard-pinned to more flexible. * Example of production DB configuration added to docs. 0.1.6 (2016-06-01) ------------------ * Python 3.4 compatibility of the codebase. Tox configuration extended with python 3.4 environment run. * Flake8 full style compatibility. Flake8 check added to Tox configuration. * SQLAlchemy session scope issue fixed (related to `#38 <https://github.com/EclecticIQ/OpenTAXII/issues/38>`_). * `opentaxii-delete-blocks` CLI command added (related to `#45 <https://github.com/EclecticIQ/OpenTAXII/issues/45>`_). * `delete_content_blocks` method `added <https://github.com/EclecticIQ/OpenTAXII/commit/dc6fddc27a98e8450c7e05e583b2bfb741f6e197#diff-6814849ac352b2b74132f8fa52e0ec4eR213>`_ to Persistence API. * Collection's name is `required <https://github.com/EclecticIQ/OpenTAXII/commit/dc6fddc27a98e8450c7e05e583b2bfb741f6e197#diff-ce3f7b939e5c540480ac655aef32c513R116>`_ to be unique in default SQL DB Auth API implementation. 0.1.5 (2016-03-15) ------------------ * Fix for the issue with persistence backend returning ``None`` instead of ``InboxMessage`` object 0.1.4 (2016-02-25) ------------------ * Hard-coded dependencies in ``setup.py`` removed. 0.1.3 (2016-02-25) ------------------ * Versions of dependencies are pinned. * Code adjusted for a new version of `anyconfig <https://pypi.python.org/pypi/anyconfig>`_ API. * Test for configuration loading added. 0.1.2 (2015-07-24) ------------------ * Docker configuration added. * Health check endpoint added. * Basic authentication support added. * Temporary workaround for `Issue #191 <https://github.com/TAXIIProject/libtaxii/issues/191>`_. * Method ``get_domain`` in Persistence API returns domain value configured for ``service_id``. If nothing returned, value set in configuration file will be used. * Performance optimisations. * Bug fixes and style improvements. 0.1.1 (2015-04-08) ------------------ * Alias for Root Logger added to logging configuration. * Context object in a request scope that holds account and token added. * Support for OPTIONS HTTP request to enable auto version negotiation added. * Documentation improved. 0.1.0 (2015-03-31) ------------------ * Initial release 2022-08-26T12:58:36+00:00 ursadb v1.5.0 ursadb v1.5.0 2022-08-29T16:39:10+00:00 Features: - Alternatives (like {(41 | 42)}) implemented in the ursadb query syntax (#65) - Better support for wildcards (#23) - Syntax for indexing with taints (#31) Performance: - Query graph pruning (#67) Correctness: - Some improvements for thread safety (#32) Refactoring and maintenance: - Ursacli rewritten to C++ (#48) - Documentation improvements (#33) 2022-08-29T16:39:10+00:00 TheHive 4.1.23 TheHive 4.1.23 2022-08-30T07:28:56+00:00 ## [4.1.23](https://github.com/TheHive-Project/TheHive/milestone/94) (2022-07-22) **Fixed bugs:** - [Bug] system user can be deleted by integrity checks [\#2406](https://github.com/TheHive-Project/TheHive/issues/2406) 2022-08-30T07:28:56+00:00 MONARC v2.12.2-p4 MONARC v2.12.2-p4 2022-08-30T07:41:11+00:00 Updated zm-client to [v2.12.1-p1](https://github.com/monarc-project/zm-client/releases/tag/v2.12.1-p1). 2022-08-30T07:41:11+00:00 pandora v1.1.0 pandora v1.1.0 2022-08-31T09:25:44+00:00 # Breaking change This release requires poetry v1.2.0 or more recent. Run the following command to upgrade it: `poetry self update` # New feature * HTML documents can be submitted to [Lookyloo](https://github.com/Lookyloo/lookyloo) (requires v1.15.0 or more recent) ![submit to lookyloo](https://user-images.githubusercontent.com/248875/187422078-f601b1f7-0cbf-47f0-aa9f-31353d3ee4d6.png) ![lookyloo capture](https://user-images.githubusercontent.com/248875/187422923-a75474e7-269f-413e-ae43-1437d6dcc59b.png) # Changes * Improvements in the modules (archives, ISO, EML) * Improvements on the stats page * Configure the links on the index * Bump dependencies # Fixes * Support ingesting a file downloaded from a Pandora instance * Automatically restart unoserver when it crashes (makes previews with libreoffice more reliable) # Notes * Many have reported issues with the reviews generated by LibreOffice. A seemingly universal fix seems to be installing the [full package from the PPA](https://github.com/pandora-analysis/pandora#important-notes-regarding-libreoffice). 2022-08-31T09:25:44+00:00 maltrail 0.49 maltrail 0.49 2022-08-31T22:11:05+00:00 Start-of-month release 2022-08-31T22:11:05+00:00 caddy v2.6.0-beta.3 caddy v2.6.0-beta.3 2022-09-05T22:07:59+00:00 This is the _first beta_ release for Caddy 2.6. Please try it out and report any regressions you notice! Thanks to everyone who helped out! :blush: Beta 1 and beta 2 were trial runs for our CI upgrades, so this is technically beta 3. There are no code changes from beta 1 to beta 3. Thank you @mohammed90 for figuring out the CI magic! These are abbreviated release notes. The full release notes (23.5 KB of them, to be precise) will be published with the final 2.6 release, explaining in detail all the new few features, enhancements, and bug fixes. ## What's Changed * reverseproxy: Implement retry count, alternative to try_duration by @francislavoie in https://github.com/caddyserver/caddy/pull/4756 * cmd: Fix reload with stdin by @francislavoie in https://github.com/caddyserver/caddy/pull/4900 * reverseproxy: Implement read & write timeouts for HTTP transport by @mholt in https://github.com/caddyserver/caddy/pull/4905 * ci: Run golangci-lint on multiple os(#4875) by @u5surf in https://github.com/caddyserver/caddy/pull/4913 * caddyhttp: Clear out matcher error immediately after grabbing it by @francislavoie in https://github.com/caddyserver/caddy/pull/4916 * chore: Add .gitattributes to force *.go to LF by @francislavoie in https://github.com/caddyserver/caddy/pull/4919 * core: Windows service integration by @WingLim in https://github.com/caddyserver/caddy/pull/4790 * fileserver: Support virtual file systems by @mholt in https://github.com/caddyserver/caddy/pull/4909 * caddyhttp: Implement `caddy respond` command by @mholt in https://github.com/caddyserver/caddy/pull/4870 * chore: Bump up to Go 1.19, minimum 1.18 by @francislavoie in https://github.com/caddyserver/caddy/pull/4925 * httpserver: Configurable shutdown delay by @mholt in https://github.com/caddyserver/caddy/pull/4906 * cmd: Use newly-available version information by @mholt in https://github.com/caddyserver/caddy/pull/4931 * Replace strings.Index usages with strings.Cut by @WilczynskiT in https://github.com/caddyserver/caddy/pull/4930 * optimization: Replaced strings.Index with strings.Cut by @chir4gm in https://github.com/caddyserver/caddy/pull/4932 * go.mod: Upgrade OpenTelemetry dependencies by @lewandowski-stripe in https://github.com/caddyserver/caddy/pull/4937 * logging: Fix `cookie` filter by @francislavoie in https://github.com/caddyserver/caddy/pull/4943 * reverseproxy: Support 1xx status codes (HTTP early hints) by @dunglas in https://github.com/caddyserver/caddy/pull/4882 * reverseproxy: Fix H2C dialer using new stdlib `DialTLSContext` by @francislavoie in https://github.com/caddyserver/caddy/pull/4951 * reverseproxy: Ignore context cancel in stream mode by @mholt in https://github.com/caddyserver/caddy/pull/4952 * reverseproxy: Add `unix+h2c` Caddyfile network shortcut by @francislavoie in https://github.com/caddyserver/caddy/pull/4953 * caddyhttp: Enable HTTP/3 by default by @mholt in https://github.com/caddyserver/caddy/pull/4707 * fileserver: reset buffer before using it (#4962) by @abdusco in https://github.com/caddyserver/caddy/pull/4963 * caddyhttp: Smarter path matching and rewriting by @mholt in https://github.com/caddyserver/caddy/pull/4948 * core: Change net.IP to netip.Addr; use netip.Prefix by @WilczynskiT in https://github.com/caddyserver/caddy/pull/4966 * caddyhttp: Set `http.error.message` to the HandlerError message by @francislavoie in https://github.com/caddyserver/caddy/pull/4971 * caddyhttp: Fix for nil `handlerErr.Err` by @francislavoie in https://github.com/caddyserver/caddy/pull/4977 * templates: cap of slice should not be smaller than length by @jedy in https://github.com/caddyserver/caddy/pull/4975 * ci: Increase linter timeout by @mholt in https://github.com/caddyserver/caddy/pull/4981 * httpcaddyfile: Add ocsp_interval global option by @benburkert in https://github.com/caddyserver/caddy/pull/4980 * zstd: fix typo in comment by @simon04 in https://github.com/caddyserver/caddy/pull/4985 * admin: Don't stop old server if new one fails by @WeidiDeng in https://github.com/caddyserver/caddy/pull/4964 * reverseproxy: Add upstreams healthy metrics by @david-szabo97 in https://github.com/caddyserver/caddy/pull/4935 * Remove duplicate words in comments by @Abirdcfly in https://github.com/caddyserver/caddy/pull/4986 * cmd: Migrate to `spf13/cobra`, remove single-dash arg support by @mohammed90 in https://github.com/caddyserver/caddy/pull/4565 * cmd: add completion command by @mohammed90 in https://github.com/caddyserver/caddy/pull/4994 * caddyhttp: Set Content-Type for static response by @mholt in https://github.com/caddyserver/caddy/pull/4999 * httpcaddyfile: Add `{cookie.*}` placeholder shortcut by @francislavoie in https://github.com/caddyserver/caddy/pull/5001 * events: Implement event system by @francislavoie in https://github.com/caddyserver/caddy/pull/4912 * core: Plugins can register listener networks by @mholt in https://github.com/caddyserver/caddy/pull/5002 * httpcaddyfile: Add shortcut for expression matchers by @francislavoie in https://github.com/caddyserver/caddy/pull/4976 * caddyhttp: Copy logger config to HTTP server during AutoHTTPS by @francislavoie in https://github.com/caddyserver/caddy/pull/4990 * dist: deb package manpages and bash completion scripts by @mohammed90 in https://github.com/caddyserver/caddy/pull/5007 * fastcgi: allow users to log stderr output (#4967) by @flga in https://github.com/caddyserver/caddy/pull/5004 * templates: Document `httpError` function by @Malankar in https://github.com/caddyserver/caddy/pull/4972 * fastcgi: Optimize FastCGI transport by @WeidiDeng in https://github.com/caddyserver/caddy/pull/4978 * core: Refactor listeners; use SO_REUSEPORT on Unix by @mholt in https://github.com/caddyserver/caddy/pull/4705 * reverseproxy: Close hijacked conns on reload/quit by @mholt in https://github.com/caddyserver/caddy/pull/4895 * ci: generate SBOM and sign artifacts using cosign by @mohammed90 in https://github.com/caddyserver/caddy/pull/4910 * caddyauth: Speed up basicauth provision, deprecate `scrypt` by @francislavoie in https://github.com/caddyserver/caddy/pull/4720 * caddyhttp: Support `respond` with HTTP 103 Early Hints by @mholt in https://github.com/caddyserver/caddy/pull/5006 * fileserver: Support glob expansion in file matcher by @mholt in https://github.com/caddyserver/caddy/pull/4993 * ci: add `id-token` permission and update the signing command by @mohammed90 in https://github.com/caddyserver/caddy/pull/5016 * ci: grant the `release` workflow the `write` permission to `contents` by @mohammed90 in https://github.com/caddyserver/caddy/pull/5017 ## New Contributors * @WingLim made their first contribution in https://github.com/caddyserver/caddy/pull/4790 * @WilczynskiT made their first contribution in https://github.com/caddyserver/caddy/pull/4930 * @chir4gm made their first contribution in https://github.com/caddyserver/caddy/pull/4932 * @lewandowski-stripe made their first contribution in https://github.com/caddyserver/caddy/pull/4937 * @abdusco made their first contribution in https://github.com/caddyserver/caddy/pull/4963 * @jedy made their first contribution in https://github.com/caddyserver/caddy/pull/4975 * @benburkert made their first contribution in https://github.com/caddyserver/caddy/pull/4980 * @WeidiDeng made their first contribution in https://github.com/caddyserver/caddy/pull/4964 * @david-szabo97 made their first contribution in https://github.com/caddyserver/caddy/pull/4935 * @Abirdcfly made their first contribution in https://github.com/caddyserver/caddy/pull/4986 * @flga made their first contribution in https://github.com/caddyserver/caddy/pull/5004 * @Malankar made their first contribution in https://github.com/caddyserver/caddy/pull/4972 **Full Changelog**: https://github.com/caddyserver/caddy/compare/v2.5.2...v2.6.0-beta.3 2022-09-05T22:07:59+00:00 FIR django4 FIR django4 2022-09-13T07:37:07+00:00 2022-09-13T07:37:07+00:00 TheHive 4.1.24 TheHive 4.1.24 2022-09-13T07:48:11+00:00 ## [4.1.24](https://github.com/TheHive-Project/TheHive/milestone/95) (2022-09-12) **Fixed bugs:** - [Bug] All analyzers become unavailable when an analyzer is updated [\#2420](https://github.com/TheHive-Project/TheHive/issues/2420) 2022-09-13T07:48:11+00:00 MISP v2.4.162 MISP v2.4.162 2022-09-13T08:42:19+00:00 ![](https://www.misp-project.org/img/blog/workflow.png ) We are pleased to announce the immediate availability of [MISP v2.4.162](https://github.com/MISP/MISP/releases/tag/v2.4.162) with a new periodic notification system, workflow updates and many improvements. In addition to the MISP v2.4.162 release, [misp-guard](https://github.com/MISP/misp-guard) has been released which is a [mitmproxy](https://mitmproxy.org/) addon that inspects the events that MISP is attempting to synchronize with external MISP instances via `PUSH` or `PULL` and applies a set of customizable rules defined in a JSON file. This is a complementary tool to support MISP users having to interconenct MISP instances between highly sensitive networks. # Periodic notification system As of version 2.4.162, MISP includes a **periodic summary** feature allowing users to consult a summary based on a requested time-frame for data the user has access to. Currently, the summaries can be generated for 3 different periods: `daily`, `weekly` and `monthly` and then sent to all users that subscribed one of these periods. In addition to choose which period users want to subscribed to, they can also specify filtering options such as tags or distribution level to be used to generate the summary. The summary can be sent via email in addition to the User-Interface view. ![Periodic summary](https://www.misp-project.org/img/blog/periodic-summary/periodic-summary-2.png) ![Periodic summary](https://www.misp-project.org/img/blog/periodic-summary/periodic-summary-3.png) For more information, check out the [Periodic summaries - Visualize summaries of MISP data](/2022/09/12/2022-09-12_periodic_notifications.html/) blog. # Workflow improvements - Added diagnostic support and support of arbitrary URL for webhook module. - New Microsoft teams module based on the webhook module. - New email notification module to send email to a list of MISP users including [Jinja templating](https://jinja.palletsprojects.com/en/3.1.x/). - Tag name can now be used in workflows. For more details about MISP Workflow, check out the [training materials](https://www.misp-project.org/misp-training/a.12-misp-workflows.pdf). # MISP core improvements - Allow option to delete tags on event sync prior to soft-delete tag implementation. - API/[Event:restSearch] Added option `event_tags` to filter for eventTag only. - API/RestSearch - Added support of `static` parameter to produce a static HTML output. - Syslog/logging for certain log entries vital information was omitted by the syslog. If no custom message is specifically set for the log entry, the change field is included. - Enforce UUIDs uniqueness on MISP data back-end. # Bugs fixed - [correlations] save the distribution state of the event before/after saving it, fixes #8528. - [attribute tags] removal broken, fixes #8567. - Class 'Folder' not found #8544. - Create unique SIDs for email attributes in NIDS export. Thanks to all the contributors and users reporting bugs to make the software better. As always, a detailed and [complete changelog is available](https://www.misp-project.org/Changelog.txt) with all the fixes, changes and improvements in MISP core. Many improvements in the MISP galaxy and especially the threat-actor galaxy. There is a detailed [changelog](https://www.misp-project.org/Changelog-misp-galaxy.txt). Improvement in the `false-positive` taxonomy and many other taxonomies. There is a detailed [changelog](https://www.misp-project.org/Changelog-misp-taxonomies.txt). Multiple objects were updated and added, for [more details](https://www.misp-project.org/Changelog-misp-objects.txt). 2022-09-13T08:42:19+00:00 DC3-MWCP 3.8.0 DC3-MWCP 3.8.0 2022-09-16T12:29:56+00:00 2022-09-16T12:29:56+00:00 caddy v2.6.0-beta.5 caddy v2.6.0-beta.5 2022-09-18T00:23:34+00:00 This release and beta.4 are mainly tests of our CI, but also contains a bunch of small fixes or enhancements, including, notably, the use of `sendfile` and other optimizations. See the release notes for beta.3 for everything else until 2.6.0 lands soon. Full release notes coming! ## Changelog * e6c58fdc caddyfile: Prevent infinite nesting on fmt (fix #4175) * 9ad0ebc9 caddyhttp: Add 'skip_log' var to omit request from logs (#4691) * b6cec378 caddyhttp: Add --debug flag to commands * 61c75f74 caddyhttp: Explicitly disallow multiple regexp matchers (#5030) * 00beec2e caddyhttp: Fix write header on responseRecorder * 74547f5b caddyhttp: Make metrics opt-in (#5042) * 61641828 caddyhttp: Support TLS key logging for debugging (#4808) * bffc2587 caddyhttp: Support configuring Server from handler provisioning (#4933) * 20d487be caddyhttp: Very minor optimization to path matcher * dd9813c6 caddyhttp: ensure ResponseWriterWrapper and ResponseRecorder use ReadFrom if the underlying response writer implements it. (#5022) * 258071d8 caddytls: Debug log on implicit tailscale error (#5041) * d35f618b caddytls: Error if placeholder is empty in 'ask' * 0b3161ae cmd: Customizable user agent (close #2795) * 1c9c8f6a cmd: Enhance some help text * 62b06853 cmd: Improve error message if config missing * 50748e19 core: Check error on ListenQUIC * e43b6d81 core: Variadic Context.Logger(); soft deprecation * 48d723c0 encode: Fix Accept-Ranges header; HEAD requests (#5039) * c19f2072 fileserver: Ignore EOF when browsing empty dir * c5df7bb6 go.mod: Update truststore * 8cc8f9fd httpcaddyfile: Add a couple more placeholder shortcuts (#5015) * a1ad20e4 httpcaddyfile: Fix bind when IPv6 is specified with network (#4950) * 754fe4f7 httpcaddyfile: Fix sorting of repeated directives * e338648f reverseproxy: Support repeated --to flags in command (#4693) * 9fe4f93b supplychain: publish signing cert, sbom, and signatures of sbom (#5027) 2022-09-18T00:23:34+00:00 caddy v2.6.0 caddy v2.6.0 2022-09-20T18:12:38+00:00 Caddy 2.6 ======== This is our biggest release since Caddy 2. Caddy 2 changed the way the world serves the Web. By providing an online config API, automatic HTTPS, unlimited extensibility, certificate automation at scale, modern protocols, sane defaults, and an unrivaled developer experience, we boldly raised the bar for web servers. Now with Caddy 2.6, we're doing it again. Caddy 2.6 is the first general-purpose web server to seamlessly enable the newly-standardized HTTP/3 protocol for all configurations by default. We've virtualized the file system so you can serve content from anywhere or anything. New event features let you observe and control Caddy's internals with custom actions. Caddy is more useful than ever for developers with its enhanced CLI tooling and features. And it's faster than ever with non-trivial performance improvements. We think you will love this release. [Watch the livestream](https://youtu.be/iJZqFpY2D14) Special dedication ------------------ This release is dedicated to the late Peter Eckersley, who passed away September 2, 2022. Peter is one of the brilliant minds behind Let's Encrypt; his work has benefited billions of people. I met Peter at the Let's Encrypt launch party in a little bar in San Francisco in 2015 and have never forgotten that occasion. He later co-authored a published research paper called [_Let’s Encrypt: An Automated Certificate Authority to Encrypt the Entire Web_](https://jhalderm.com/pub/papers/letsencrypt-ccs19.pdf), which highly espoused Caddy's ACME integration: "We hope to see other popular server software follow Caddy’s lead." We look forward to when other servers do that, and we hope to honor Peter's work and influence which will live on through his memory and the encrypted Web he made possible. --- [Sponsors](https://github.com/sponsors/mholt) -------- [ZeroSSL](https://zerossl.com) remains Caddy's executive sponsor. [We were thrilled](https://twitter.com/caddyserver/status/1559591673511813120) to welcome [Stripe](https://stripe.com) recently as an enterprise sponsor! Other notable sponsors include [AppCove](https://appcove.com/), [Dukaan](https://mydukaan.io/), [Suborbital](https://suborbital.dev), [Tailscale](https://tailscale.com), plus [Bubble](https://bubble.io/) and [GitHub](https://github.com) which both made generous one-time donations. We have many other vital sponsors and donors on which we also rely. [Our sponsors](https://github.com/sponsors/mholt) come from all over the world and include independent professionals, startups, and small companies -- and they are the absolute best. Thank you for making a more secure Web possible! _Personal note from Matt: Recent life upgrades mean that your sponsorships now sustain a family of 5 so that I can continue to maintain Caddy. Two years ago, I don't think I would have taken this risk because I'd need to find other work to provide for a family. Thank you for coming together as a professional community to make the Caddy project possible!_ We strongly recommend that companies who -- or companies whose customers -- use or benefit from Caddy [become a sponsor](https://github.com/sponsors/mholt) to ensure ongoing maintenance, priority development, private support, and more. Sponsorship tiers can be tailored to your requirements! Highlights ---------- :warning: **_Don't miss deprecations / breaking changes at the bottom. Notably, if you use metrics, you will now need to turn them on._** ### HTTP/3 is here (#4707) Caddy now enables [RFC 9114](https://datatracker.ietf.org/doc/rfc9114/)-compliant HTTP/3 by default. The `experimental_http3` option has graduated and been removed. We've removed another experimental option, `allow_h2c`, and individual HTTP versions (`h1 h2 h2c h3`) can now be toggled with the new `protocols` setting. Note that HTTP/3 utilizes the QUIC transport, which requires UDP. If your network or firewall configuration only allows TCP, HTTP/3 connections will fail and clients (should) fall back to HTTP/2. For servers with properly-configured UDP networks, HTTP/3 should "just work" for enabled clients. HTTP/3 clients can connect by reading Caddy's Alt-Svc header to know how to connect to Caddy via UDP. This header is now emitted automatically and by default. Other than that, there are no other changes needed to existing servers, as Caddy opens a separate UDP socket for HTTP/3. Our HTTP/3 server attempts to mitigate amplification and reflection attacks by requiring address validation when the server is under load. This adds one round-trip for clients, but is only done as a defensive measure when necessary. Serious thanks to @marten-seemann who builds and maintains the [quic-go](https://github.com/lucas-clemente/quic-go) library we depend on for this. (Go has not announced any plans to officially support or implement HTTP/3.) We expect numerous QUIC and HTTP/3 improvements to come as implementations and best practices mature with more production experience. ### Virtual file systems (#4909) Caddy's `file_server` module now supports _virtual_ file systems. We've replaced all hard-coded `os.Open()`, `os.Stat()`, etc. calls with Go's relatively new `io/fs` package, and introduced a new Caddy module namespace `caddy.fs` for implementations of such file systems. Some examples of what is possible: - Serve content from S3 or other blob/cloud storage services - Serve dynamically-generated content that "feels" static - [Embed your site directly into your `caddy` binary](https://github.com/mholt/caddy-embed) and serve it from memory - Serve content directly from an archive file (e.g. `.zip` or `.tar.gz`) - Load files from a database instead of disk Basically, instead of serving files from the local disk, you can have Caddy serve the "files" from somewhere or something else. The default is still the local file system. Note that this feature isn't limited to just Caddy's `file_server` module. Potentially any module that reads the local disk may benefit from using `caddy.fs` modules instead. I wrote [a module that lets you embed your site](https://github.com/mholt/caddy-embed) within your `caddy` binary -- wherever your server goes, your site goes! We encourage the community to implement and publish new file system modules for Caddy. (From an [early tweet](https://twitter.com/mholt6/status/1551434889358032897) there seems to be quite high demand.) ### Events (#4912 and #4984) Not surprisingly, many people prefer Caddy to automate certificates used with other software/services. Until now, there hasn't been a great way to know when Caddy has obtained or renewed a certificate (deferred in part by our opinion that certificate management should be baked into the software using the certificate in the first place). Cron jobs generally work for reloading new certificates into services because certificate expiry is _mostly_ predictable, but now there is a better way with one of our most requested features: events! We thought about events in general for a long time and discussed questions like, "What makes an event different from a log?" "Are events synchronous?" "Do self-initiated events get emitted before or after their code (are they past-tense or future-tense) -- or both? or neither (asynchronous)?" "What do we like from existing event systems?" "What do we wish event systems did differently?" While we think we have pretty good answers to these questions now, we won't be sure until we gather more production experience. For this reason, events are implemented as an experimental app module -- not as part of the core. (Remember, Caddy's core currently only loads config and sets up logging/storage.) This means that Caddy's core cannot emit events.[^1] So even though our event implementation may change, it is likely to be only slight and gradual changes; and we encourage anyone and everyone to start using events as soon as possible and to **give us your feedback**. We think we have the start of a great event system, but we need you to prove it! Caddy modules can emit events when interesting things happen. For example, the reverse proxy emits `healthy` and `unhealthy` events when backends go up and down. The TLS app emits `cert_obtaining`, `cert_obtained`, and `cert_failed` before and after obtaining a certificate or after the operation failed, respectively; and `cert_ocsp_revoked` after a certificate is discovered to be revoked by OCSP. There are several more events already, with even more to be added later. Events can have data associated with them. For example, `healthy`/`unhealthy` come with the address of the host; `cert_obtained` has the domain name, issuer, and storage path. You can access this from config in placeholders, e.g. `{event.data.identifier}`. Caddy modules can subscribe to events by specifying the name(s) of events to bind to, and the Caddy module ID(s) or namespace(s) to watch. When an event is emitted, it propagates from the module that emitted it up the provisioning heirarchy. This means that an event emitted by `http.handlers.reverse_proxy` will fire for `http.handlers` and `http` as well, similar to the DOM in HTML/JavaScript. Event handlers are invoked synchronously. We chose this for several reasons. First, despite how easy Go makes concurrency, there are many subtleties to concurrency in a server. Goroutines may be lightweight, but their operations might not be; and if event goroutines are starting more quickly than they are stopping, we either drop events arbitrarily or run out of memory/CPU. Also, we think one of the qualities that differentiates events from logs is the ability for an event to influence the emitting code's flow: a true "hook" in that sense. Instead of simply observing that something is happening (which is what a log tells you), you can influence its behavior. Maybe you want to run a command before a certificate is obtained to see if it _should_ be obtained. Or maybe you want to change how a TLS handshake is completed on-the-fly. Asynchronous event handlers cannot do this. For simple behavioral changes, synchronous events can be a powerful and useful tool for customizing your server. The new `event` app lets you easily configure subscriptions and event handlers. Event handling is modular, so you will need to plug in a module that does what you want: run a command, reload a service, make an HTTP request, or anything else! Because this feature is experimental and new, we don't yet know how people will be using it, so currently, Caddy does not ship with any event handler plugins. However, we're pretty sure based on feedback over the years that many of you would like to run commands on certain events (one of our top feature requests is to trigger a daemon reload after certificate renewals). So I went ahead and implemented an [`exec` event handler plugin](https://github.com/mholt/caddy-events-exec) that can run commands. We almost included it in Caddy's standard distribution, but out of an abundance of caution we decided to keep it a separate plugin for now until we learn more about real production use cases from experience. Here's an example of handling events. In JSON, you configure the `events` app: ```json { "apps": { "events": { "subscriptions": [ { "events": ["cert_obtained"], "handlers": [ { "handler": "exec", "command": "systemctl", "args": ["reload", "mydaemon"] } ] } ] } } } ``` or the equivalent Caddyfile global option: ``` { events { on cert_obtained exec systemctl reload mydaemon } } ``` It's that simple! Just make sure you have your event handler modules plugged in. **We hope you will provide feedback, report bugs, and request features related to events.** [^1]: Compilation fails with an import cycle. If Caddy core uses any feature of Caddy, it must also be in the core or another package not imported by any modules! ### Smarter path matching and rewriting (#4948) Is the URI path `/a/b/c` the same as `/a/b%2Fc`? What about `/a/b//c`? Turns out, _it depends._ What these questions illustrate is a famously frustrating problem, and has largely gone unsolved until now. All existing solutions I investigated in other products were unsatisfactory: - Nginx (and Caddy until now) always does path comparisons in unescaped/normalized space. This makes it impossible to route on literal escape sequences unless you double-encode your pattern, which violates specification. - Apache outright rejects valid[^2] HTTP requests containing encoded slashes. This behavior can either be disabled completely (creating a security problem known as unsafe paths) or tweaked to never decode encoded slashes (creating ambiguities when comparing against route patterns). - Laravel, like nginx, always decodes slashes, but routing such requests [mangles application data](https://github.com/laravel/framework/issues/22125) that contains slashes. The process of decoding a URI and collapsing slashes in the path is called normalization. Normalization has to occur for safe, reliable routing (imagine `//secret` bypassing auth checks configured for `/secret`), but at the same time, raw paths are sometimes needed to preserve application data (imagine a route `/bands/:name` which succeeds for `/bands/AC&2fDC` but fails for the normalized `/bands/AC/DC`). And it's not just routing; servers like Caddy often rewrite/manipulate paths. Because normalizing URIs creates a Many:1 mapping (there are multiple encoded forms of a single URI), normalizing is inherently lossy: the original input cannot be recovered with certainty, so we can't reconstruct the original or intended URI with complete fidelity. Other solutions with coarse on-off knobs can't balance both security and application correctness: it seems you have to trade one for the other. The crux of the problem seems to be that the server/framework/router doesn't know which parts of the path are application data and which parts are path components, so it just "plays it safe" and decodes the whole thing. I think Caddy's solution to this is quite novel. **Our solution is to interpret encoded characters and multiple slashes in a path pattern literally as a hint of the developer's intent.** For example, if you write a path matcher `/a/b/c`, it will still match `/a/b/c` and `/a/b%2Fc`. However, if your path matcher is `/a/b%2Fc`, Caddy will _only_ match `/a/b%2Fc`. This extends to wildcards with our new "escape-wildcard" feature: `/bands/%*/` will match `/bands/AC%2fDC` but `/bands/*/` won't. This works for multiple slashes too. If your path matcher uses `//`, Caddy will require the request path to contain those slashes literally at that position. We've also implemented this for prefix and suffix manipulations. For example, if you wanted to strip a prefix of `//prefix` from `//prefix/foo`, it will now work, whereas before it wouldn't because it would look at a fully-normalized URI. Essentially, we use the configured path pattern as a cue for whether to decode/merge a character or leave it raw when normalizing. This is a complex and subtle change, so please be sure to read the full PR in #4948 and the linked Laravel issue. It's very informative! [^2]: The "validity" of such a URI based on spec compliance is debatable. RFC 9110 says, "distinct resources SHOULD NOT be identified by HTTP URIs that are equivalent after normalization." ### HTTP 103 Early Hints (#4882 and #5006) HTTP Early Hints ([RFC 8297](https://datatracker.ietf.org/doc/rfc8297/)) is the effective successor to HTTP/2 Server Push. When 103 is emitted with relevant Link headers, web pages will load faster than normal. 1xx responses are precursors to the final response; clients must be able to support receiving multiple responses to a single request (nearly all modern clients do; and it almost certainly shouldn't break any HTTP/2 clients). Early hints are a great way to speed up page loads where the main content may take a while to generate (a slow DB query, for example) but the subresources can start being loaded right away. In those cases it is often beneficial to send early hints. Caddy can both originate and proxy 103 responses. To send early hints from Caddy, simply set the `Link` headers as the hints, then write the response with a 103 status code: ``` route /slow-pages/* { header Link "</style.css>; rel=preload; as=style" header Link "</script.js>; rel=preload; as=script" respond 103 } ``` Unlike normal responses, after writing HTTP 103, Caddy's middleware chain will _continue_ to execute and invoke the next handlers (for example, `reverse_proxy`) since 103 is not the final response. Multiple 103s can be sent. Caddy's reverse proxy also supports HTTP 103 responses, meaning that backends can send early hints and Caddy will proxy them to the client immediately as you'd expect., Note that browser support is still limited (only Chrome implements it at this time) and Caddy must be built with Go 1.19 (our builds use the latest Go version; but we still support Go 1.18 for now). Thank you to @dunglas with API Platform for contributing this feature to both Go and Caddy! ### Improved command line interface (#4565 and #4994) Caddy has always used Go's standard `flag` package for its CLI, which has served us quite well. However, recent improvements in the [Cobra](https://github.com/spf13/cobra) library make it possible for our CLI to gain worthwhile features without incurring a heavy dependency. The new `caddy manpage` command generates man pages, and the `caddy completion` command generates shell completions. Both are installed automatically as part of our official Linux packages, so your next `apt upgrade` (etc.) should take care of that. Additionally, short options (e.g. `-c`) are now supported. And if you typo a command, Caddy will helpfully suggest a correction (e.g. `caddy adpt` will suggest `caddy adapt`). Note that long-form flags must now use double-hyphen syntax (e.g. `--config`) even though the single-hyphen syntax (`-config`) was previously accepted. The standard library's flag parser treats `-` and `--` the same, but Cobra's does not. Our online documentation has always used `--` for flags, so we do not consider this a breaking change, but it's good to be aware of this change if you're used to how Go's parser works. Very many thanks to @mohammed90 for contributing these features! ### New `caddy respond` command (#4870) For rapid development needing a local HTTP server, the [`caddy respond` command](https://caddyserver.com/docs/command-line#caddy-respond) might be just what you need: hard-coded HTTP responses for one or more servers so that you can effortlessly have a custom HTTP endpoint to test with. A plain `caddy respond` command will listen on a random port and reply with HTTP 200. (The port or address is printed to the terminal for you.) You can set a custom status code like `caddy respond 401` or a custom body like `caddy respond "Hello world!"` -- or both: `caddy respond --status 401 "Hello world!"` Or you can pipe in a response body, for example serving a maintenance page: ```bash $ cat maintenance.html | caddy respond --status 503 --header "Content-Type: text/html; charset=utf-8" ``` You can even spin up multiple servers at once and use basic template features to configure each server with a different response: ```bash $ echo "I'm server {{.N}} on port {{.Port}}" | caddy respond --listen :2000-2004 Server address: [::]:2000 Server address: [::]:2001 Server address: [::]:2002 Server address: [::]:2003 Server address: [::]:2004 $ curl 127.0.0.1:2002 I'm server 2 on port 2002 ``` You can debug HTTP clients easier by enabling access logging with the `--access-log` flag. The `--header` flag can be used multiple times to set custom HTTP headers, and `--debug` enables debug mode for more verbose logging. We hope you find this feature useful! ### Multiple dynamic upstream sources (5fb5b81) In Caddy 2.5(.1) we introduced dynamic upstreams, which allow you to configure the `reverse_proxy` to get the list of backends on-the-fly during requests. This very popular feature's development was sponsored by Stripe, who we are thrilled to welcome as an enterprise sponsor. Stripe uses Caddy heavily for their internal systems, and for greater redundancy they need to be able to fail over to secondary upstreams if a primary cluster is down. This is where the new `multi` dynamic upstreams module comes in. Now you can configure, for example, two SRV lookups for aggregated results: ```json { "handler": "reverse_proxy", "dynamic_upstreams": { "source": "multi", "sources": [ { "source": "srv", "name": "primary" }, { "source": "srv", "name": "secondary" } ] } } ``` This appends the backends returned from the secondary SRV lookup to the results of the primary SRV lookup (order preserved). To implement failover, simply use the `first` load balancing policy which chooses the first available upstream. ### Configurable shutdown delay (#4906) A shutdown can now be scheduled for a later time using the `shutdown_delay` option. This is useful for giving advance notice to health checkers that this server will be closing soon. The shutdown delay happens _before_ the grace period where new connections are no longer accepted and existing ones are gracefully closed. During the shutdown delay, the server operates normally with the exception of the value of two placeholders. During the delay: - `{http.shutting_down}` placeholder equals `true`. - `{http.time_until_shutdown}` returns the duration that remains until server close. This allows health check endpoints to announce that they will soon be going down so that this instance can be moved out of the rotation or a replacement instance can be spun up in the meantime. For example: ``` { shutdown_delay 10s } example.com { handle /health-check { @goingDown `{http.shutting_down}` respond @goingDown "Bye-bye in {http.time_until_shutdown}" 503 respond 200 } } ``` By the way, the syntax of that `@goingDown` named matcher is new in 2.6: if a named matcher consists only of a CEL expression string, the type of matcher can be omitted; i.e. what you see above is equivalent to `@goingDown expression "{http.shutting_down} == true"`. (A shutdown is defined as a config unload where there is no new config to load, or the new config does not have a server configured at the same address as the current server. In other words, a shutdown of a server means a particular HTTP socket will be closed.) Speaking of grace periods, config changes no longer block while waiting on servers' grace periods. This means faster, more responsive config reloads; just beware that, depending on the length of your grace period, your reload command or config API request may return before the old servers have completely finished shutting down. ### Faster FastCGI transport (#4978) PHP apps, rejoice! The round-trip between Caddy and php-fpm just got a lot faster. Thanks to contributions by @WeidiDeng, the FastCGI transport has been rewritten to be more efficient. This is some of the oldest and most unique code in Caddy's code base. When Caddy was rewritten for v2 in 2019, _everything_ was rewritten or refactored... except this, the FastCGI transport. This is the first time this part of the code has been improved since it was first implemented[^3] in 2014! During tests, profiling showed the new code spends 86% less CPU time in GC (`gcDrain`) thanks to significantly fewer allocations. This is largely in part due to pooling buffers, which required a non-trivial refactoring to implement. ![CPU profile](https://user-images.githubusercontent.com/1128849/188224782-572c877d-42ea-4241-927f-346000512a75.png) A very rough benchmark using `php_info()` yielded a 25% increase in requests per second. Before the rewrite, Caddy almost always performed worse than nginx even with `fastcgi_keep_conn off`. Our new code performs competitively with nginx, and in some tests Caddy even outperformed nginx with `fastcgi_keep_conn on` -- and we have not implemented connection pooling/reuse into the new transport yet. Because every setup is different, your actual results will vary. In general though, you can expect busy servers to handle PHP faster. [^3]: I didn't know how to write a FastCGI client back then (I'm still too scared to do much with it myself); Go's standard library implements only the responder role, not the web server (client). Fortunately there was a [random repository on BitBucket](http://bitbucket.org/PinIdea/fcgi_client) that was forked from [a random repository on Google Code written in 2012](https://code.google.com/archive/p/go-fastcgi-client/) that modified the Go std lib's `fcgi` package. It was rough around the edges, but with a little TLC we got it to do what we needed. The copyright had the name Junqing Tan in it, which we still retain in our source code to this day. ### Faster file server (#5022) In a patch contributed by @flga, we've reduced copying between buffers and even eliminated it altogether in some cases using [`sendfile(2)`](https://man7.org/linux/man-pages/man2/sendfile.2.html). This has shown to have a 25-50% performance boost. It's automatic and no configuration is required to benefit. In [some tests](https://blog.tjll.net/reverse-proxy-hot-dog-eating-contest-caddy-vs-nginx/), Caddy's new defaults are even faster than optimized nginx. Static files over 512 bytes being served over plaintext HTTP sockets may now be served directly by the Linux kernel, which is much faster than copying the file to user-space. Static files are faster over HTTPS, too. In addition to sendfile (which we can't[^4] use over TLS), we now utilize the `io.ReaderFrom` interface to reuse existing buffers and further reduce copying within user space. Our tests show that this significantly enhances performance even over TLS. [^4]: This is possible with kTLS, but [the Go standard library doesn't support it](https://github.com/golang/go/issues/44506) and it's [a bit tedious](https://words.filippo.io/playing-with-kernel-tls-in-linux-4-13-and-go/) to make it work, although @FiloSottile was successful with his [spike code](https://github.com/FiloSottile/go/commit/dbed9972d9947eb0001e9f5b639e0df05acec8bd). ### Signed release assets Thanks to heroic efforts by @mohammed90, our [GitHub release](https://github.com/caddyserver/caddy/releases) assets are now signed and certified. Mohammed [wrote an excellent Twitter thread](https://twitter.com/MohammedSahaf/status/1572022375247663105) explaining the whole thing better than I can here! So if you're wondering why the number of assets shot from 28 to 134... that's why. Other notable enhancements -------------------------- - More efficient `query` matcher. (04a14ee37ac6192d734518fa9082d6eb93971bc6) - A new Caddyfile placeholder `{cookie.*}` grants easy access to cookie values. (#5001) - Windows service integration: Caddy can now be controlled with `sc.exe`. (#4790) - Replace `net.IP` type with leaner `netip.Addr` type. (#4966) - Caddyfile-configurable OCSP check interval with `ocsp_interval` global option. (#4980) - The reverse proxy now supports `retry_count` as an alternative to `try_duration`; i.e. try backends up to a fixed number of times, rather than up to a time limit. (#4756) - The reverse proxy closes both ends of "hijacked" connections when shutting down or reloading. (#4895) - The reverse proxy gracefully closes both ends of websocket connections on shutdown or reload. (#4895) - The reverse proxy emits metrics regarding the health of upstreams. (#4935) - The reverse-proxy command can accept repeated --to flags and load balance. (#4693) - The reverse proxy's HTTP transport now supports distinct read and write timeouts. (#4905) - Simpler and more reliable config reloads on Linux with SO_REUSEPORT. (#4705) - Templates can access reverse proxy responses if used within `handle_response`. (#4871) - Builds now include git revision information when using `go build`. (#4931) - The file matcher (and `try_files`) now supports glob patterns. (#4993) - Named matchers in the Caddyfile can use CEL expressions without specifying `expression` first. (#4976) - The FastCGI transport can now capture and print stderr output. (#5004) - Listeners can be provided by plugins, enabling new network types. (#5002) - Caddy can write TLS secrets to a file for debugging purposes. (#4808) - Sites declared as `http://` in the Caddyfile will no longer be overridden by auto-HTTPS redirects. (#5051) - Config reloads no longer block while the prior servers are shutting down. (#5043) ## :warning: Deprecations/breaks - **Metrics are now opt-in.** Due to [multiple confirmed reports](https://github.com/caddyserver/caddy/issues/4644) of non-trivial performance regressions with metrics, we are making them opt-in. (Technically, this is not a breaking change, as Caddy will still function normally and your old configs won't be rejected -- but your metrics will stop being produced unless you enable them.) If you rely on metrics, you can enable them globally in the Caddyfile with global options: ``` { servers { metrics } } ``` As with other server-scoped global options, you can selectively customize which servers to enable metrics (e.g. `servers :8080`). _Note that this change is experimental and might be temporary: if we can reduce the performance impact or find a better way to enable and configure metrics, this could change._ - The signature of `caddy.Context.Logger()` has changed, but in a backwards-compatible way. Modules use this function to obtain a logger they can use within Caddy; until now, modules had to pass themselves in as an argument. Now, the context can figure out which module to associate the logger with, so the sole parameter has been made variadic. It may be removed in the future. Plugins should update their code to not pass in a pointer to themselves. - Basic auth deprecates `scrypt` because it was seldom used and error-prone; use `bcrypt` instead (#4720) - Several changes to experimental `servers` global options: removed the `protocol` sub-option, which has been replaced with the `protocols` sub-option; `strict_sni_host` is its own separate sub-option; `allow_h2c` and `experimental_http3` have been removed, as both H2C (`h2c`) and HTTP/3 (`h3`) can be toggled in `protocols` (HTTP/3 is now enabled by default and no longer experimental). As a reminder, features, parameters, and APIs marked as experimental are subject to change or removal. We strive to keep breaking changes of stable features to a minimum and gracefully deprecate whenever possible with emphasis in release notes, warnings in logs, etc. Most breaking changes are motivated or necessitated by bugs/regressions, security, or wrong/unclear documentation. Thank you --------- As usual, a huge thank-you to all our [sponsors](https://github.com/sponsors/mholt) and those who contributed both code and feedback. We also acknowledge the many people who participated in discussions and helped others on the forum. Thank you! ## New Contributors * @WingLim made their first contribution in https://github.com/caddyserver/caddy/pull/4790 * @WilczynskiT made their first contribution in https://github.com/caddyserver/caddy/pull/4930 * @chir4gm made their first contribution in https://github.com/caddyserver/caddy/pull/4932 * @lewandowski-stripe made their first contribution in https://github.com/caddyserver/caddy/pull/4937 * @abdusco made their first contribution in https://github.com/caddyserver/caddy/pull/4963 * @jedy made their first contribution in https://github.com/caddyserver/caddy/pull/4975 * @benburkert made their first contribution in https://github.com/caddyserver/caddy/pull/4980 * @WeidiDeng made their first contribution in https://github.com/caddyserver/caddy/pull/4964 * @david-szabo97 made their first contribution in https://github.com/caddyserver/caddy/pull/4935 * @Abirdcfly made their first contribution in https://github.com/caddyserver/caddy/pull/4986 * @flga made their first contribution in https://github.com/caddyserver/caddy/pull/5004 * @Malankar made their first contribution in https://github.com/caddyserver/caddy/pull/4972 * @stapelberg made their first contribution in https://github.com/caddyserver/caddy/pull/4950 * @parrotmac made their first contribution in https://github.com/caddyserver/caddy/pull/4693 * @Manouchehri made their first contribution in https://github.com/caddyserver/caddy/pull/4808 **Full Changelog**: https://github.com/caddyserver/caddy/compare/v2.5.2...v2.6.0 --- 2022-09-20T18:12:38+00:00 rocksdb v7.6.0 rocksdb v7.6.0 2022-09-20T19:20:37+00:00 ### New Features * Added `prepopulate_blob_cache` to ColumnFamilyOptions. If enabled, prepopulate warm/hot blobs which are already in memory into blob cache at the time of flush. On a flush, the blob that is in memory (in memtables) get flushed to the device. If using Direct IO, additional IO is incurred to read this blob back into memory again, which is avoided by enabling this option. This further helps if the workload exhibits high temporal locality, where most of the reads go to recently written data. This also helps in case of the remote file system since it involves network traffic and higher latencies. * Support using secondary cache with the blob cache. When creating a blob cache, the user can set a secondary blob cache by configuring `secondary_cache` in LRUCacheOptions. * Charge memory usage of blob cache when the backing cache of the blob cache and the block cache are different. If an operation reserving memory for blob cache exceeds the avaible space left in the block cache at some point (i.e, causing a cache full under `LRUCacheOptions::strict_capacity_limit` = true), creation will fail with `Status::MemoryLimit()`. To opt in this feature, enable charging `CacheEntryRole::kBlobCache` in `BlockBasedTableOptions::cache_usage_options`. * Improve subcompaction range partition so that it is likely to be more even. More evenly distribution of subcompaction will improve compaction throughput for some workloads. All input files' index blocks to sample some anchor key points from which we pick positions to partition the input range. This would introduce some CPU overhead in compaction preparation phase, if subcompaction is enabled, but it should be a small fraction of the CPU usage of the whole compaction process. This also brings a behavier change: subcompaction number is much more likely to maxed out than before. * Add CompactionPri::kRoundRobin, a compaction picking mode that cycles through all the files with a compact cursor in a round-robin manner. This feature is available since 7.5. * Provide support for subcompactions for user_defined_timestamp. * Added an option `memtable_protection_bytes_per_key` that turns on memtable per key-value checksum protection. Each memtable entry will be suffixed by a checksum that is computed during writes, and verified in reads/compaction. Detected corruption will be logged and with corruption status returned to user. * Added a blob-specific cache priority level - bottom level. Blobs are typically lower-value targets for caching than data blocks, since 1) with BlobDB, data blocks containing blob references conceptually form an index structure which has to be consulted before we can read the blob value, and 2) cached blobs represent only a single key-value, while cached data blocks generally contain multiple KVs. The user can specify the new option `low_pri_pool_ratio` in `LRUCacheOptions` to configure the ratio of capacity reserved for low priority cache entries (and therefore the remaining ratio is the space reserved for the bottom level), or configuring the new argument `low_pri_pool_ratio` in `NewLRUCache()` to achieve the same effect. ### Public API changes * Removed Customizable support for RateLimiter and removed its CreateFromString() and Type() functions. * `CompactRangeOptions::exclusive_manual_compaction` is now false by default. This ensures RocksDB does not introduce artificial parallelism limitations by default. * Tiered Storage: change `bottommost_temperture` to `last_level_temperture`. The old option name is kept only for migration, please use the new option. The behavior is changed to apply temperature for the `last_level` SST files only. * Added a new experimental ReadOption flag called optimize_multiget_for_io, which when set attempts to reduce MultiGet latency by spawning coroutines for keys in multiple levels. ### Bug Fixes * Fix a bug starting in 7.4.0 in which some fsync operations might be skipped in a DB after any DropColumnFamily on that DB, until it is re-opened. This can lead to data loss on power loss. (For custom FileSystem implementations, this could lead to `FSDirectory::Fsync` or `FSDirectory::Close` after the first `FSDirectory::Close`; Also, valgrind could report call to `close()` with `fd=-1`.) * Fix a bug where `GenericRateLimiter` could revert the bandwidth set dynamically using `SetBytesPerSecond()` when a user configures a structure enclosing it, e.g., using `GetOptionsFromString()` to configure an `Options` that references an existing `RateLimiter` object. * Fix race conditions in `GenericRateLimiter`. * Fix a bug in `FIFOCompactionPicker::PickTTLCompaction` where total_size calculating might cause underflow * Fix data race bug in hash linked list memtable. With this bug, read request might temporarily miss an old record in the memtable in a race condition to the hash bucket. * Fix a bug that `best_efforts_recovery` may fail to open the db with mmap read. * Fixed a bug where blobs read during compaction would pollute the cache. * Fixed a data race in LRUCache when used with a secondary_cache. * Fixed a bug where blobs read by iterators would be inserted into the cache even with the `fill_cache` read option set to false. * Fixed the segfault caused by `AllocateData()` in `CompressedSecondaryCache::SplitValueIntoChunks()` and `MergeChunksIntoValueTest`. * Fixed a bug in BlobDB where a mix of inlined and blob values could result in an incorrect value being passed to the compaction filter (see #10391). * Fixed a memory leak bug in stress tests caused by `FaultInjectionSecondaryCache`. ### Behavior Change * Added checksum handshake during the copying of decompressed WAL fragment. This together with #9875, #10037, #10212, #10114 and #10319 provides end-to-end integrity protection for write batch during recovery. * To minimize the internal fragmentation caused by the variable size of the compressed blocks in `CompressedSecondaryCache`, the original block is split according to the jemalloc bin size in `Insert()` and then merged back in `Lookup()`. * PosixLogger is removed and by default EnvLogger will be used for info logging. The behavior of the two loggers should be very similar when using the default Posix Env. * Remove [min|max]_timestamp from VersionEdit for now since they are not tracked in MANIFEST anyway but consume two empty std::string (up to 64 bytes) for each file. Should they be added back in the future, we should store them more compactly. * Improve universal tiered storage compaction picker to avoid extra major compaction triggered by size amplification. If `preclude_last_level_data_seconds` is enabled, the size amplification is calculated within non last_level data only which skip the last level and use the penultimate level as the size base. * If an error is hit when writing to a file (append, sync, etc), RocksDB is more strict with not issuing more operations to it, except closing the file, with exceptions of some WAL file operations in error recovery path. * A `WriteBufferManager` constructed with `allow_stall == false` will no longer trigger write stall implicitly by thrashing until memtable count limit is reached. Instead, a column family can continue accumulating writes while that CF is flushing, which means memory may increase. Users who prefer stalling writes must now explicitly set `allow_stall == true`. * Add `CompressedSecondaryCache` into the stress tests. * Block cache keys have changed, which will cause any persistent caches to miss between versions. ### Performance Improvements * Instead of constructing `FragmentedRangeTombstoneList` during every read operation, it is now constructed once and stored in immutable memtables. This improves speed of querying range tombstones from immutable memtables. * When using iterators with the integrated BlobDB implementation, blob cache handles are now released immediately when the iterator's position changes. * MultiGet can now do more IO in parallel by reading data blocks from SST files in multiple levels, if the optimize_multiget_for_io ReadOption flag is set. 2022-09-20T19:20:37+00:00 caddy v2.6.1 caddy v2.6.1 2022-09-21T22:21:28+00:00 Hotfix for unix sockets, the `encode` handler, and the `caddy file-server` command. Please see [the release notes for v2.6.0](https://github.com/caddyserver/caddy/releases/tag/v2.6.0) for other important information if you're coming from < 2.6! ## Changelog * 1426c97d core: Reuse unix sockets (UDS) and don't try to serve HTTP/3 over UDS (#5063) * 44ad0ced encode: don't WriteHeader unless called (#5060) * beb7dcbf fileserver: Reinstate --debug flag 2022-09-21T22:21:28+00:00 MISP v2.4.163 MISP v2.4.163 2022-09-26T14:12:21+00:00 ![](https://www.misp-project.org/img/blog/periodic-summary/periodic-summary-2.png ) We are pleased to announce the immediate availability of [MISP v2.4.163](https://github.com/MISP/MISP/releases/tag/v2.4.163) with an updated [periodic notification system](https://www.misp-project.org/2022/09/12/2022-09-12_periodic_notifications.html/) and many improvements. # Updated periodic notification system - A new option has been added to set the number of days for the trending calculation. - New correlation are now showed in the periodic notification. - Only the top 10 MITRE ATT&CK techniques are displayed and sorted by number of occurrences. - Layout has been improved in the UI and also in the static email rendering. - Only show data in the chart for tags having changes over time. For more information, check out the [Periodic summaries - Visualize summaries of MISP data](https://www.misp-project.org/2022/09/12/2022-09-12_periodic_notifications.html/) blog. # Fixes - MISP [OpenAPI description file](https://www.misp-project.org/openapi/) has been improved. - [community] Clarification concerning the NATO process. - [ssdeep] Check if the ssdeep contains newline characters. - Many code clean-up and speed-up included. - Improvements and bugs fixed in the correlation engine. - Many bugs fixed. Thanks to all the contributors and users reporting bugs to make the software better. As always, a detailed and [complete changelog is available](https://www.misp-project.org/Changelog.txt) with all the fixes, changes and improvements in MISP core. # misp-stix v.2.4.163 misp-stix has been released too and now in-line with the MISP release schedule. The full [changelog is available](https://www.misp-project.org/Changelog-misp-stix.txt). Many improvements in the MISP galaxy and especially the threat-actor galaxy, [360.net Threat Actors](https://www.misp-project.org/galaxy.html#_360_net_threat_actors) added. There is a detailed [changelog](https://www.misp-project.org/Changelog-misp-galaxy.txt). New `financial` taxonomy and many other taxonomies. There is a detailed [changelog](https://www.misp-project.org/Changelog-misp-taxonomies.txt). Multiple objects were updated and added, for [more details](https://www.misp-project.org/Changelog-misp-objects.txt). Various fixes in [misp-modules](https://misp.github.io/misp-modules/) for more [details](https://www.misp-project.org/Changelog-misp-modules.txt). 2022-09-26T14:12:21+00:00 dalton v3.2.5 dalton v3.2.5 2022-09-27T23:51:25+00:00 - Faster compile time for Zeek - Fixed bug where Zeek wouldn't run if there were not Zeek scripts 2022-09-27T23:51:25+00:00