http://open-source-security-software.net/releases.atom Recent releases 2022-12-10T06:35:21.365308+00:00 python-feedgen OpenTAXII 0.6.0 OpenTAXII 0.6.0 2022-05-25T11:15:18+00:00 Changelog ========= 0.6.0 (2022-05-25 ------------------ * Add `public_discovery` option to taxii2 config * Add support for publicly readable taxii 2 api roots 0.5.0 (2022-05-24) ------------------ * Add support for publicly readable taxii 2 collections 0.4.0 (2022-05-20) ------------------ * Move next_param handling into `OpenTAXII2PersistenceAPI` 0.3.0 (2022-04-13) ------------------ * Implement taxii2.1 support 0.3.0a4 (2022-04-13) -------------------- * Merge changes from 0.2.4 maintenance release 0.3.0a3 (2022-01-21) -------------------- * Fix bug that prevented booting with only taxii1 config (`#217 <https://github.com/eclecticiq/OpenTAXII/issues/217>`_ thanks `@azurekid <https://github.com/azurekid>`_ for the report) 0.3.0a2 (2021-12-27) -------------------- * Merge changes from 0.2.3 maintenance release 0.3.0a1 ------- * Add python 3.10 support 0.3.0a0 ------- * Enablement for future taxii2 implementation * Fix documentation build issues 0.2.4 (2022-04-13) ------------------ * Make sure werkzeug <2.1 and >=2.1 work correctly with auth system 0.2.3 (2021-12-22) ------------------ * Fix bug in multithreaded use of sqlite (`#210 <https://github.com/eclecticiq/OpenTAXII/issues/210>`_ thanks `@rohits144 <https://github.com/rohits144>`_ for the report) 0.2.2 (2021-11-05) ------------------ * Fix readthedocs build 0.2.1 (2021-11-03) ------------------ * Add tests for python 3.6, 3.7, 3.8, 3.9, pypy * Add tests for sqlite, mysql, mariadb, postgresql * Fix bug that broke ``delete_content_blocks`` when using mysql on sqlalchemy 1.3 * Docs: Add db schema diagram * Docs: Clarify how to get default data in a default (development) docker instance * Fix implicit routing in TAXII 1.1 Inboxes * Update jwt usage to pyjwt >= 2.0 (thanks `@SanyaKapoor <https://github.com/SanyaKapoor>`_) 0.2.0 (2020-06-30) ------------------ * Enforce UTC usage in datetime fields in SQL DB Persistence API. * `Fix for #114 <https://github.com/eclecticiq/OpenTAXII/issues/114>`_: reintroduce ``opentaxii-create-account`` CLI command. * `Fix for #153 <https://github.com/eclecticiq/OpenTAXII/issues/152>`_: check if user can modify a collection before advertising it over inbox service. * Multiple coding style fixes. * Various documentation updates. 0.1.12 (2019-03-06) ------------------- * Remove unnecessary print statements. 0.1.11 (2019-02-13) ------------------- * Make JSON logging consistent when the application is run via Gunicorn. * Set ``acceptable_destination`` key in status details instead of extended headers * Allow passing ``engine_parameters`` to ``SQLDatabaseAPI`` for those who want to customize SQLAlchemy engine parameters. * Require recent version of ``lxml`` for security reasons. * Various test and Docker infrastructure improvements. 0.1.10 (2018-06-03) ------------------- * Replace separate service/collection/account creation process with single ``opentaxii-sync-data`` CLI command. * Persistence and Auth APIs extended with missing CRUD methods, that are used by ``opentaxii-sync-data``. * Read/modify collection level ACL added. * DB models for default implementation of Persistence API and Auth API were changed. No automatic migration code is provided (sorry!), so upgrading might require manual DB migration. * Drop python2.7 from testing scope. * Various bug fixes and improvements. 0.1.9 (2017-06-19) ------------------ * `libtaxii <https://github.com/TAXIIProject/libtaxii>`_ dependency upgraded to 1.1.111. * Various bug fixes and improvements (thanks to `@bjigmp <https://github.com/bjigmp>`_, `@chorsley <https://github.com/chorsley>`_, `@rjprins <https://github.com/rjprins>`_). 0.1.8 (2017-02-21) ------------------ * Ability to enable/disable "huge trees" support in XML parser. Configuration property ``xml_parser_supports_huge_tree`` set to ``yes`` or ``true`` will disable security restrictions and force XML parser to support very deep trees and very long text content. * Adjust SQL Persistence API implemetation so it works smoothly with MySQL backend. * Use Python 3.5 instead of Python 3.4 for tests. 0.1.7 (2016-10-18) ------------------ * Minor fixes. * Dependencies were changed from hard-pinned to more flexible. * Example of production DB configuration added to docs. 0.1.6 (2016-06-01) ------------------ * Python 3.4 compatibility of the codebase. Tox configuration extended with python 3.4 environment run. * Flake8 full style compatibility. Flake8 check added to Tox configuration. * SQLAlchemy session scope issue fixed (related to `#38 <https://github.com/EclecticIQ/OpenTAXII/issues/38>`_). * `opentaxii-delete-blocks` CLI command added (related to `#45 <https://github.com/EclecticIQ/OpenTAXII/issues/45>`_). * `delete_content_blocks` method `added <https://github.com/EclecticIQ/OpenTAXII/commit/dc6fddc27a98e8450c7e05e583b2bfb741f6e197#diff-6814849ac352b2b74132f8fa52e0ec4eR213>`_ to Persistence API. * Collection's name is `required <https://github.com/EclecticIQ/OpenTAXII/commit/dc6fddc27a98e8450c7e05e583b2bfb741f6e197#diff-ce3f7b939e5c540480ac655aef32c513R116>`_ to be unique in default SQL DB Auth API implementation. 0.1.5 (2016-03-15) ------------------ * Fix for the issue with persistence backend returning ``None`` instead of ``InboxMessage`` object 0.1.4 (2016-02-25) ------------------ * Hard-coded dependencies in ``setup.py`` removed. 0.1.3 (2016-02-25) ------------------ * Versions of dependencies are pinned. * Code adjusted for a new version of `anyconfig <https://pypi.python.org/pypi/anyconfig>`_ API. * Test for configuration loading added. 0.1.2 (2015-07-24) ------------------ * Docker configuration added. * Health check endpoint added. * Basic authentication support added. * Temporary workaround for `Issue #191 <https://github.com/TAXIIProject/libtaxii/issues/191>`_. * Method ``get_domain`` in Persistence API returns domain value configured for ``service_id``. If nothing returned, value set in configuration file will be used. * Performance optimisations. * Bug fixes and style improvements. 0.1.1 (2015-04-08) ------------------ * Alias for Root Logger added to logging configuration. * Context object in a request scope that holds account and token added. * Support for OPTIONS HTTP request to enable auto version negotiation added. * Documentation improved. 0.1.0 (2015-03-31) ------------------ * Initial release 2022-05-25T11:15:18+00:00 dalton v3.2.3 dalton v3.2.3 2022-05-26T14:51:26+00:00 - Added share link on job results page to recreate job automatically - Added PCAP drag and drop to UI when creating a new job 2022-05-26T14:51:26+00:00 OpenTAXII 0.7.0 OpenTAXII 0.7.0 2022-05-27T18:01:28+00:00 Changelog ========= 0.7.0 (2022-05-27) ------------------ * Nest taxii2 endpoints under `/taxii2/` 0.6.0 (2022-05-25) ------------------ * Add `public_discovery` option to taxii2 config * Add support for publicly readable taxii 2 api roots 0.5.0 (2022-05-24) ------------------ * Add support for publicly readable taxii 2 collections 0.4.0 (2022-05-20) ------------------ * Move next_param handling into `OpenTAXII2PersistenceAPI` 0.3.0 (2022-04-13) ------------------ * Implement taxii2.1 support 0.3.0a4 (2022-04-13) -------------------- * Merge changes from 0.2.4 maintenance release 0.3.0a3 (2022-01-21) -------------------- * Fix bug that prevented booting with only taxii1 config (`#217 <https://github.com/eclecticiq/OpenTAXII/issues/217>`_ thanks `@azurekid <https://github.com/azurekid>`_ for the report) 0.3.0a2 (2021-12-27) -------------------- * Merge changes from 0.2.3 maintenance release 0.3.0a1 ------- * Add python 3.10 support 0.3.0a0 ------- * Enablement for future taxii2 implementation * Fix documentation build issues 0.2.4 (2022-04-13) ------------------ * Make sure werkzeug <2.1 and >=2.1 work correctly with auth system 0.2.3 (2021-12-22) ------------------ * Fix bug in multithreaded use of sqlite (`#210 <https://github.com/eclecticiq/OpenTAXII/issues/210>`_ thanks `@rohits144 <https://github.com/rohits144>`_ for the report) 0.2.2 (2021-11-05) ------------------ * Fix readthedocs build 0.2.1 (2021-11-03) ------------------ * Add tests for python 3.6, 3.7, 3.8, 3.9, pypy * Add tests for sqlite, mysql, mariadb, postgresql * Fix bug that broke ``delete_content_blocks`` when using mysql on sqlalchemy 1.3 * Docs: Add db schema diagram * Docs: Clarify how to get default data in a default (development) docker instance * Fix implicit routing in TAXII 1.1 Inboxes * Update jwt usage to pyjwt >= 2.0 (thanks `@SanyaKapoor <https://github.com/SanyaKapoor>`_) 0.2.0 (2020-06-30) ------------------ * Enforce UTC usage in datetime fields in SQL DB Persistence API. * `Fix for #114 <https://github.com/eclecticiq/OpenTAXII/issues/114>`_: reintroduce ``opentaxii-create-account`` CLI command. * `Fix for #153 <https://github.com/eclecticiq/OpenTAXII/issues/152>`_: check if user can modify a collection before advertising it over inbox service. * Multiple coding style fixes. * Various documentation updates. 0.1.12 (2019-03-06) ------------------- * Remove unnecessary print statements. 0.1.11 (2019-02-13) ------------------- * Make JSON logging consistent when the application is run via Gunicorn. * Set ``acceptable_destination`` key in status details instead of extended headers * Allow passing ``engine_parameters`` to ``SQLDatabaseAPI`` for those who want to customize SQLAlchemy engine parameters. * Require recent version of ``lxml`` for security reasons. * Various test and Docker infrastructure improvements. 0.1.10 (2018-06-03) ------------------- * Replace separate service/collection/account creation process with single ``opentaxii-sync-data`` CLI command. * Persistence and Auth APIs extended with missing CRUD methods, that are used by ``opentaxii-sync-data``. * Read/modify collection level ACL added. * DB models for default implementation of Persistence API and Auth API were changed. No automatic migration code is provided (sorry!), so upgrading might require manual DB migration. * Drop python2.7 from testing scope. * Various bug fixes and improvements. 0.1.9 (2017-06-19) ------------------ * `libtaxii <https://github.com/TAXIIProject/libtaxii>`_ dependency upgraded to 1.1.111. * Various bug fixes and improvements (thanks to `@bjigmp <https://github.com/bjigmp>`_, `@chorsley <https://github.com/chorsley>`_, `@rjprins <https://github.com/rjprins>`_). 0.1.8 (2017-02-21) ------------------ * Ability to enable/disable "huge trees" support in XML parser. Configuration property ``xml_parser_supports_huge_tree`` set to ``yes`` or ``true`` will disable security restrictions and force XML parser to support very deep trees and very long text content. * Adjust SQL Persistence API implemetation so it works smoothly with MySQL backend. * Use Python 3.5 instead of Python 3.4 for tests. 0.1.7 (2016-10-18) ------------------ * Minor fixes. * Dependencies were changed from hard-pinned to more flexible. * Example of production DB configuration added to docs. 0.1.6 (2016-06-01) ------------------ * Python 3.4 compatibility of the codebase. Tox configuration extended with python 3.4 environment run. * Flake8 full style compatibility. Flake8 check added to Tox configuration. * SQLAlchemy session scope issue fixed (related to `#38 <https://github.com/EclecticIQ/OpenTAXII/issues/38>`_). * `opentaxii-delete-blocks` CLI command added (related to `#45 <https://github.com/EclecticIQ/OpenTAXII/issues/45>`_). * `delete_content_blocks` method `added <https://github.com/EclecticIQ/OpenTAXII/commit/dc6fddc27a98e8450c7e05e583b2bfb741f6e197#diff-6814849ac352b2b74132f8fa52e0ec4eR213>`_ to Persistence API. * Collection's name is `required <https://github.com/EclecticIQ/OpenTAXII/commit/dc6fddc27a98e8450c7e05e583b2bfb741f6e197#diff-ce3f7b939e5c540480ac655aef32c513R116>`_ to be unique in default SQL DB Auth API implementation. 0.1.5 (2016-03-15) ------------------ * Fix for the issue with persistence backend returning ``None`` instead of ``InboxMessage`` object 0.1.4 (2016-02-25) ------------------ * Hard-coded dependencies in ``setup.py`` removed. 0.1.3 (2016-02-25) ------------------ * Versions of dependencies are pinned. * Code adjusted for a new version of `anyconfig <https://pypi.python.org/pypi/anyconfig>`_ API. * Test for configuration loading added. 0.1.2 (2015-07-24) ------------------ * Docker configuration added. * Health check endpoint added. * Basic authentication support added. * Temporary workaround for `Issue #191 <https://github.com/TAXIIProject/libtaxii/issues/191>`_. * Method ``get_domain`` in Persistence API returns domain value configured for ``service_id``. If nothing returned, value set in configuration file will be used. * Performance optimisations. * Bug fixes and style improvements. 0.1.1 (2015-04-08) ------------------ * Alias for Root Logger added to logging configuration. * Context object in a request scope that holds account and token added. * Support for OPTIONS HTTP request to enable auto version negotiation added. * Documentation improved. 0.1.0 (2015-03-31) ------------------ * Initial release 2022-05-27T18:01:28+00:00 MISP v2.4.159 MISP v2.4.159 2022-05-30T17:05:38+00:00 We are pleased to announce the immediate availability of MISP v2.4.159. This releases includes many improvements, bugs fixing and improvement concerning performance on large dataset. ![](https://www.misp-project.org/img/blog/graph-syria.png) # Performance Improvements - [DB] Add MysqlExtended DboSource to support index query hints. - [Query] Add new setting to disable taxonomy checks when browsing data. - We discovered that some MISP users are still using slow file-based session handling in PHP. Now, we added in the diagnostic, if session is file based. We recommend everyone to use the Redis session. - Many additional speed-up and faster functions in the MISP internals. - Reduce memory usage when generating all correlations. # Improvements - [Feed] Allow option to disable correlations for all events coming from a feed. This can be useful when correlation requires to be disabled for an imported feed. - [UI] Allow to upload MISP event by pasting data to textarea in addition to the file upload. - An optional feature `clusters:attachMultipleClusters` is now available to allow the mirroring of attribute clusters to event. - [auditlog] Support for fetch event changes from specific time. - [UI] Allow to filter attributes from Related Events box. - [UI] Allow to filter attributes from warninglist box. - [UI] Many UI improvements to make the interface easier to read. - [UI] Disable correlation checkbox for non correlating types. - [STIX 2 import] Better Galaxies parsing by looking for the ATT&CK technique id. - [API] Enable sharing group filter for Event controller not just attribute. # Fixes - [STIX] Avoiding non RFC-4122 UUIDs to be imported (and therefore skipped) - [STIX 1 import] Save process network connections. - [STIX 1 import] Fixed galaxy tag_names fetching from TTP names. # Knowledge Bases ## MISP Taxonomies - [dga] First version of the DGA taxonomy based on https://www.usenix.org/system/files/conference/usenixsecurity16/sec16_paper_plohmann.pdf. - GrayZone of Active Defense, originaly published by Washington University, v2 created and updated by DCG420 - Various fixes to existing taxonomies. ## MISP Objects Template - A new PaloAlto Threat Event object template has been added. - A updated security playbook has been added. - A new ransom negotiation object has been added. - An improved Passive SSH template object. - Various fixes and improvements to different object templates such as email, virustotal-submissions and others. ## MISP Galaxy - Improved Cryptominers galaxy. - Improved backdoors galaxy. - Threat Actor galaxy updated and extended with new threat-actors. - MISP Galaxy updated for MITRE ATT&CK v11.2. # Acknowledgement We would like to thank all the [contributors](https://www.misp-project.org/contributors), reporters and users who have helped us in the past months to improve MISP and information sharing at large. This release includes multiple updates in [misp-objects](https://www.misp-project.org/objects.html), [misp-taxonomies](https://www.misp-project.org/taxonomies.html) and [misp-galaxy](https://www.misp-project.org/galaxy.html). As always, a detailed and [complete changelog is available](https://www.misp-project.org/Changelog.txt) with all the fixes, changes and improvements in MISP core. Additional changelogs are available for [misp galaxy](https://www.misp-project.org/Changelog-misp-galaxy.txt), [misp-taxonomies](https://www.misp-project.org/Changelog-misp-taxonomies.txt), [misp-objects](https://www.misp-project.org/Changelog-misp-objects.txt) and [misp-modules](https://www.misp-project.org/Changelog-misp-modules.txt) 2022-05-30T17:05:38+00:00 HyperDbg v0.1.0 HyperDbg v0.1.0 2022-05-31T14:08:32+00:00 # HyperDbg v0.1 is released! **If you’re enjoying HyperDbg, don’t forget to give a star 🌟 on GitHub!** Please visit [Build & Install](https://docs.hyperdbg.org/getting-started/build-and-install) to configure the environment for running **HyperDbg**. Check out the [Quick Start](https://docs.hyperdbg.org/getting-started/quick-start) and [Frequently Asked Questions (FAQs)](https://docs.hyperdbg.org/getting-started/faq) to learn more. You can use the examples of [using the debugger](https://docs.hyperdbg.org/using-hyperdbg/kernel-mode-debugging/examples) and the [script engine](https://docs.hyperdbg.org/commands/scripting-language/examples) to get started with **HyperDbg**. ## New Fearues * Advanced Hypervisor-based Kernel Mode Debugger [<a href="https://docs.hyperdbg.org/using-hyperdbg/kernel-mode-debugging" target="_blank">link</a>][<a href="https://docs.hyperdbg.org/getting-started/attach-to-hyperdbg/debug" target="_blank">link</a>][<a href="https://docs.hyperdbg.org/getting-started/attach-to-hyperdbg/local-debugging" target="_blank">link</a>] * Classic EPT Hook (Hidden Breakpoint) [<a href="https://docs.hyperdbg.org/commands/extension-commands/epthook" target="_blank">link</a>][<a href="https://docs.hyperdbg.org/design/features/vmm-module/design-of-epthook" target="_blank">link</a>][<a href="https://docs.hyperdbg.org/using-hyperdbg/kernel-mode-debugging/examples/events/hooking-any-function" target="_blank">link</a>] * Inline EPT Hook (Inline Hook) [<a href="https://docs.hyperdbg.org/commands/extension-commands/epthook2" target="_blank">link</a>][<a href="https://docs.hyperdbg.org/design/features/vmm-module/design-of-epthook2" target="_blank">link</a>] * Monitor Memory For R/W (Emulating Hardware Debug Registers Without Limitation) [<a href="https://docs.hyperdbg.org/commands/extension-commands/monitor" target="_blank">link</a>][<a href="https://docs.hyperdbg.org/design/features/vmm-module/design-of-monitor" target="_blank">link</a>][<a href="https://docs.hyperdbg.org/using-hyperdbg/kernel-mode-debugging/examples/events/monitoring-accesses-to-structures" target="_blank">link</a>] * SYSCALL Hook (Disable EFER & Handle #UD) [<a href="https://docs.hyperdbg.org/commands/extension-commands/syscall" target="_blank">link</a>][<a href="https://docs.hyperdbg.org/design/features/vmm-module/design-of-syscall-and-sysret" target="_blank">link</a>][<a href="https://docs.hyperdbg.org/using-hyperdbg/kernel-mode-debugging/examples/events/intercepting-all-syscalls" target="_blank">link</a>] * SYSRET Hook (Disable EFER & Handle #UD) [<a href="https://docs.hyperdbg.org/commands/extension-commands/sysret" target="_blank">link</a>][<a href="https://docs.hyperdbg.org/design/features/vmm-module/design-of-syscall-and-sysret" target="_blank">link</a>] * CPUID Hook & Monitor [<a href="https://docs.hyperdbg.org/commands/extension-commands/cpuid" target="_blank">link</a>][<a href="https://docs.hyperdbg.org/using-hyperdbg/kernel-mode-debugging/examples/events/triggering-special-instructions" target="_blank">link</a>] * RDMSR Hook & Monitor [<a href="https://docs.hyperdbg.org/commands/extension-commands/msrread" target="_blank">link</a>][<a href="https://docs.hyperdbg.org/using-hyperdbg/kernel-mode-debugging/examples/events/identifying-system-behavior" target="_blank">link</a>] * WRMSR Hook & Monitor [<a href="https://docs.hyperdbg.org/commands/extension-commands/msrwrite" target="_blank">link</a>][<a href="https://docs.hyperdbg.org/using-hyperdbg/kernel-mode-debugging/examples/events/identifying-system-behavior" target="_blank">link</a>] * RDTSC/RDTSCP Hook & Monitor [<a href="https://docs.hyperdbg.org/commands/extension-commands/tsc" target="_blank">link</a>] * RDPMC Hook & Monitor [<a href="https://docs.hyperdbg.org/commands/extension-commands/pmc" target="_blank">link</a>] * VMCALL Hook & Monitor [<a href="https://docs.hyperdbg.org/commands/extension-commands/vmcall" target="_blank">link</a>] * Debug Registers Hook & Monitor [<a href="https://docs.hyperdbg.org/commands/extension-commands/dr" target="_blank">link</a>] * I/O Port (In Instruction) Hook & Monitor [<a href="https://docs.hyperdbg.org/commands/extension-commands/ioin" target="_blank">link</a>][<a href="https://docs.hyperdbg.org/using-hyperdbg/kernel-mode-debugging/examples/events/triggering-special-instructions" target="_blank">link</a>] * I/O Port (Out Instruction) Hook & Monitor [<a href="https://docs.hyperdbg.org/commands/extension-commands/ioout" target="_blank">link</a>][<a href="https://docs.hyperdbg.org/using-hyperdbg/kernel-mode-debugging/examples/events/triggering-special-instructions" target="_blank">link</a>] * MMIO Monitor [<a href="https://docs.hyperdbg.org/commands/extension-commands/monitor" target="_blank">link</a>] * Exception (IDT < 32) Monitor [<a href="https://docs.hyperdbg.org/commands/extension-commands/exception" target="_blank">link</a>][<a href="https://docs.hyperdbg.org/design/features/vmm-module/design-of-exception-and-interrupt" target="_blank">link</a>][<a href="https://docs.hyperdbg.org/using-hyperdbg/kernel-mode-debugging/examples/events/identifying-system-behavior" target="_blank">link</a>] * External-Interrupt (IDT > 32) Monitor [<a href="https://docs.hyperdbg.org/commands/extension-commands/interrupt" target="_blank">link</a>][<a href="https://docs.hyperdbg.org/design/features/vmm-module/design-of-exception-and-interrupt" target="_blank">link</a>][<a href="https://docs.hyperdbg.org/using-hyperdbg/kernel-mode-debugging/examples/events/identifying-system-behavior" target="_blank">link</a>] * Running Automated Scripts [<a href="https://docs.hyperdbg.org/commands/scripting-language/hyperdbg-scripts" target="_blank">link</a>] * Transparent-mode (Anti-debugging and Anti-hypervisor Resistance) [<a href="https://docs.hyperdbg.org/tips-and-tricks/considerations/transparent-mode" target="_blank">link</a>][<a href="https://docs.hyperdbg.org/using-hyperdbg/kernel-mode-debugging/examples/misc/defeating-anti-debug-and-anti-hypervisor-methods" target="_blank">link</a>] * Running Custom Assembly In Both VMX-root, VMX non-root (Kernel & User) [<a href="https://docs.hyperdbg.org/using-hyperdbg/prerequisites/how-to-create-an-action" target="_blank">link</a>] * Checking For Custom Conditions [<a href="https://docs.hyperdbg.org/using-hyperdbg/prerequisites/how-to-create-a-condition" target="_blank">link</a>][<a href="https://docs.hyperdbg.org/design/debugger-internals/conditions" target="_blank">link</a>] * Process-specific & Thread-specific Debugging [<a href="https://docs.hyperdbg.org/commands/meta-commands/.process" target="_blank">link</a>][<a href="https://docs.hyperdbg.org/commands/meta-commands/.thread" target="_blank">link</a>][<a href="https://docs.hyperdbg.org/using-hyperdbg/user-mode-debugging/examples/basics/switching-to-a-specific-process-or-thread" target="_blank">link</a>] * VMX-root Compatible Message Tracing [<a href="https://docs.hyperdbg.org/design/features/vmm-module/vmx-root-mode-compatible-message-tracing" target="_blank">link</a>] * Powerful Kernel Side Scripting Engine [<a href="https://docs.hyperdbg.org/commands/scripting-language" target="_blank">link</a>][<a href="https://docs.hyperdbg.org/design/script-engine" target="_blank">link</a>] * Support To Symbols (Parsing PDB Files) [<a href="https://docs.hyperdbg.org/commands/meta-commands/.sympath" target="_blank">link</a>][<a href="https://docs.hyperdbg.org/commands/meta-commands/.sym" target="_blank">link</a>] * Event Forwarding (#DFIR) [<a href="https://docs.hyperdbg.org/tips-and-tricks/misc/event-forwarding" target="_blank">link</a>][<a href="https://docs.hyperdbg.org/commands/debugging-commands/output" target="_blank">link</a>] * Transparent Breakpoint Handler [<a href="https://docs.hyperdbg.org/commands/debugging-commands/bp" target="_blank">link</a>][<a href="https://docs.hyperdbg.org/using-hyperdbg/kernel-mode-debugging/examples/basics/setting-breakpoints-and-stepping-instructions" target="_blank">link</a>] * Various Custom Scripts [<a href="https://github.com/HyperDbg/scripts" target="_blank">link</a>] **Note**: community contributions are always welcomed and appreciated. If you plan to contribute a new feature, it's best to [discuss](https://github.com/HyperDbg/HyperDbg/discussions) it first. Bug fixes, tests, and documentation improvements are greatly appreciated. 2022-05-31T14:08:32+00:00 maltrail 0.46 maltrail 0.46 2022-05-31T22:11:06+00:00 Start-of-month release 2022-05-31T22:11:06+00:00 syncthing v1.20.2-rc.3 syncthing v1.20.2-rc.3 2022-06-01T05:34:59+00:00 Bugfixes: - #7289: TCP port 0 is announced in the LAN beacon - #8314: https://relays.syncthing.net/endpoint contains non-valid URLs with spaces - #8355: Upgrading from v1.19.2 to v1.20.x now requires chmod for syncing files Enhancements: - #8264: Include default ignore patterns in the GUI's advanced configuration editor - #8310: Indicate folders / devices where the remote end is paused. 2022-06-01T05:34:59+00:00 PyPCAPKit v0.16.0 PyPCAPKit v0.16.0 2022-06-01T05:35:59+00:00 ## What's Changed * revised entire project * type annotations * protocol redesign * linter compliance * added and revised tests (see #82, bugfix help wanted) * general bugfix in the project (#101 and many more) * reviewed and rearranged APIs, especially * added `pcapkit.corekit.multidict` module based on `Werkzeug` project * removed `validators` & some decorators from `pcapkit.utilities` * moved PCAP & aux protocols under `pcapkit.misc` * redesigned `Protocol` class to integrate parsing & construction at the same time, as well as better subclass protocol implementation experience * initialised PyPCAPKit Enhancement Proposals discussion channel (see #106) **Full Changelog**: https://github.com/JarryShaw/PyPCAPKit/compare/v0.15.5...v0.16.0 2022-06-01T05:35:59+00:00 OpenTAXII 0.8.0 OpenTAXII 0.8.0 2022-06-05T14:57:50+00:00 Changelog ========= 0.8.0 (2022-06-05) ------------------ * Nest details inside taxii2 job and allow counts without details 0.7.0 (2022-05-27) ------------------ * Nest taxii2 endpoints under `/taxii2/` 0.6.0 (2022-05-25) ------------------ * Add `public_discovery` option to taxii2 config * Add support for publicly readable taxii 2 api roots 0.5.0 (2022-05-24) ------------------ * Add support for publicly readable taxii 2 collections 0.4.0 (2022-05-20) ------------------ * Move next_param handling into `OpenTAXII2PersistenceAPI` 0.3.0 (2022-04-13) ------------------ * Implement taxii2.1 support 0.3.0a4 (2022-04-13) -------------------- * Merge changes from 0.2.4 maintenance release 0.3.0a3 (2022-01-21) -------------------- * Fix bug that prevented booting with only taxii1 config (`#217 <https://github.com/eclecticiq/OpenTAXII/issues/217>`_ thanks `@azurekid <https://github.com/azurekid>`_ for the report) 0.3.0a2 (2021-12-27) -------------------- * Merge changes from 0.2.3 maintenance release 0.3.0a1 ------- * Add python 3.10 support 0.3.0a0 ------- * Enablement for future taxii2 implementation * Fix documentation build issues 0.2.4 (2022-04-13) ------------------ * Make sure werkzeug <2.1 and >=2.1 work correctly with auth system 0.2.3 (2021-12-22) ------------------ * Fix bug in multithreaded use of sqlite (`#210 <https://github.com/eclecticiq/OpenTAXII/issues/210>`_ thanks `@rohits144 <https://github.com/rohits144>`_ for the report) 0.2.2 (2021-11-05) ------------------ * Fix readthedocs build 0.2.1 (2021-11-03) ------------------ * Add tests for python 3.6, 3.7, 3.8, 3.9, pypy * Add tests for sqlite, mysql, mariadb, postgresql * Fix bug that broke ``delete_content_blocks`` when using mysql on sqlalchemy 1.3 * Docs: Add db schema diagram * Docs: Clarify how to get default data in a default (development) docker instance * Fix implicit routing in TAXII 1.1 Inboxes * Update jwt usage to pyjwt >= 2.0 (thanks `@SanyaKapoor <https://github.com/SanyaKapoor>`_) 0.2.0 (2020-06-30) ------------------ * Enforce UTC usage in datetime fields in SQL DB Persistence API. * `Fix for #114 <https://github.com/eclecticiq/OpenTAXII/issues/114>`_: reintroduce ``opentaxii-create-account`` CLI command. * `Fix for #153 <https://github.com/eclecticiq/OpenTAXII/issues/152>`_: check if user can modify a collection before advertising it over inbox service. * Multiple coding style fixes. * Various documentation updates. 0.1.12 (2019-03-06) ------------------- * Remove unnecessary print statements. 0.1.11 (2019-02-13) ------------------- * Make JSON logging consistent when the application is run via Gunicorn. * Set ``acceptable_destination`` key in status details instead of extended headers * Allow passing ``engine_parameters`` to ``SQLDatabaseAPI`` for those who want to customize SQLAlchemy engine parameters. * Require recent version of ``lxml`` for security reasons. * Various test and Docker infrastructure improvements. 0.1.10 (2018-06-03) ------------------- * Replace separate service/collection/account creation process with single ``opentaxii-sync-data`` CLI command. * Persistence and Auth APIs extended with missing CRUD methods, that are used by ``opentaxii-sync-data``. * Read/modify collection level ACL added. * DB models for default implementation of Persistence API and Auth API were changed. No automatic migration code is provided (sorry!), so upgrading might require manual DB migration. * Drop python2.7 from testing scope. * Various bug fixes and improvements. 0.1.9 (2017-06-19) ------------------ * `libtaxii <https://github.com/TAXIIProject/libtaxii>`_ dependency upgraded to 1.1.111. * Various bug fixes and improvements (thanks to `@bjigmp <https://github.com/bjigmp>`_, `@chorsley <https://github.com/chorsley>`_, `@rjprins <https://github.com/rjprins>`_). 0.1.8 (2017-02-21) ------------------ * Ability to enable/disable "huge trees" support in XML parser. Configuration property ``xml_parser_supports_huge_tree`` set to ``yes`` or ``true`` will disable security restrictions and force XML parser to support very deep trees and very long text content. * Adjust SQL Persistence API implemetation so it works smoothly with MySQL backend. * Use Python 3.5 instead of Python 3.4 for tests. 0.1.7 (2016-10-18) ------------------ * Minor fixes. * Dependencies were changed from hard-pinned to more flexible. * Example of production DB configuration added to docs. 0.1.6 (2016-06-01) ------------------ * Python 3.4 compatibility of the codebase. Tox configuration extended with python 3.4 environment run. * Flake8 full style compatibility. Flake8 check added to Tox configuration. * SQLAlchemy session scope issue fixed (related to `#38 <https://github.com/EclecticIQ/OpenTAXII/issues/38>`_). * `opentaxii-delete-blocks` CLI command added (related to `#45 <https://github.com/EclecticIQ/OpenTAXII/issues/45>`_). * `delete_content_blocks` method `added <https://github.com/EclecticIQ/OpenTAXII/commit/dc6fddc27a98e8450c7e05e583b2bfb741f6e197#diff-6814849ac352b2b74132f8fa52e0ec4eR213>`_ to Persistence API. * Collection's name is `required <https://github.com/EclecticIQ/OpenTAXII/commit/dc6fddc27a98e8450c7e05e583b2bfb741f6e197#diff-ce3f7b939e5c540480ac655aef32c513R116>`_ to be unique in default SQL DB Auth API implementation. 0.1.5 (2016-03-15) ------------------ * Fix for the issue with persistence backend returning ``None`` instead of ``InboxMessage`` object 0.1.4 (2016-02-25) ------------------ * Hard-coded dependencies in ``setup.py`` removed. 0.1.3 (2016-02-25) ------------------ * Versions of dependencies are pinned. * Code adjusted for a new version of `anyconfig <https://pypi.python.org/pypi/anyconfig>`_ API. * Test for configuration loading added. 0.1.2 (2015-07-24) ------------------ * Docker configuration added. * Health check endpoint added. * Basic authentication support added. * Temporary workaround for `Issue #191 <https://github.com/TAXIIProject/libtaxii/issues/191>`_. * Method ``get_domain`` in Persistence API returns domain value configured for ``service_id``. If nothing returned, value set in configuration file will be used. * Performance optimisations. * Bug fixes and style improvements. 0.1.1 (2015-04-08) ------------------ * Alias for Root Logger added to logging configuration. * Context object in a request scope that holds account and token added. * Support for OPTIONS HTTP request to enable auto version negotiation added. * Documentation improved. 0.1.0 (2015-03-31) ------------------ * Initial release 2022-06-05T14:57:50+00:00 syncthing v1.20.2 syncthing v1.20.2 2022-06-07T10:13:46+00:00 Bugfixes: - #7289: TCP port 0 is announced in the LAN beacon - #8314: https://relays.syncthing.net/endpoint contains non-valid URLs with spaces - #8355: Upgrading from v1.19.2 to v1.20.x now requires chmod for syncing files Enhancements: - #8264: Include default ignore patterns in the GUI's advanced configuration editor - #8310: Indicate folders / devices where the remote end is paused. 2022-06-07T10:13:46+00:00 PyPCAPKit v0.16.1 PyPCAPKit v0.16.1 2022-06-08T19:59:55+00:00 ## What's Changed * changed `warnings.warn` calls to `pcapkit.utilities.warnings.warn` * revised missing optional dependency warnings (add for CLI & vendor) * updated version strings in repository * `make isort` **Full Changelog**: https://github.com/JarryShaw/PyPCAPKit/compare/v0.16.0...v0.16.1 2022-06-08T19:59:55+00:00 rocksdb v7.3.1 rocksdb v7.3.1 2022-06-10T23:08:05+00:00 ## 7.3.1 (06/08/2022) ### Bug Fixes * Fix a bug in WAL tracking. Before this PR (#10087), calling `SyncWAL()` on the only WAL file of the db will not log the event in MANIFEST, thus allowing a subsequent `DB::Open` even if the WAL file is missing or corrupted. * Fixed a bug for non-TransactionDB with avoid_flush_during_recovery = true and TransactionDB where in case of crash, min_log_number_to_keep may not change on recovery and persisting a new MANIFEST with advanced log_numbers for some column families, results in "column family inconsistency" error on second recovery. As a solution, RocksDB will persist the new MANIFEST after successfully syncing the new WAL. If a future recovery starts from the new MANIFEST, then it means the new WAL is successfully synced. Due to the sentinel empty write batch at the beginning, kPointInTimeRecovery of WAL is guaranteed to go after this point. If future recovery starts from the old MANIFEST, it means the writing the new MANIFEST failed. We won't have the "SST ahead of WAL" error. * Fixed a bug where RocksDB DB::Open() may creates and writes to two new MANIFEST files even before recovery succeeds. Now writes to MANIFEST are persisted only after recovery is successful. ## 7.3.0 (05/20/2022) ### Bug Fixes * Fixed a bug where manual flush would block forever even though flush options had wait=false. * Fixed a bug where RocksDB could corrupt DBs with `avoid_flush_during_recovery == true` by removing valid WALs, leading to `Status::Corruption` with message like "SST file is ahead of WALs" when attempting to reopen. * Fixed a bug in async_io path where incorrect length of data is read by FilePrefetchBuffer if data is consumed from two populated buffers and request for more data is sent. * Fixed a CompactionFilter bug. Compaction filter used to use `Delete` to remove keys, even if the keys should be removed with `SingleDelete`. Mixing `Delete` and `SingleDelete` may cause undefined behavior. * Fixed a bug in `WritableFileWriter::WriteDirect` and `WritableFileWriter::WriteDirectWithChecksum`. The rate_limiter_priority specified in ReadOptions was not passed to the RateLimiter when requesting a token. * Fixed a bug which might cause process crash when I/O error happens when reading an index block in MultiGet(). ### New Features * DB::GetLiveFilesStorageInfo is ready for production use. * Add new stats PREFETCHED_BYTES_DISCARDED which records number of prefetched bytes discarded by RocksDB FilePrefetchBuffer on destruction and POLL_WAIT_MICROS records wait time for FS::Poll API completion. * RemoteCompaction supports table_properties_collector_factories override on compaction worker. * Start tracking SST unique id in MANIFEST, which will be used to verify with SST properties during DB open to make sure the SST file is not overwritten or misplaced. A db option `verify_sst_unique_id_in_manifest` is introduced to enable/disable the verification, if enabled all SST files will be opened during DB-open to verify the unique id (default is false), so it's recommended to use it with `max_open_files = -1` to pre-open the files. * Added the ability to concurrently read data blocks from multiple files in a level in batched MultiGet. This can be enabled by setting the async_io option in ReadOptions. Using this feature requires a FileSystem that supports ReadAsync (PosixFileSystem is not supported yet for this), and for RocksDB to be compiled with folly and c++20. * Add FileSystem::ReadAsync API in io_tracing. ### Public API changes * Add rollback_deletion_type_callback to TransactionDBOptions so that write-prepared transactions know whether to issue a Delete or SingleDelete to cancel a previous key written during prior prepare phase. The PR aims to prevent mixing SingleDeletes and Deletes for the same key that can lead to undefined behaviors for write-prepared transactions. * EXPERIMENTAL: Add new API AbortIO in file_system to abort the read requests submitted asynchronously. * CompactionFilter::Decision has a new value: kRemoveWithSingleDelete. If CompactionFilter returns this decision, then CompactionIterator will use `SingleDelete` to mark a key as removed. * Renamed CompactionFilter::Decision::kRemoveWithSingleDelete to kPurge since the latter sounds more general and hides the implementation details of how compaction iterator handles keys. * Added ability to specify functions for Prepare and Validate to OptionsTypeInfo. Added methods to OptionTypeInfo to set the functions via an API. These methods are intended for RocksDB plugin developers for configuration management. * Added a new immutable db options, enforce_single_del_contracts. If set to false (default is true), compaction will NOT fail due to a single delete followed by a delete for the same key. The purpose of this temporay option is to help existing use cases migrate. * Introduce `BlockBasedTableOptions::cache_usage_options` and use that to replace `BlockBasedTableOptions::reserve_table_builder_memory` and `BlockBasedTableOptions::reserve_table_reader_memory`. * Changed `GetUniqueIdFromTableProperties` to return a 128-bit unique identifier, which will be the standard size now. The old functionality (192-bit) is available from `GetExtendedUniqueIdFromTableProperties`. Both functions are no longer "experimental" and are ready for production use. * In IOOptions, mark `prio` as deprecated for future removal. * In `file_system.h`, mark `IOPriority` as deprecated for future removal. * Add an option, `CompressionOptions::use_zstd_dict_trainer`, to indicate whether zstd dictionary trainer should be used for generating zstd compression dictionaries. The default value of this option is true for backward compatibility. When this option is set to false, zstd API `ZDICT_finalizeDictionary` is used to generate compression dictionaries. * Seek API which positions itself every LevelIterator on the correct data block in the correct SST file which can be parallelized if ReadOptions.async_io option is enabled. * Add new stat number_async_seek in PerfContext that indicates number of async calls made by seek to prefetch data. ### Bug Fixes * RocksDB calls FileSystem::Poll API during FilePrefetchBuffer destruction which impacts performance as it waits for read requets completion which is not needed anymore. Calling FileSystem::AbortIO to abort those requests instead fixes that performance issue. * Fixed unnecessary block cache contention when queries within a MultiGet batch and across parallel batches access the same data block, which previously could cause severely degraded performance in this unusual case. (In more typical MultiGet cases, this fix is expected to yield a small or negligible performance improvement.) ### Behavior changes * Enforce the existing contract of SingleDelete so that SingleDelete cannot be mixed with Delete because it leads to undefined behavior. Fix a number of unit tests that violate the contract but happen to pass. * ldb `--try_load_options` default to true if `--db` is specified and not creating a new DB, the user can still explicitly disable that by `--try_load_options=false` (or explicitly enable that by `--try_load_options`). * During Flush write or Compaction write/read, the WriteController is used to determine whether DB writes are stalled or slowed down. The priority (Env::IOPriority) can then be determined accordingly and be passed in IOOptions to the file system. 2022-06-10T23:08:05+00:00 cwe_checker v0.6 cwe_checker v0.6 2022-06-13T05:48:56+00:00 Version 0.6 contains improved abstract domains able to represent data more precise and more complete. Furthermore, the Pointer Inference analysis was reworked to be a bottom-up analysis and an additional function signature analysis step was added to the analysis pipeline. These improvements allow all analyses depending on the Pointer Inference to be both more precise and more complete. Other highlights include: - A new command line flag for analyzing bare-metal binaries. - The check for *CWE-78: Command line injections* was completely rewritten using abstract domains for strings. - The check for *CWE-119: Buffer Overflow* was completely rewritten and now emits additional data flow information in the JSON output to help with root cause analysis. - The check for *CWE-416: Use After Free* was completely rewritten and now emits additional data flow information in the JSON output to help with root cause analysis. See the CHANGES.md for more details. 2022-06-13T05:48:56+00:00 OpenTAXII 0.9.0 OpenTAXII 0.9.0 2022-06-13T14:44:11+00:00 Changelog ========= 0.9.0 (2022-06-13) ------------------ * Allow custom properties. This can be disabled by config option ``allow_custom_properties`` 0.8.0 (2022-06-05) ------------------ * Nest details inside taxii2 job and allow counts without details 0.7.0 (2022-05-27) ------------------ * Nest taxii2 endpoints under `/taxii2/` 0.6.0 (2022-05-25) ------------------ * Add `public_discovery` option to taxii2 config * Add support for publicly readable taxii 2 api roots 0.5.0 (2022-05-24) ------------------ * Add support for publicly readable taxii 2 collections 0.4.0 (2022-05-20) ------------------ * Move next_param handling into `OpenTAXII2PersistenceAPI` 0.3.0 (2022-04-13) ------------------ * Implement taxii2.1 support 0.3.0a4 (2022-04-13) -------------------- * Merge changes from 0.2.4 maintenance release 0.3.0a3 (2022-01-21) -------------------- * Fix bug that prevented booting with only taxii1 config (`#217 <https://github.com/eclecticiq/OpenTAXII/issues/217>`_ thanks `@azurekid <https://github.com/azurekid>`_ for the report) 0.3.0a2 (2021-12-27) -------------------- * Merge changes from 0.2.3 maintenance release 0.3.0a1 ------- * Add python 3.10 support 0.3.0a0 ------- * Enablement for future taxii2 implementation * Fix documentation build issues 0.2.4 (2022-04-13) ------------------ * Make sure werkzeug <2.1 and >=2.1 work correctly with auth system 0.2.3 (2021-12-22) ------------------ * Fix bug in multithreaded use of sqlite (`#210 <https://github.com/eclecticiq/OpenTAXII/issues/210>`_ thanks `@rohits144 <https://github.com/rohits144>`_ for the report) 0.2.2 (2021-11-05) ------------------ * Fix readthedocs build 0.2.1 (2021-11-03) ------------------ * Add tests for python 3.6, 3.7, 3.8, 3.9, pypy * Add tests for sqlite, mysql, mariadb, postgresql * Fix bug that broke ``delete_content_blocks`` when using mysql on sqlalchemy 1.3 * Docs: Add db schema diagram * Docs: Clarify how to get default data in a default (development) docker instance * Fix implicit routing in TAXII 1.1 Inboxes * Update jwt usage to pyjwt >= 2.0 (thanks `@SanyaKapoor <https://github.com/SanyaKapoor>`_) 0.2.0 (2020-06-30) ------------------ * Enforce UTC usage in datetime fields in SQL DB Persistence API. * `Fix for #114 <https://github.com/eclecticiq/OpenTAXII/issues/114>`_: reintroduce ``opentaxii-create-account`` CLI command. * `Fix for #153 <https://github.com/eclecticiq/OpenTAXII/issues/152>`_: check if user can modify a collection before advertising it over inbox service. * Multiple coding style fixes. * Various documentation updates. 0.1.12 (2019-03-06) ------------------- * Remove unnecessary print statements. 0.1.11 (2019-02-13) ------------------- * Make JSON logging consistent when the application is run via Gunicorn. * Set ``acceptable_destination`` key in status details instead of extended headers * Allow passing ``engine_parameters`` to ``SQLDatabaseAPI`` for those who want to customize SQLAlchemy engine parameters. * Require recent version of ``lxml`` for security reasons. * Various test and Docker infrastructure improvements. 0.1.10 (2018-06-03) ------------------- * Replace separate service/collection/account creation process with single ``opentaxii-sync-data`` CLI command. * Persistence and Auth APIs extended with missing CRUD methods, that are used by ``opentaxii-sync-data``. * Read/modify collection level ACL added. * DB models for default implementation of Persistence API and Auth API were changed. No automatic migration code is provided (sorry!), so upgrading might require manual DB migration. * Drop python2.7 from testing scope. * Various bug fixes and improvements. 0.1.9 (2017-06-19) ------------------ * `libtaxii <https://github.com/TAXIIProject/libtaxii>`_ dependency upgraded to 1.1.111. * Various bug fixes and improvements (thanks to `@bjigmp <https://github.com/bjigmp>`_, `@chorsley <https://github.com/chorsley>`_, `@rjprins <https://github.com/rjprins>`_). 0.1.8 (2017-02-21) ------------------ * Ability to enable/disable "huge trees" support in XML parser. Configuration property ``xml_parser_supports_huge_tree`` set to ``yes`` or ``true`` will disable security restrictions and force XML parser to support very deep trees and very long text content. * Adjust SQL Persistence API implemetation so it works smoothly with MySQL backend. * Use Python 3.5 instead of Python 3.4 for tests. 0.1.7 (2016-10-18) ------------------ * Minor fixes. * Dependencies were changed from hard-pinned to more flexible. * Example of production DB configuration added to docs. 0.1.6 (2016-06-01) ------------------ * Python 3.4 compatibility of the codebase. Tox configuration extended with python 3.4 environment run. * Flake8 full style compatibility. Flake8 check added to Tox configuration. * SQLAlchemy session scope issue fixed (related to `#38 <https://github.com/EclecticIQ/OpenTAXII/issues/38>`_). * `opentaxii-delete-blocks` CLI command added (related to `#45 <https://github.com/EclecticIQ/OpenTAXII/issues/45>`_). * `delete_content_blocks` method `added <https://github.com/EclecticIQ/OpenTAXII/commit/dc6fddc27a98e8450c7e05e583b2bfb741f6e197#diff-6814849ac352b2b74132f8fa52e0ec4eR213>`_ to Persistence API. * Collection's name is `required <https://github.com/EclecticIQ/OpenTAXII/commit/dc6fddc27a98e8450c7e05e583b2bfb741f6e197#diff-ce3f7b939e5c540480ac655aef32c513R116>`_ to be unique in default SQL DB Auth API implementation. 0.1.5 (2016-03-15) ------------------ * Fix for the issue with persistence backend returning ``None`` instead of ``InboxMessage`` object 0.1.4 (2016-02-25) ------------------ * Hard-coded dependencies in ``setup.py`` removed. 0.1.3 (2016-02-25) ------------------ * Versions of dependencies are pinned. * Code adjusted for a new version of `anyconfig <https://pypi.python.org/pypi/anyconfig>`_ API. * Test for configuration loading added. 0.1.2 (2015-07-24) ------------------ * Docker configuration added. * Health check endpoint added. * Basic authentication support added. * Temporary workaround for `Issue #191 <https://github.com/TAXIIProject/libtaxii/issues/191>`_. * Method ``get_domain`` in Persistence API returns domain value configured for ``service_id``. If nothing returned, value set in configuration file will be used. * Performance optimisations. * Bug fixes and style improvements. 0.1.1 (2015-04-08) ------------------ * Alias for Root Logger added to logging configuration. * Context object in a request scope that holds account and token added. * Support for OPTIONS HTTP request to enable auto version negotiation added. * Documentation improved. 0.1.0 (2015-03-31) ------------------ * Initial release 2022-06-13T14:44:11+00:00 syncthing v1.20.3-rc.1 syncthing v1.20.3-rc.1 2022-06-14T12:24:10+00:00 Bugfixes: - #8376: Chrome Autofill Breaks Authentication 2022-06-14T12:24:10+00:00 MONARC v2.11.1-p2 MONARC v2.11.1-p2 2022-06-15T06:56:04+00:00 2022-06-15T06:56:04+00:00 monarc-stats-service v0.5.0 monarc-stats-service v0.5.0 2022-06-20T12:27:47+00:00 - chg: [documentation] Updated documentation (9f38db11b382d8516fb71b60154aa0c7ba77004c); - chg: [API] CLIENT_REGISTRATION_OPEN is now set to True by default (b277436f81bbac1445822f0399dc348c5e283f70); - fix: [security] prevent the creation of new admin users (even by an admin) (257c16fed890bda8974594238a743f8afda0ff5a); - fix: replaced after_request by before_request for the API (88a276bf4d5f35e4e5da6ac065e1eb62f2892670); - small codebase fix for container image (82cdeaa714dbff14b9068c0b65c302ec4d02b3c4); - dockerfile and build pipeline (f8c663b61e1c8475d0f17060690d9920a1cc9e90); - updated Python dependencies. 2022-06-20T12:27:47+00:00 MONARC v2.12.0 MONARC v2.12.0 2022-06-20T12:44:13+00:00 ### New - [compliance scale](https://github.com/monarc-project/MonarcAppFO/discussions/439) - [metadata assets](https://github.com/monarc-project/MonarcAppFO/discussions/437) - [two-factor authentication](https://github.com/monarc-project/MonarcAppFO/discussions/442) - new build deployment is available and based on GitHub Actions ### Fix - [Stats provider] removed the leading slash in the URI ([e7dfba1](https://github.com/monarc-project/zm-client/commit/e7dfba1cf64322bc3e83630df6729b525d7d5c8d)) Details about upcoming related releases: https://github.com/orgs/monarc-project/projects/3 2022-06-20T12:44:13+00:00 monarc-stats-service v0.5.1 monarc-stats-service v0.5.1 2022-06-21T21:59:16+00:00 Changes ~~~~~~~ - [dependenvies] Updated Python dependencies. 2022-06-21T21:59:16+00:00 MONARC v2.12.1 MONARC v2.12.1 2022-06-22T07:39:21+00:00 2022-06-22T07:39:21+00:00 TheHive 4.1.21 TheHive 4.1.21 2022-06-22T11:21:10+00:00 ## [4.1.21](https://github.com/TheHive-Project/TheHive/milestone/91) (2022-06-22) **Fixed bugs:** - [Bug] S3 storage fails with old version of Minio [\#2388](https://github.com/TheHive-Project/TheHive/issues/2388) - [Bug ] Authentication Bypass Vulnerability [\#2391](https://github.com/TheHive-Project/TheHive/issues/2391) 2022-06-22T11:21:10+00:00 Lookyloo v1.13.0 Lookyloo v1.13.0 2022-06-26T16:06:50+00:00 # Maintenance and bug-fixes release All releases don't need to contain new features, sometimes, it is just some cleanup, and it is okay. * Properly handle exceptions in some edge cases (fixes in har2tree) * Properly display an error message if the capture fails * Use the same default User-Agent in when a capture is submitted via the API as via the web interface. * Cleanup some legacy code * Bump all dependencies (JS/CSS and Python) # Still, there is a new-ish thing We revamped the package generator, and it should be [more usable](https://github.com/Lookyloo/lookyloo/pkgs/container/lookyloo). If it is not, let us know! 2022-06-26T16:06:50+00:00 DC3-MWCP 3.7.0 DC3-MWCP 3.7.0 2022-06-28T13:39:42+00:00 2022-06-28T13:39:42+00:00 syncthing v1.20.3-rc.2 syncthing v1.20.3-rc.2 2022-06-29T06:28:24+00:00 Bugfixes: - #8369: Shared device names are missing from "Edit Folder -> Sharing" - #8376: Chrome Autofill Breaks Authentication - #8386: Ignore patterns with wildcard and non-ASCII characters don't work as expected Enhancements: - #8393: Warn if two devices are introducers to each other 2022-06-29T06:28:24+00:00 MONARC v2.12.2 MONARC v2.12.2 2022-06-29T07:20:25+00:00 2022-06-29T07:20:25+00:00 pandora v1.0.0 pandora v1.0.0 2022-06-29T16:30:21+00:00 This is the first official stable open source release of [Pandora](https://github.com/pandora-analysis/pandora). Pandora is an analysis framework to discover if a file is suspicious and conveniently show the results. The solution can be installed on-prem to avoid information leak in organisations. It has been tested relatively extensively over the last few months, but there might still be issues. If anything goes wrong, please open an issue and we will do our best to solve it. If you want to test Pandora without installing it, the online version is available at [pandora.circl.lu](https://pandora.circl.lu/). # Core functionalities * Easy to implement workers to analyze specific file formats, or connect to third party services * Admin interface * Session-based user management interface, and sharing * Generate a preview of the submitted document (if applicable) * Extract indicators/observables from submitted files * Extract content of archives * Extract attachments from email in EML and MSG format * Extract text content (if applicable) * Extract EXIF metadata * Pool service to fetch emails from an IMAP mailbox * Notify Administrator * MISP export and submission (admin only) * Statistics (admin only) * Role management (admin only) * Locally defined observables (legitimate/suspicious) (admin only) # Screenshots ## Submission interface ![pandora1](https://user-images.githubusercontent.com/248875/176490605-28763966-f924-4fad-855e-017f88f1954a.png) ## Result page ![pandora2](https://user-images.githubusercontent.com/248875/176490653-d0a1c658-1dc5-4b90-90e5-a565404f2447.png) --------------- ![pandora3](https://user-images.githubusercontent.com/248875/176492185-b4bb61cc-78c0-4146-8ff8-447fbf719a0c.png) 2022-06-29T16:30:21+00:00 maltrail 0.47 maltrail 0.47 2022-06-30T22:11:05+00:00 Start-of-month release 2022-06-30T22:11:05+00:00 TheHive 4.1.22 TheHive 4.1.22 2022-07-01T15:00:56+00:00 ## [4.1.22](https://github.com/TheHive-Project/TheHive/milestone/93) (2022-07-01) **Implemented enhancements:** - [Enhancement] Add check on user role [\#2401](https://github.com/TheHive-Project/TheHive/issues/2401) **Fixed bugs:** - [Bug] Use dedicated stream topic for stream dispatcher subscription [\#2400](https://github.com/TheHive-Project/TheHive/issues/2400) 2022-07-01T15:00:56+00:00 monarc-stats-service v0.5.2 monarc-stats-service v0.5.2 2022-07-04T08:41:37+00:00 ## Changes - [API] patch on client now expects again a model from Namespace (client_ns). - Updated Python dependencies. - [documentation] Updated links to documentation. - [documentation] Updated information about installation. - [deployment] added docker-compose.yml file. ## Fix - [API] enable patch method for enabling/disabling stats sharing. 2022-07-04T08:41:37+00:00 MONARC v2.12.2-p1 MONARC v2.12.2-p1 2022-07-04T08:56:31+00:00 2022-07-04T08:56:31+00:00 osquery 5.4.0 osquery 5.4.0 2022-07-06T21:20:21+00:00 2022-07-06T21:20:21+00:00 GDPRDPIAT v3.0.1 GDPRDPIAT v3.0.1 2022-07-07T10:35:43+00:00 ## What's Changed * Bugfixes/surveyjs cdn by @simonarnell in https://github.com/simonarnell/GDPRDPIAT/pull/6 **Full Changelog**: https://github.com/simonarnell/GDPRDPIAT/compare/v3.0.0...v3.0.1 2022-07-07T10:35:43+00:00 MONARC v2.12.2-p2 MONARC v2.12.2-p2 2022-07-07T10:44:13+00:00 Fixed an issue with sortable 1.15.0 https://github.com/monarc-project/ng-client/commit/4d8e61e56edf7b858db68381f2b99389b3921866 2022-07-07T10:44:13+00:00 MONARC v2.12.2-p3 MONARC v2.12.2-p3 2022-07-07T12:42:37+00:00 2022-07-07T12:42:37+00:00 pandora v1.0.1 pandora v1.0.1 2022-07-11T14:44:26+00:00 Quick release to improve handling of submissions with passwords. # New features * Support password on submit via API. # Maintenance * Bugfixes * Dependencies update. 2022-07-11T14:44:26+00:00 OpenTAXII 0.9.1 OpenTAXII 0.9.1 2022-07-11T20:28:28+00:00 Changelog ========= 0.9.1 (2022-07-11) ------------------ * Implement `raise_unauthorized` for taxii2, this was missing and lead to 500 errors. 0.9.0 (2022-06-13) ------------------ * Allow custom properties. This can be disabled by config option ``allow_custom_properties`` 0.8.0 (2022-06-05) ------------------ * Nest details inside taxii2 job and allow counts without details 0.7.0 (2022-05-27) ------------------ * Nest taxii2 endpoints under `/taxii2/` 0.6.0 (2022-05-25) ------------------ * Add `public_discovery` option to taxii2 config * Add support for publicly readable taxii 2 api roots 0.5.0 (2022-05-24) ------------------ * Add support for publicly readable taxii 2 collections 0.4.0 (2022-05-20) ------------------ * Move next_param handling into `OpenTAXII2PersistenceAPI` 0.3.0 (2022-04-13) ------------------ * Implement taxii2.1 support 0.3.0a4 (2022-04-13) -------------------- * Merge changes from 0.2.4 maintenance release 0.3.0a3 (2022-01-21) -------------------- * Fix bug that prevented booting with only taxii1 config (`#217 <https://github.com/eclecticiq/OpenTAXII/issues/217>`_ thanks `@azurekid <https://github.com/azurekid>`_ for the report) 0.3.0a2 (2021-12-27) -------------------- * Merge changes from 0.2.3 maintenance release 0.3.0a1 ------- * Add python 3.10 support 0.3.0a0 ------- * Enablement for future taxii2 implementation * Fix documentation build issues 0.2.4 (2022-04-13) ------------------ * Make sure werkzeug <2.1 and >=2.1 work correctly with auth system 0.2.3 (2021-12-22) ------------------ * Fix bug in multithreaded use of sqlite (`#210 <https://github.com/eclecticiq/OpenTAXII/issues/210>`_ thanks `@rohits144 <https://github.com/rohits144>`_ for the report) 0.2.2 (2021-11-05) ------------------ * Fix readthedocs build 0.2.1 (2021-11-03) ------------------ * Add tests for python 3.6, 3.7, 3.8, 3.9, pypy * Add tests for sqlite, mysql, mariadb, postgresql * Fix bug that broke ``delete_content_blocks`` when using mysql on sqlalchemy 1.3 * Docs: Add db schema diagram * Docs: Clarify how to get default data in a default (development) docker instance * Fix implicit routing in TAXII 1.1 Inboxes * Update jwt usage to pyjwt >= 2.0 (thanks `@SanyaKapoor <https://github.com/SanyaKapoor>`_) 0.2.0 (2020-06-30) ------------------ * Enforce UTC usage in datetime fields in SQL DB Persistence API. * `Fix for #114 <https://github.com/eclecticiq/OpenTAXII/issues/114>`_: reintroduce ``opentaxii-create-account`` CLI command. * `Fix for #153 <https://github.com/eclecticiq/OpenTAXII/issues/152>`_: check if user can modify a collection before advertising it over inbox service. * Multiple coding style fixes. * Various documentation updates. 0.1.12 (2019-03-06) ------------------- * Remove unnecessary print statements. 0.1.11 (2019-02-13) ------------------- * Make JSON logging consistent when the application is run via Gunicorn. * Set ``acceptable_destination`` key in status details instead of extended headers * Allow passing ``engine_parameters`` to ``SQLDatabaseAPI`` for those who want to customize SQLAlchemy engine parameters. * Require recent version of ``lxml`` for security reasons. * Various test and Docker infrastructure improvements. 0.1.10 (2018-06-03) ------------------- * Replace separate service/collection/account creation process with single ``opentaxii-sync-data`` CLI command. * Persistence and Auth APIs extended with missing CRUD methods, that are used by ``opentaxii-sync-data``. * Read/modify collection level ACL added. * DB models for default implementation of Persistence API and Auth API were changed. No automatic migration code is provided (sorry!), so upgrading might require manual DB migration. * Drop python2.7 from testing scope. * Various bug fixes and improvements. 0.1.9 (2017-06-19) ------------------ * `libtaxii <https://github.com/TAXIIProject/libtaxii>`_ dependency upgraded to 1.1.111. * Various bug fixes and improvements (thanks to `@bjigmp <https://github.com/bjigmp>`_, `@chorsley <https://github.com/chorsley>`_, `@rjprins <https://github.com/rjprins>`_). 0.1.8 (2017-02-21) ------------------ * Ability to enable/disable "huge trees" support in XML parser. Configuration property ``xml_parser_supports_huge_tree`` set to ``yes`` or ``true`` will disable security restrictions and force XML parser to support very deep trees and very long text content. * Adjust SQL Persistence API implemetation so it works smoothly with MySQL backend. * Use Python 3.5 instead of Python 3.4 for tests. 0.1.7 (2016-10-18) ------------------ * Minor fixes. * Dependencies were changed from hard-pinned to more flexible. * Example of production DB configuration added to docs. 0.1.6 (2016-06-01) ------------------ * Python 3.4 compatibility of the codebase. Tox configuration extended with python 3.4 environment run. * Flake8 full style compatibility. Flake8 check added to Tox configuration. * SQLAlchemy session scope issue fixed (related to `#38 <https://github.com/EclecticIQ/OpenTAXII/issues/38>`_). * `opentaxii-delete-blocks` CLI command added (related to `#45 <https://github.com/EclecticIQ/OpenTAXII/issues/45>`_). * `delete_content_blocks` method `added <https://github.com/EclecticIQ/OpenTAXII/commit/dc6fddc27a98e8450c7e05e583b2bfb741f6e197#diff-6814849ac352b2b74132f8fa52e0ec4eR213>`_ to Persistence API. * Collection's name is `required <https://github.com/EclecticIQ/OpenTAXII/commit/dc6fddc27a98e8450c7e05e583b2bfb741f6e197#diff-ce3f7b939e5c540480ac655aef32c513R116>`_ to be unique in default SQL DB Auth API implementation. 0.1.5 (2016-03-15) ------------------ * Fix for the issue with persistence backend returning ``None`` instead of ``InboxMessage`` object 0.1.4 (2016-02-25) ------------------ * Hard-coded dependencies in ``setup.py`` removed. 0.1.3 (2016-02-25) ------------------ * Versions of dependencies are pinned. * Code adjusted for a new version of `anyconfig <https://pypi.python.org/pypi/anyconfig>`_ API. * Test for configuration loading added. 0.1.2 (2015-07-24) ------------------ * Docker configuration added. * Health check endpoint added. * Basic authentication support added. * Temporary workaround for `Issue #191 <https://github.com/TAXIIProject/libtaxii/issues/191>`_. * Method ``get_domain`` in Persistence API returns domain value configured for ``service_id``. If nothing returned, value set in configuration file will be used. * Performance optimisations. * Bug fixes and style improvements. 0.1.1 (2015-04-08) ------------------ * Alias for Root Logger added to logging configuration. * Context object in a request scope that holds account and token added. * Support for OPTIONS HTTP request to enable auto version negotiation added. * Documentation improved. 0.1.0 (2015-03-31) ------------------ * Initial release 2022-07-11T20:28:28+00:00 caddy v2.5.2 caddy v2.5.2 2022-07-12T19:06:28+00:00 This patch release fixes bugs, adds some new features, and makes worthwhile enhancements. We recommend everyone test and upgrade! Many improvements have been made to the `reverse_proxy` module. Highlights: - **New [`/adapt` admin endpoint](https://caddyserver.com/docs/api#post-adapt):** Use your installed config adapters via API in addition to the existing `caddy adapt` CLI command. - **New `Etag`/`If-Match` support for config API:** Safely update your config concurrently and avoid collisions by using [our unique Etag implementation](https://caddyserver.com/docs/api#concurrent-config-changes). - **Rename copied headers from reverse_proxy:** If you're using `handle_response`, you can more easily map headers to a different name for clients. - **Many HTTP matchers have been added to CEL:** You can now use the logic of our HTTP request matchers in CEL expressions. - **Notable bug fixes:** EAB reuse, various QUIC & HTTP/3 fixes, more specific HTTP status codes, various reverse proxy fixes. ## Changelog * 660c59b6 admin: Implement /adapt endpoint (close #4465) (#4846) * ad3a83fb admin: expect quoted ETags (#4879) * f259ed52 admin: support ETag on config endpoints (#4579) * 1498132e caddyhttp: Log error from CEL evaluation (fix #4832) * 0a14f97e caddytls: Make peer certificate verification pluggable (#4389) * 412dcc07 caddytls: Reuse issuer between PreCheck and Issue (#4866) * 499ad6d1 core: Micro-optim in run() (#4810) * c0f76e9e fileserver: Use safe redirects in file browser * 58e05cab forwardauth: Fix case when `copy_headers` is omitted (#4856) * 0b6f7643 forwardauth: Support renaming copied headers, block support (#4783) * 8bac134f go.mod: Bump up quic-go to v0.28.0, fixes for BC breaks (#4867) * 3d18bc56 go.mod: Update go-yaml to v3 * 56013934 go.mod: Update some dependencies * 8e6bc360 go.mod: Upgrade some dependencies * 53c4d788 headers: Only replace known placeholders (#4880) * 0bcd02d5 headers: Support wildcards for delete ops (close #4830) (#4831) * 58970cae httpcaddyfile: Add `{err.*}` placeholder shortcut (#4798) * b687d7b9 httpcaddyfile: Support multiple values for `default_bind` (#4774) * a9267791 reverseproxy: Add --internal-certs CLI flag #3589 (#4817) * aaf6794b reverseproxy: Add renegotiation param in TLS client (#4784) * 54d1923c reverseproxy: Adjust new TLS Caddyfile directive names (#4872) * 7f9b1f43 reverseproxy: Correct the `tls_server_name` docs (#4827) * c82fe911 reverseproxy: Dynamic ServerName for TLS upstreams (#4836) * d6bc9e0b reverseproxy: Err 503 if all upstreams unavailable * 98468af8 reverseproxy: Fix double headers in response handlers (#4847) * 25f10511 reverseproxy: Fix panic when TLS is not configured (#4848) * 5e729c1e reverseproxy: HTTP 504 for upstream timeouts (#4824) * f9b42c37 reverseproxy: Make TLS renegotiation optional * b6e96fa3 reverseproxy: Skip TLS for certain configured ports (#4843) * 57d27c1b reverseproxy: Support http1.1>h2c (close #4777) (#4778) * 9864b138 reverseproxy: api: Remove misleading 'healthy' value * 693e9b52 rewrite: Handle fragment before query (fix #4775) * 6891f7f4 templates: Add `humanize` function (#4767) * 9e760e2e templates: Documentation consistency (#4796) ## New Contributors * @nekohasekai made their first contribution in https://github.com/caddyserver/caddy/pull/4782 * @davidbgk made their first contribution in https://github.com/caddyserver/caddy/pull/4796 * @git001 made their first contribution in https://github.com/caddyserver/caddy/pull/4767 * @varianone made their first contribution in https://github.com/caddyserver/caddy/pull/4817 * @Gr33nbl00d made their first contribution in https://github.com/caddyserver/caddy/pull/4389 * @yaslama made their first contribution in https://github.com/caddyserver/caddy/pull/4784 * @kresike made their first contribution in https://github.com/caddyserver/caddy/pull/4836 * @TristonianJones made their first contribution in https://github.com/caddyserver/caddy/pull/4715 * @jhwz made their first contribution in https://github.com/caddyserver/caddy/pull/4579 **Full Changelog**: https://github.com/caddyserver/caddy/compare/v2.5.1...v2.5.2 2022-07-12T19:06:28+00:00 AIL-framework v4.2 AIL-framework v4.2 2022-07-16T08:40:47+00:00 ## AIL Framework version 4.2 released including typo squatting tracker, improved AIL2AIL sync, zerobinz fetcher and many bugs fixes v4.2 (2022-06-24) AIL Framework version 4.2 has been released including: - A new tracker for tracking potential typo squatted domains. This feature relies on the new [ail-typo-squatting ](https://github.com/ail-project/ail-typo-squatting) library which can be also used outside of AIL framework. This contribution is from @DavidCruciani - Many improvement and bugs fixed for the AIL2AIL sync. A huge thanks to @aaronkaplan from EU Directorate-General for Informatics (DIGIT) for support and tests during the long debugging sessions. - A new module for zerobinz to create an immediate crawler request if a zerobinz link appears in an item. The module can be used for other services with ephemeral content. Thanks to @gallypette for the contribution and the improvement ideas. - A new hosts detection module has been introduced. - Multiple bugs were fixed. ### Detailed Changes * [Tracker} Tracker_Typo_Squatting. [David Cruciani] * [v4.2] add v4.2 update. [Terrtia] * [investigation] fix investigation by user + delete an obj from all investigation. [Terrtia] * [install vitualenv] remove travis env. [Terrtia] * [Retro Hunt] add logs. [Terrtia] * [Retro Hunt] add logs. [Terrtia] * [Retro Hunt] add logs. [Terrtia] * [AIL2AIL Sync] update exchange format. [Terrtia] * [AIL2AIL Sync] update exchange format. [Terrtia] * [add Hosts module] [Terrtia] * [sync module] debug. [Terrtia] * [sync client] debug. [Terrtia] * [websockets client] bind client ip. [Terrtia] * [websocket server] add host and port config. [Terrtia] * [telegram importer] add username correlation. [Terrtia] * [UI subtype objs] get obj by subtype + name. [Terrtia] * [misp export] add username. [Terrtia] ### Fix * [typosquatting] remove unused import. [Thirion Aurélien] * [tracker] clean import. [Thirion Aurélien] * [tracker term] fix typosquatting key. [Thirion Aurélien] * [Typo] tracker typo. [David Cruciani] * [tracker] UI for other than typosquat. [David Cruciani] * [typo] UI. [David Cruciani] * [Language] fix cld3 import. [Terrtia] * [launcher] kill AIL_2_AIL screen. [Terrtia] * [cld3] enable cld3. [Terrtia] * [cld3 python3.10] temp disable cld3. [Terrtia] * [launcher] remove Travis test. [Terrtia] * [Retro Hunt] item directory. [Terrtia] * [Retro Hunt] item directory. [Terrtia] * [Retro Hunt] fix item directory. [Terrtia] * [AIL exchange mime-type] [Terrtia] * [Hosts module] module + launcher. [Terrtia] * [abstract module] exception traceback #145. [Terrtia] * [ui tag selector] force custom tags. [Terrtia] * [installer] remove old tor install. [Terrtia] * [sync module] fix redis tag queue. [Terrtia] * [sync module] fix tags filter. [Terrtia] * [sync client] debug. [Terrtia] * [sync client] debug. [Terrtia] * [sync module] debug. [Terrtia] * [websockets client] fix client bind. [Terrtia] * [websockets] remove size limit. [Terrtia] * [UI subtype objs] fix form. [Terrtia] * [misp config] https. [Thirion Aurélien] ### Other * Merge pull request #147 from ail-project/typo. [Thirion Aurélien] Integration of the typo-squatting tracker * Fix; [set tracker] missing function. [Thirion Aurélien] * Merge branch 'master' into typo. [David Cruciani] * Add: [tracker] typo-squatting. [David Cruciani] * Merge branch 'master' of github.com:ail-project/ail-framework. [Terrtia] * Merge pull request #146 from gallypette/master. [Thirion Aurélien] add: [modules] zerobinz * Add: [modules] zerobinz. [huynenjl@gmail.com] * Merge branch 'master' of github.com:ail-project/ail-framework. [Terrtia] 2022-07-16T08:40:47+00:00 rocksdb v7.4.3 rocksdb v7.4.3 2022-07-18T15:18:20+00:00 ## 7.4.3 (07/13/2022) ### Behavior Changes * For track_and_verify_wals_in_manifest, revert to the original behavior before #10087: syncing of live WAL file is not tracked, and we track only the synced sizes of **closed** WALs. (PR #10330). ## 7.4.2 (06/30/2022) ### Bug Fixes * Fix a bug in Logger where if dbname and db_log_dir are on different filesystems, dbname creation would fail wrt to db_log_dir path returning an error and fails to open the DB. ## 7.4.1 (06/28/2022) ### Bug Fixes * Pass `rate_limiter_priority` through filter block reader functions to `FileSystem`. ## 7.4.0 (06/19/2022) ### Bug Fixes * Fixed a bug in calculating key-value integrity protection for users of in-place memtable updates. In particular, the affected users would be those who configure `protection_bytes_per_key > 0` on `WriteBatch` or `WriteOptions`, and configure `inplace_callback != nullptr`. * Fixed a bug where a snapshot taken during SST file ingestion would be unstable. * Fixed a bug for non-TransactionDB with avoid_flush_during_recovery = true and TransactionDB where in case of crash, min_log_number_to_keep may not change on recovery and persisting a new MANIFEST with advanced log_numbers for some column families, results in "column family inconsistency" error on second recovery. As a solution, RocksDB will persist the new MANIFEST after successfully syncing the new WAL. If a future recovery starts from the new MANIFEST, then it means the new WAL is successfully synced. Due to the sentinel empty write batch at the beginning, kPointInTimeRecovery of WAL is guaranteed to go after this point. If future recovery starts from the old MANIFEST, it means the writing the new MANIFEST failed. We won't have the "SST ahead of WAL" error. * Fixed a bug where RocksDB DB::Open() may creates and writes to two new MANIFEST files even before recovery succeeds. Now writes to MANIFEST are persisted only after recovery is successful. * Fix a race condition in WAL size tracking which is caused by an unsafe iterator access after container is changed. * Fix unprotected concurrent accesses to `WritableFileWriter::filesize_` by `DB::SyncWAL()` and `DB::Put()` in two write queue mode. * Fix a bug in WAL tracking. Before this PR (#10087), calling `SyncWAL()` on the only WAL file of the db will not log the event in MANIFEST, thus allowing a subsequent `DB::Open` even if the WAL file is missing or corrupted. * Fix a bug that could return wrong results with `index_type=kHashSearch` and using `SetOptions` to change the `prefix_extractor`. * Fixed a bug in WAL tracking with wal_compression. WAL compression writes a kSetCompressionType record which is not associated with any sequence number. As result, WalManager::GetSortedWalsOfType() will skip these WALs and not return them to caller, e.g. Checkpoint, Backup, causing the operations to fail. * Avoid a crash if the IDENTITY file is accidentally truncated to empty. A new DB ID will be written and generated on Open. * Fixed a possible corruption for users of `manual_wal_flush` and/or `FlushWAL(true /* sync */)`, together with `track_and_verify_wals_in_manifest == true`. For those users, losing unsynced data (e.g., due to power loss) could make future DB opens fail with a `Status::Corruption` complaining about missing WAL data. * Fixed a bug in `WriteBatchInternal::Append()` where WAL termination point in write batch was not considered and the function appends an incorrect number of checksums. * Fixed a crash bug introduced in 7.3.0 affecting users of MultiGet with `kDataBlockBinaryAndHash`. * Add some fixes in async_io which was doing extra prefetching in shorter scans. ### Public API changes * Add new API GetUnixTime in Snapshot class which returns the unix time at which Snapshot is taken. * Add transaction `get_pinned` and `multi_get` to C API. * Add two-phase commit support to C API. * Add `rocksdb_transaction_get_writebatch_wi` and `rocksdb_transaction_rebuild_from_writebatch` to C API. * Add `rocksdb_options_get_blob_file_starting_level` and `rocksdb_options_set_blob_file_starting_level` to C API. * Add `blobFileStartingLevel` and `setBlobFileStartingLevel` to Java API. * Add SingleDelete for DB in C API * Add User Defined Timestamp in C API. * `rocksdb_comparator_with_ts_create` to create timestamp aware comparator * Put, Get, Delete, SingleDelete, MultiGet APIs has corresponding timestamp aware APIs with suffix `with_ts` * And Add C API's for Transaction, SstFileWriter, Compaction as mentioned [here](https://github.com/facebook/rocksdb/wiki/User-defined-Timestamp-(Experimental)) * The contract for implementations of Comparator::IsSameLengthImmediateSuccessor has been updated to work around a design bug in `auto_prefix_mode`. * The API documentation for `auto_prefix_mode` now notes some corner cases in which it returns different results than `total_order_seek`, due to design bugs that are not easily fixed. Users using built-in comparators and keys at least the size of a fixed prefix length are not affected. * Obsoleted the NUM_DATA_BLOCKS_READ_PER_LEVEL stat and introduced the NUM_LEVEL_READ_PER_MULTIGET and MULTIGET_COROUTINE_COUNT stats * Introduced `WriteOptions::protection_bytes_per_key`, which can be used to enable key-value integrity protection for live updates. ### New Features * Add FileSystem::ReadAsync API in io_tracing * Add blob garbage collection parameters `blob_garbage_collection_policy` and `blob_garbage_collection_age_cutoff` to both force-enable and force-disable GC, as well as selectively override age cutoff when using CompactRange. * Add an extra sanity check in `GetSortedWalFiles()` (also used by `GetLiveFilesStorageInfo()`, `BackupEngine`, and `Checkpoint`) to reduce risk of successfully created backup or checkpoint failing to open because of missing WAL file. * Add a new column family option `blob_file_starting_level` to enable writing blob files during flushes and compactions starting from the specified LSM tree level. * Add support for timestamped snapshots (#9879) * Provide support for AbortIO in posix to cancel submitted asynchronous requests using io_uring. * Add support for rate-limiting batched `MultiGet()` APIs ### Behavior changes * DB::Open(), DB::OpenAsSecondary() will fail if a Logger cannot be created (#9984) * Removed support for reading Bloom filters using obsolete block-based filter format. (Support for writing such filters was dropped in 7.0.) For good read performance on old DBs using these filters, a full compaction is required. * Per KV checksum in write batch is verified before a write batch is written to WAL to detect any corruption to the write batch (#10114). ### Performance Improvements * When compiled with folly (Meta-internal integration; experimental in open source build), improve the locking performance (CPU efficiency) of LRUCache by using folly DistributedMutex in place of standard mutex. 2022-07-18T15:18:20+00:00 rocksdb v7.4.4 rocksdb v7.4.4 2022-07-28T18:34:26+00:00 ## 7.4.4 (07/19/2022) ### Public API changes * Removed Customizable support for RateLimiter and removed its CreateFromString() and Type() functions. ### Bug Fixes * Fix a bug where `GenericRateLimiter` could revert the bandwidth set dynamically using `SetBytesPerSecond()` when a user configures a structure enclosing it, e.g., using `GetOptionsFromString()` to configure an `Options` that references an existing `RateLimiter` object. 2022-07-28T18:34:26+00:00 rocksdb v7.4.5 rocksdb v7.4.5 2022-08-02T23:17:54+00:00 ## 7.4.5 (08/02/2022) ### Bug Fixes * Fix a bug starting in 7.4.0 in which some fsync operations might be skipped in a DB after any DropColumnFamily on that DB, until it is re-opened. This can lead to data loss on power loss. (For custom FileSystem implementations, this could lead to `FSDirectory::Fsync` or `FSDirectory::Close` after the first `FSDirectory::Close`; Also, valgrind could report call to `close()` with `fd=-1`.) 2022-08-02T23:17:54+00:00 whids v1.8.0-beta.7 whids v1.8.0-beta.7 2022-08-03T12:33:11+00:00 2022-08-03T12:33:11+00:00 PyPCAPKit v0.16.2 PyPCAPKit v0.16.2 2022-08-04T02:55:47+00:00 2022-08-04T02:55:47+00:00 maltrail 0.48 maltrail 0.48 2022-08-04T06:36:06+00:00 Start-of-month release 2022-08-04T06:36:06+00:00 pcileech v4.15 pcileech v4.15 2022-08-04T06:39:53+00:00 * Support for MemProcFS v5.0 2022-08-04T06:39:53+00:00 MISP v2.4.160 MISP v2.4.160 2022-08-08T12:32:32+00:00 We are pleased to announce the immediate availability of MISP v2.4.160. With the August summer-holiday season kicking into high gear, we have a very special release for you all, containing a long list of major new features, improvements and general quality of life improvements. Unlike we do normally, this time around we're preparing separate blog posts for some of those major features, so follow the links below to read up on in-depth descriptions of each. # Workflows Something that has been in the works for quite some time now is finally hitting a release version of MISP, as of 2.4.160, we have the first release of the built in workflow system released. This system allows you to use an easy to use, yet extremely powerful graphical interface to modify how MISP handles certain tasks such as event publishing, user enrollment, synchronisation, etc., by adding additional logical steps in their respective executions, utilising a module system similar to what was already common to MISP from enrichment subsystems, exports as well as imports. This is merely the first step (or leap rather) towards customising and sharing custom workflows, stay tuned for new features, improvements as well as triggers and modules in the near future. Head over to the [README](https://github.com/MISP/misp-workflow-blueprints/blob/main/README.md) as well as a nifty [slide deck](https://www.misp-project.org/misp-training/a.12-misp-workflows.pdf), to find out what this incredibly powerful can do for you and your community. # New correlation engine One of the biggest pain points as of recently has been our dated and rather bloated correlation engine, which could easily bring a long running MISP instance to its knees when certain highly correlated data sources were synchronised. As of 2.4.160, we now have 2 brand new correlation engines at your disposal, with the old engine being retired immediately. Please be aware that upgrading to the current version will regenerate your correlations using the new engine, something that can take quite a long time (on our largest instance it took a whopping 40 hours!). With that said, we can assure you it's well worth the wait and should resolve several long standing performance bottlenecks as well as heavily cut down on the space requirements for your data. For more information, on the new engines, their differences, the various new support tools as well as what benefits you should expect, head over to the [dedicated blog post](https://github.com/MISP/MISP/blob/2.4/docs/correlations.rework.md). # STIX 2 library reworks There has been a massive amount of work going into the STIX 2.x library rework, bringing us closer and closer to having a full mapping of everything expressable. We're collaborating with CISA and Mitre to ensure that MISP can both express and understand STIX to its fullest extent. For more information, head over to the [release notes](https://github.com/MISP/misp-stix/releases/tag/v2.4.160) over on the MISP STIX library's repo. # Mermaid support for Event reports added Writing custom reports has become more and more popular, but one annoyance has been the lack of a way to depict graphs and flow charts without relying on external tools to create those (and share them as images for example). Using Mermaid, you now have a nifty tool to build graphs out of simple markdown directly in the event report editor. # Various other improvements A long list of other improvements, affecting the performance and stability of the platform as well as improvements to existing features. Head over to the changelog for a detailed list of changes. # Acknowledgement We would like to thank all the [contributors](https://www.misp-project.org/contributors), reporters and users who have helped us in the past months to improve MISP and information sharing at large. This release includes multiple updates in [misp-objects](https://www.misp-project.org/objects.html), [misp-taxonomies](https://www.misp-project.org/taxonomies.html) and [misp-galaxy](https://www.misp-project.org/galaxy.html). As always, a detailed and [complete changelog is available](https://www.misp-project.org/Changelog.txt) with all the fixes, changes and improvements in MISP core. Additional changelogs are available for [misp galaxy](https://www.misp-project.org/Changelog-misp-galaxy.txt), [misp-taxonomies](https://www.misp-project.org/Changelog-misp-taxonomies.txt), [misp-objects](https://www.misp-project.org/Changelog-misp-objects.txt) and [misp-modules](https://www.misp-project.org/Changelog-misp-modules.txt) 2022-08-08T12:32:32+00:00 Lookyloo v1.14.0 Lookyloo v1.14.0 2022-08-08T13:53:34+00:00 # New features * Trigger a capture on a web enabled document provided as a file instead of a URL. Useful for HTML files attached to emails, or HTML body in email. ![Screenshot_20220808_131607](https://user-images.githubusercontent.com/248875/183406016-e02f99bd-fe8c-47ea-906e-39e9da2280b6.png) -------- * Compress (gzip) the HAR file in archived captures - saves a lot of disk space. * Support for RiskIQ Passive DNS (requires API key) * Display SSL/TLS information available in the HAR dump from Playwright ![Screenshot_20220808_132643](https://user-images.githubusercontent.com/248875/183407809-4475d6ce-2311-43fe-bb79-8a0697bae78c.png) -------- * Optional DoNotTrack HTTP header in capture ![Screenshot_20220808_132302](https://user-images.githubusercontent.com/248875/183407193-06aaf9a0-8377-49e7-bceb-5cb9dadad6fb.png) -------- * Display size of rendered page on hostnode popup. * [WiP] Download files when the URL captured to a downloadable file (PDF, Office doc, ...) (**Important note** the downloaded file is not exposed to the user yet) * [WiP] List all hashes available in the capture, sort them by frequency. Makes it easier to find phishing sides using the same resources. ![Screenshot_20220808_132149](https://user-images.githubusercontent.com/248875/183407015-daf83393-e605-4f3c-a1a2-0d3885023422.png) # Fixes * Major speed improvements when displaying the hostnode popups (only show the recent cached captures by default) * Improvements in the caching mechanism * Cleanup data showed by monitoring script * Avoid crashes when RiskIQ isn't reachable # Changes * Update dependencies (js, python) * Improve logging in archiver * Improve config file 2022-08-08T13:53:34+00:00 whids v1.8.0-beta.8 whids v1.8.0-beta.8 2022-08-08T16:38:56+00:00 2022-08-08T16:38:56+00:00 MISP v2.4.161 MISP v2.4.161 2022-08-11T15:30:58+00:00 We are pleased to announce the immediate availability of [MISP v2.4.161](https://github.com/MISP/MISP/releases/tag/v2.4.161). ![](https://www.misp-project.org//img/blog/workflow.png) # Small improvements - A new option added to log the last API request of an API key. (Thanks to Tom King for the contribution) - Overcorrelation features have some new improvements such as: - A new tool to generate occurrence counts (real numbers this time) - A hook to truncate the over-correlating value table on recorrelation - We no longer store the partial counts as occurrences when generating correlations - Performance improvements in event fetching - Various performance tuning in the new correlation engine including the full recorrelation # Bugs fixed - `tlp:amber+strict` and `tlp:clear` are now valid tags - [stix2 import] Better `external_references` parsing for attack patterns objects Thanks to all the contributors and users reporting bugs to make the software better. As always, a detailed and [complete changelog is available](https://www.misp-project.org/Changelog.txt) with all the fixes, changes and improvements in MISP core. 2022-08-11T15:30:58+00:00 osquery 5.5.0 osquery 5.5.0 2022-08-12T17:47:19+00:00 draft 2022-08-12T17:47:19+00:00 dnstwist 20220815 dnstwist 20220815 2022-08-15T15:39:49+00:00 2022-08-15T15:39:49+00:00 dalton v3.2.4 dalton v3.2.4 2022-08-16T18:51:57+00:00 - New URL for downloading Suricata source code 2022-08-16T18:51:57+00:00 osquery 5.5.1 osquery 5.5.1 2022-08-18T13:24:43+00:00 Draft! (think 5.5.0 plus sqlite) 2022-08-18T13:24:43+00:00 rocksdb v7.5.3 rocksdb v7.5.3 2022-08-24T22:39:19+00:00 ## 7.5.2 (08/02/2022) ### Bug Fixes * Fix a bug starting in 7.4.0 in which some fsync operations might be skipped in a DB after any DropColumnFamily on that DB, until it is re-opened. This can lead to data loss on power loss. (For custom FileSystem implementations, this could lead to `FSDirectory::Fsync` or `FSDirectory::Close` after the first `FSDirectory::Close`; Also, valgrind could report call to `close()` with `fd=-1`.) ## 7.5.1 (08/01/2022) ### Bug Fixes * Fix a bug where rate_limiter_parameter is not passed into `PartitionedFilterBlockReader::GetFilterPartitionBlock`. ## 7.5.0 (07/15/2022) ### New Features * Mempurge option flag `experimental_mempurge_threshold` is now a ColumnFamilyOptions and can now be dynamically configured using `SetOptions()`. * Support backward iteration when `ReadOptions::iter_start_ts` is set. * Provide support for ReadOptions.async_io with direct_io to improve Seek latency by using async IO to parallelize child iterator seek and doing asynchronous prefetching on sequential scans. * Added support for blob caching in order to cache frequently used blobs for BlobDB. * User can configure the new ColumnFamilyOptions `blob_cache` to enable/disable blob caching. * Either sharing the backend cache with the block cache or using a completely separate cache is supported. * A new abstraction interface called `BlobSource` for blob read logic gives all users access to blobs, whether they are in the blob cache, secondary cache, or (remote) storage. Blobs can be potentially read both while handling user reads (`Get`, `MultiGet`, or iterator) and during compaction (while dealing with compaction filters, Merges, or garbage collection) but eventually all blob reads go through `Version::GetBlob` or, for MultiGet, `Version::MultiGetBlob` (and then get dispatched to the interface -- `BlobSource`). * Add experimental tiered compaction feature `AdvancedColumnFamilyOptions::preclude_last_level_data_seconds`, which makes sure the new data inserted within preclude_last_level_data_seconds won't be placed on cold tier (the feature is not complete). ### Public API changes * Add metadata related structs and functions in C API, including * `rocksdb_get_column_family_metadata()` and `rocksdb_get_column_family_metadata_cf()` to obtain `rocksdb_column_family_metadata_t`. * `rocksdb_column_family_metadata_t` and its get functions & destroy function. * `rocksdb_level_metadata_t` and its and its get functions & destroy function. * `rocksdb_file_metadata_t` and its and get functions & destroy functions. * Add suggest_compact_range() and suggest_compact_range_cf() to C API. * When using block cache strict capacity limit (`LRUCache` with `strict_capacity_limit=true`), DB operations now fail with Status code `kAborted` subcode `kMemoryLimit` (`IsMemoryLimit()`) instead of `kIncomplete` (`IsIncomplete()`) when the capacity limit is reached, because Incomplete can mean other specific things for some operations. In more detail, `Cache::Insert()` now returns the updated Status code and this usually propagates through RocksDB to the user on failure. * NewClockCache calls temporarily return an LRUCache (with similar characteristics as the desired ClockCache). This is because ClockCache is being replaced by a new version (the old one had unknown bugs) but this is still under development. * Add two functions `int ReserveThreads(int threads_to_be_reserved)` and `int ReleaseThreads(threads_to_be_released)` into `Env` class. In the default implementation, both return 0. Newly added `xxxEnv` class that inherits `Env` should implement these two functions for thread reservation/releasing features. * Removed Customizable support for RateLimiter and removed its CreateFromString() and Type() functions. ### Bug Fixes * Fix a bug in which backup/checkpoint can include a WAL deleted by RocksDB. * Fix a bug where concurrent compactions might cause unnecessary further write stalling. In some cases, this might cause write rate to drop to minimum. * Fix a bug in Logger where if dbname and db_log_dir are on different filesystems, dbname creation would fail wrt to db_log_dir path returning an error and fails to open the DB. * Fix a CPU and memory efficiency issue introduce by https://github.com/facebook/rocksdb/pull/8336 which made InternalKeyComparator configurable as an unintended side effect * Fix a bug where `GenericRateLimiter` could revert the bandwidth set dynamically using `SetBytesPerSecond()` when a user configures a structure enclosing it, e.g., using `GetOptionsFromString()` to configure an `Options` that references an existing `RateLimiter` object. ## Behavior Change * In leveled compaction with dynamic levelling, level multiplier is not anymore adjusted due to oversized L0. Instead, compaction score is adjusted by increasing size level target by adding incoming bytes from upper levels. This would deprioritize compactions from upper levels if more data from L0 is coming. This is to fix some unnecessary full stalling due to drastic change of level targets, while not wasting write bandwidth for compaction while writes are overloaded. * For track_and_verify_wals_in_manifest, revert to the original behavior before #10087: syncing of live WAL file is not tracked, and we track only the synced sizes of **closed** WALs. (PR #10330). * WAL compression now computes/verifies checksum during compression/decompression. ### Performance Improvements * Rather than doing total sort against all files in a level, SortFileByOverlappingRatio() to only find the top 50 files based on score. This can improve write throughput for the use cases where data is loaded in increasing key order and there are a lot of files in one LSM-tree, where applying compaction results is the bottleneck. * In leveled compaction, L0->L1 trivial move will allow more than one file to be moved in one compaction. This would allow L0 files to be moved down faster when data is loaded in sequential order, making slowdown or stop condition harder to hit. Also seek L0->L1 trivial move when only some files qualify. * In leveled compaction, try to trivial move more than one files if possible, up to 4 files or max_compaction_bytes. This is to allow higher write throughput for some use cases where data is loaded in sequential order, where appying compaction results is the bottleneck. 2022-08-24T22:39:19+00:00 Lookyloo v1.15.0 Lookyloo v1.15.0 2022-08-25T12:43:29+00:00 # Breaking change * Lookyloo requires Redis 7.0 or more decent. The upgrade process is as follows: 1. Go to the Redis directory (should be in the same directory as where you cloned Lookyloo) 2. Run the following commands ``` git fetch git checkout 7.0 make distclean make -j4 make test ``` 3. You now have the new version of redis in place, you can update lookyloo as usual. # New features * Use pre-configured devices from Playwright (mobile only for now) ![Device select for mobile](https://user-images.githubusercontent.com/248875/186662401-b6486584-ea7f-4f83-8e6f-f0d67d191e77.png) * Download files when the URLs points to a downloadable content ![Download file and submit to pandora](https://user-images.githubusercontent.com/248875/186667605-a5c0c667-cdbf-4fc2-ac84-e0a7b51c405c.png) * Submit downloadable content to [Pandora](https://pandora.circl.lu/submit) (if available) * Automatically select the most appropriate browser engine based on the user-agent # Fixes * Make sure all the gunicorn instances displays all the recent captures * Other bugfixes and GUI improvements # Changes * Improve capture page with radio button to select which user-agent to submit * Bump dependencies 2022-08-25T12:43:29+00:00 OpenTAXII 0.9.2 OpenTAXII 0.9.2 2022-08-26T12:58:36+00:00 Changelog ========= 0.9.2 (2022-08-26) ------------------ * Improve readability and navigation of docs (`#238 <https://github.com/eclecticiq/OpenTAXII/pull/238>`_ thanks `@zed-eiq <https://github.com/zed-eiq>`_ for the improvement). 0.9.1 (2022-07-11) ------------------ * Implement `raise_unauthorized` for taxii2, this was missing and lead to 500 errors. 0.9.0 (2022-06-13) ------------------ * Allow custom properties. This can be disabled by config option ``allow_custom_properties`` 0.8.0 (2022-06-05) ------------------ * Nest details inside taxii2 job and allow counts without details 0.7.0 (2022-05-27) ------------------ * Nest taxii2 endpoints under `/taxii2/` 0.6.0 (2022-05-25) ------------------ * Add `public_discovery` option to taxii2 config * Add support for publicly readable taxii 2 api roots 0.5.0 (2022-05-24) ------------------ * Add support for publicly readable taxii 2 collections 0.4.0 (2022-05-20) ------------------ * Move next_param handling into `OpenTAXII2PersistenceAPI` 0.3.0 (2022-04-13) ------------------ * Implement taxii2.1 support 0.3.0a4 (2022-04-13) -------------------- * Merge changes from 0.2.4 maintenance release 0.3.0a3 (2022-01-21) -------------------- * Fix bug that prevented booting with only taxii1 config (`#217 <https://github.com/eclecticiq/OpenTAXII/issues/217>`_ thanks `@azurekid <https://github.com/azurekid>`_ for the report) 0.3.0a2 (2021-12-27) -------------------- * Merge changes from 0.2.3 maintenance release 0.3.0a1 ------- * Add python 3.10 support 0.3.0a0 ------- * Enablement for future taxii2 implementation * Fix documentation build issues 0.2.4 (2022-04-13) ------------------ * Make sure werkzeug <2.1 and >=2.1 work correctly with auth system 0.2.3 (2021-12-22) ------------------ * Fix bug in multithreaded use of sqlite (`#210 <https://github.com/eclecticiq/OpenTAXII/issues/210>`_ thanks `@rohits144 <https://github.com/rohits144>`_ for the report) 0.2.2 (2021-11-05) ------------------ * Fix readthedocs build 0.2.1 (2021-11-03) ------------------ * Add tests for python 3.6, 3.7, 3.8, 3.9, pypy * Add tests for sqlite, mysql, mariadb, postgresql * Fix bug that broke ``delete_content_blocks`` when using mysql on sqlalchemy 1.3 * Docs: Add db schema diagram * Docs: Clarify how to get default data in a default (development) docker instance * Fix implicit routing in TAXII 1.1 Inboxes * Update jwt usage to pyjwt >= 2.0 (thanks `@SanyaKapoor <https://github.com/SanyaKapoor>`_) 0.2.0 (2020-06-30) ------------------ * Enforce UTC usage in datetime fields in SQL DB Persistence API. * `Fix for #114 <https://github.com/eclecticiq/OpenTAXII/issues/114>`_: reintroduce ``opentaxii-create-account`` CLI command. * `Fix for #153 <https://github.com/eclecticiq/OpenTAXII/issues/152>`_: check if user can modify a collection before advertising it over inbox service. * Multiple coding style fixes. * Various documentation updates. 0.1.12 (2019-03-06) ------------------- * Remove unnecessary print statements. 0.1.11 (2019-02-13) ------------------- * Make JSON logging consistent when the application is run via Gunicorn. * Set ``acceptable_destination`` key in status details instead of extended headers * Allow passing ``engine_parameters`` to ``SQLDatabaseAPI`` for those who want to customize SQLAlchemy engine parameters. * Require recent version of ``lxml`` for security reasons. * Various test and Docker infrastructure improvements. 0.1.10 (2018-06-03) ------------------- * Replace separate service/collection/account creation process with single ``opentaxii-sync-data`` CLI command. * Persistence and Auth APIs extended with missing CRUD methods, that are used by ``opentaxii-sync-data``. * Read/modify collection level ACL added. * DB models for default implementation of Persistence API and Auth API were changed. No automatic migration code is provided (sorry!), so upgrading might require manual DB migration. * Drop python2.7 from testing scope. * Various bug fixes and improvements. 0.1.9 (2017-06-19) ------------------ * `libtaxii <https://github.com/TAXIIProject/libtaxii>`_ dependency upgraded to 1.1.111. * Various bug fixes and improvements (thanks to `@bjigmp <https://github.com/bjigmp>`_, `@chorsley <https://github.com/chorsley>`_, `@rjprins <https://github.com/rjprins>`_). 0.1.8 (2017-02-21) ------------------ * Ability to enable/disable "huge trees" support in XML parser. Configuration property ``xml_parser_supports_huge_tree`` set to ``yes`` or ``true`` will disable security restrictions and force XML parser to support very deep trees and very long text content. * Adjust SQL Persistence API implemetation so it works smoothly with MySQL backend. * Use Python 3.5 instead of Python 3.4 for tests. 0.1.7 (2016-10-18) ------------------ * Minor fixes. * Dependencies were changed from hard-pinned to more flexible. * Example of production DB configuration added to docs. 0.1.6 (2016-06-01) ------------------ * Python 3.4 compatibility of the codebase. Tox configuration extended with python 3.4 environment run. * Flake8 full style compatibility. Flake8 check added to Tox configuration. * SQLAlchemy session scope issue fixed (related to `#38 <https://github.com/EclecticIQ/OpenTAXII/issues/38>`_). * `opentaxii-delete-blocks` CLI command added (related to `#45 <https://github.com/EclecticIQ/OpenTAXII/issues/45>`_). * `delete_content_blocks` method `added <https://github.com/EclecticIQ/OpenTAXII/commit/dc6fddc27a98e8450c7e05e583b2bfb741f6e197#diff-6814849ac352b2b74132f8fa52e0ec4eR213>`_ to Persistence API. * Collection's name is `required <https://github.com/EclecticIQ/OpenTAXII/commit/dc6fddc27a98e8450c7e05e583b2bfb741f6e197#diff-ce3f7b939e5c540480ac655aef32c513R116>`_ to be unique in default SQL DB Auth API implementation. 0.1.5 (2016-03-15) ------------------ * Fix for the issue with persistence backend returning ``None`` instead of ``InboxMessage`` object 0.1.4 (2016-02-25) ------------------ * Hard-coded dependencies in ``setup.py`` removed. 0.1.3 (2016-02-25) ------------------ * Versions of dependencies are pinned. * Code adjusted for a new version of `anyconfig <https://pypi.python.org/pypi/anyconfig>`_ API. * Test for configuration loading added. 0.1.2 (2015-07-24) ------------------ * Docker configuration added. * Health check endpoint added. * Basic authentication support added. * Temporary workaround for `Issue #191 <https://github.com/TAXIIProject/libtaxii/issues/191>`_. * Method ``get_domain`` in Persistence API returns domain value configured for ``service_id``. If nothing returned, value set in configuration file will be used. * Performance optimisations. * Bug fixes and style improvements. 0.1.1 (2015-04-08) ------------------ * Alias for Root Logger added to logging configuration. * Context object in a request scope that holds account and token added. * Support for OPTIONS HTTP request to enable auto version negotiation added. * Documentation improved. 0.1.0 (2015-03-31) ------------------ * Initial release 2022-08-26T12:58:36+00:00 ursadb v1.5.0 ursadb v1.5.0 2022-08-29T16:39:10+00:00 Features: - Alternatives (like {(41 | 42)}) implemented in the ursadb query syntax (#65) - Better support for wildcards (#23) - Syntax for indexing with taints (#31) Performance: - Query graph pruning (#67) Correctness: - Some improvements for thread safety (#32) Refactoring and maintenance: - Ursacli rewritten to C++ (#48) - Documentation improvements (#33) 2022-08-29T16:39:10+00:00 TheHive 4.1.23 TheHive 4.1.23 2022-08-30T07:28:56+00:00 ## [4.1.23](https://github.com/TheHive-Project/TheHive/milestone/94) (2022-07-22) **Fixed bugs:** - [Bug] system user can be deleted by integrity checks [\#2406](https://github.com/TheHive-Project/TheHive/issues/2406) 2022-08-30T07:28:56+00:00 MONARC v2.12.2-p4 MONARC v2.12.2-p4 2022-08-30T07:41:11+00:00 Updated zm-client to [v2.12.1-p1](https://github.com/monarc-project/zm-client/releases/tag/v2.12.1-p1). 2022-08-30T07:41:11+00:00 pandora v1.1.0 pandora v1.1.0 2022-08-31T09:25:44+00:00 # Breaking change This release requires poetry v1.2.0 or more recent. Run the following command to upgrade it: `poetry self update` # New feature * HTML documents can be submitted to [Lookyloo](https://github.com/Lookyloo/lookyloo) (requires v1.15.0 or more recent) ![submit to lookyloo](https://user-images.githubusercontent.com/248875/187422078-f601b1f7-0cbf-47f0-aa9f-31353d3ee4d6.png) ![lookyloo capture](https://user-images.githubusercontent.com/248875/187422923-a75474e7-269f-413e-ae43-1437d6dcc59b.png) # Changes * Improvements in the modules (archives, ISO, EML) * Improvements on the stats page * Configure the links on the index * Bump dependencies # Fixes * Support ingesting a file downloaded from a Pandora instance * Automatically restart unoserver when it crashes (makes previews with libreoffice more reliable) # Notes * Many have reported issues with the reviews generated by LibreOffice. A seemingly universal fix seems to be installing the [full package from the PPA](https://github.com/pandora-analysis/pandora#important-notes-regarding-libreoffice). 2022-08-31T09:25:44+00:00 maltrail 0.49 maltrail 0.49 2022-08-31T22:11:05+00:00 Start-of-month release 2022-08-31T22:11:05+00:00 caddy v2.6.0-beta.3 caddy v2.6.0-beta.3 2022-09-05T22:07:59+00:00 This is the _first beta_ release for Caddy 2.6. Please try it out and report any regressions you notice! Thanks to everyone who helped out! :blush: Beta 1 and beta 2 were trial runs for our CI upgrades, so this is technically beta 3. There are no code changes from beta 1 to beta 3. Thank you @mohammed90 for figuring out the CI magic! These are abbreviated release notes. The full release notes (23.5 KB of them, to be precise) will be published with the final 2.6 release, explaining in detail all the new few features, enhancements, and bug fixes. ## What's Changed * reverseproxy: Implement retry count, alternative to try_duration by @francislavoie in https://github.com/caddyserver/caddy/pull/4756 * cmd: Fix reload with stdin by @francislavoie in https://github.com/caddyserver/caddy/pull/4900 * reverseproxy: Implement read & write timeouts for HTTP transport by @mholt in https://github.com/caddyserver/caddy/pull/4905 * ci: Run golangci-lint on multiple os(#4875) by @u5surf in https://github.com/caddyserver/caddy/pull/4913 * caddyhttp: Clear out matcher error immediately after grabbing it by @francislavoie in https://github.com/caddyserver/caddy/pull/4916 * chore: Add .gitattributes to force *.go to LF by @francislavoie in https://github.com/caddyserver/caddy/pull/4919 * core: Windows service integration by @WingLim in https://github.com/caddyserver/caddy/pull/4790 * fileserver: Support virtual file systems by @mholt in https://github.com/caddyserver/caddy/pull/4909 * caddyhttp: Implement `caddy respond` command by @mholt in https://github.com/caddyserver/caddy/pull/4870 * chore: Bump up to Go 1.19, minimum 1.18 by @francislavoie in https://github.com/caddyserver/caddy/pull/4925 * httpserver: Configurable shutdown delay by @mholt in https://github.com/caddyserver/caddy/pull/4906 * cmd: Use newly-available version information by @mholt in https://github.com/caddyserver/caddy/pull/4931 * Replace strings.Index usages with strings.Cut by @WilczynskiT in https://github.com/caddyserver/caddy/pull/4930 * optimization: Replaced strings.Index with strings.Cut by @chir4gm in https://github.com/caddyserver/caddy/pull/4932 * go.mod: Upgrade OpenTelemetry dependencies by @lewandowski-stripe in https://github.com/caddyserver/caddy/pull/4937 * logging: Fix `cookie` filter by @francislavoie in https://github.com/caddyserver/caddy/pull/4943 * reverseproxy: Support 1xx status codes (HTTP early hints) by @dunglas in https://github.com/caddyserver/caddy/pull/4882 * reverseproxy: Fix H2C dialer using new stdlib `DialTLSContext` by @francislavoie in https://github.com/caddyserver/caddy/pull/4951 * reverseproxy: Ignore context cancel in stream mode by @mholt in https://github.com/caddyserver/caddy/pull/4952 * reverseproxy: Add `unix+h2c` Caddyfile network shortcut by @francislavoie in https://github.com/caddyserver/caddy/pull/4953 * caddyhttp: Enable HTTP/3 by default by @mholt in https://github.com/caddyserver/caddy/pull/4707 * fileserver: reset buffer before using it (#4962) by @abdusco in https://github.com/caddyserver/caddy/pull/4963 * caddyhttp: Smarter path matching and rewriting by @mholt in https://github.com/caddyserver/caddy/pull/4948 * core: Change net.IP to netip.Addr; use netip.Prefix by @WilczynskiT in https://github.com/caddyserver/caddy/pull/4966 * caddyhttp: Set `http.error.message` to the HandlerError message by @francislavoie in https://github.com/caddyserver/caddy/pull/4971 * caddyhttp: Fix for nil `handlerErr.Err` by @francislavoie in https://github.com/caddyserver/caddy/pull/4977 * templates: cap of slice should not be smaller than length by @jedy in https://github.com/caddyserver/caddy/pull/4975 * ci: Increase linter timeout by @mholt in https://github.com/caddyserver/caddy/pull/4981 * httpcaddyfile: Add ocsp_interval global option by @benburkert in https://github.com/caddyserver/caddy/pull/4980 * zstd: fix typo in comment by @simon04 in https://github.com/caddyserver/caddy/pull/4985 * admin: Don't stop old server if new one fails by @WeidiDeng in https://github.com/caddyserver/caddy/pull/4964 * reverseproxy: Add upstreams healthy metrics by @david-szabo97 in https://github.com/caddyserver/caddy/pull/4935 * Remove duplicate words in comments by @Abirdcfly in https://github.com/caddyserver/caddy/pull/4986 * cmd: Migrate to `spf13/cobra`, remove single-dash arg support by @mohammed90 in https://github.com/caddyserver/caddy/pull/4565 * cmd: add completion command by @mohammed90 in https://github.com/caddyserver/caddy/pull/4994 * caddyhttp: Set Content-Type for static response by @mholt in https://github.com/caddyserver/caddy/pull/4999 * httpcaddyfile: Add `{cookie.*}` placeholder shortcut by @francislavoie in https://github.com/caddyserver/caddy/pull/5001 * events: Implement event system by @francislavoie in https://github.com/caddyserver/caddy/pull/4912 * core: Plugins can register listener networks by @mholt in https://github.com/caddyserver/caddy/pull/5002 * httpcaddyfile: Add shortcut for expression matchers by @francislavoie in https://github.com/caddyserver/caddy/pull/4976 * caddyhttp: Copy logger config to HTTP server during AutoHTTPS by @francislavoie in https://github.com/caddyserver/caddy/pull/4990 * dist: deb package manpages and bash completion scripts by @mohammed90 in https://github.com/caddyserver/caddy/pull/5007 * fastcgi: allow users to log stderr output (#4967) by @flga in https://github.com/caddyserver/caddy/pull/5004 * templates: Document `httpError` function by @Malankar in https://github.com/caddyserver/caddy/pull/4972 * fastcgi: Optimize FastCGI transport by @WeidiDeng in https://github.com/caddyserver/caddy/pull/4978 * core: Refactor listeners; use SO_REUSEPORT on Unix by @mholt in https://github.com/caddyserver/caddy/pull/4705 * reverseproxy: Close hijacked conns on reload/quit by @mholt in https://github.com/caddyserver/caddy/pull/4895 * ci: generate SBOM and sign artifacts using cosign by @mohammed90 in https://github.com/caddyserver/caddy/pull/4910 * caddyauth: Speed up basicauth provision, deprecate `scrypt` by @francislavoie in https://github.com/caddyserver/caddy/pull/4720 * caddyhttp: Support `respond` with HTTP 103 Early Hints by @mholt in https://github.com/caddyserver/caddy/pull/5006 * fileserver: Support glob expansion in file matcher by @mholt in https://github.com/caddyserver/caddy/pull/4993 * ci: add `id-token` permission and update the signing command by @mohammed90 in https://github.com/caddyserver/caddy/pull/5016 * ci: grant the `release` workflow the `write` permission to `contents` by @mohammed90 in https://github.com/caddyserver/caddy/pull/5017 ## New Contributors * @WingLim made their first contribution in https://github.com/caddyserver/caddy/pull/4790 * @WilczynskiT made their first contribution in https://github.com/caddyserver/caddy/pull/4930 * @chir4gm made their first contribution in https://github.com/caddyserver/caddy/pull/4932 * @lewandowski-stripe made their first contribution in https://github.com/caddyserver/caddy/pull/4937 * @abdusco made their first contribution in https://github.com/caddyserver/caddy/pull/4963 * @jedy made their first contribution in https://github.com/caddyserver/caddy/pull/4975 * @benburkert made their first contribution in https://github.com/caddyserver/caddy/pull/4980 * @WeidiDeng made their first contribution in https://github.com/caddyserver/caddy/pull/4964 * @david-szabo97 made their first contribution in https://github.com/caddyserver/caddy/pull/4935 * @Abirdcfly made their first contribution in https://github.com/caddyserver/caddy/pull/4986 * @flga made their first contribution in https://github.com/caddyserver/caddy/pull/5004 * @Malankar made their first contribution in https://github.com/caddyserver/caddy/pull/4972 **Full Changelog**: https://github.com/caddyserver/caddy/compare/v2.5.2...v2.6.0-beta.3 2022-09-05T22:07:59+00:00 FIR django4 FIR django4 2022-09-13T07:37:07+00:00 2022-09-13T07:37:07+00:00 TheHive 4.1.24 TheHive 4.1.24 2022-09-13T07:48:11+00:00 ## [4.1.24](https://github.com/TheHive-Project/TheHive/milestone/95) (2022-09-12) **Fixed bugs:** - [Bug] All analyzers become unavailable when an analyzer is updated [\#2420](https://github.com/TheHive-Project/TheHive/issues/2420) 2022-09-13T07:48:11+00:00 MISP v2.4.162 MISP v2.4.162 2022-09-13T08:42:19+00:00 ![](https://www.misp-project.org/img/blog/workflow.png ) We are pleased to announce the immediate availability of [MISP v2.4.162](https://github.com/MISP/MISP/releases/tag/v2.4.162) with a new periodic notification system, workflow updates and many improvements. In addition to the MISP v2.4.162 release, [misp-guard](https://github.com/MISP/misp-guard) has been released which is a [mitmproxy](https://mitmproxy.org/) addon that inspects the events that MISP is attempting to synchronize with external MISP instances via `PUSH` or `PULL` and applies a set of customizable rules defined in a JSON file. This is a complementary tool to support MISP users having to interconenct MISP instances between highly sensitive networks. # Periodic notification system As of version 2.4.162, MISP includes a **periodic summary** feature allowing users to consult a summary based on a requested time-frame for data the user has access to. Currently, the summaries can be generated for 3 different periods: `daily`, `weekly` and `monthly` and then sent to all users that subscribed one of these periods. In addition to choose which period users want to subscribed to, they can also specify filtering options such as tags or distribution level to be used to generate the summary. The summary can be sent via email in addition to the User-Interface view. ![Periodic summary](https://www.misp-project.org/img/blog/periodic-summary/periodic-summary-2.png) ![Periodic summary](https://www.misp-project.org/img/blog/periodic-summary/periodic-summary-3.png) For more information, check out the [Periodic summaries - Visualize summaries of MISP data](/2022/09/12/2022-09-12_periodic_notifications.html/) blog. # Workflow improvements - Added diagnostic support and support of arbitrary URL for webhook module. - New Microsoft teams module based on the webhook module. - New email notification module to send email to a list of MISP users including [Jinja templating](https://jinja.palletsprojects.com/en/3.1.x/). - Tag name can now be used in workflows. For more details about MISP Workflow, check out the [training materials](https://www.misp-project.org/misp-training/a.12-misp-workflows.pdf). # MISP core improvements - Allow option to delete tags on event sync prior to soft-delete tag implementation. - API/[Event:restSearch] Added option `event_tags` to filter for eventTag only. - API/RestSearch - Added support of `static` parameter to produce a static HTML output. - Syslog/logging for certain log entries vital information was omitted by the syslog. If no custom message is specifically set for the log entry, the change field is included. - Enforce UUIDs uniqueness on MISP data back-end. # Bugs fixed - [correlations] save the distribution state of the event before/after saving it, fixes #8528. - [attribute tags] removal broken, fixes #8567. - Class 'Folder' not found #8544. - Create unique SIDs for email attributes in NIDS export. Thanks to all the contributors and users reporting bugs to make the software better. As always, a detailed and [complete changelog is available](https://www.misp-project.org/Changelog.txt) with all the fixes, changes and improvements in MISP core. Many improvements in the MISP galaxy and especially the threat-actor galaxy. There is a detailed [changelog](https://www.misp-project.org/Changelog-misp-galaxy.txt). Improvement in the `false-positive` taxonomy and many other taxonomies. There is a detailed [changelog](https://www.misp-project.org/Changelog-misp-taxonomies.txt). Multiple objects were updated and added, for [more details](https://www.misp-project.org/Changelog-misp-objects.txt). 2022-09-13T08:42:19+00:00 DC3-MWCP 3.8.0 DC3-MWCP 3.8.0 2022-09-16T12:29:56+00:00 2022-09-16T12:29:56+00:00 caddy v2.6.0-beta.5 caddy v2.6.0-beta.5 2022-09-18T00:23:34+00:00 This release and beta.4 are mainly tests of our CI, but also contains a bunch of small fixes or enhancements, including, notably, the use of `sendfile` and other optimizations. See the release notes for beta.3 for everything else until 2.6.0 lands soon. Full release notes coming! ## Changelog * e6c58fdc caddyfile: Prevent infinite nesting on fmt (fix #4175) * 9ad0ebc9 caddyhttp: Add 'skip_log' var to omit request from logs (#4691) * b6cec378 caddyhttp: Add --debug flag to commands * 61c75f74 caddyhttp: Explicitly disallow multiple regexp matchers (#5030) * 00beec2e caddyhttp: Fix write header on responseRecorder * 74547f5b caddyhttp: Make metrics opt-in (#5042) * 61641828 caddyhttp: Support TLS key logging for debugging (#4808) * bffc2587 caddyhttp: Support configuring Server from handler provisioning (#4933) * 20d487be caddyhttp: Very minor optimization to path matcher * dd9813c6 caddyhttp: ensure ResponseWriterWrapper and ResponseRecorder use ReadFrom if the underlying response writer implements it. (#5022) * 258071d8 caddytls: Debug log on implicit tailscale error (#5041) * d35f618b caddytls: Error if placeholder is empty in 'ask' * 0b3161ae cmd: Customizable user agent (close #2795) * 1c9c8f6a cmd: Enhance some help text * 62b06853 cmd: Improve error message if config missing * 50748e19 core: Check error on ListenQUIC * e43b6d81 core: Variadic Context.Logger(); soft deprecation * 48d723c0 encode: Fix Accept-Ranges header; HEAD requests (#5039) * c19f2072 fileserver: Ignore EOF when browsing empty dir * c5df7bb6 go.mod: Update truststore * 8cc8f9fd httpcaddyfile: Add a couple more placeholder shortcuts (#5015) * a1ad20e4 httpcaddyfile: Fix bind when IPv6 is specified with network (#4950) * 754fe4f7 httpcaddyfile: Fix sorting of repeated directives * e338648f reverseproxy: Support repeated --to flags in command (#4693) * 9fe4f93b supplychain: publish signing cert, sbom, and signatures of sbom (#5027) 2022-09-18T00:23:34+00:00 caddy v2.6.0 caddy v2.6.0 2022-09-20T18:12:38+00:00 Caddy 2.6 ======== This is our biggest release since Caddy 2. Caddy 2 changed the way the world serves the Web. By providing an online config API, automatic HTTPS, unlimited extensibility, certificate automation at scale, modern protocols, sane defaults, and an unrivaled developer experience, we boldly raised the bar for web servers. Now with Caddy 2.6, we're doing it again. Caddy 2.6 is the first general-purpose web server to seamlessly enable the newly-standardized HTTP/3 protocol for all configurations by default. We've virtualized the file system so you can serve content from anywhere or anything. New event features let you observe and control Caddy's internals with custom actions. Caddy is more useful than ever for developers with its enhanced CLI tooling and features. And it's faster than ever with non-trivial performance improvements. We think you will love this release. [Watch the livestream](https://youtu.be/iJZqFpY2D14) Special dedication ------------------ This release is dedicated to the late Peter Eckersley, who passed away September 2, 2022. Peter is one of the brilliant minds behind Let's Encrypt; his work has benefited billions of people. I met Peter at the Let's Encrypt launch party in a little bar in San Francisco in 2015 and have never forgotten that occasion. He later co-authored a published research paper called [_Let’s Encrypt: An Automated Certificate Authority to Encrypt the Entire Web_](https://jhalderm.com/pub/papers/letsencrypt-ccs19.pdf), which highly espoused Caddy's ACME integration: "We hope to see other popular server software follow Caddy’s lead." We look forward to when other servers do that, and we hope to honor Peter's work and influence which will live on through his memory and the encrypted Web he made possible. --- [Sponsors](https://github.com/sponsors/mholt) -------- [ZeroSSL](https://zerossl.com) remains Caddy's executive sponsor. [We were thrilled](https://twitter.com/caddyserver/status/1559591673511813120) to welcome [Stripe](https://stripe.com) recently as an enterprise sponsor! Other notable sponsors include [AppCove](https://appcove.com/), [Dukaan](https://mydukaan.io/), [Suborbital](https://suborbital.dev), [Tailscale](https://tailscale.com), plus [Bubble](https://bubble.io/) and [GitHub](https://github.com) which both made generous one-time donations. We have many other vital sponsors and donors on which we also rely. [Our sponsors](https://github.com/sponsors/mholt) come from all over the world and include independent professionals, startups, and small companies -- and they are the absolute best. Thank you for making a more secure Web possible! _Personal note from Matt: Recent life upgrades mean that your sponsorships now sustain a family of 5 so that I can continue to maintain Caddy. Two years ago, I don't think I would have taken this risk because I'd need to find other work to provide for a family. Thank you for coming together as a professional community to make the Caddy project possible!_ We strongly recommend that companies who -- or companies whose customers -- use or benefit from Caddy [become a sponsor](https://github.com/sponsors/mholt) to ensure ongoing maintenance, priority development, private support, and more. Sponsorship tiers can be tailored to your requirements! Highlights ---------- :warning: **_Don't miss deprecations / breaking changes at the bottom. Notably, if you use metrics, you will now need to turn them on._** ### HTTP/3 is here (#4707) Caddy now enables [RFC 9114](https://datatracker.ietf.org/doc/rfc9114/)-compliant HTTP/3 by default. The `experimental_http3` option has graduated and been removed. We've removed another experimental option, `allow_h2c`, and individual HTTP versions (`h1 h2 h2c h3`) can now be toggled with the new `protocols` setting. Note that HTTP/3 utilizes the QUIC transport, which requires UDP. If your network or firewall configuration only allows TCP, HTTP/3 connections will fail and clients (should) fall back to HTTP/2. For servers with properly-configured UDP networks, HTTP/3 should "just work" for enabled clients. HTTP/3 clients can connect by reading Caddy's Alt-Svc header to know how to connect to Caddy via UDP. This header is now emitted automatically and by default. Other than that, there are no other changes needed to existing servers, as Caddy opens a separate UDP socket for HTTP/3. Our HTTP/3 server attempts to mitigate amplification and reflection attacks by requiring address validation when the server is under load. This adds one round-trip for clients, but is only done as a defensive measure when necessary. Serious thanks to @marten-seemann who builds and maintains the [quic-go](https://github.com/lucas-clemente/quic-go) library we depend on for this. (Go has not announced any plans to officially support or implement HTTP/3.) We expect numerous QUIC and HTTP/3 improvements to come as implementations and best practices mature with more production experience. ### Virtual file systems (#4909) Caddy's `file_server` module now supports _virtual_ file systems. We've replaced all hard-coded `os.Open()`, `os.Stat()`, etc. calls with Go's relatively new `io/fs` package, and introduced a new Caddy module namespace `caddy.fs` for implementations of such file systems. Some examples of what is possible: - Serve content from S3 or other blob/cloud storage services - Serve dynamically-generated content that "feels" static - [Embed your site directly into your `caddy` binary](https://github.com/mholt/caddy-embed) and serve it from memory - Serve content directly from an archive file (e.g. `.zip` or `.tar.gz`) - Load files from a database instead of disk Basically, instead of serving files from the local disk, you can have Caddy serve the "files" from somewhere or something else. The default is still the local file system. Note that this feature isn't limited to just Caddy's `file_server` module. Potentially any module that reads the local disk may benefit from using `caddy.fs` modules instead. I wrote [a module that lets you embed your site](https://github.com/mholt/caddy-embed) within your `caddy` binary -- wherever your server goes, your site goes! We encourage the community to implement and publish new file system modules for Caddy. (From an [early tweet](https://twitter.com/mholt6/status/1551434889358032897) there seems to be quite high demand.) ### Events (#4912 and #4984) Not surprisingly, many people prefer Caddy to automate certificates used with other software/services. Until now, there hasn't been a great way to know when Caddy has obtained or renewed a certificate (deferred in part by our opinion that certificate management should be baked into the software using the certificate in the first place). Cron jobs generally work for reloading new certificates into services because certificate expiry is _mostly_ predictable, but now there is a better way with one of our most requested features: events! We thought about events in general for a long time and discussed questions like, "What makes an event different from a log?" "Are events synchronous?" "Do self-initiated events get emitted before or after their code (are they past-tense or future-tense) -- or both? or neither (asynchronous)?" "What do we like from existing event systems?" "What do we wish event systems did differently?" While we think we have pretty good answers to these questions now, we won't be sure until we gather more production experience. For this reason, events are implemented as an experimental app module -- not as part of the core. (Remember, Caddy's core currently only loads config and sets up logging/storage.) This means that Caddy's core cannot emit events.[^1] So even though our event implementation may change, it is likely to be only slight and gradual changes; and we encourage anyone and everyone to start using events as soon as possible and to **give us your feedback**. We think we have the start of a great event system, but we need you to prove it! Caddy modules can emit events when interesting things happen. For example, the reverse proxy emits `healthy` and `unhealthy` events when backends go up and down. The TLS app emits `cert_obtaining`, `cert_obtained`, and `cert_failed` before and after obtaining a certificate or after the operation failed, respectively; and `cert_ocsp_revoked` after a certificate is discovered to be revoked by OCSP. There are several more events already, with even more to be added later. Events can have data associated with them. For example, `healthy`/`unhealthy` come with the address of the host; `cert_obtained` has the domain name, issuer, and storage path. You can access this from config in placeholders, e.g. `{event.data.identifier}`. Caddy modules can subscribe to events by specifying the name(s) of events to bind to, and the Caddy module ID(s) or namespace(s) to watch. When an event is emitted, it propagates from the module that emitted it up the provisioning heirarchy. This means that an event emitted by `http.handlers.reverse_proxy` will fire for `http.handlers` and `http` as well, similar to the DOM in HTML/JavaScript. Event handlers are invoked synchronously. We chose this for several reasons. First, despite how easy Go makes concurrency, there are many subtleties to concurrency in a server. Goroutines may be lightweight, but their operations might not be; and if event goroutines are starting more quickly than they are stopping, we either drop events arbitrarily or run out of memory/CPU. Also, we think one of the qualities that differentiates events from logs is the ability for an event to influence the emitting code's flow: a true "hook" in that sense. Instead of simply observing that something is happening (which is what a log tells you), you can influence its behavior. Maybe you want to run a command before a certificate is obtained to see if it _should_ be obtained. Or maybe you want to change how a TLS handshake is completed on-the-fly. Asynchronous event handlers cannot do this. For simple behavioral changes, synchronous events can be a powerful and useful tool for customizing your server. The new `event` app lets you easily configure subscriptions and event handlers. Event handling is modular, so you will need to plug in a module that does what you want: run a command, reload a service, make an HTTP request, or anything else! Because this feature is experimental and new, we don't yet know how people will be using it, so currently, Caddy does not ship with any event handler plugins. However, we're pretty sure based on feedback over the years that many of you would like to run commands on certain events (one of our top feature requests is to trigger a daemon reload after certificate renewals). So I went ahead and implemented an [`exec` event handler plugin](https://github.com/mholt/caddy-events-exec) that can run commands. We almost included it in Caddy's standard distribution, but out of an abundance of caution we decided to keep it a separate plugin for now until we learn more about real production use cases from experience. Here's an example of handling events. In JSON, you configure the `events` app: ```json { "apps": { "events": { "subscriptions": [ { "events": ["cert_obtained"], "handlers": [ { "handler": "exec", "command": "systemctl", "args": ["reload", "mydaemon"] } ] } ] } } } ``` or the equivalent Caddyfile global option: ``` { events { on cert_obtained exec systemctl reload mydaemon } } ``` It's that simple! Just make sure you have your event handler modules plugged in. **We hope you will provide feedback, report bugs, and request features related to events.** [^1]: Compilation fails with an import cycle. If Caddy core uses any feature of Caddy, it must also be in the core or another package not imported by any modules! ### Smarter path matching and rewriting (#4948) Is the URI path `/a/b/c` the same as `/a/b%2Fc`? What about `/a/b//c`? Turns out, _it depends._ What these questions illustrate is a famously frustrating problem, and has largely gone unsolved until now. All existing solutions I investigated in other products were unsatisfactory: - Nginx (and Caddy until now) always does path comparisons in unescaped/normalized space. This makes it impossible to route on literal escape sequences unless you double-encode your pattern, which violates specification. - Apache outright rejects valid[^2] HTTP requests containing encoded slashes. This behavior can either be disabled completely (creating a security problem known as unsafe paths) or tweaked to never decode encoded slashes (creating ambiguities when comparing against route patterns). - Laravel, like nginx, always decodes slashes, but routing such requests [mangles application data](https://github.com/laravel/framework/issues/22125) that contains slashes. The process of decoding a URI and collapsing slashes in the path is called normalization. Normalization has to occur for safe, reliable routing (imagine `//secret` bypassing auth checks configured for `/secret`), but at the same time, raw paths are sometimes needed to preserve application data (imagine a route `/bands/:name` which succeeds for `/bands/AC&2fDC` but fails for the normalized `/bands/AC/DC`). And it's not just routing; servers like Caddy often rewrite/manipulate paths. Because normalizing URIs creates a Many:1 mapping (there are multiple encoded forms of a single URI), normalizing is inherently lossy: the original input cannot be recovered with certainty, so we can't reconstruct the original or intended URI with complete fidelity. Other solutions with coarse on-off knobs can't balance both security and application correctness: it seems you have to trade one for the other. The crux of the problem seems to be that the server/framework/router doesn't know which parts of the path are application data and which parts are path components, so it just "plays it safe" and decodes the whole thing. I think Caddy's solution to this is quite novel. **Our solution is to interpret encoded characters and multiple slashes in a path pattern literally as a hint of the developer's intent.** For example, if you write a path matcher `/a/b/c`, it will still match `/a/b/c` and `/a/b%2Fc`. However, if your path matcher is `/a/b%2Fc`, Caddy will _only_ match `/a/b%2Fc`. This extends to wildcards with our new "escape-wildcard" feature: `/bands/%*/` will match `/bands/AC%2fDC` but `/bands/*/` won't. This works for multiple slashes too. If your path matcher uses `//`, Caddy will require the request path to contain those slashes literally at that position. We've also implemented this for prefix and suffix manipulations. For example, if you wanted to strip a prefix of `//prefix` from `//prefix/foo`, it will now work, whereas before it wouldn't because it would look at a fully-normalized URI. Essentially, we use the configured path pattern as a cue for whether to decode/merge a character or leave it raw when normalizing. This is a complex and subtle change, so please be sure to read the full PR in #4948 and the linked Laravel issue. It's very informative! [^2]: The "validity" of such a URI based on spec compliance is debatable. RFC 9110 says, "distinct resources SHOULD NOT be identified by HTTP URIs that are equivalent after normalization." ### HTTP 103 Early Hints (#4882 and #5006) HTTP Early Hints ([RFC 8297](https://datatracker.ietf.org/doc/rfc8297/)) is the effective successor to HTTP/2 Server Push. When 103 is emitted with relevant Link headers, web pages will load faster than normal. 1xx responses are precursors to the final response; clients must be able to support receiving multiple responses to a single request (nearly all modern clients do; and it almost certainly shouldn't break any HTTP/2 clients). Early hints are a great way to speed up page loads where the main content may take a while to generate (a slow DB query, for example) but the subresources can start being loaded right away. In those cases it is often beneficial to send early hints. Caddy can both originate and proxy 103 responses. To send early hints from Caddy, simply set the `Link` headers as the hints, then write the response with a 103 status code: ``` route /slow-pages/* { header Link "</style.css>; rel=preload; as=style" header Link "</script.js>; rel=preload; as=script" respond 103 } ``` Unlike normal responses, after writing HTTP 103, Caddy's middleware chain will _continue_ to execute and invoke the next handlers (for example, `reverse_proxy`) since 103 is not the final response. Multiple 103s can be sent. Caddy's reverse proxy also supports HTTP 103 responses, meaning that backends can send early hints and Caddy will proxy them to the client immediately as you'd expect., Note that browser support is still limited (only Chrome implements it at this time) and Caddy must be built with Go 1.19 (our builds use the latest Go version; but we still support Go 1.18 for now). Thank you to @dunglas with API Platform for contributing this feature to both Go and Caddy! ### Improved command line interface (#4565 and #4994) Caddy has always used Go's standard `flag` package for its CLI, which has served us quite well. However, recent improvements in the [Cobra](https://github.com/spf13/cobra) library make it possible for our CLI to gain worthwhile features without incurring a heavy dependency. The new `caddy manpage` command generates man pages, and the `caddy completion` command generates shell completions. Both are installed automatically as part of our official Linux packages, so your next `apt upgrade` (etc.) should take care of that. Additionally, short options (e.g. `-c`) are now supported. And if you typo a command, Caddy will helpfully suggest a correction (e.g. `caddy adpt` will suggest `caddy adapt`). Note that long-form flags must now use double-hyphen syntax (e.g. `--config`) even though the single-hyphen syntax (`-config`) was previously accepted. The standard library's flag parser treats `-` and `--` the same, but Cobra's does not. Our online documentation has always used `--` for flags, so we do not consider this a breaking change, but it's good to be aware of this change if you're used to how Go's parser works. Very many thanks to @mohammed90 for contributing these features! ### New `caddy respond` command (#4870) For rapid development needing a local HTTP server, the [`caddy respond` command](https://caddyserver.com/docs/command-line#caddy-respond) might be just what you need: hard-coded HTTP responses for one or more servers so that you can effortlessly have a custom HTTP endpoint to test with. A plain `caddy respond` command will listen on a random port and reply with HTTP 200. (The port or address is printed to the terminal for you.) You can set a custom status code like `caddy respond 401` or a custom body like `caddy respond "Hello world!"` -- or both: `caddy respond --status 401 "Hello world!"` Or you can pipe in a response body, for example serving a maintenance page: ```bash $ cat maintenance.html | caddy respond --status 503 --header "Content-Type: text/html; charset=utf-8" ``` You can even spin up multiple servers at once and use basic template features to configure each server with a different response: ```bash $ echo "I'm server {{.N}} on port {{.Port}}" | caddy respond --listen :2000-2004 Server address: [::]:2000 Server address: [::]:2001 Server address: [::]:2002 Server address: [::]:2003 Server address: [::]:2004 $ curl 127.0.0.1:2002 I'm server 2 on port 2002 ``` You can debug HTTP clients easier by enabling access logging with the `--access-log` flag. The `--header` flag can be used multiple times to set custom HTTP headers, and `--debug` enables debug mode for more verbose logging. We hope you find this feature useful! ### Multiple dynamic upstream sources (5fb5b81) In Caddy 2.5(.1) we introduced dynamic upstreams, which allow you to configure the `reverse_proxy` to get the list of backends on-the-fly during requests. This very popular feature's development was sponsored by Stripe, who we are thrilled to welcome as an enterprise sponsor. Stripe uses Caddy heavily for their internal systems, and for greater redundancy they need to be able to fail over to secondary upstreams if a primary cluster is down. This is where the new `multi` dynamic upstreams module comes in. Now you can configure, for example, two SRV lookups for aggregated results: ```json { "handler": "reverse_proxy", "dynamic_upstreams": { "source": "multi", "sources": [ { "source": "srv", "name": "primary" }, { "source": "srv", "name": "secondary" } ] } } ``` This appends the backends returned from the secondary SRV lookup to the results of the primary SRV lookup (order preserved). To implement failover, simply use the `first` load balancing policy which chooses the first available upstream. ### Configurable shutdown delay (#4906) A shutdown can now be scheduled for a later time using the `shutdown_delay` option. This is useful for giving advance notice to health checkers that this server will be closing soon. The shutdown delay happens _before_ the grace period where new connections are no longer accepted and existing ones are gracefully closed. During the shutdown delay, the server operates normally with the exception of the value of two placeholders. During the delay: - `{http.shutting_down}` placeholder equals `true`. - `{http.time_until_shutdown}` returns the duration that remains until server close. This allows health check endpoints to announce that they will soon be going down so that this instance can be moved out of the rotation or a replacement instance can be spun up in the meantime. For example: ``` { shutdown_delay 10s } example.com { handle /health-check { @goingDown `{http.shutting_down}` respond @goingDown "Bye-bye in {http.time_until_shutdown}" 503 respond 200 } } ``` By the way, the syntax of that `@goingDown` named matcher is new in 2.6: if a named matcher consists only of a CEL expression string, the type of matcher can be omitted; i.e. what you see above is equivalent to `@goingDown expression "{http.shutting_down} == true"`. (A shutdown is defined as a config unload where there is no new config to load, or the new config does not have a server configured at the same address as the current server. In other words, a shutdown of a server means a particular HTTP socket will be closed.) Speaking of grace periods, config changes no longer block while waiting on servers' grace periods. This means faster, more responsive config reloads; just beware that, depending on the length of your grace period, your reload command or config API request may return before the old servers have completely finished shutting down. ### Faster FastCGI transport (#4978) PHP apps, rejoice! The round-trip between Caddy and php-fpm just got a lot faster. Thanks to contributions by @WeidiDeng, the FastCGI transport has been rewritten to be more efficient. This is some of the oldest and most unique code in Caddy's code base. When Caddy was rewritten for v2 in 2019, _everything_ was rewritten or refactored... except this, the FastCGI transport. This is the first time this part of the code has been improved since it was first implemented[^3] in 2014! During tests, profiling showed the new code spends 86% less CPU time in GC (`gcDrain`) thanks to significantly fewer allocations. This is largely in part due to pooling buffers, which required a non-trivial refactoring to implement. ![CPU profile](https://user-images.githubusercontent.com/1128849/188224782-572c877d-42ea-4241-927f-346000512a75.png) A very rough benchmark using `php_info()` yielded a 25% increase in requests per second. Before the rewrite, Caddy almost always performed worse than nginx even with `fastcgi_keep_conn off`. Our new code performs competitively with nginx, and in some tests Caddy even outperformed nginx with `fastcgi_keep_conn on` -- and we have not implemented connection pooling/reuse into the new transport yet. Because every setup is different, your actual results will vary. In general though, you can expect busy servers to handle PHP faster. [^3]: I didn't know how to write a FastCGI client back then (I'm still too scared to do much with it myself); Go's standard library implements only the responder role, not the web server (client). Fortunately there was a [random repository on BitBucket](http://bitbucket.org/PinIdea/fcgi_client) that was forked from [a random repository on Google Code written in 2012](https://code.google.com/archive/p/go-fastcgi-client/) that modified the Go std lib's `fcgi` package. It was rough around the edges, but with a little TLC we got it to do what we needed. The copyright had the name Junqing Tan in it, which we still retain in our source code to this day. ### Faster file server (#5022) In a patch contributed by @flga, we've reduced copying between buffers and even eliminated it altogether in some cases using [`sendfile(2)`](https://man7.org/linux/man-pages/man2/sendfile.2.html). This has shown to have a 25-50% performance boost. It's automatic and no configuration is required to benefit. In [some tests](https://blog.tjll.net/reverse-proxy-hot-dog-eating-contest-caddy-vs-nginx/), Caddy's new defaults are even faster than optimized nginx. Static files over 512 bytes being served over plaintext HTTP sockets may now be served directly by the Linux kernel, which is much faster than copying the file to user-space. Static files are faster over HTTPS, too. In addition to sendfile (which we can't[^4] use over TLS), we now utilize the `io.ReaderFrom` interface to reuse existing buffers and further reduce copying within user space. Our tests show that this significantly enhances performance even over TLS. [^4]: This is possible with kTLS, but [the Go standard library doesn't support it](https://github.com/golang/go/issues/44506) and it's [a bit tedious](https://words.filippo.io/playing-with-kernel-tls-in-linux-4-13-and-go/) to make it work, although @FiloSottile was successful with his [spike code](https://github.com/FiloSottile/go/commit/dbed9972d9947eb0001e9f5b639e0df05acec8bd). ### Signed release assets Thanks to heroic efforts by @mohammed90, our [GitHub release](https://github.com/caddyserver/caddy/releases) assets are now signed and certified. Mohammed [wrote an excellent Twitter thread](https://twitter.com/MohammedSahaf/status/1572022375247663105) explaining the whole thing better than I can here! So if you're wondering why the number of assets shot from 28 to 134... that's why. Other notable enhancements -------------------------- - More efficient `query` matcher. (04a14ee37ac6192d734518fa9082d6eb93971bc6) - A new Caddyfile placeholder `{cookie.*}` grants easy access to cookie values. (#5001) - Windows service integration: Caddy can now be controlled with `sc.exe`. (#4790) - Replace `net.IP` type with leaner `netip.Addr` type. (#4966) - Caddyfile-configurable OCSP check interval with `ocsp_interval` global option. (#4980) - The reverse proxy now supports `retry_count` as an alternative to `try_duration`; i.e. try backends up to a fixed number of times, rather than up to a time limit. (#4756) - The reverse proxy closes both ends of "hijacked" connections when shutting down or reloading. (#4895) - The reverse proxy gracefully closes both ends of websocket connections on shutdown or reload. (#4895) - The reverse proxy emits metrics regarding the health of upstreams. (#4935) - The reverse-proxy command can accept repeated --to flags and load balance. (#4693) - The reverse proxy's HTTP transport now supports distinct read and write timeouts. (#4905) - Simpler and more reliable config reloads on Linux with SO_REUSEPORT. (#4705) - Templates can access reverse proxy responses if used within `handle_response`. (#4871) - Builds now include git revision information when using `go build`. (#4931) - The file matcher (and `try_files`) now supports glob patterns. (#4993) - Named matchers in the Caddyfile can use CEL expressions without specifying `expression` first. (#4976) - The FastCGI transport can now capture and print stderr output. (#5004) - Listeners can be provided by plugins, enabling new network types. (#5002) - Caddy can write TLS secrets to a file for debugging purposes. (#4808) - Sites declared as `http://` in the Caddyfile will no longer be overridden by auto-HTTPS redirects. (#5051) - Config reloads no longer block while the prior servers are shutting down. (#5043) ## :warning: Deprecations/breaks - **Metrics are now opt-in.** Due to [multiple confirmed reports](https://github.com/caddyserver/caddy/issues/4644) of non-trivial performance regressions with metrics, we are making them opt-in. (Technically, this is not a breaking change, as Caddy will still function normally and your old configs won't be rejected -- but your metrics will stop being produced unless you enable them.) If you rely on metrics, you can enable them globally in the Caddyfile with global options: ``` { servers { metrics } } ``` As with other server-scoped global options, you can selectively customize which servers to enable metrics (e.g. `servers :8080`). _Note that this change is experimental and might be temporary: if we can reduce the performance impact or find a better way to enable and configure metrics, this could change._ - The signature of `caddy.Context.Logger()` has changed, but in a backwards-compatible way. Modules use this function to obtain a logger they can use within Caddy; until now, modules had to pass themselves in as an argument. Now, the context can figure out which module to associate the logger with, so the sole parameter has been made variadic. It may be removed in the future. Plugins should update their code to not pass in a pointer to themselves. - Basic auth deprecates `scrypt` because it was seldom used and error-prone; use `bcrypt` instead (#4720) - Several changes to experimental `servers` global options: removed the `protocol` sub-option, which has been replaced with the `protocols` sub-option; `strict_sni_host` is its own separate sub-option; `allow_h2c` and `experimental_http3` have been removed, as both H2C (`h2c`) and HTTP/3 (`h3`) can be toggled in `protocols` (HTTP/3 is now enabled by default and no longer experimental). As a reminder, features, parameters, and APIs marked as experimental are subject to change or removal. We strive to keep breaking changes of stable features to a minimum and gracefully deprecate whenever possible with emphasis in release notes, warnings in logs, etc. Most breaking changes are motivated or necessitated by bugs/regressions, security, or wrong/unclear documentation. Thank you --------- As usual, a huge thank-you to all our [sponsors](https://github.com/sponsors/mholt) and those who contributed both code and feedback. We also acknowledge the many people who participated in discussions and helped others on the forum. Thank you! ## New Contributors * @WingLim made their first contribution in https://github.com/caddyserver/caddy/pull/4790 * @WilczynskiT made their first contribution in https://github.com/caddyserver/caddy/pull/4930 * @chir4gm made their first contribution in https://github.com/caddyserver/caddy/pull/4932 * @lewandowski-stripe made their first contribution in https://github.com/caddyserver/caddy/pull/4937 * @abdusco made their first contribution in https://github.com/caddyserver/caddy/pull/4963 * @jedy made their first contribution in https://github.com/caddyserver/caddy/pull/4975 * @benburkert made their first contribution in https://github.com/caddyserver/caddy/pull/4980 * @WeidiDeng made their first contribution in https://github.com/caddyserver/caddy/pull/4964 * @david-szabo97 made their first contribution in https://github.com/caddyserver/caddy/pull/4935 * @Abirdcfly made their first contribution in https://github.com/caddyserver/caddy/pull/4986 * @flga made their first contribution in https://github.com/caddyserver/caddy/pull/5004 * @Malankar made their first contribution in https://github.com/caddyserver/caddy/pull/4972 * @stapelberg made their first contribution in https://github.com/caddyserver/caddy/pull/4950 * @parrotmac made their first contribution in https://github.com/caddyserver/caddy/pull/4693 * @Manouchehri made their first contribution in https://github.com/caddyserver/caddy/pull/4808 **Full Changelog**: https://github.com/caddyserver/caddy/compare/v2.5.2...v2.6.0 --- 2022-09-20T18:12:38+00:00 rocksdb v7.6.0 rocksdb v7.6.0 2022-09-20T19:20:37+00:00 ### New Features * Added `prepopulate_blob_cache` to ColumnFamilyOptions. If enabled, prepopulate warm/hot blobs which are already in memory into blob cache at the time of flush. On a flush, the blob that is in memory (in memtables) get flushed to the device. If using Direct IO, additional IO is incurred to read this blob back into memory again, which is avoided by enabling this option. This further helps if the workload exhibits high temporal locality, where most of the reads go to recently written data. This also helps in case of the remote file system since it involves network traffic and higher latencies. * Support using secondary cache with the blob cache. When creating a blob cache, the user can set a secondary blob cache by configuring `secondary_cache` in LRUCacheOptions. * Charge memory usage of blob cache when the backing cache of the blob cache and the block cache are different. If an operation reserving memory for blob cache exceeds the avaible space left in the block cache at some point (i.e, causing a cache full under `LRUCacheOptions::strict_capacity_limit` = true), creation will fail with `Status::MemoryLimit()`. To opt in this feature, enable charging `CacheEntryRole::kBlobCache` in `BlockBasedTableOptions::cache_usage_options`. * Improve subcompaction range partition so that it is likely to be more even. More evenly distribution of subcompaction will improve compaction throughput for some workloads. All input files' index blocks to sample some anchor key points from which we pick positions to partition the input range. This would introduce some CPU overhead in compaction preparation phase, if subcompaction is enabled, but it should be a small fraction of the CPU usage of the whole compaction process. This also brings a behavier change: subcompaction number is much more likely to maxed out than before. * Add CompactionPri::kRoundRobin, a compaction picking mode that cycles through all the files with a compact cursor in a round-robin manner. This feature is available since 7.5. * Provide support for subcompactions for user_defined_timestamp. * Added an option `memtable_protection_bytes_per_key` that turns on memtable per key-value checksum protection. Each memtable entry will be suffixed by a checksum that is computed during writes, and verified in reads/compaction. Detected corruption will be logged and with corruption status returned to user. * Added a blob-specific cache priority level - bottom level. Blobs are typically lower-value targets for caching than data blocks, since 1) with BlobDB, data blocks containing blob references conceptually form an index structure which has to be consulted before we can read the blob value, and 2) cached blobs represent only a single key-value, while cached data blocks generally contain multiple KVs. The user can specify the new option `low_pri_pool_ratio` in `LRUCacheOptions` to configure the ratio of capacity reserved for low priority cache entries (and therefore the remaining ratio is the space reserved for the bottom level), or configuring the new argument `low_pri_pool_ratio` in `NewLRUCache()` to achieve the same effect. ### Public API changes * Removed Customizable support for RateLimiter and removed its CreateFromString() and Type() functions. * `CompactRangeOptions::exclusive_manual_compaction` is now false by default. This ensures RocksDB does not introduce artificial parallelism limitations by default. * Tiered Storage: change `bottommost_temperture` to `last_level_temperture`. The old option name is kept only for migration, please use the new option. The behavior is changed to apply temperature for the `last_level` SST files only. * Added a new experimental ReadOption flag called optimize_multiget_for_io, which when set attempts to reduce MultiGet latency by spawning coroutines for keys in multiple levels. ### Bug Fixes * Fix a bug starting in 7.4.0 in which some fsync operations might be skipped in a DB after any DropColumnFamily on that DB, until it is re-opened. This can lead to data loss on power loss. (For custom FileSystem implementations, this could lead to `FSDirectory::Fsync` or `FSDirectory::Close` after the first `FSDirectory::Close`; Also, valgrind could report call to `close()` with `fd=-1`.) * Fix a bug where `GenericRateLimiter` could revert the bandwidth set dynamically using `SetBytesPerSecond()` when a user configures a structure enclosing it, e.g., using `GetOptionsFromString()` to configure an `Options` that references an existing `RateLimiter` object. * Fix race conditions in `GenericRateLimiter`. * Fix a bug in `FIFOCompactionPicker::PickTTLCompaction` where total_size calculating might cause underflow * Fix data race bug in hash linked list memtable. With this bug, read request might temporarily miss an old record in the memtable in a race condition to the hash bucket. * Fix a bug that `best_efforts_recovery` may fail to open the db with mmap read. * Fixed a bug where blobs read during compaction would pollute the cache. * Fixed a data race in LRUCache when used with a secondary_cache. * Fixed a bug where blobs read by iterators would be inserted into the cache even with the `fill_cache` read option set to false. * Fixed the segfault caused by `AllocateData()` in `CompressedSecondaryCache::SplitValueIntoChunks()` and `MergeChunksIntoValueTest`. * Fixed a bug in BlobDB where a mix of inlined and blob values could result in an incorrect value being passed to the compaction filter (see #10391). * Fixed a memory leak bug in stress tests caused by `FaultInjectionSecondaryCache`. ### Behavior Change * Added checksum handshake during the copying of decompressed WAL fragment. This together with #9875, #10037, #10212, #10114 and #10319 provides end-to-end integrity protection for write batch during recovery. * To minimize the internal fragmentation caused by the variable size of the compressed blocks in `CompressedSecondaryCache`, the original block is split according to the jemalloc bin size in `Insert()` and then merged back in `Lookup()`. * PosixLogger is removed and by default EnvLogger will be used for info logging. The behavior of the two loggers should be very similar when using the default Posix Env. * Remove [min|max]_timestamp from VersionEdit for now since they are not tracked in MANIFEST anyway but consume two empty std::string (up to 64 bytes) for each file. Should they be added back in the future, we should store them more compactly. * Improve universal tiered storage compaction picker to avoid extra major compaction triggered by size amplification. If `preclude_last_level_data_seconds` is enabled, the size amplification is calculated within non last_level data only which skip the last level and use the penultimate level as the size base. * If an error is hit when writing to a file (append, sync, etc), RocksDB is more strict with not issuing more operations to it, except closing the file, with exceptions of some WAL file operations in error recovery path. * A `WriteBufferManager` constructed with `allow_stall == false` will no longer trigger write stall implicitly by thrashing until memtable count limit is reached. Instead, a column family can continue accumulating writes while that CF is flushing, which means memory may increase. Users who prefer stalling writes must now explicitly set `allow_stall == true`. * Add `CompressedSecondaryCache` into the stress tests. * Block cache keys have changed, which will cause any persistent caches to miss between versions. ### Performance Improvements * Instead of constructing `FragmentedRangeTombstoneList` during every read operation, it is now constructed once and stored in immutable memtables. This improves speed of querying range tombstones from immutable memtables. * When using iterators with the integrated BlobDB implementation, blob cache handles are now released immediately when the iterator's position changes. * MultiGet can now do more IO in parallel by reading data blocks from SST files in multiple levels, if the optimize_multiget_for_io ReadOption flag is set. 2022-09-20T19:20:37+00:00 caddy v2.6.1 caddy v2.6.1 2022-09-21T22:21:28+00:00 Hotfix for unix sockets, the `encode` handler, and the `caddy file-server` command. Please see [the release notes for v2.6.0](https://github.com/caddyserver/caddy/releases/tag/v2.6.0) for other important information if you're coming from < 2.6! ## Changelog * 1426c97d core: Reuse unix sockets (UDS) and don't try to serve HTTP/3 over UDS (#5063) * 44ad0ced encode: don't WriteHeader unless called (#5060) * beb7dcbf fileserver: Reinstate --debug flag 2022-09-21T22:21:28+00:00 MISP v2.4.163 MISP v2.4.163 2022-09-26T14:12:21+00:00 ![](https://www.misp-project.org/img/blog/periodic-summary/periodic-summary-2.png ) We are pleased to announce the immediate availability of [MISP v2.4.163](https://github.com/MISP/MISP/releases/tag/v2.4.163) with an updated [periodic notification system](https://www.misp-project.org/2022/09/12/2022-09-12_periodic_notifications.html/) and many improvements. # Updated periodic notification system - A new option has been added to set the number of days for the trending calculation. - New correlation are now showed in the periodic notification. - Only the top 10 MITRE ATT&CK techniques are displayed and sorted by number of occurrences. - Layout has been improved in the UI and also in the static email rendering. - Only show data in the chart for tags having changes over time. For more information, check out the [Periodic summaries - Visualize summaries of MISP data](https://www.misp-project.org/2022/09/12/2022-09-12_periodic_notifications.html/) blog. # Fixes - MISP [OpenAPI description file](https://www.misp-project.org/openapi/) has been improved. - [community] Clarification concerning the NATO process. - [ssdeep] Check if the ssdeep contains newline characters. - Many code clean-up and speed-up included. - Improvements and bugs fixed in the correlation engine. - Many bugs fixed. Thanks to all the contributors and users reporting bugs to make the software better. As always, a detailed and [complete changelog is available](https://www.misp-project.org/Changelog.txt) with all the fixes, changes and improvements in MISP core. # misp-stix v.2.4.163 misp-stix has been released too and now in-line with the MISP release schedule. The full [changelog is available](https://www.misp-project.org/Changelog-misp-stix.txt). Many improvements in the MISP galaxy and especially the threat-actor galaxy, [360.net Threat Actors](https://www.misp-project.org/galaxy.html#_360_net_threat_actors) added. There is a detailed [changelog](https://www.misp-project.org/Changelog-misp-galaxy.txt). New `financial` taxonomy and many other taxonomies. There is a detailed [changelog](https://www.misp-project.org/Changelog-misp-taxonomies.txt). Multiple objects were updated and added, for [more details](https://www.misp-project.org/Changelog-misp-objects.txt). Various fixes in [misp-modules](https://misp.github.io/misp-modules/) for more [details](https://www.misp-project.org/Changelog-misp-modules.txt). 2022-09-26T14:12:21+00:00 dalton v3.2.5 dalton v3.2.5 2022-09-27T23:51:25+00:00 - Faster compile time for Zeek - Fixed bug where Zeek wouldn't run if there were not Zeek scripts 2022-09-27T23:51:25+00:00 maltrail 0.50 maltrail 0.50 2022-09-30T22:11:05+00:00 Start-of-month release 2022-09-30T22:11:05+00:00 MONARC v2.12.3 MONARC v2.12.3 2022-10-05T15:18:41+00:00 ### New features: * Link multiple specific models per client. * Added the possibility to import assets in the library from CSV files. 2022-10-05T15:18:41+00:00 rocksdb v7.7.2 rocksdb v7.7.2 2022-10-07T16:28:46+00:00 ## 7.7.2 (10/05/2022) ### Bug Fixes * Fixed a bug in iterator refresh that was not freeing up SuperVersion, which could cause excessive resource pinniung (#10770). * Fixed a bug where RocksDB could be doing compaction endlessly when allow_ingest_behind is true and the bottommost level is not filled (#10767). ### Behavior Changes * Sanitize min_write_buffer_number_to_merge to 1 if atomic flush is enabled to prevent unexpected data loss when WAL is disabled in a multi-column-family setting (#10773). ## 7.7.1 (09/26/2022) ### Bug Fixes * Fixed an optimistic transaction validation bug caused by DBImpl::GetLatestSequenceForKey() returning non-latest seq for merge (#10724). * Fixed a bug in iterator refresh which could segfault for DeleteRange users (#10739). ## 7.7.0 (09/18/2022) ### Bug Fixes * Fixed a hang when an operation such as `GetLiveFiles` or `CreateNewBackup` is asked to trigger and wait for memtable flush on a read-only DB. Such indirect requests for memtable flush are now ignored on a read-only DB. * Fixed bug where `FlushWAL(true /* sync */)` (used by `GetLiveFilesStorageInfo()`, which is used by checkpoint and backup) could cause parallel writes at the tail of a WAL file to never be synced. * Fix periodic_task unable to re-register the same task type, which may cause `SetOptions()` fail to update periodical_task time like: `stats_dump_period_sec`, `stats_persist_period_sec`. * Fixed a bug in the rocksdb.prefetched.bytes.discarded stat. It was counting the prefetch buffer size, rather than the actual number of bytes discarded from the buffer. * Fix bug where the directory containing CURRENT can left unsynced after CURRENT is updated to point to the latest MANIFEST, which leads to risk of unsync data loss of CURRENT. * Update rocksdb.multiget.io.batch.size stat in non-async MultiGet as well. * Fix a bug in key range overlap checking with concurrent compactions when user-defined timestamp is enabled. User-defined timestamps should be EXCLUDED when checking if two ranges overlap. * Fixed a bug where the blob cache prepopulating logic did not consider the secondary cache (see #10603). * Fixed the rocksdb.num.sst.read.per.level, rocksdb.num.index.and.filter.blocks.read.per.level and rocksdb.num.level.read.per.multiget stats in the MultiGet coroutines * Fix a bug in io_uring_prep_cancel in AbortIO API for posix which expects sqe->addr to match with read request submitted and wrong paramter was being passed. * Fixed a regression in iterator performance when the entire DB is a single memtable introduced in #10449. The fix is in #10705 and #10716. * Fix a bug in io_uring_prep_cancel in AbortIO API for posix which expects sqe->addr to match with read request submitted and wrong paramter was being passed. * Fixed a regression in iterator performance when the entire DB is a single memtable introduced in #10449. The fix is in #10705 and #10716. ### Public API changes * Add `rocksdb_column_family_handle_get_id`, `rocksdb_column_family_handle_get_name` to get name, id of column family in C API * Add a new stat rocksdb.async.prefetch.abort.micros to measure time spent waiting for async prefetch reads to abort ### Java API Changes * Add CompactionPriority.RoundRobin. * Revert to using the default metadata charge policy when creating an LRU cache via the Java API. ### Behavior Change * DBOptions::verify_sst_unique_id_in_manifest is now an on-by-default feature that verifies SST file identity whenever they are opened by a DB, rather than only at DB::Open time. * Right now, when the option migration tool (OptionChangeMigration()) migrates to FIFO compaction, it compacts all the data into one single SST file and move to L0. This might create a problem for some users: the giant file may be soon deleted to satisfy max_table_files_size, and might cayse the DB to be almost empty. We change the behavior so that the files are cut to be smaller, but these files might not follow the data insertion order. With the change, after the migration, migrated data might not be dropped by insertion order by FIFO compaction. * When a block is firstly found from `CompressedSecondaryCache`, we just insert a dummy block into the primary cache and don’t erase the block from `CompressedSecondaryCache`. A standalone handle is returned to the caller. Only if the block is found again from `CompressedSecondaryCache` before the dummy block is evicted, we erase the block from `CompressedSecondaryCache` and insert it into the primary cache. * When a block is firstly evicted from the primary cache to `CompressedSecondaryCache`, we just insert a dummy block in `CompressedSecondaryCache`. Only if it is evicted again before the dummy block is evicted from the cache, it is treated as a hot block and is inserted into `CompressedSecondaryCache`. * Improved the estimation of memory used by cached blobs by taking into account the size of the object owning the blob value and also the allocator overhead if `malloc_usable_size` is available (see #10583). * Blob values now have their own category in the cache occupancy statistics, as opposed to being lumped into the "Misc" bucket (see #10601). * Change the optimize_multiget_for_io experimental ReadOptions flag to default on. ### New Features * RocksDB does internal auto prefetching if it notices 2 sequential reads if readahead_size is not specified. New option `num_file_reads_for_auto_readahead` is added in BlockBasedTableOptions which indicates after how many sequential reads internal auto prefetching should be start (default is 2). * Added new perf context counters `block_cache_standalone_handle_count`, `block_cache_real_handle_count`,`compressed_sec_cache_insert_real_count`, `compressed_sec_cache_insert_dummy_count`, `compressed_sec_cache_uncompressed_bytes`, and `compressed_sec_cache_compressed_bytes`. * Memory for blobs which are to be inserted into the blob cache is now allocated using the cache's allocator (see #10628 and #10647). * HyperClockCache is an experimental, lock-free Cache alternative for block cache that offers much improved CPU efficiency under high parallel load or high contention, with some caveats. As much as 4.5x higher ops/sec vs. LRUCache has been seen in db_bench under high parallel load. * `CompressedSecondaryCacheOptions::enable_custom_split_merge` is added for enabling the custom split and merge feature, which split the compressed value into chunks so that they may better fit jemalloc bins. ### Performance Improvements * Iterator performance is improved for `DeleteRange()` users. Internally, iterator will skip to the end of a range tombstone when possible, instead of looping through each key and check individually if a key is range deleted. * Eliminated some allocations and copies in the blob read path. Also, `PinnableSlice` now only points to the blob value and pins the backing resource (cache entry or buffer) in all cases, instead of containing a copy of the blob value. See #10625 and #10647. * In case of scans with async_io enabled, few optimizations have been added to issue more asynchronous requests in parallel in order to avoid synchronous prefetching. * `DeleteRange()` users should see improvement in get/iterator performance from mutable memtable (see #10547). 2022-10-07T16:28:46+00:00 dnstwist 20221008 dnstwist 20221008 2022-10-08T16:33:43+00:00 2022-10-08T16:33:43+00:00 MISP v2.4.164 MISP v2.4.164 2022-10-10T14:45:54+00:00 ![](https://www.misp-project.org/img/blog/periodic-summary/periodic-summary-2.png) We are pleased to announce the immediate availability of [MISP v2.4.164](https://github.com/MISP/MISP/releases/tag/v2.4.164) with a new tag relationship features, many improvements and a security fix. # New tag relationship feature Relationship can now be added to any attribute tag or event tag. This works with tags and galaxy clusters. The new feature is available in event view. The tag relationship feature is also exposed in the API under the endpoint `/tags/modifyTagRelationship/[scope]/[id]` where `scope` is the attribute/event and `id` is the id of the EventTag / AttributeTag object. # Improvements and bug fixes - [periodic_report] Added security recommendations section showing course of actions related to attack techniques. - [workflow] add support for `local` and `relationship` in workflow. - [API/galaxyCluster/restSearch] Allow multiple filtering conditions to be used at once. - [EventGraph] Added entity comment in the graph as tooltip and support of comment in searches. - [UI] Many improvements and optimisation. # CVE-2022-42724 This release fix a security vulnerabilities ([CVE-2022-42724](https://cvepremium.circl.lu/cve/CVE-2022-42724)) which allows org admin to discover role names which should have been restricted to site admin. We strongly recommend MISP administrator to update to this latest version. For a more detailed changelog, please see the online [Changelog](https://www.misp-project.org/Changelog.txt). 2022-10-10T14:45:54+00:00 osquery 5.6.0 osquery 5.6.0 2022-10-10T16:57:28+00:00 Draft! 2022-10-10T16:57:28+00:00 dnstwist 20221011 dnstwist 20221011 2022-10-11T10:22:48+00:00 2022-10-11T10:22:48+00:00 OpenTAXII 0.9.3 OpenTAXII 0.9.3 2022-10-11T13:46:48+00:00 Changelog ========= 0.9.3 (2022-10-11) ------------------ * Add public write support. 0.9.2 (2022-08-26) ------------------ * Improve readability and navigation of docs (`#238 <https://github.com/eclecticiq/OpenTAXII/pull/238>`_ thanks `@zed-eiq <https://github.com/zed-eiq>`_ for the improvement). 0.9.1 (2022-07-11) ------------------ * Implement `raise_unauthorized` for taxii2, this was missing and lead to 500 errors. 0.9.0 (2022-06-13) ------------------ * Allow custom properties. This can be disabled by config option ``allow_custom_properties`` 0.8.0 (2022-06-05) ------------------ * Nest details inside taxii2 job and allow counts without details 0.7.0 (2022-05-27) ------------------ * Nest taxii2 endpoints under `/taxii2/` 0.6.0 (2022-05-25) ------------------ * Add `public_discovery` option to taxii2 config * Add support for publicly readable taxii 2 api roots 0.5.0 (2022-05-24) ------------------ * Add support for publicly readable taxii 2 collections 0.4.0 (2022-05-20) ------------------ * Move next_param handling into `OpenTAXII2PersistenceAPI` 0.3.0 (2022-04-13) ------------------ * Implement taxii2.1 support 0.3.0a4 (2022-04-13) -------------------- * Merge changes from 0.2.4 maintenance release 0.3.0a3 (2022-01-21) -------------------- * Fix bug that prevented booting with only taxii1 config (`#217 <https://github.com/eclecticiq/OpenTAXII/issues/217>`_ thanks `@azurekid <https://github.com/azurekid>`_ for the report) 0.3.0a2 (2021-12-27) -------------------- * Merge changes from 0.2.3 maintenance release 0.3.0a1 ------- * Add python 3.10 support 0.3.0a0 ------- * Enablement for future taxii2 implementation * Fix documentation build issues 0.2.4 (2022-04-13) ------------------ * Make sure werkzeug <2.1 and >=2.1 work correctly with auth system 0.2.3 (2021-12-22) ------------------ * Fix bug in multithreaded use of sqlite (`#210 <https://github.com/eclecticiq/OpenTAXII/issues/210>`_ thanks `@rohits144 <https://github.com/rohits144>`_ for the report) 0.2.2 (2021-11-05) ------------------ * Fix readthedocs build 0.2.1 (2021-11-03) ------------------ * Add tests for python 3.6, 3.7, 3.8, 3.9, pypy * Add tests for sqlite, mysql, mariadb, postgresql * Fix bug that broke ``delete_content_blocks`` when using mysql on sqlalchemy 1.3 * Docs: Add db schema diagram * Docs: Clarify how to get default data in a default (development) docker instance * Fix implicit routing in TAXII 1.1 Inboxes * Update jwt usage to pyjwt >= 2.0 (thanks `@SanyaKapoor <https://github.com/SanyaKapoor>`_) 0.2.0 (2020-06-30) ------------------ * Enforce UTC usage in datetime fields in SQL DB Persistence API. * `Fix for #114 <https://github.com/eclecticiq/OpenTAXII/issues/114>`_: reintroduce ``opentaxii-create-account`` CLI command. * `Fix for #153 <https://github.com/eclecticiq/OpenTAXII/issues/152>`_: check if user can modify a collection before advertising it over inbox service. * Multiple coding style fixes. * Various documentation updates. 0.1.12 (2019-03-06) ------------------- * Remove unnecessary print statements. 0.1.11 (2019-02-13) ------------------- * Make JSON logging consistent when the application is run via Gunicorn. * Set ``acceptable_destination`` key in status details instead of extended headers * Allow passing ``engine_parameters`` to ``SQLDatabaseAPI`` for those who want to customize SQLAlchemy engine parameters. * Require recent version of ``lxml`` for security reasons. * Various test and Docker infrastructure improvements. 0.1.10 (2018-06-03) ------------------- * Replace separate service/collection/account creation process with single ``opentaxii-sync-data`` CLI command. * Persistence and Auth APIs extended with missing CRUD methods, that are used by ``opentaxii-sync-data``. * Read/modify collection level ACL added. * DB models for default implementation of Persistence API and Auth API were changed. No automatic migration code is provided (sorry!), so upgrading might require manual DB migration. * Drop python2.7 from testing scope. * Various bug fixes and improvements. 0.1.9 (2017-06-19) ------------------ * `libtaxii <https://github.com/TAXIIProject/libtaxii>`_ dependency upgraded to 1.1.111. * Various bug fixes and improvements (thanks to `@bjigmp <https://github.com/bjigmp>`_, `@chorsley <https://github.com/chorsley>`_, `@rjprins <https://github.com/rjprins>`_). 0.1.8 (2017-02-21) ------------------ * Ability to enable/disable "huge trees" support in XML parser. Configuration property ``xml_parser_supports_huge_tree`` set to ``yes`` or ``true`` will disable security restrictions and force XML parser to support very deep trees and very long text content. * Adjust SQL Persistence API implemetation so it works smoothly with MySQL backend. * Use Python 3.5 instead of Python 3.4 for tests. 0.1.7 (2016-10-18) ------------------ * Minor fixes. * Dependencies were changed from hard-pinned to more flexible. * Example of production DB configuration added to docs. 0.1.6 (2016-06-01) ------------------ * Python 3.4 compatibility of the codebase. Tox configuration extended with python 3.4 environment run. * Flake8 full style compatibility. Flake8 check added to Tox configuration. * SQLAlchemy session scope issue fixed (related to `#38 <https://github.com/EclecticIQ/OpenTAXII/issues/38>`_). * `opentaxii-delete-blocks` CLI command added (related to `#45 <https://github.com/EclecticIQ/OpenTAXII/issues/45>`_). * `delete_content_blocks` method `added <https://github.com/EclecticIQ/OpenTAXII/commit/dc6fddc27a98e8450c7e05e583b2bfb741f6e197#diff-6814849ac352b2b74132f8fa52e0ec4eR213>`_ to Persistence API. * Collection's name is `required <https://github.com/EclecticIQ/OpenTAXII/commit/dc6fddc27a98e8450c7e05e583b2bfb741f6e197#diff-ce3f7b939e5c540480ac655aef32c513R116>`_ to be unique in default SQL DB Auth API implementation. 0.1.5 (2016-03-15) ------------------ * Fix for the issue with persistence backend returning ``None`` instead of ``InboxMessage`` object 0.1.4 (2016-02-25) ------------------ * Hard-coded dependencies in ``setup.py`` removed. 0.1.3 (2016-02-25) ------------------ * Versions of dependencies are pinned. * Code adjusted for a new version of `anyconfig <https://pypi.python.org/pypi/anyconfig>`_ API. * Test for configuration loading added. 0.1.2 (2015-07-24) ------------------ * Docker configuration added. * Health check endpoint added. * Basic authentication support added. * Temporary workaround for `Issue #191 <https://github.com/TAXIIProject/libtaxii/issues/191>`_. * Method ``get_domain`` in Persistence API returns domain value configured for ``service_id``. If nothing returned, value set in configuration file will be used. * Performance optimisations. * Bug fixes and style improvements. 0.1.1 (2015-04-08) ------------------ * Alias for Root Logger added to logging configuration. * Context object in a request scope that holds account and token added. * Support for OPTIONS HTTP request to enable auto version negotiation added. * Documentation improved. 0.1.0 (2015-03-31) ------------------ * Initial release 2022-10-11T13:46:48+00:00 MONARC v2.12.3-p1 MONARC v2.12.3-p1 2022-10-12T11:07:12+00:00 [FrontOffice] Fixed the analysis creation issue. 2022-10-12T11:07:12+00:00 rocksdb v7.7.3 rocksdb v7.7.3 2022-10-12T21:58:10+00:00 ## 7.7.3 (10/11/2022) ### Bug Fixes * Fixed a memory safety bug in experimental HyperClockCache (#10768) 2022-10-12T21:58:10+00:00 MONARC v2.12.3-p2 MONARC v2.12.3-p2 2022-10-13T14:25:34+00:00 [FrontOffice] Fixed the object categories language validation. 2022-10-13T14:25:34+00:00 caddy v2.6.2 caddy v2.6.2 2022-10-13T18:44:05+00:00 This release brings a number of bug fixes and minor enhancements. All users should upgrade after testing and verifying their setups. Thank you to all who contributed! If you are coming from < 2.6, please see [the 2.6 release notes](https://github.com/caddyserver/caddy/releases/tag/v2.6.0) because a lot is new! ## Changelog * 037dc23c admin: Use replacer on listen addresses (#5071) * 498f32ba caddyconfig: Implement retries into HTTPLoader (#5077) * 9873ff99 caddyhttp: Remote IP prefix placeholders * 61822f12 caddyhttp: replace placeholders in map defaults (#5081) * e07a2672 caddytest: Revise sleep durations * 253d97c9 core: Chdir to executable location on Windows (#5115) * ab720fb7 core: Fix ListenQUIC listener key conflict * e3e8aabb core: Refactor and improve listener logic (#5089) * e4fac129 core: Set version manually via CustomVersion (#5072) * f7c1a51e fastcgi: Redirect using original URI path (fix #5073) * 2be56c52 fileserver: Treat invalid file path as NotFound (#5099) * b1d04f5b fileserver: better dark mode visited link contrast (#5105) * 33f60da9 fileserver: stop listing dir when request context is cancelled (#5131) * 2153a81e forwardauth: Canonicalize header fields (fix #5038) (#5097) * fe91de67 go.mod: Upgrade select dependencies * 70419700 headers: Support repeated WriteHeader if 1xx (fix #5074) * d46ba2e2 httpcaddyfile: Fix `metrics` global option parsing (#5126) * 6bad878a httpcaddyfile: Improve detection of indistinguishable TLS automation policies (#5120) * 2808de1e httpcaddyfile: Skip `automate` when `auto_https off` is specified (#5110) * 3e1fd2a8 httpcaddyfile: Wrap site block in subroute if host matcher used (#5130) * 9e1d964b logging: Add `time_local` option to use local time instead of UTC (#5108) * 01e192ed logging: Better `console` encoder defaults (#5109) * 99ffe933 logging: Fix `skip_hosts` with wildcards (#5102) * ea58d519 logging: Perform filtering on arrays of strings (where possible) (#5101) * 5e52bbb1 map: Remove infinite recursion check (#5094) * b4e28af9 replacer: working directory global placeholder (#5127) * e2991eb0 reverseproxy: On 103 don't delete own headers (#5091) * 2a8c458f reverseproxy: Parse humanized byte size (fix #5095) * d0556929 reverseproxy: fix upstream scheme handling in command (#5088) * 013b5103 rewrite: Only trim prefix if matched ## New Contributors * @lemmi made their first contribution in https://github.com/caddyserver/caddy/pull/5088 * @willnorris made their first contribution in https://github.com/caddyserver/caddy/pull/5081 * @yroc92 made their first contribution in https://github.com/caddyserver/caddy/pull/5071 * @iliana made their first contribution in https://github.com/caddyserver/caddy/pull/5105 * @TobiX made their first contribution in https://github.com/caddyserver/caddy/pull/5106 * @likev made their first contribution in https://github.com/caddyserver/caddy/pull/5099 * @cherouvim made their first contribution in https://github.com/caddyserver/caddy/pull/5121 **Full Changelog**: https://github.com/caddyserver/caddy/compare/v2.6.1...v2.6.2 2022-10-13T18:44:05+00:00 MONARC v2.12.3-p3 MONARC v2.12.3-p3 2022-10-20T11:40:40+00:00 * [Fixed the import error of the assets, threats or vulnerabilities codes duplication.](https://github.com/monarc-project/zm-client/commit/e96a3e5a823eb9b68d4bf4ec6a74de7c77f18d09) 2022-10-20T11:40:40+00:00 dnstwist 20221022 dnstwist 20221022 2022-10-22T12:53:46+00:00 2022-10-22T12:53:46+00:00 Lookyloo v1.16.0 Lookyloo v1.16.0 2022-10-29T13:19:39+00:00 # Breaking change This release requires poetry v1.2.0 or more recent. Run the following command to upgrade it: `poetry self update` # New Features * Move to Lacus/LacusCore, many changes to make lookyloo's code compatible with it * [Lacus](https://github.com/ail-project/lacus) and [PyLacus](https://github.com/ail-project/pylacus): use this mode to trigger the capture from an other machine than the one you run Lookyloo from * [LacusCore](https://github.com/ail-project/lacuscore): (the default) keep triggering the captures from the same machine as the one lookyloo is running on With lacus, the captures are more reliable and using Lacus as a web service allows to monitor them better. If you want to use the webservice, you'll need to: 1. [Install Lacus](https://github.com/ail-project/Lacus#install-guide) * make sure it is running by loading http://127.0.0.1:7100 on the machine you have it running on (7100 is the default port, you can of course change it) 2. Edit the config file `config/generic.json` (key `remote_lacus`): * set `enable` to `true` * set `url` to the url your loolyloo instance can use to connect to lacus: `http://<ip>:<port>` 3. Restart lookyloo & try it # Changes * Make hashlookup visible to everyone * Improve loggingv1.15.0 * Maintenance: use poetry 1.2, bump deps (Python and JS), bump Github actions * Improve caching with Lacus * Retry failing captures when it might be a temporary issue (typically domain resolution) # Fixes * Avoid triggering a capture (and failing) when the URL and documents are missing * Issue with urlscan when the capture had no referer * Better handling of exceptions in VT module * Better handling of devices offered by Playwright and their user-agents 2022-10-29T13:19:39+00:00 PyPCAPKit v0.16.3 PyPCAPKit v0.16.3 2022-10-31T22:16:01+00:00 New Distribution [0.16.3] * bugfix for #114 & #116 (hopefully) * bugfix for distro workflow 2022-10-31T22:16:01+00:00 maltrail 0.51 maltrail 0.51 2022-10-31T23:11:05+00:00 Start-of-month release 2022-10-31T23:11:05+00:00 MONARC v2.12.4 MONARC v2.12.4 2022-11-11T15:53:31+00:00 ### New features - Implemented new dashboard chart views and a curve line that represents the average number tendency. - Added a possibility to enforce all the instance's users to enable 2-Factor Authentication. 2022-11-11T15:53:31+00:00 MONARC v2.12.4-p1 MONARC v2.12.4-p1 2022-11-17T10:46:50+00:00 * Fixed the threats' codes duplication issue during the import. * Fixed the dashboard export with the new chart changes. 2022-11-17T10:46:50+00:00 Stegano v0.11.0 Stegano v0.11.0 2022-11-20T00:02:32+00:00 Reduced memory footprint and processing speed. The modules ``lsb`` and ``lsbset`` have been merged ([PR #34](https://github.com/cedricbonhomme/Stegano/pull/34)). 2022-11-20T00:02:32+00:00 Stegano v0.11.1 Stegano v0.11.1 2022-11-20T21:03:28+00:00 Fixed a bug in the command line when no sub-command is specified. 2022-11-20T21:03:28+00:00 MISP v2.4.165 MISP v2.4.165 2022-11-22T09:28:31+00:00 ![](https://www.misp-project.org/img/blog/workflow.png) We are pleased to announce the immediate availability of [MISP v2.4.165](https://github.com/MISP/MISP/releases/tag/v2.4.165) with many improvements to the workflow subsystem along with various performance improvements. # Improvements - [workflow] Module to toggle/remove the to_ids flag - [workflow] Added generic module to support attribute edition - [workflow] [triggers:event_after_save_new] Added 2 new triggers for new events and new events from pull. - [workflow:execute_module] Allow to ignore format conversion before executing module. - [workflows:triggers] Added filtering capability on the index - [CLI] Feed management added - [CLI] Pretty and JSON output added in list and view feeds - [Auth] OpenID connect improved - [freetext] Fetch security vendor domains from [warninglist](https://github.com/MISP/misp-warninglists) - [UI] Allow to disable PGP key fetching - [UI] Show warning if user don't have permission to use API - [tool:evengraph] Include relationships when using pivot key - [UI] Show servers where event will be pushed # Performance improvements - [feed] Store freetext feed compressed in cache - [internal] Store some data in Redis compressed to save memory - [correlation] Do not correlate over correlating value again for full correlation - [internal] Add support for [simdjson](https://github.com/simdjson/simdjson) extension - [warninglist] Load warninglist from Redis for TLDs and security vendors # Bugs fixed - [tags] not passing name, filter, search all together would lead to the search not working # Security issues - [security] Permission for tag collections - [security] Check user permission when attaching clusters We strongly recommend MISP administrators to update to this latest version. For a more detailed changelog, please see the online [Changelog](https://www.misp-project.org/Changelog.txt). # New workflow blueprints available New [workflow blueprints](https://github.com/MISP/misp-workflow-blueprints/) were added to support new use-cases. - [Attach `tlp:clear` on `tlp:white`](./blueprints/blueprint_attach-tlp_clear-on-tlp_white_1661328256.json) - Attach the `tlp:clear` tag on elements having the `tlp:white` tag. - [`PAP:RED` and `tlp:red` Blocking](./blueprints/blueprint_pap_red-and-tlp_red-blocking_1661328258.json) - Block actions if any attributes have the `PAP:RED` or `tlp:red` tag. - [Remote `to_ids` flag if the indicator appears in known file list](https://github.com/MISP/misp-workflow-blueprints/blob/main/blueprints/blueprint_disable-to_ids-flag-for-existing-hash-in-hashlookup_1667228944.json) - Disable to_ids flag for existing hash in [hashlookup](https://www.hashlookup.io/). - [Set tag based on BGP Ranking maliciousness level](https://github.com/MISP/misp-workflow-blueprints/blob/main/blueprints/blueprint_set-tag-based-on-bgp-ranking-maliciousness-level_1668498668.json) - Set tag based on [BGP Ranking](https://bgpranking.circl.lu) maliciousness level. # New MISP modules - [expansion] Added extract_url_components module to create an object from an URL attribute. - [expansion] New [crowdsec](https://www.crowdsec.net/) expansion module added. - [expansion] New [VARIoT IoT exploits database](https://www.variotdbs.pl/exploits/) expansion module added. - [expansion] Updates on hyasinsight expansion module. # MISP taxonomies - new misp-workflow taxonomy to have a consistent tag message for the MISP workflow. - Taxonomy in support of integrating MISP with Sentinel. Sentinel indicator threat types added. For more [details](https://www.misp-project.org/Changelog-misp-taxonomies.txt). # MISP galaxy - Many updates to the threat actor database. - Update to the MITRE ATT&CK framework to version 12.0. For more [details](https://www.misp-project.org/Changelog-misp-galaxy.txt). # MISP objects - New object to describe Telegram bots. - Updated exploit object. For more [details](https://www.misp-project.org/Changelog-misp-objects.txt). # Social network - Mastodon MISP project is also now reachable via Mastodon. Feel free to follow us at @misp@misp-community.org 2022-11-22T09:28:31+00:00 MONARC v2.12.4-p2 MONARC v2.12.4-p2 2022-11-24T15:49:39+00:00 ### Fix * Fixed the import issue when more than 1 theme exists with the same label. 2022-11-24T15:49:39+00:00 MISP v2.4.166 MISP v2.4.166 2022-11-30T17:21:47+00:00 ![Workflow screenshot](https://www.misp-project.org/img/blog/workflow.png) We are pleased to announce the immediate availability of [MISP v2.4.166](https://github.com/MISP/MISP/releases/tag/v2.4.166) with new features and fixes, including two critical security fixes. # TAXII 2.1 server push integration With the collaboration of CISA and MITRE, we have included the first version of the [TAXII](https://docs.oasis-open.org/cti/taxii/v2.1/taxii-v2.1.html) integration in MISP, allowing administrators to configure their MISPs to push content to TAXII 2.1 servers. For more informatia new dedicated will be posted soon. On server side, the [taxii2-client Python library](https://pypi.org/project/taxii2-client/) is required to be installed. The conversion is performed by the wonderful and efficient [misp-stix library](https://github.com/MISP/misp-stix). # Logging rework The logging of MISP has been severely reworked by Jakub Onderka, including a separate Access log subsystem as well as multiple improvements and cleanups to the system at large. # Security fixes Two critical vulnerabilities have been patched allowing for the tampering with data shared in the community via galaxy clusters and tags. It is **HIGHLY** recommended to update to 2.4.166 as soon as possible to avoid information tampering. We also encourage everyone to consider informing peered MISP instance owners to do the same. CVEs have been requested and are pending for both. Thanks to Jakub Onderka for discovering and fixing the vulnerabilities. # Allowing for working around the edge cases introduced by TLP v2.0 Even though [TLP 2.0](https://www.first.org/tlp/) has been supported by MISP for a while, in order to cope with both tools old and new as well as older information sources, we see the need to often attach both TLP:WHITE and TLP:CLEAR to data points. This has however been blocked by the taxonomy exclusivity rules - something that we've now added exceptions for. Let's hope that we can avoid similar surprises in the future. For more [details](https://www.misp-project.org/Changelog.txt) about changes in the MISP core software. # Other updates and changes ## MISP Objects - [passport object] Updated to include passport-creation field. ## MISP Galaxy - MITRE ATT&CK updated and fixing the missing reference - Many improvements and fixes in all the meta fields - Tool galaxy updated - [Ransomware groups](https://www.misp-project.org/galaxy.html#_ransomware) updated 2022-11-30T17:21:47+00:00 maltrail 0.52 maltrail 0.52 2022-11-30T23:11:06+00:00 Start-of-month release 2022-11-30T23:11:06+00:00 reko version-0.11.2 reko version-0.11.2 2022-12-05T15:51:49+00:00 My, has it been that long already since last release? Here's an overview of what's happened since. The Reko solution was moved to .NET 6. As expected, performance and memory footprint was impoved. The partially completed, still very much work-in-progress Avalonia development branch was merged into `master`. Moving forward, any changes in the GUI components will be mirrored in Windows Forms and Avalonia. For instance, the new Base Address finder tool window now exists for both GUI platforms. Another big push is the move to support multithreaded scanning and analysis. Today, Reko uses only a single CPU thread for its CPU-intensive work. Partitioning the work as mutually independent workloads across multiple threads should see a great improvement in execution time. In order to accomplish this, several classes have been altered to only use read-only interfaces to global data, to prevent data race conditions. The work to eliminate globally visible mutable state continues. Several new metadata annotations were added ( with the gentle prodding of @Elthial). You can now annotate C functions with the `[[noreturn]]` , `[[reko::arg(seq...)]]`, and `[[reko::address]]` attributes. Other bits and bobs: * Improvements to `ArgumentGuesser` to handle call sites where Reko can't prove the target address. * The codebase's namespaces were refactored to make them more easy to understand. * Added or improved support for the following architectures: ARM, AArch64, BlackBin, Fujitsu F2MC16FX, MIPS, PDP-7, PowrPC, RiscV, MicroBlaze, SuperH SH2..SH4A, TriCore and x86 EVEX * Support for discovering and parsing x86 and x86-64 MSVC RTTI . * Fix calculation of alignment of unions (@ptomin) * Refactored intrinsics for improved performance * PharLap "MP" Exp file loader (#1169) * MzExe: allow reading PEs without IAT (@smx-smx) * Improved support for XEX executables (@smx-smx) * Handle '*' in `*scanf` format strings. Special thanks to @ptomin and @smx for their dedication and contributions. 2022-12-05T15:51:49+00:00 osquery 5.7.0 osquery 5.7.0 2022-12-06T19:00:16+00:00 Draft 2022-12-06T19:00:16+00:00 DC3-MWCP 3.9.0 DC3-MWCP 3.9.0 2022-12-08T23:32:47+00:00 2022-12-08T23:32:47+00:00