http://open-source-security-software.net/releases.atom 2023-03-31T04:13:35.155717+00:00 python-feedgen MONARC v2.12.2-p3 2022-07-07T12:42:37+00:00 2022-07-07T12:42:37+00:00 pandora v1.0.1 2022-07-11T14:44:26+00:00 Quick release to improve handling of submissions with passwords. # New features * Support password on submit via API. # Maintenance * Bugfixes * Dependencies update. 2022-07-11T14:44:26+00:00 OpenTAXII 0.9.1 2022-07-11T20:28:28+00:00 Changelog ========= 0.9.1 (2022-07-11) ------------------ * Implement `raise_unauthorized` for taxii2, this was missing and lead to 500 errors. 0.9.0 (2022-06-13) ------------------ * Allow custom properties. This can be disabled by config option ``allow_custom_properties`` 0.8.0 (2022-06-05) ------------------ * Nest details inside taxii2 job and allow counts without details 0.7.0 (2022-05-27) ------------------ * Nest taxii2 endpoints under `/taxii2/` 0.6.0 (2022-05-25) ------------------ * Add `public_discovery` option to taxii2 config * Add support for publicly readable taxii 2 api roots 0.5.0 (2022-05-24) ------------------ * Add support for publicly readable taxii 2 collections 0.4.0 (2022-05-20) ------------------ * Move next_param handling into `OpenTAXII2PersistenceAPI` 0.3.0 (2022-04-13) ------------------ * Implement taxii2.1 support 0.3.0a4 (2022-04-13) -------------------- * Merge changes from 0.2.4 maintenance release 0.3.0a3 (2022-01-21) -------------------- * Fix bug that prevented booting with only taxii1 config (`#217 <https://github.com/eclecticiq/OpenTAXII/issues/217>`_ thanks `@azurekid <https://github.com/azurekid>`_ for the report) 0.3.0a2 (2021-12-27) -------------------- * Merge changes from 0.2.3 maintenance release 0.3.0a1 ------- * Add python 3.10 support 0.3.0a0 ------- * Enablement for future taxii2 implementation * Fix documentation build issues 0.2.4 (2022-04-13) ------------------ * Make sure werkzeug <2.1 and >=2.1 work correctly with auth system 0.2.3 (2021-12-22) ------------------ * Fix bug in multithreaded use of sqlite (`#210 <https://github.com/eclecticiq/OpenTAXII/issues/210>`_ thanks `@rohits144 <https://github.com/rohits144>`_ for the report) 0.2.2 (2021-11-05) ------------------ * Fix readthedocs build 0.2.1 (2021-11-03) ------------------ * Add tests for python 3.6, 3.7, 3.8, 3.9, pypy * Add tests for sqlite, mysql, mariadb, postgresql * Fix bug that broke ``delete_content_blocks`` when using mysql on sqlalchemy 1.3 * Docs: Add db schema diagram * Docs: Clarify how to get default data in a default (development) docker instance * Fix implicit routing in TAXII 1.1 Inboxes * Update jwt usage to pyjwt >= 2.0 (thanks `@SanyaKapoor <https://github.com/SanyaKapoor>`_) 0.2.0 (2020-06-30) ------------------ * Enforce UTC usage in datetime fields in SQL DB Persistence API. * `Fix for #114 <https://github.com/eclecticiq/OpenTAXII/issues/114>`_: reintroduce ``opentaxii-create-account`` CLI command. * `Fix for #153 <https://github.com/eclecticiq/OpenTAXII/issues/152>`_: check if user can modify a collection before advertising it over inbox service. * Multiple coding style fixes. * Various documentation updates. 0.1.12 (2019-03-06) ------------------- * Remove unnecessary print statements. 0.1.11 (2019-02-13) ------------------- * Make JSON logging consistent when the application is run via Gunicorn. * Set ``acceptable_destination`` key in status details instead of extended headers * Allow passing ``engine_parameters`` to ``SQLDatabaseAPI`` for those who want to customize SQLAlchemy engine parameters. * Require recent version of ``lxml`` for security reasons. * Various test and Docker infrastructure improvements. 0.1.10 (2018-06-03) ------------------- * Replace separate service/collection/account creation process with single ``opentaxii-sync-data`` CLI command. * Persistence and Auth APIs extended with missing CRUD methods, that are used by ``opentaxii-sync-data``. * Read/modify collection level ACL added. * DB models for default implementation of Persistence API and Auth API were changed. No automatic migration code is provided (sorry!), so upgrading might require manual DB migration. * Drop python2.7 from testing scope. * Various bug fixes and improvements. 0.1.9 (2017-06-19) ------------------ * `libtaxii <https://github.com/TAXIIProject/libtaxii>`_ dependency upgraded to 1.1.111. * Various bug fixes and improvements (thanks to `@bjigmp <https://github.com/bjigmp>`_, `@chorsley <https://github.com/chorsley>`_, `@rjprins <https://github.com/rjprins>`_). 0.1.8 (2017-02-21) ------------------ * Ability to enable/disable "huge trees" support in XML parser. Configuration property ``xml_parser_supports_huge_tree`` set to ``yes`` or ``true`` will disable security restrictions and force XML parser to support very deep trees and very long text content. * Adjust SQL Persistence API implemetation so it works smoothly with MySQL backend. * Use Python 3.5 instead of Python 3.4 for tests. 0.1.7 (2016-10-18) ------------------ * Minor fixes. * Dependencies were changed from hard-pinned to more flexible. * Example of production DB configuration added to docs. 0.1.6 (2016-06-01) ------------------ * Python 3.4 compatibility of the codebase. Tox configuration extended with python 3.4 environment run. * Flake8 full style compatibility. Flake8 check added to Tox configuration. * SQLAlchemy session scope issue fixed (related to `#38 <https://github.com/EclecticIQ/OpenTAXII/issues/38>`_). * `opentaxii-delete-blocks` CLI command added (related to `#45 <https://github.com/EclecticIQ/OpenTAXII/issues/45>`_). * `delete_content_blocks` method `added <https://github.com/EclecticIQ/OpenTAXII/commit/dc6fddc27a98e8450c7e05e583b2bfb741f6e197#diff-6814849ac352b2b74132f8fa52e0ec4eR213>`_ to Persistence API. * Collection's name is `required <https://github.com/EclecticIQ/OpenTAXII/commit/dc6fddc27a98e8450c7e05e583b2bfb741f6e197#diff-ce3f7b939e5c540480ac655aef32c513R116>`_ to be unique in default SQL DB Auth API implementation. 0.1.5 (2016-03-15) ------------------ * Fix for the issue with persistence backend returning ``None`` instead of ``InboxMessage`` object 0.1.4 (2016-02-25) ------------------ * Hard-coded dependencies in ``setup.py`` removed. 0.1.3 (2016-02-25) ------------------ * Versions of dependencies are pinned. * Code adjusted for a new version of `anyconfig <https://pypi.python.org/pypi/anyconfig>`_ API. * Test for configuration loading added. 0.1.2 (2015-07-24) ------------------ * Docker configuration added. * Health check endpoint added. * Basic authentication support added. * Temporary workaround for `Issue #191 <https://github.com/TAXIIProject/libtaxii/issues/191>`_. * Method ``get_domain`` in Persistence API returns domain value configured for ``service_id``. If nothing returned, value set in configuration file will be used. * Performance optimisations. * Bug fixes and style improvements. 0.1.1 (2015-04-08) ------------------ * Alias for Root Logger added to logging configuration. * Context object in a request scope that holds account and token added. * Support for OPTIONS HTTP request to enable auto version negotiation added. * Documentation improved. 0.1.0 (2015-03-31) ------------------ * Initial release 2022-07-11T20:28:28+00:00 caddy v2.5.2 2022-07-12T19:06:28+00:00 This patch release fixes bugs, adds some new features, and makes worthwhile enhancements. We recommend everyone test and upgrade! Many improvements have been made to the `reverse_proxy` module. Highlights: - **New [`/adapt` admin endpoint](https://caddyserver.com/docs/api#post-adapt):** Use your installed config adapters via API in addition to the existing `caddy adapt` CLI command. - **New `Etag`/`If-Match` support for config API:** Safely update your config concurrently and avoid collisions by using [our unique Etag implementation](https://caddyserver.com/docs/api#concurrent-config-changes). - **Rename copied headers from reverse_proxy:** If you're using `handle_response`, you can more easily map headers to a different name for clients. - **Many HTTP matchers have been added to CEL:** You can now use the logic of our HTTP request matchers in CEL expressions. - **Notable bug fixes:** EAB reuse, various QUIC & HTTP/3 fixes, more specific HTTP status codes, various reverse proxy fixes. ## Changelog * 660c59b6 admin: Implement /adapt endpoint (close #4465) (#4846) * ad3a83fb admin: expect quoted ETags (#4879) * f259ed52 admin: support ETag on config endpoints (#4579) * 1498132e caddyhttp: Log error from CEL evaluation (fix #4832) * 0a14f97e caddytls: Make peer certificate verification pluggable (#4389) * 412dcc07 caddytls: Reuse issuer between PreCheck and Issue (#4866) * 499ad6d1 core: Micro-optim in run() (#4810) * c0f76e9e fileserver: Use safe redirects in file browser * 58e05cab forwardauth: Fix case when `copy_headers` is omitted (#4856) * 0b6f7643 forwardauth: Support renaming copied headers, block support (#4783) * 8bac134f go.mod: Bump up quic-go to v0.28.0, fixes for BC breaks (#4867) * 3d18bc56 go.mod: Update go-yaml to v3 * 56013934 go.mod: Update some dependencies * 8e6bc360 go.mod: Upgrade some dependencies * 53c4d788 headers: Only replace known placeholders (#4880) * 0bcd02d5 headers: Support wildcards for delete ops (close #4830) (#4831) * 58970cae httpcaddyfile: Add `{err.*}` placeholder shortcut (#4798) * b687d7b9 httpcaddyfile: Support multiple values for `default_bind` (#4774) * a9267791 reverseproxy: Add --internal-certs CLI flag #3589 (#4817) * aaf6794b reverseproxy: Add renegotiation param in TLS client (#4784) * 54d1923c reverseproxy: Adjust new TLS Caddyfile directive names (#4872) * 7f9b1f43 reverseproxy: Correct the `tls_server_name` docs (#4827) * c82fe911 reverseproxy: Dynamic ServerName for TLS upstreams (#4836) * d6bc9e0b reverseproxy: Err 503 if all upstreams unavailable * 98468af8 reverseproxy: Fix double headers in response handlers (#4847) * 25f10511 reverseproxy: Fix panic when TLS is not configured (#4848) * 5e729c1e reverseproxy: HTTP 504 for upstream timeouts (#4824) * f9b42c37 reverseproxy: Make TLS renegotiation optional * b6e96fa3 reverseproxy: Skip TLS for certain configured ports (#4843) * 57d27c1b reverseproxy: Support http1.1>h2c (close #4777) (#4778) * 9864b138 reverseproxy: api: Remove misleading 'healthy' value * 693e9b52 rewrite: Handle fragment before query (fix #4775) * 6891f7f4 templates: Add `humanize` function (#4767) * 9e760e2e templates: Documentation consistency (#4796) ## New Contributors * @nekohasekai made their first contribution in https://github.com/caddyserver/caddy/pull/4782 * @davidbgk made their first contribution in https://github.com/caddyserver/caddy/pull/4796 * @git001 made their first contribution in https://github.com/caddyserver/caddy/pull/4767 * @varianone made their first contribution in https://github.com/caddyserver/caddy/pull/4817 * @Gr33nbl00d made their first contribution in https://github.com/caddyserver/caddy/pull/4389 * @yaslama made their first contribution in https://github.com/caddyserver/caddy/pull/4784 * @kresike made their first contribution in https://github.com/caddyserver/caddy/pull/4836 * @TristonianJones made their first contribution in https://github.com/caddyserver/caddy/pull/4715 * @jhwz made their first contribution in https://github.com/caddyserver/caddy/pull/4579 **Full Changelog**: https://github.com/caddyserver/caddy/compare/v2.5.1...v2.5.2 2022-07-12T19:06:28+00:00 AIL-framework v4.2 2022-07-16T08:40:47+00:00 ## AIL Framework version 4.2 released including typo squatting tracker, improved AIL2AIL sync, zerobinz fetcher and many bugs fixes v4.2 (2022-06-24) AIL Framework version 4.2 has been released including: - A new tracker for tracking potential typo squatted domains. This feature relies on the new [ail-typo-squatting ](https://github.com/ail-project/ail-typo-squatting) library which can be also used outside of AIL framework. This contribution is from @DavidCruciani - Many improvement and bugs fixed for the AIL2AIL sync. A huge thanks to @aaronkaplan from EU Directorate-General for Informatics (DIGIT) for support and tests during the long debugging sessions. - A new module for zerobinz to create an immediate crawler request if a zerobinz link appears in an item. The module can be used for other services with ephemeral content. Thanks to @gallypette for the contribution and the improvement ideas. - A new hosts detection module has been introduced. - Multiple bugs were fixed. ### Detailed Changes * [Tracker} Tracker_Typo_Squatting. [David Cruciani] * [v4.2] add v4.2 update. [Terrtia] * [investigation] fix investigation by user + delete an obj from all investigation. [Terrtia] * [install vitualenv] remove travis env. [Terrtia] * [Retro Hunt] add logs. [Terrtia] * [Retro Hunt] add logs. [Terrtia] * [Retro Hunt] add logs. [Terrtia] * [AIL2AIL Sync] update exchange format. [Terrtia] * [AIL2AIL Sync] update exchange format. [Terrtia] * [add Hosts module] [Terrtia] * [sync module] debug. [Terrtia] * [sync client] debug. [Terrtia] * [websockets client] bind client ip. [Terrtia] * [websocket server] add host and port config. [Terrtia] * [telegram importer] add username correlation. [Terrtia] * [UI subtype objs] get obj by subtype + name. [Terrtia] * [misp export] add username. [Terrtia] ### Fix * [typosquatting] remove unused import. [Thirion Aurélien] * [tracker] clean import. [Thirion Aurélien] * [tracker term] fix typosquatting key. [Thirion Aurélien] * [Typo] tracker typo. [David Cruciani] * [tracker] UI for other than typosquat. [David Cruciani] * [typo] UI. [David Cruciani] * [Language] fix cld3 import. [Terrtia] * [launcher] kill AIL_2_AIL screen. [Terrtia] * [cld3] enable cld3. [Terrtia] * [cld3 python3.10] temp disable cld3. [Terrtia] * [launcher] remove Travis test. [Terrtia] * [Retro Hunt] item directory. [Terrtia] * [Retro Hunt] item directory. [Terrtia] * [Retro Hunt] fix item directory. [Terrtia] * [AIL exchange mime-type] [Terrtia] * [Hosts module] module + launcher. [Terrtia] * [abstract module] exception traceback #145. [Terrtia] * [ui tag selector] force custom tags. [Terrtia] * [installer] remove old tor install. [Terrtia] * [sync module] fix redis tag queue. [Terrtia] * [sync module] fix tags filter. [Terrtia] * [sync client] debug. [Terrtia] * [sync client] debug. [Terrtia] * [sync module] debug. [Terrtia] * [websockets client] fix client bind. [Terrtia] * [websockets] remove size limit. [Terrtia] * [UI subtype objs] fix form. [Terrtia] * [misp config] https. [Thirion Aurélien] ### Other * Merge pull request #147 from ail-project/typo. [Thirion Aurélien] Integration of the typo-squatting tracker * Fix; [set tracker] missing function. [Thirion Aurélien] * Merge branch 'master' into typo. [David Cruciani] * Add: [tracker] typo-squatting. [David Cruciani] * Merge branch 'master' of github.com:ail-project/ail-framework. [Terrtia] * Merge pull request #146 from gallypette/master. [Thirion Aurélien] add: [modules] zerobinz * Add: [modules] zerobinz. [huynenjl@gmail.com] * Merge branch 'master' of github.com:ail-project/ail-framework. [Terrtia] 2022-07-16T08:40:47+00:00 rocksdb v7.4.3 2022-07-18T15:18:20+00:00 ## 7.4.3 (07/13/2022) ### Behavior Changes * For track_and_verify_wals_in_manifest, revert to the original behavior before #10087: syncing of live WAL file is not tracked, and we track only the synced sizes of **closed** WALs. (PR #10330). ## 7.4.2 (06/30/2022) ### Bug Fixes * Fix a bug in Logger where if dbname and db_log_dir are on different filesystems, dbname creation would fail wrt to db_log_dir path returning an error and fails to open the DB. ## 7.4.1 (06/28/2022) ### Bug Fixes * Pass `rate_limiter_priority` through filter block reader functions to `FileSystem`. ## 7.4.0 (06/19/2022) ### Bug Fixes * Fixed a bug in calculating key-value integrity protection for users of in-place memtable updates. In particular, the affected users would be those who configure `protection_bytes_per_key > 0` on `WriteBatch` or `WriteOptions`, and configure `inplace_callback != nullptr`. * Fixed a bug where a snapshot taken during SST file ingestion would be unstable. * Fixed a bug for non-TransactionDB with avoid_flush_during_recovery = true and TransactionDB where in case of crash, min_log_number_to_keep may not change on recovery and persisting a new MANIFEST with advanced log_numbers for some column families, results in "column family inconsistency" error on second recovery. As a solution, RocksDB will persist the new MANIFEST after successfully syncing the new WAL. If a future recovery starts from the new MANIFEST, then it means the new WAL is successfully synced. Due to the sentinel empty write batch at the beginning, kPointInTimeRecovery of WAL is guaranteed to go after this point. If future recovery starts from the old MANIFEST, it means the writing the new MANIFEST failed. We won't have the "SST ahead of WAL" error. * Fixed a bug where RocksDB DB::Open() may creates and writes to two new MANIFEST files even before recovery succeeds. Now writes to MANIFEST are persisted only after recovery is successful. * Fix a race condition in WAL size tracking which is caused by an unsafe iterator access after container is changed. * Fix unprotected concurrent accesses to `WritableFileWriter::filesize_` by `DB::SyncWAL()` and `DB::Put()` in two write queue mode. * Fix a bug in WAL tracking. Before this PR (#10087), calling `SyncWAL()` on the only WAL file of the db will not log the event in MANIFEST, thus allowing a subsequent `DB::Open` even if the WAL file is missing or corrupted. * Fix a bug that could return wrong results with `index_type=kHashSearch` and using `SetOptions` to change the `prefix_extractor`. * Fixed a bug in WAL tracking with wal_compression. WAL compression writes a kSetCompressionType record which is not associated with any sequence number. As result, WalManager::GetSortedWalsOfType() will skip these WALs and not return them to caller, e.g. Checkpoint, Backup, causing the operations to fail. * Avoid a crash if the IDENTITY file is accidentally truncated to empty. A new DB ID will be written and generated on Open. * Fixed a possible corruption for users of `manual_wal_flush` and/or `FlushWAL(true /* sync */)`, together with `track_and_verify_wals_in_manifest == true`. For those users, losing unsynced data (e.g., due to power loss) could make future DB opens fail with a `Status::Corruption` complaining about missing WAL data. * Fixed a bug in `WriteBatchInternal::Append()` where WAL termination point in write batch was not considered and the function appends an incorrect number of checksums. * Fixed a crash bug introduced in 7.3.0 affecting users of MultiGet with `kDataBlockBinaryAndHash`. * Add some fixes in async_io which was doing extra prefetching in shorter scans. ### Public API changes * Add new API GetUnixTime in Snapshot class which returns the unix time at which Snapshot is taken. * Add transaction `get_pinned` and `multi_get` to C API. * Add two-phase commit support to C API. * Add `rocksdb_transaction_get_writebatch_wi` and `rocksdb_transaction_rebuild_from_writebatch` to C API. * Add `rocksdb_options_get_blob_file_starting_level` and `rocksdb_options_set_blob_file_starting_level` to C API. * Add `blobFileStartingLevel` and `setBlobFileStartingLevel` to Java API. * Add SingleDelete for DB in C API * Add User Defined Timestamp in C API. * `rocksdb_comparator_with_ts_create` to create timestamp aware comparator * Put, Get, Delete, SingleDelete, MultiGet APIs has corresponding timestamp aware APIs with suffix `with_ts` * And Add C API's for Transaction, SstFileWriter, Compaction as mentioned [here](https://github.com/facebook/rocksdb/wiki/User-defined-Timestamp-(Experimental)) * The contract for implementations of Comparator::IsSameLengthImmediateSuccessor has been updated to work around a design bug in `auto_prefix_mode`. * The API documentation for `auto_prefix_mode` now notes some corner cases in which it returns different results than `total_order_seek`, due to design bugs that are not easily fixed. Users using built-in comparators and keys at least the size of a fixed prefix length are not affected. * Obsoleted the NUM_DATA_BLOCKS_READ_PER_LEVEL stat and introduced the NUM_LEVEL_READ_PER_MULTIGET and MULTIGET_COROUTINE_COUNT stats * Introduced `WriteOptions::protection_bytes_per_key`, which can be used to enable key-value integrity protection for live updates. ### New Features * Add FileSystem::ReadAsync API in io_tracing * Add blob garbage collection parameters `blob_garbage_collection_policy` and `blob_garbage_collection_age_cutoff` to both force-enable and force-disable GC, as well as selectively override age cutoff when using CompactRange. * Add an extra sanity check in `GetSortedWalFiles()` (also used by `GetLiveFilesStorageInfo()`, `BackupEngine`, and `Checkpoint`) to reduce risk of successfully created backup or checkpoint failing to open because of missing WAL file. * Add a new column family option `blob_file_starting_level` to enable writing blob files during flushes and compactions starting from the specified LSM tree level. * Add support for timestamped snapshots (#9879) * Provide support for AbortIO in posix to cancel submitted asynchronous requests using io_uring. * Add support for rate-limiting batched `MultiGet()` APIs ### Behavior changes * DB::Open(), DB::OpenAsSecondary() will fail if a Logger cannot be created (#9984) * Removed support for reading Bloom filters using obsolete block-based filter format. (Support for writing such filters was dropped in 7.0.) For good read performance on old DBs using these filters, a full compaction is required. * Per KV checksum in write batch is verified before a write batch is written to WAL to detect any corruption to the write batch (#10114). ### Performance Improvements * When compiled with folly (Meta-internal integration; experimental in open source build), improve the locking performance (CPU efficiency) of LRUCache by using folly DistributedMutex in place of standard mutex. 2022-07-18T15:18:20+00:00 rocksdb v7.4.4 2022-07-28T18:34:26+00:00 ## 7.4.4 (07/19/2022) ### Public API changes * Removed Customizable support for RateLimiter and removed its CreateFromString() and Type() functions. ### Bug Fixes * Fix a bug where `GenericRateLimiter` could revert the bandwidth set dynamically using `SetBytesPerSecond()` when a user configures a structure enclosing it, e.g., using `GetOptionsFromString()` to configure an `Options` that references an existing `RateLimiter` object. 2022-07-28T18:34:26+00:00 rocksdb v7.4.5 2022-08-02T23:17:54+00:00 ## 7.4.5 (08/02/2022) ### Bug Fixes * Fix a bug starting in 7.4.0 in which some fsync operations might be skipped in a DB after any DropColumnFamily on that DB, until it is re-opened. This can lead to data loss on power loss. (For custom FileSystem implementations, this could lead to `FSDirectory::Fsync` or `FSDirectory::Close` after the first `FSDirectory::Close`; Also, valgrind could report call to `close()` with `fd=-1`.) 2022-08-02T23:17:54+00:00 whids v1.8.0-beta.7 2022-08-03T12:33:11+00:00 2022-08-03T12:33:11+00:00 PyPCAPKit v0.16.2 2022-08-04T02:55:47+00:00 2022-08-04T02:55:47+00:00 maltrail 0.48 2022-08-04T06:36:06+00:00 Start-of-month release 2022-08-04T06:36:06+00:00 pcileech v4.15 2022-08-04T06:39:53+00:00 * Support for MemProcFS v5.0 2022-08-04T06:39:53+00:00 MISP v2.4.160 2022-08-08T12:32:32+00:00 We are pleased to announce the immediate availability of MISP v2.4.160. With the August summer-holiday season kicking into high gear, we have a very special release for you all, containing a long list of major new features, improvements and general quality of life improvements. Unlike we do normally, this time around we're preparing separate blog posts for some of those major features, so follow the links below to read up on in-depth descriptions of each. # Workflows Something that has been in the works for quite some time now is finally hitting a release version of MISP, as of 2.4.160, we have the first release of the built in workflow system released. This system allows you to use an easy to use, yet extremely powerful graphical interface to modify how MISP handles certain tasks such as event publishing, user enrollment, synchronisation, etc., by adding additional logical steps in their respective executions, utilising a module system similar to what was already common to MISP from enrichment subsystems, exports as well as imports. This is merely the first step (or leap rather) towards customising and sharing custom workflows, stay tuned for new features, improvements as well as triggers and modules in the near future. Head over to the [README](https://github.com/MISP/misp-workflow-blueprints/blob/main/README.md) as well as a nifty [slide deck](https://www.misp-project.org/misp-training/a.12-misp-workflows.pdf), to find out what this incredibly powerful can do for you and your community. # New correlation engine One of the biggest pain points as of recently has been our dated and rather bloated correlation engine, which could easily bring a long running MISP instance to its knees when certain highly correlated data sources were synchronised. As of 2.4.160, we now have 2 brand new correlation engines at your disposal, with the old engine being retired immediately. Please be aware that upgrading to the current version will regenerate your correlations using the new engine, something that can take quite a long time (on our largest instance it took a whopping 40 hours!). With that said, we can assure you it's well worth the wait and should resolve several long standing performance bottlenecks as well as heavily cut down on the space requirements for your data. For more information, on the new engines, their differences, the various new support tools as well as what benefits you should expect, head over to the [dedicated blog post](https://github.com/MISP/MISP/blob/2.4/docs/correlations.rework.md). # STIX 2 library reworks There has been a massive amount of work going into the STIX 2.x library rework, bringing us closer and closer to having a full mapping of everything expressable. We're collaborating with CISA and Mitre to ensure that MISP can both express and understand STIX to its fullest extent. For more information, head over to the [release notes](https://github.com/MISP/misp-stix/releases/tag/v2.4.160) over on the MISP STIX library's repo. # Mermaid support for Event reports added Writing custom reports has become more and more popular, but one annoyance has been the lack of a way to depict graphs and flow charts without relying on external tools to create those (and share them as images for example). Using Mermaid, you now have a nifty tool to build graphs out of simple markdown directly in the event report editor. # Various other improvements A long list of other improvements, affecting the performance and stability of the platform as well as improvements to existing features. Head over to the changelog for a detailed list of changes. # Acknowledgement We would like to thank all the [contributors](https://www.misp-project.org/contributors), reporters and users who have helped us in the past months to improve MISP and information sharing at large. This release includes multiple updates in [misp-objects](https://www.misp-project.org/objects.html), [misp-taxonomies](https://www.misp-project.org/taxonomies.html) and [misp-galaxy](https://www.misp-project.org/galaxy.html). As always, a detailed and [complete changelog is available](https://www.misp-project.org/Changelog.txt) with all the fixes, changes and improvements in MISP core. Additional changelogs are available for [misp galaxy](https://www.misp-project.org/Changelog-misp-galaxy.txt), [misp-taxonomies](https://www.misp-project.org/Changelog-misp-taxonomies.txt), [misp-objects](https://www.misp-project.org/Changelog-misp-objects.txt) and [misp-modules](https://www.misp-project.org/Changelog-misp-modules.txt) 2022-08-08T12:32:32+00:00 Lookyloo v1.14.0 2022-08-08T13:53:34+00:00 # New features * Trigger a capture on a web enabled document provided as a file instead of a URL. Useful for HTML files attached to emails, or HTML body in email.  -------- * Compress (gzip) the HAR file in archived captures - saves a lot of disk space. * Support for RiskIQ Passive DNS (requires API key) * Display SSL/TLS information available in the HAR dump from Playwright  -------- * Optional DoNotTrack HTTP header in capture  -------- * Display size of rendered page on hostnode popup. * [WiP] Download files when the URL captured to a downloadable file (PDF, Office doc, ...) (**Important note** the downloaded file is not exposed to the user yet) * [WiP] List all hashes available in the capture, sort them by frequency. Makes it easier to find phishing sides using the same resources.  # Fixes * Major speed improvements when displaying the hostnode popups (only show the recent cached captures by default) * Improvements in the caching mechanism * Cleanup data showed by monitoring script * Avoid crashes when RiskIQ isn't reachable # Changes * Update dependencies (js, python) * Improve logging in archiver * Improve config file 2022-08-08T13:53:34+00:00 whids v1.8.0-beta.8 2022-08-08T16:38:56+00:00 2022-08-08T16:38:56+00:00 MISP v2.4.161 2022-08-11T15:30:58+00:00 We are pleased to announce the immediate availability of [MISP v2.4.161](https://github.com/MISP/MISP/releases/tag/v2.4.161).  # Small improvements - A new option added to log the last API request of an API key. (Thanks to Tom King for the contribution) - Overcorrelation features have some new improvements such as: - A new tool to generate occurrence counts (real numbers this time) - A hook to truncate the over-correlating value table on recorrelation - We no longer store the partial counts as occurrences when generating correlations - Performance improvements in event fetching - Various performance tuning in the new correlation engine including the full recorrelation # Bugs fixed - `tlp:amber+strict` and `tlp:clear` are now valid tags - [stix2 import] Better `external_references` parsing for attack patterns objects Thanks to all the contributors and users reporting bugs to make the software better. As always, a detailed and [complete changelog is available](https://www.misp-project.org/Changelog.txt) with all the fixes, changes and improvements in MISP core. 2022-08-11T15:30:58+00:00 osquery 5.5.0 2022-08-12T17:47:19+00:00 draft 2022-08-12T17:47:19+00:00 dnstwist 20220815 2022-08-15T15:39:49+00:00 2022-08-15T15:39:49+00:00 dalton v3.2.4 2022-08-16T18:51:57+00:00 - New URL for downloading Suricata source code 2022-08-16T18:51:57+00:00 osquery 5.5.1 2022-08-18T13:24:43+00:00 Draft! (think 5.5.0 plus sqlite) 2022-08-18T13:24:43+00:00 rocksdb v7.5.3 2022-08-24T22:39:19+00:00 ## 7.5.2 (08/02/2022) ### Bug Fixes * Fix a bug starting in 7.4.0 in which some fsync operations might be skipped in a DB after any DropColumnFamily on that DB, until it is re-opened. This can lead to data loss on power loss. (For custom FileSystem implementations, this could lead to `FSDirectory::Fsync` or `FSDirectory::Close` after the first `FSDirectory::Close`; Also, valgrind could report call to `close()` with `fd=-1`.) ## 7.5.1 (08/01/2022) ### Bug Fixes * Fix a bug where rate_limiter_parameter is not passed into `PartitionedFilterBlockReader::GetFilterPartitionBlock`. ## 7.5.0 (07/15/2022) ### New Features * Mempurge option flag `experimental_mempurge_threshold` is now a ColumnFamilyOptions and can now be dynamically configured using `SetOptions()`. * Support backward iteration when `ReadOptions::iter_start_ts` is set. * Provide support for ReadOptions.async_io with direct_io to improve Seek latency by using async IO to parallelize child iterator seek and doing asynchronous prefetching on sequential scans. * Added support for blob caching in order to cache frequently used blobs for BlobDB. * User can configure the new ColumnFamilyOptions `blob_cache` to enable/disable blob caching. * Either sharing the backend cache with the block cache or using a completely separate cache is supported. * A new abstraction interface called `BlobSource` for blob read logic gives all users access to blobs, whether they are in the blob cache, secondary cache, or (remote) storage. Blobs can be potentially read both while handling user reads (`Get`, `MultiGet`, or iterator) and during compaction (while dealing with compaction filters, Merges, or garbage collection) but eventually all blob reads go through `Version::GetBlob` or, for MultiGet, `Version::MultiGetBlob` (and then get dispatched to the interface -- `BlobSource`). * Add experimental tiered compaction feature `AdvancedColumnFamilyOptions::preclude_last_level_data_seconds`, which makes sure the new data inserted within preclude_last_level_data_seconds won't be placed on cold tier (the feature is not complete). ### Public API changes * Add metadata related structs and functions in C API, including * `rocksdb_get_column_family_metadata()` and `rocksdb_get_column_family_metadata_cf()` to obtain `rocksdb_column_family_metadata_t`. * `rocksdb_column_family_metadata_t` and its get functions & destroy function. * `rocksdb_level_metadata_t` and its and its get functions & destroy function. * `rocksdb_file_metadata_t` and its and get functions & destroy functions. * Add suggest_compact_range() and suggest_compact_range_cf() to C API. * When using block cache strict capacity limit (`LRUCache` with `strict_capacity_limit=true`), DB operations now fail with Status code `kAborted` subcode `kMemoryLimit` (`IsMemoryLimit()`) instead of `kIncomplete` (`IsIncomplete()`) when the capacity limit is reached, because Incomplete can mean other specific things for some operations. In more detail, `Cache::Insert()` now returns the updated Status code and this usually propagates through RocksDB to the user on failure. * NewClockCache calls temporarily return an LRUCache (with similar characteristics as the desired ClockCache). This is because ClockCache is being replaced by a new version (the old one had unknown bugs) but this is still under development. * Add two functions `int ReserveThreads(int threads_to_be_reserved)` and `int ReleaseThreads(threads_to_be_released)` into `Env` class. In the default implementation, both return 0. Newly added `xxxEnv` class that inherits `Env` should implement these two functions for thread reservation/releasing features. * Removed Customizable support for RateLimiter and removed its CreateFromString() and Type() functions. ### Bug Fixes * Fix a bug in which backup/checkpoint can include a WAL deleted by RocksDB. * Fix a bug where concurrent compactions might cause unnecessary further write stalling. In some cases, this might cause write rate to drop to minimum. * Fix a bug in Logger where if dbname and db_log_dir are on different filesystems, dbname creation would fail wrt to db_log_dir path returning an error and fails to open the DB. * Fix a CPU and memory efficiency issue introduce by https://github.com/facebook/rocksdb/pull/8336 which made InternalKeyComparator configurable as an unintended side effect * Fix a bug where `GenericRateLimiter` could revert the bandwidth set dynamically using `SetBytesPerSecond()` when a user configures a structure enclosing it, e.g., using `GetOptionsFromString()` to configure an `Options` that references an existing `RateLimiter` object. ## Behavior Change * In leveled compaction with dynamic levelling, level multiplier is not anymore adjusted due to oversized L0. Instead, compaction score is adjusted by increasing size level target by adding incoming bytes from upper levels. This would deprioritize compactions from upper levels if more data from L0 is coming. This is to fix some unnecessary full stalling due to drastic change of level targets, while not wasting write bandwidth for compaction while writes are overloaded. * For track_and_verify_wals_in_manifest, revert to the original behavior before #10087: syncing of live WAL file is not tracked, and we track only the synced sizes of **closed** WALs. (PR #10330). * WAL compression now computes/verifies checksum during compression/decompression. ### Performance Improvements * Rather than doing total sort against all files in a level, SortFileByOverlappingRatio() to only find the top 50 files based on score. This can improve write throughput for the use cases where data is loaded in increasing key order and there are a lot of files in one LSM-tree, where applying compaction results is the bottleneck. * In leveled compaction, L0->L1 trivial move will allow more than one file to be moved in one compaction. This would allow L0 files to be moved down faster when data is loaded in sequential order, making slowdown or stop condition harder to hit. Also seek L0->L1 trivial move when only some files qualify. * In leveled compaction, try to trivial move more than one files if possible, up to 4 files or max_compaction_bytes. This is to allow higher write throughput for some use cases where data is loaded in sequential order, where appying compaction results is the bottleneck. 2022-08-24T22:39:19+00:00 Lookyloo v1.15.0 2022-08-25T12:43:29+00:00 # Breaking change * Lookyloo requires Redis 7.0 or more decent. The upgrade process is as follows: 1. Go to the Redis directory (should be in the same directory as where you cloned Lookyloo) 2. Run the following commands ``` git fetch git checkout 7.0 make distclean make -j4 make test ``` 3. You now have the new version of redis in place, you can update lookyloo as usual. # New features * Use pre-configured devices from Playwright (mobile only for now)  * Download files when the URLs points to a downloadable content  * Submit downloadable content to [Pandora](https://pandora.circl.lu/submit) (if available) * Automatically select the most appropriate browser engine based on the user-agent # Fixes * Make sure all the gunicorn instances displays all the recent captures * Other bugfixes and GUI improvements # Changes * Improve capture page with radio button to select which user-agent to submit * Bump dependencies 2022-08-25T12:43:29+00:00 OpenTAXII 0.9.2 2022-08-26T12:58:36+00:00 Changelog ========= 0.9.2 (2022-08-26) ------------------ * Improve readability and navigation of docs (`#238 <https://github.com/eclecticiq/OpenTAXII/pull/238>`_ thanks `@zed-eiq <https://github.com/zed-eiq>`_ for the improvement). 0.9.1 (2022-07-11) ------------------ * Implement `raise_unauthorized` for taxii2, this was missing and lead to 500 errors. 0.9.0 (2022-06-13) ------------------ * Allow custom properties. This can be disabled by config option ``allow_custom_properties`` 0.8.0 (2022-06-05) ------------------ * Nest details inside taxii2 job and allow counts without details 0.7.0 (2022-05-27) ------------------ * Nest taxii2 endpoints under `/taxii2/` 0.6.0 (2022-05-25) ------------------ * Add `public_discovery` option to taxii2 config * Add support for publicly readable taxii 2 api roots 0.5.0 (2022-05-24) ------------------ * Add support for publicly readable taxii 2 collections 0.4.0 (2022-05-20) ------------------ * Move next_param handling into `OpenTAXII2PersistenceAPI` 0.3.0 (2022-04-13) ------------------ * Implement taxii2.1 support 0.3.0a4 (2022-04-13) -------------------- * Merge changes from 0.2.4 maintenance release 0.3.0a3 (2022-01-21) -------------------- * Fix bug that prevented booting with only taxii1 config (`#217 <https://github.com/eclecticiq/OpenTAXII/issues/217>`_ thanks `@azurekid <https://github.com/azurekid>`_ for the report) 0.3.0a2 (2021-12-27) -------------------- * Merge changes from 0.2.3 maintenance release 0.3.0a1 ------- * Add python 3.10 support 0.3.0a0 ------- * Enablement for future taxii2 implementation * Fix documentation build issues 0.2.4 (2022-04-13) ------------------ * Make sure werkzeug <2.1 and >=2.1 work correctly with auth system 0.2.3 (2021-12-22) ------------------ * Fix bug in multithreaded use of sqlite (`#210 <https://github.com/eclecticiq/OpenTAXII/issues/210>`_ thanks `@rohits144 <https://github.com/rohits144>`_ for the report) 0.2.2 (2021-11-05) ------------------ * Fix readthedocs build 0.2.1 (2021-11-03) ------------------ * Add tests for python 3.6, 3.7, 3.8, 3.9, pypy * Add tests for sqlite, mysql, mariadb, postgresql * Fix bug that broke ``delete_content_blocks`` when using mysql on sqlalchemy 1.3 * Docs: Add db schema diagram * Docs: Clarify how to get default data in a default (development) docker instance * Fix implicit routing in TAXII 1.1 Inboxes * Update jwt usage to pyjwt >= 2.0 (thanks `@SanyaKapoor <https://github.com/SanyaKapoor>`_) 0.2.0 (2020-06-30) ------------------ * Enforce UTC usage in datetime fields in SQL DB Persistence API. * `Fix for #114 <https://github.com/eclecticiq/OpenTAXII/issues/114>`_: reintroduce ``opentaxii-create-account`` CLI command. * `Fix for #153 <https://github.com/eclecticiq/OpenTAXII/issues/152>`_: check if user can modify a collection before advertising it over inbox service. * Multiple coding style fixes. * Various documentation updates. 0.1.12 (2019-03-06) ------------------- * Remove unnecessary print statements. 0.1.11 (2019-02-13) ------------------- * Make JSON logging consistent when the application is run via Gunicorn. * Set ``acceptable_destination`` key in status details instead of extended headers * Allow passing ``engine_parameters`` to ``SQLDatabaseAPI`` for those who want to customize SQLAlchemy engine parameters. * Require recent version of ``lxml`` for security reasons. * Various test and Docker infrastructure improvements. 0.1.10 (2018-06-03) ------------------- * Replace separate service/collection/account creation process with single ``opentaxii-sync-data`` CLI command. * Persistence and Auth APIs extended with missing CRUD methods, that are used by ``opentaxii-sync-data``. * Read/modify collection level ACL added. * DB models for default implementation of Persistence API and Auth API were changed. No automatic migration code is provided (sorry!), so upgrading might require manual DB migration. * Drop python2.7 from testing scope. * Various bug fixes and improvements. 0.1.9 (2017-06-19) ------------------ * `libtaxii <https://github.com/TAXIIProject/libtaxii>`_ dependency upgraded to 1.1.111. * Various bug fixes and improvements (thanks to `@bjigmp <https://github.com/bjigmp>`_, `@chorsley <https://github.com/chorsley>`_, `@rjprins <https://github.com/rjprins>`_). 0.1.8 (2017-02-21) ------------------ * Ability to enable/disable "huge trees" support in XML parser. Configuration property ``xml_parser_supports_huge_tree`` set to ``yes`` or ``true`` will disable security restrictions and force XML parser to support very deep trees and very long text content. * Adjust SQL Persistence API implemetation so it works smoothly with MySQL backend. * Use Python 3.5 instead of Python 3.4 for tests. 0.1.7 (2016-10-18) ------------------ * Minor fixes. * Dependencies were changed from hard-pinned to more flexible. * Example of production DB configuration added to docs. 0.1.6 (2016-06-01) ------------------ * Python 3.4 compatibility of the codebase. Tox configuration extended with python 3.4 environment run. * Flake8 full style compatibility. Flake8 check added to Tox configuration. * SQLAlchemy session scope issue fixed (related to `#38 <https://github.com/EclecticIQ/OpenTAXII/issues/38>`_). * `opentaxii-delete-blocks` CLI command added (related to `#45 <https://github.com/EclecticIQ/OpenTAXII/issues/45>`_). * `delete_content_blocks` method `added <https://github.com/EclecticIQ/OpenTAXII/commit/dc6fddc27a98e8450c7e05e583b2bfb741f6e197#diff-6814849ac352b2b74132f8fa52e0ec4eR213>`_ to Persistence API. * Collection's name is `required <https://github.com/EclecticIQ/OpenTAXII/commit/dc6fddc27a98e8450c7e05e583b2bfb741f6e197#diff-ce3f7b939e5c540480ac655aef32c513R116>`_ to be unique in default SQL DB Auth API implementation. 0.1.5 (2016-03-15) ------------------ * Fix for the issue with persistence backend returning ``None`` instead of ``InboxMessage`` object 0.1.4 (2016-02-25) ------------------ * Hard-coded dependencies in ``setup.py`` removed. 0.1.3 (2016-02-25) ------------------ * Versions of dependencies are pinned. * Code adjusted for a new version of `anyconfig <https://pypi.python.org/pypi/anyconfig>`_ API. * Test for configuration loading added. 0.1.2 (2015-07-24) ------------------ * Docker configuration added. * Health check endpoint added. * Basic authentication support added. * Temporary workaround for `Issue #191 <https://github.com/TAXIIProject/libtaxii/issues/191>`_. * Method ``get_domain`` in Persistence API returns domain value configured for ``service_id``. If nothing returned, value set in configuration file will be used. * Performance optimisations. * Bug fixes and style improvements. 0.1.1 (2015-04-08) ------------------ * Alias for Root Logger added to logging configuration. * Context object in a request scope that holds account and token added. * Support for OPTIONS HTTP request to enable auto version negotiation added. * Documentation improved. 0.1.0 (2015-03-31) ------------------ * Initial release 2022-08-26T12:58:36+00:00 ursadb v1.5.0 2022-08-29T16:39:10+00:00 Features: - Alternatives (like {(41 | 42)}) implemented in the ursadb query syntax (#65) - Better support for wildcards (#23) - Syntax for indexing with taints (#31) Performance: - Query graph pruning (#67) Correctness: - Some improvements for thread safety (#32) Refactoring and maintenance: - Ursacli rewritten to C++ (#48) - Documentation improvements (#33) 2022-08-29T16:39:10+00:00 TheHive 4.1.23 2022-08-30T07:28:56+00:00 ## [4.1.23](https://github.com/TheHive-Project/TheHive/milestone/94) (2022-07-22) **Fixed bugs:** - [Bug] system user can be deleted by integrity checks [\#2406](https://github.com/TheHive-Project/TheHive/issues/2406) 2022-08-30T07:28:56+00:00 MONARC v2.12.2-p4 2022-08-30T07:41:11+00:00 Updated zm-client to [v2.12.1-p1](https://github.com/monarc-project/zm-client/releases/tag/v2.12.1-p1). 2022-08-30T07:41:11+00:00 pandora v1.1.0 2022-08-31T09:25:44+00:00 # Breaking change This release requires poetry v1.2.0 or more recent. Run the following command to upgrade it: `poetry self update` # New feature * HTML documents can be submitted to [Lookyloo](https://github.com/Lookyloo/lookyloo) (requires v1.15.0 or more recent)   # Changes * Improvements in the modules (archives, ISO, EML) * Improvements on the stats page * Configure the links on the index * Bump dependencies # Fixes * Support ingesting a file downloaded from a Pandora instance * Automatically restart unoserver when it crashes (makes previews with libreoffice more reliable) # Notes * Many have reported issues with the reviews generated by LibreOffice. A seemingly universal fix seems to be installing the [full package from the PPA](https://github.com/pandora-analysis/pandora#important-notes-regarding-libreoffice). 2022-08-31T09:25:44+00:00 maltrail 0.49 2022-08-31T22:11:05+00:00 Start-of-month release 2022-08-31T22:11:05+00:00 caddy v2.6.0-beta.3 2022-09-05T22:07:59+00:00 This is the _first beta_ release for Caddy 2.6. Please try it out and report any regressions you notice! Thanks to everyone who helped out! :blush: Beta 1 and beta 2 were trial runs for our CI upgrades, so this is technically beta 3. There are no code changes from beta 1 to beta 3. Thank you @mohammed90 for figuring out the CI magic! These are abbreviated release notes. The full release notes (23.5 KB of them, to be precise) will be published with the final 2.6 release, explaining in detail all the new few features, enhancements, and bug fixes. ## What's Changed * reverseproxy: Implement retry count, alternative to try_duration by @francislavoie in https://github.com/caddyserver/caddy/pull/4756 * cmd: Fix reload with stdin by @francislavoie in https://github.com/caddyserver/caddy/pull/4900 * reverseproxy: Implement read & write timeouts for HTTP transport by @mholt in https://github.com/caddyserver/caddy/pull/4905 * ci: Run golangci-lint on multiple os(#4875) by @u5surf in https://github.com/caddyserver/caddy/pull/4913 * caddyhttp: Clear out matcher error immediately after grabbing it by @francislavoie in https://github.com/caddyserver/caddy/pull/4916 * chore: Add .gitattributes to force *.go to LF by @francislavoie in https://github.com/caddyserver/caddy/pull/4919 * core: Windows service integration by @WingLim in https://github.com/caddyserver/caddy/pull/4790 * fileserver: Support virtual file systems by @mholt in https://github.com/caddyserver/caddy/pull/4909 * caddyhttp: Implement `caddy respond` command by @mholt in https://github.com/caddyserver/caddy/pull/4870 * chore: Bump up to Go 1.19, minimum 1.18 by @francislavoie in https://github.com/caddyserver/caddy/pull/4925 * httpserver: Configurable shutdown delay by @mholt in https://github.com/caddyserver/caddy/pull/4906 * cmd: Use newly-available version information by @mholt in https://github.com/caddyserver/caddy/pull/4931 * Replace strings.Index usages with strings.Cut by @WilczynskiT in https://github.com/caddyserver/caddy/pull/4930 * optimization: Replaced strings.Index with strings.Cut by @chir4gm in https://github.com/caddyserver/caddy/pull/4932 * go.mod: Upgrade OpenTelemetry dependencies by @lewandowski-stripe in https://github.com/caddyserver/caddy/pull/4937 * logging: Fix `cookie` filter by @francislavoie in https://github.com/caddyserver/caddy/pull/4943 * reverseproxy: Support 1xx status codes (HTTP early hints) by @dunglas in https://github.com/caddyserver/caddy/pull/4882 * reverseproxy: Fix H2C dialer using new stdlib `DialTLSContext` by @francislavoie in https://github.com/caddyserver/caddy/pull/4951 * reverseproxy: Ignore context cancel in stream mode by @mholt in https://github.com/caddyserver/caddy/pull/4952 * reverseproxy: Add `unix+h2c` Caddyfile network shortcut by @francislavoie in https://github.com/caddyserver/caddy/pull/4953 * caddyhttp: Enable HTTP/3 by default by @mholt in https://github.com/caddyserver/caddy/pull/4707 * fileserver: reset buffer before using it (#4962) by @abdusco in https://github.com/caddyserver/caddy/pull/4963 * caddyhttp: Smarter path matching and rewriting by @mholt in https://github.com/caddyserver/caddy/pull/4948 * core: Change net.IP to netip.Addr; use netip.Prefix by @WilczynskiT in https://github.com/caddyserver/caddy/pull/4966 * caddyhttp: Set `http.error.message` to the HandlerError message by @francislavoie in https://github.com/caddyserver/caddy/pull/4971 * caddyhttp: Fix for nil `handlerErr.Err` by @francislavoie in https://github.com/caddyserver/caddy/pull/4977 * templates: cap of slice should not be smaller than length by @jedy in https://github.com/caddyserver/caddy/pull/4975 * ci: Increase linter timeout by @mholt in https://github.com/caddyserver/caddy/pull/4981 * httpcaddyfile: Add ocsp_interval global option by @benburkert in https://github.com/caddyserver/caddy/pull/4980 * zstd: fix typo in comment by @simon04 in https://github.com/caddyserver/caddy/pull/4985 * admin: Don't stop old server if new one fails by @WeidiDeng in https://github.com/caddyserver/caddy/pull/4964 * reverseproxy: Add upstreams healthy metrics by @david-szabo97 in https://github.com/caddyserver/caddy/pull/4935 * Remove duplicate words in comments by @Abirdcfly in https://github.com/caddyserver/caddy/pull/4986 * cmd: Migrate to `spf13/cobra`, remove single-dash arg support by @mohammed90 in https://github.com/caddyserver/caddy/pull/4565 * cmd: add completion command by @mohammed90 in https://github.com/caddyserver/caddy/pull/4994 * caddyhttp: Set Content-Type for static response by @mholt in https://github.com/caddyserver/caddy/pull/4999 * httpcaddyfile: Add `{cookie.*}` placeholder shortcut by @francislavoie in https://github.com/caddyserver/caddy/pull/5001 * events: Implement event system by @francislavoie in https://github.com/caddyserver/caddy/pull/4912 * core: Plugins can register listener networks by @mholt in https://github.com/caddyserver/caddy/pull/5002 * httpcaddyfile: Add shortcut for expression matchers by @francislavoie in https://github.com/caddyserver/caddy/pull/4976 * caddyhttp: Copy logger config to HTTP server during AutoHTTPS by @francislavoie in https://github.com/caddyserver/caddy/pull/4990 * dist: deb package manpages and bash completion scripts by @mohammed90 in https://github.com/caddyserver/caddy/pull/5007 * fastcgi: allow users to log stderr output (#4967) by @flga in https://github.com/caddyserver/caddy/pull/5004 * templates: Document `httpError` function by @Malankar in https://github.com/caddyserver/caddy/pull/4972 * fastcgi: Optimize FastCGI transport by @WeidiDeng in https://github.com/caddyserver/caddy/pull/4978 * core: Refactor listeners; use SO_REUSEPORT on Unix by @mholt in https://github.com/caddyserver/caddy/pull/4705 * reverseproxy: Close hijacked conns on reload/quit by @mholt in https://github.com/caddyserver/caddy/pull/4895 * ci: generate SBOM and sign artifacts using cosign by @mohammed90 in https://github.com/caddyserver/caddy/pull/4910 * caddyauth: Speed up basicauth provision, deprecate `scrypt` by @francislavoie in https://github.com/caddyserver/caddy/pull/4720 * caddyhttp: Support `respond` with HTTP 103 Early Hints by @mholt in https://github.com/caddyserver/caddy/pull/5006 * fileserver: Support glob expansion in file matcher by @mholt in https://github.com/caddyserver/caddy/pull/4993 * ci: add `id-token` permission and update the signing command by @mohammed90 in https://github.com/caddyserver/caddy/pull/5016 * ci: grant the `release` workflow the `write` permission to `contents` by @mohammed90 in https://github.com/caddyserver/caddy/pull/5017 ## New Contributors * @WingLim made their first contribution in https://github.com/caddyserver/caddy/pull/4790 * @WilczynskiT made their first contribution in https://github.com/caddyserver/caddy/pull/4930 * @chir4gm made their first contribution in https://github.com/caddyserver/caddy/pull/4932 * @lewandowski-stripe made their first contribution in https://github.com/caddyserver/caddy/pull/4937 * @abdusco made their first contribution in https://github.com/caddyserver/caddy/pull/4963 * @jedy made their first contribution in https://github.com/caddyserver/caddy/pull/4975 * @benburkert made their first contribution in https://github.com/caddyserver/caddy/pull/4980 * @WeidiDeng made their first contribution in https://github.com/caddyserver/caddy/pull/4964 * @david-szabo97 made their first contribution in https://github.com/caddyserver/caddy/pull/4935 * @Abirdcfly made their first contribution in https://github.com/caddyserver/caddy/pull/4986 * @flga made their first contribution in https://github.com/caddyserver/caddy/pull/5004 * @Malankar made their first contribution in https://github.com/caddyserver/caddy/pull/4972 **Full Changelog**: https://github.com/caddyserver/caddy/compare/v2.5.2...v2.6.0-beta.3 2022-09-05T22:07:59+00:00 FIR django4 2022-09-13T07:37:07+00:00 2022-09-13T07:37:07+00:00 TheHive 4.1.24 2022-09-13T07:48:11+00:00 ## [4.1.24](https://github.com/TheHive-Project/TheHive/milestone/95) (2022-09-12) **Fixed bugs:** - [Bug] All analyzers become unavailable when an analyzer is updated [\#2420](https://github.com/TheHive-Project/TheHive/issues/2420) 2022-09-13T07:48:11+00:00 MISP v2.4.162 2022-09-13T08:42:19+00:00  We are pleased to announce the immediate availability of [MISP v2.4.162](https://github.com/MISP/MISP/releases/tag/v2.4.162) with a new periodic notification system, workflow updates and many improvements. In addition to the MISP v2.4.162 release, [misp-guard](https://github.com/MISP/misp-guard) has been released which is a [mitmproxy](https://mitmproxy.org/) addon that inspects the events that MISP is attempting to synchronize with external MISP instances via `PUSH` or `PULL` and applies a set of customizable rules defined in a JSON file. This is a complementary tool to support MISP users having to interconenct MISP instances between highly sensitive networks. # Periodic notification system As of version 2.4.162, MISP includes a **periodic summary** feature allowing users to consult a summary based on a requested time-frame for data the user has access to. Currently, the summaries can be generated for 3 different periods: `daily`, `weekly` and `monthly` and then sent to all users that subscribed one of these periods. In addition to choose which period users want to subscribed to, they can also specify filtering options such as tags or distribution level to be used to generate the summary. The summary can be sent via email in addition to the User-Interface view.   For more information, check out the [Periodic summaries - Visualize summaries of MISP data](/2022/09/12/2022-09-12_periodic_notifications.html/) blog. # Workflow improvements - Added diagnostic support and support of arbitrary URL for webhook module. - New Microsoft teams module based on the webhook module. - New email notification module to send email to a list of MISP users including [Jinja templating](https://jinja.palletsprojects.com/en/3.1.x/). - Tag name can now be used in workflows. For more details about MISP Workflow, check out the [training materials](https://www.misp-project.org/misp-training/a.12-misp-workflows.pdf). # MISP core improvements - Allow option to delete tags on event sync prior to soft-delete tag implementation. - API/[Event:restSearch] Added option `event_tags` to filter for eventTag only. - API/RestSearch - Added support of `static` parameter to produce a static HTML output. - Syslog/logging for certain log entries vital information was omitted by the syslog. If no custom message is specifically set for the log entry, the change field is included. - Enforce UUIDs uniqueness on MISP data back-end. # Bugs fixed - [correlations] save the distribution state of the event before/after saving it, fixes #8528. - [attribute tags] removal broken, fixes #8567. - Class 'Folder' not found #8544. - Create unique SIDs for email attributes in NIDS export. Thanks to all the contributors and users reporting bugs to make the software better. As always, a detailed and [complete changelog is available](https://www.misp-project.org/Changelog.txt) with all the fixes, changes and improvements in MISP core. Many improvements in the MISP galaxy and especially the threat-actor galaxy. There is a detailed [changelog](https://www.misp-project.org/Changelog-misp-galaxy.txt). Improvement in the `false-positive` taxonomy and many other taxonomies. There is a detailed [changelog](https://www.misp-project.org/Changelog-misp-taxonomies.txt). Multiple objects were updated and added, for [more details](https://www.misp-project.org/Changelog-misp-objects.txt). 2022-09-13T08:42:19+00:00 DC3-MWCP 3.8.0 2022-09-16T12:29:56+00:00 2022-09-16T12:29:56+00:00 caddy v2.6.0-beta.5 2022-09-18T00:23:34+00:00 This release and beta.4 are mainly tests of our CI, but also contains a bunch of small fixes or enhancements, including, notably, the use of `sendfile` and other optimizations. See the release notes for beta.3 for everything else until 2.6.0 lands soon. Full release notes coming! ## Changelog * e6c58fdc caddyfile: Prevent infinite nesting on fmt (fix #4175) * 9ad0ebc9 caddyhttp: Add 'skip_log' var to omit request from logs (#4691) * b6cec378 caddyhttp: Add --debug flag to commands * 61c75f74 caddyhttp: Explicitly disallow multiple regexp matchers (#5030) * 00beec2e caddyhttp: Fix write header on responseRecorder * 74547f5b caddyhttp: Make metrics opt-in (#5042) * 61641828 caddyhttp: Support TLS key logging for debugging (#4808) * bffc2587 caddyhttp: Support configuring Server from handler provisioning (#4933) * 20d487be caddyhttp: Very minor optimization to path matcher * dd9813c6 caddyhttp: ensure ResponseWriterWrapper and ResponseRecorder use ReadFrom if the underlying response writer implements it. (#5022) * 258071d8 caddytls: Debug log on implicit tailscale error (#5041) * d35f618b caddytls: Error if placeholder is empty in 'ask' * 0b3161ae cmd: Customizable user agent (close #2795) * 1c9c8f6a cmd: Enhance some help text * 62b06853 cmd: Improve error message if config missing * 50748e19 core: Check error on ListenQUIC * e43b6d81 core: Variadic Context.Logger(); soft deprecation * 48d723c0 encode: Fix Accept-Ranges header; HEAD requests (#5039) * c19f2072 fileserver: Ignore EOF when browsing empty dir * c5df7bb6 go.mod: Update truststore * 8cc8f9fd httpcaddyfile: Add a couple more placeholder shortcuts (#5015) * a1ad20e4 httpcaddyfile: Fix bind when IPv6 is specified with network (#4950) * 754fe4f7 httpcaddyfile: Fix sorting of repeated directives * e338648f reverseproxy: Support repeated --to flags in command (#4693) * 9fe4f93b supplychain: publish signing cert, sbom, and signatures of sbom (#5027) 2022-09-18T00:23:34+00:00 caddy v2.6.0 2022-09-20T18:12:38+00:00 Caddy 2.6 ======== This is our biggest release since Caddy 2. Caddy 2 changed the way the world serves the Web. By providing an online config API, automatic HTTPS, unlimited extensibility, certificate automation at scale, modern protocols, sane defaults, and an unrivaled developer experience, we boldly raised the bar for web servers. Now with Caddy 2.6, we're doing it again. Caddy 2.6 is the first general-purpose web server to seamlessly enable the newly-standardized HTTP/3 protocol for all configurations by default. We've virtualized the file system so you can serve content from anywhere or anything. New event features let you observe and control Caddy's internals with custom actions. Caddy is more useful than ever for developers with its enhanced CLI tooling and features. And it's faster than ever with non-trivial performance improvements. We think you will love this release. [Watch the livestream](https://youtu.be/iJZqFpY2D14) Special dedication ------------------ This release is dedicated to the late Peter Eckersley, who passed away September 2, 2022. Peter is one of the brilliant minds behind Let's Encrypt; his work has benefited billions of people. I met Peter at the Let's Encrypt launch party in a little bar in San Francisco in 2015 and have never forgotten that occasion. He later co-authored a published research paper called [_Let’s Encrypt: An Automated Certificate Authority to Encrypt the Entire Web_](https://jhalderm.com/pub/papers/letsencrypt-ccs19.pdf), which highly espoused Caddy's ACME integration: "We hope to see other popular server software follow Caddy’s lead." We look forward to when other servers do that, and we hope to honor Peter's work and influence which will live on through his memory and the encrypted Web he made possible. --- [Sponsors](https://github.com/sponsors/mholt) -------- [ZeroSSL](https://zerossl.com) remains Caddy's executive sponsor. [We were thrilled](https://twitter.com/caddyserver/status/1559591673511813120) to welcome [Stripe](https://stripe.com) recently as an enterprise sponsor! Other notable sponsors include [AppCove](https://appcove.com/), [Dukaan](https://mydukaan.io/), [Suborbital](https://suborbital.dev), [Tailscale](https://tailscale.com), plus [Bubble](https://bubble.io/) and [GitHub](https://github.com) which both made generous one-time donations. We have many other vital sponsors and donors on which we also rely. [Our sponsors](https://github.com/sponsors/mholt) come from all over the world and include independent professionals, startups, and small companies -- and they are the absolute best. Thank you for making a more secure Web possible! _Personal note from Matt: Recent life upgrades mean that your sponsorships now sustain a family of 5 so that I can continue to maintain Caddy. Two years ago, I don't think I would have taken this risk because I'd need to find other work to provide for a family. Thank you for coming together as a professional community to make the Caddy project possible!_ We strongly recommend that companies who -- or companies whose customers -- use or benefit from Caddy [become a sponsor](https://github.com/sponsors/mholt) to ensure ongoing maintenance, priority development, private support, and more. Sponsorship tiers can be tailored to your requirements! Highlights ---------- :warning: **_Don't miss deprecations / breaking changes at the bottom. Notably, if you use metrics, you will now need to turn them on._** ### HTTP/3 is here (#4707) Caddy now enables [RFC 9114](https://datatracker.ietf.org/doc/rfc9114/)-compliant HTTP/3 by default. The `experimental_http3` option has graduated and been removed. We've removed another experimental option, `allow_h2c`, and individual HTTP versions (`h1 h2 h2c h3`) can now be toggled with the new `protocols` setting. Note that HTTP/3 utilizes the QUIC transport, which requires UDP. If your network or firewall configuration only allows TCP, HTTP/3 connections will fail and clients (should) fall back to HTTP/2. For servers with properly-configured UDP networks, HTTP/3 should "just work" for enabled clients. HTTP/3 clients can connect by reading Caddy's Alt-Svc header to know how to connect to Caddy via UDP. This header is now emitted automatically and by default. Other than that, there are no other changes needed to existing servers, as Caddy opens a separate UDP socket for HTTP/3. Our HTTP/3 server attempts to mitigate amplification and reflection attacks by requiring address validation when the server is under load. This adds one round-trip for clients, but is only done as a defensive measure when necessary. Serious thanks to @marten-seemann who builds and maintains the [quic-go](https://github.com/lucas-clemente/quic-go) library we depend on for this. (Go has not announced any plans to officially support or implement HTTP/3.) We expect numerous QUIC and HTTP/3 improvements to come as implementations and best practices mature with more production experience. ### Virtual file systems (#4909) Caddy's `file_server` module now supports _virtual_ file systems. We've replaced all hard-coded `os.Open()`, `os.Stat()`, etc. calls with Go's relatively new `io/fs` package, and introduced a new Caddy module namespace `caddy.fs` for implementations of such file systems. Some examples of what is possible: - Serve content from S3 or other blob/cloud storage services - Serve dynamically-generated content that "feels" static - [Embed your site directly into your `caddy` binary](https://github.com/mholt/caddy-embed) and serve it from memory - Serve content directly from an archive file (e.g. `.zip` or `.tar.gz`) - Load files from a database instead of disk Basically, instead of serving files from the local disk, you can have Caddy serve the "files" from somewhere or something else. The default is still the local file system. Note that this feature isn't limited to just Caddy's `file_server` module. Potentially any module that reads the local disk may benefit from using `caddy.fs` modules instead. I wrote [a module that lets you embed your site](https://github.com/mholt/caddy-embed) within your `caddy` binary -- wherever your server goes, your site goes! We encourage the community to implement and publish new file system modules for Caddy. (From an [early tweet](https://twitter.com/mholt6/status/1551434889358032897) there seems to be quite high demand.) ### Events (#4912 and #4984) Not surprisingly, many people prefer Caddy to automate certificates used with other software/services. Until now, there hasn't been a great way to know when Caddy has obtained or renewed a certificate (deferred in part by our opinion that certificate management should be baked into the software using the certificate in the first place). Cron jobs generally work for reloading new certificates into services because certificate expiry is _mostly_ predictable, but now there is a better way with one of our most requested features: events! We thought about events in general for a long time and discussed questions like, "What makes an event different from a log?" "Are events synchronous?" "Do self-initiated events get emitted before or after their code (are they past-tense or future-tense) -- or both? or neither (asynchronous)?" "What do we like from existing event systems?" "What do we wish event systems did differently?" While we think we have pretty good answers to these questions now, we won't be sure until we gather more production experience. For this reason, events are implemented as an experimental app module -- not as part of the core. (Remember, Caddy's core currently only loads config and sets up logging/storage.) This means that Caddy's core cannot emit events.[^1] So even though our event implementation may change, it is likely to be only slight and gradual changes; and we encourage anyone and everyone to start using events as soon as possible and to **give us your feedback**. We think we have the start of a great event system, but we need you to prove it! Caddy modules can emit events when interesting things happen. For example, the reverse proxy emits `healthy` and `unhealthy` events when backends go up and down. The TLS app emits `cert_obtaining`, `cert_obtained`, and `cert_failed` before and after obtaining a certificate or after the operation failed, respectively; and `cert_ocsp_revoked` after a certificate is discovered to be revoked by OCSP. There are several more events already, with even more to be added later. Events can have data associated with them. For example, `healthy`/`unhealthy` come with the address of the host; `cert_obtained` has the domain name, issuer, and storage path. You can access this from config in placeholders, e.g. `{event.data.identifier}`. Caddy modules can subscribe to events by specifying the name(s) of events to bind to, and the Caddy module ID(s) or namespace(s) to watch. When an event is emitted, it propagates from the module that emitted it up the provisioning heirarchy. This means that an event emitted by `http.handlers.reverse_proxy` will fire for `http.handlers` and `http` as well, similar to the DOM in HTML/JavaScript. Event handlers are invoked synchronously. We chose this for several reasons. First, despite how easy Go makes concurrency, there are many subtleties to concurrency in a server. Goroutines may be lightweight, but their operations might not be; and if event goroutines are starting more quickly than they are stopping, we either drop events arbitrarily or run out of memory/CPU. Also, we think one of the qualities that differentiates events from logs is the ability for an event to influence the emitting code's flow: a true "hook" in that sense. Instead of simply observing that something is happening (which is what a log tells you), you can influence its behavior. Maybe you want to run a command before a certificate is obtained to see if it _should_ be obtained. Or maybe you want to change how a TLS handshake is completed on-the-fly. Asynchronous event handlers cannot do this. For simple behavioral changes, synchronous events can be a powerful and useful tool for customizing your server. The new `event` app lets you easily configure subscriptions and event handlers. Event handling is modular, so you will need to plug in a module that does what you want: run a command, reload a service, make an HTTP request, or anything else! Because this feature is experimental and new, we don't yet know how people will be using it, so currently, Caddy does not ship with any event handler plugins. However, we're pretty sure based on feedback over the years that many of you would like to run commands on certain events (one of our top feature requests is to trigger a daemon reload after certificate renewals). So I went ahead and implemented an [`exec` event handler plugin](https://github.com/mholt/caddy-events-exec) that can run commands. We almost included it in Caddy's standard distribution, but out of an abundance of caution we decided to keep it a separate plugin for now until we learn more about real production use cases from experience. Here's an example of handling events. In JSON, you configure the `events` app: ```json { "apps": { "events": { "subscriptions": [ { "events": ["cert_obtained"], "handlers": [ { "handler": "exec", "command": "systemctl", "args": ["reload", "mydaemon"] } ] } ] } } } ``` or the equivalent Caddyfile global option: ``` { events { on cert_obtained exec systemctl reload mydaemon } } ``` It's that simple! Just make sure you have your event handler modules plugged in. **We hope you will provide feedback, report bugs, and request features related to events.** [^1]: Compilation fails with an import cycle. If Caddy core uses any feature of Caddy, it must also be in the core or another package not imported by any modules! ### Smarter path matching and rewriting (#4948) Is the URI path `/a/b/c` the same as `/a/b%2Fc`? What about `/a/b//c`? Turns out, _it depends._ What these questions illustrate is a famously frustrating problem, and has largely gone unsolved until now. All existing solutions I investigated in other products were unsatisfactory: - Nginx (and Caddy until now) always does path comparisons in unescaped/normalized space. This makes it impossible to route on literal escape sequences unless you double-encode your pattern, which violates specification. - Apache outright rejects valid[^2] HTTP requests containing encoded slashes. This behavior can either be disabled completely (creating a security problem known as unsafe paths) or tweaked to never decode encoded slashes (creating ambiguities when comparing against route patterns). - Laravel, like nginx, always decodes slashes, but routing such requests [mangles application data](https://github.com/laravel/framework/issues/22125) that contains slashes. The process of decoding a URI and collapsing slashes in the path is called normalization. Normalization has to occur for safe, reliable routing (imagine `//secret` bypassing auth checks configured for `/secret`), but at the same time, raw paths are sometimes needed to preserve application data (imagine a route `/bands/:name` which succeeds for `/bands/AC&2fDC` but fails for the normalized `/bands/AC/DC`). And it's not just routing; servers like Caddy often rewrite/manipulate paths. Because normalizing URIs creates a Many:1 mapping (there are multiple encoded forms of a single URI), normalizing is inherently lossy: the original input cannot be recovered with certainty, so we can't reconstruct the original or intended URI with complete fidelity. Other solutions with coarse on-off knobs can't balance both security and application correctness: it seems you have to trade one for the other. The crux of the problem seems to be that the server/framework/router doesn't know which parts of the path are application data and which parts are path components, so it just "plays it safe" and decodes the whole thing. I think Caddy's solution to this is quite novel. **Our solution is to interpret encoded characters and multiple slashes in a path pattern literally as a hint of the developer's intent.** For example, if you write a path matcher `/a/b/c`, it will still match `/a/b/c` and `/a/b%2Fc`. However, if your path matcher is `/a/b%2Fc`, Caddy will _only_ match `/a/b%2Fc`. This extends to wildcards with our new "escape-wildcard" feature: `/bands/%*/` will match `/bands/AC%2fDC` but `/bands/*/` won't. This works for multiple slashes too. If your path matcher uses `//`, Caddy will require the request path to contain those slashes literally at that position. We've also implemented this for prefix and suffix manipulations. For example, if you wanted to strip a prefix of `//prefix` from `//prefix/foo`, it will now work, whereas before it wouldn't because it would look at a fully-normalized URI. Essentially, we use the configured path pattern as a cue for whether to decode/merge a character or leave it raw when normalizing. This is a complex and subtle change, so please be sure to read the full PR in #4948 and the linked Laravel issue. It's very informative! [^2]: The "validity" of such a URI based on spec compliance is debatable. RFC 9110 says, "distinct resources SHOULD NOT be identified by HTTP URIs that are equivalent after normalization." ### HTTP 103 Early Hints (#4882 and #5006) HTTP Early Hints ([RFC 8297](https://datatracker.ietf.org/doc/rfc8297/)) is the effective successor to HTTP/2 Server Push. When 103 is emitted with relevant Link headers, web pages will load faster than normal. 1xx responses are precursors to the final response; clients must be able to support receiving multiple responses to a single request (nearly all modern clients do; and it almost certainly shouldn't break any HTTP/2 clients). Early hints are a great way to speed up page loads where the main content may take a while to generate (a slow DB query, for example) but the subresources can start being loaded right away. In those cases it is often beneficial to send early hints. Caddy can both originate and proxy 103 responses. To send early hints from Caddy, simply set the `Link` headers as the hints, then write the response with a 103 status code: ``` route /slow-pages/* { header Link "</style.css>; rel=preload; as=style" header Link "</script.js>; rel=preload; as=script" respond 103 } ``` Unlike normal responses, after writing HTTP 103, Caddy's middleware chain will _continue_ to execute and invoke the next handlers (for example, `reverse_proxy`) since 103 is not the final response. Multiple 103s can be sent. Caddy's reverse proxy also supports HTTP 103 responses, meaning that backends can send early hints and Caddy will proxy them to the client immediately as you'd expect., Note that browser support is still limited (only Chrome implements it at this time) and Caddy must be built with Go 1.19 (our builds use the latest Go version; but we still support Go 1.18 for now). Thank you to @dunglas with API Platform for contributing this feature to both Go and Caddy! ### Improved command line interface (#4565 and #4994) Caddy has always used Go's standard `flag` package for its CLI, which has served us quite well. However, recent improvements in the [Cobra](https://github.com/spf13/cobra) library make it possible for our CLI to gain worthwhile features without incurring a heavy dependency. The new `caddy manpage` command generates man pages, and the `caddy completion` command generates shell completions. Both are installed automatically as part of our official Linux packages, so your next `apt upgrade` (etc.) should take care of that. Additionally, short options (e.g. `-c`) are now supported. And if you typo a command, Caddy will helpfully suggest a correction (e.g. `caddy adpt` will suggest `caddy adapt`). Note that long-form flags must now use double-hyphen syntax (e.g. `--config`) even though the single-hyphen syntax (`-config`) was previously accepted. The standard library's flag parser treats `-` and `--` the same, but Cobra's does not. Our online documentation has always used `--` for flags, so we do not consider this a breaking change, but it's good to be aware of this change if you're used to how Go's parser works. Very many thanks to @mohammed90 for contributing these features! ### New `caddy respond` command (#4870) For rapid development needing a local HTTP server, the [`caddy respond` command](https://caddyserver.com/docs/command-line#caddy-respond) might be just what you need: hard-coded HTTP responses for one or more servers so that you can effortlessly have a custom HTTP endpoint to test with. A plain `caddy respond` command will listen on a random port and reply with HTTP 200. (The port or address is printed to the terminal for you.) You can set a custom status code like `caddy respond 401` or a custom body like `caddy respond "Hello world!"` -- or both: `caddy respond --status 401 "Hello world!"` Or you can pipe in a response body, for example serving a maintenance page: ```bash $ cat maintenance.html | caddy respond --status 503 --header "Content-Type: text/html; charset=utf-8" ``` You can even spin up multiple servers at once and use basic template features to configure each server with a different response: ```bash $ echo "I'm server {{.N}} on port {{.Port}}" | caddy respond --listen :2000-2004 Server address: [::]:2000 Server address: [::]:2001 Server address: [::]:2002 Server address: [::]:2003 Server address: [::]:2004 $ curl 127.0.0.1:2002 I'm server 2 on port 2002 ``` You can debug HTTP clients easier by enabling access logging with the `--access-log` flag. The `--header` flag can be used multiple times to set custom HTTP headers, and `--debug` enables debug mode for more verbose logging. We hope you find this feature useful! ### Multiple dynamic upstream sources (5fb5b81) In Caddy 2.5(.1) we introduced dynamic upstreams, which allow you to configure the `reverse_proxy` to get the list of backends on-the-fly during requests. This very popular feature's development was sponsored by Stripe, who we are thrilled to welcome as an enterprise sponsor. Stripe uses Caddy heavily for their internal systems, and for greater redundancy they need to be able to fail over to secondary upstreams if a primary cluster is down. This is where the new `multi` dynamic upstreams module comes in. Now you can configure, for example, two SRV lookups for aggregated results: ```json { "handler": "reverse_proxy", "dynamic_upstreams": { "source": "multi", "sources": [ { "source": "srv", "name": "primary" }, { "source": "srv", "name": "secondary" } ] } } ``` This appends the backends returned from the secondary SRV lookup to the results of the primary SRV lookup (order preserved). To implement failover, simply use the `first` load balancing policy which chooses the first available upstream. ### Configurable shutdown delay (#4906) A shutdown can now be scheduled for a later time using the `shutdown_delay` option. This is useful for giving advance notice to health checkers that this server will be closing soon. The shutdown delay happens _before_ the grace period where new connections are no longer accepted and existing ones are gracefully closed. During the shutdown delay, the server operates normally with the exception of the value of two placeholders. During the delay: - `{http.shutting_down}` placeholder equals `true`. - `{http.time_until_shutdown}` returns the duration that remains until server close. This allows health check endpoints to announce that they will soon be going down so that this instance can be moved out of the rotation or a replacement instance can be spun up in the meantime. For example: ``` { shutdown_delay 10s } example.com { handle /health-check { @goingDown `{http.shutting_down}` respond @goingDown "Bye-bye in {http.time_until_shutdown}" 503 respond 200 } } ``` By the way, the syntax of that `@goingDown` named matcher is new in 2.6: if a named matcher consists only of a CEL expression string, the type of matcher can be omitted; i.e. what you see above is equivalent to `@goingDown expression "{http.shutting_down} == true"`. (A shutdown is defined as a config unload where there is no new config to load, or the new config does not have a server configured at the same address as the current server. In other words, a shutdown of a server means a particular HTTP socket will be closed.) Speaking of grace periods, config changes no longer block while waiting on servers' grace periods. This means faster, more responsive config reloads; just beware that, depending on the length of your grace period, your reload command or config API request may return before the old servers have completely finished shutting down. ### Faster FastCGI transport (#4978) PHP apps, rejoice! The round-trip between Caddy and php-fpm just got a lot faster. Thanks to contributions by @WeidiDeng, the FastCGI transport has been rewritten to be more efficient. This is some of the oldest and most unique code in Caddy's code base. When Caddy was rewritten for v2 in 2019, _everything_ was rewritten or refactored... except this, the FastCGI transport. This is the first time this part of the code has been improved since it was first implemented[^3] in 2014! During tests, profiling showed the new code spends 86% less CPU time in GC (`gcDrain`) thanks to significantly fewer allocations. This is largely in part due to pooling buffers, which required a non-trivial refactoring to implement.  A very rough benchmark using `php_info()` yielded a 25% increase in requests per second. Before the rewrite, Caddy almost always performed worse than nginx even with `fastcgi_keep_conn off`. Our new code performs competitively with nginx, and in some tests Caddy even outperformed nginx with `fastcgi_keep_conn on` -- and we have not implemented connection pooling/reuse into the new transport yet. Because every setup is different, your actual results will vary. In general though, you can expect busy servers to handle PHP faster. [^3]: I didn't know how to write a FastCGI client back then (I'm still too scared to do much with it myself); Go's standard library implements only the responder role, not the web server (client). Fortunately there was a [random repository on BitBucket](http://bitbucket.org/PinIdea/fcgi_client) that was forked from [a random repository on Google Code written in 2012](https://code.google.com/archive/p/go-fastcgi-client/) that modified the Go std lib's `fcgi` package. It was rough around the edges, but with a little TLC we got it to do what we needed. The copyright had the name Junqing Tan in it, which we still retain in our source code to this day. ### Faster file server (#5022) In a patch contributed by @flga, we've reduced copying between buffers and even eliminated it altogether in some cases using [`sendfile(2)`](https://man7.org/linux/man-pages/man2/sendfile.2.html). This has shown to have a 25-50% performance boost. It's automatic and no configuration is required to benefit. In [some tests](https://blog.tjll.net/reverse-proxy-hot-dog-eating-contest-caddy-vs-nginx/), Caddy's new defaults are even faster than optimized nginx. Static files over 512 bytes being served over plaintext HTTP sockets may now be served directly by the Linux kernel, which is much faster than copying the file to user-space. Static files are faster over HTTPS, too. In addition to sendfile (which we can't[^4] use over TLS), we now utilize the `io.ReaderFrom` interface to reuse existing buffers and further reduce copying within user space. Our tests show that this significantly enhances performance even over TLS. [^4]: This is possible with kTLS, but [the Go standard library doesn't support it](https://github.com/golang/go/issues/44506) and it's [a bit tedious](https://words.filippo.io/playing-with-kernel-tls-in-linux-4-13-and-go/) to make it work, although @FiloSottile was successful with his [spike code](https://github.com/FiloSottile/go/commit/dbed9972d9947eb0001e9f5b639e0df05acec8bd). ### Signed release assets Thanks to heroic efforts by @mohammed90, our [GitHub release](https://github.com/caddyserver/caddy/releases) assets are now signed and certified. Mohammed [wrote an excellent Twitter thread](https://twitter.com/MohammedSahaf/status/1572022375247663105) explaining the whole thing better than I can here! So if you're wondering why the number of assets shot from 28 to 134... that's why. Other notable enhancements -------------------------- - More efficient `query` matcher. (04a14ee37ac6192d734518fa9082d6eb93971bc6) - A new Caddyfile placeholder `{cookie.*}` grants easy access to cookie values. (#5001) - Windows service integration: Caddy can now be controlled with `sc.exe`. (#4790) - Replace `net.IP` type with leaner `netip.Addr` type. (#4966) - Caddyfile-configurable OCSP check interval with `ocsp_interval` global option. (#4980) - The reverse proxy now supports `retry_count` as an alternative to `try_duration`; i.e. try backends up to a fixed number of times, rather than up to a time limit. (#4756) - The reverse proxy closes both ends of "hijacked" connections when shutting down or reloading. (#4895) - The reverse proxy gracefully closes both ends of websocket connections on shutdown or reload. (#4895) - The reverse proxy emits metrics regarding the health of upstreams. (#4935) - The reverse-proxy command can accept repeated --to flags and load balance. (#4693) - The reverse proxy's HTTP transport now supports distinct read and write timeouts. (#4905) - Simpler and more reliable config reloads on Linux with SO_REUSEPORT. (#4705) - Templates can access reverse proxy responses if used within `handle_response`. (#4871) - Builds now include git revision information when using `go build`. (#4931) - The file matcher (and `try_files`) now supports glob patterns. (#4993) - Named matchers in the Caddyfile can use CEL expressions without specifying `expression` first. (#4976) - The FastCGI transport can now capture and print stderr output. (#5004) - Listeners can be provided by plugins, enabling new network types. (#5002) - Caddy can write TLS secrets to a file for debugging purposes. (#4808) - Sites declared as `http://` in the Caddyfile will no longer be overridden by auto-HTTPS redirects. (#5051) - Config reloads no longer block while the prior servers are shutting down. (#5043) ## :warning: Deprecations/breaks - **Metrics are now opt-in.** Due to [multiple confirmed reports](https://github.com/caddyserver/caddy/issues/4644) of non-trivial performance regressions with metrics, we are making them opt-in. (Technically, this is not a breaking change, as Caddy will still function normally and your old configs won't be rejected -- but your metrics will stop being produced unless you enable them.) If you rely on metrics, you can enable them globally in the Caddyfile with global options: ``` { servers { metrics } } ``` As with other server-scoped global options, you can selectively customize which servers to enable metrics (e.g. `servers :8080`). _Note that this change is experimental and might be temporary: if we can reduce the performance impact or find a better way to enable and configure metrics, this could change._ - The signature of `caddy.Context.Logger()` has changed, but in a backwards-compatible way. Modules use this function to obtain a logger they can use within Caddy; until now, modules had to pass themselves in as an argument. Now, the context can figure out which module to associate the logger with, so the sole parameter has been made variadic. It may be removed in the future. Plugins should update their code to not pass in a pointer to themselves. - Basic auth deprecates `scrypt` because it was seldom used and error-prone; use `bcrypt` instead (#4720) - Several changes to experimental `servers` global options: removed the `protocol` sub-option, which has been replaced with the `protocols` sub-option; `strict_sni_host` is its own separate sub-option; `allow_h2c` and `experimental_http3` have been removed, as both H2C (`h2c`) and HTTP/3 (`h3`) can be toggled in `protocols` (HTTP/3 is now enabled by default and no longer experimental). As a reminder, features, parameters, and APIs marked as experimental are subject to change or removal. We strive to keep breaking changes of stable features to a minimum and gracefully deprecate whenever possible with emphasis in release notes, warnings in logs, etc. Most breaking changes are motivated or necessitated by bugs/regressions, security, or wrong/unclear documentation. Thank you --------- As usual, a huge thank-you to all our [sponsors](https://github.com/sponsors/mholt) and those who contributed both code and feedback. We also acknowledge the many people who participated in discussions and helped others on the forum. Thank you! ## New Contributors * @WingLim made their first contribution in https://github.com/caddyserver/caddy/pull/4790 * @WilczynskiT made their first contribution in https://github.com/caddyserver/caddy/pull/4930 * @chir4gm made their first contribution in https://github.com/caddyserver/caddy/pull/4932 * @lewandowski-stripe made their first contribution in https://github.com/caddyserver/caddy/pull/4937 * @abdusco made their first contribution in https://github.com/caddyserver/caddy/pull/4963 * @jedy made their first contribution in https://github.com/caddyserver/caddy/pull/4975 * @benburkert made their first contribution in https://github.com/caddyserver/caddy/pull/4980 * @WeidiDeng made their first contribution in https://github.com/caddyserver/caddy/pull/4964 * @david-szabo97 made their first contribution in https://github.com/caddyserver/caddy/pull/4935 * @Abirdcfly made their first contribution in https://github.com/caddyserver/caddy/pull/4986 * @flga made their first contribution in https://github.com/caddyserver/caddy/pull/5004 * @Malankar made their first contribution in https://github.com/caddyserver/caddy/pull/4972 * @stapelberg made their first contribution in https://github.com/caddyserver/caddy/pull/4950 * @parrotmac made their first contribution in https://github.com/caddyserver/caddy/pull/4693 * @Manouchehri made their first contribution in https://github.com/caddyserver/caddy/pull/4808 **Full Changelog**: https://github.com/caddyserver/caddy/compare/v2.5.2...v2.6.0 --- 2022-09-20T18:12:38+00:00 rocksdb v7.6.0 2022-09-20T19:20:37+00:00 ### New Features * Added `prepopulate_blob_cache` to ColumnFamilyOptions. If enabled, prepopulate warm/hot blobs which are already in memory into blob cache at the time of flush. On a flush, the blob that is in memory (in memtables) get flushed to the device. If using Direct IO, additional IO is incurred to read this blob back into memory again, which is avoided by enabling this option. This further helps if the workload exhibits high temporal locality, where most of the reads go to recently written data. This also helps in case of the remote file system since it involves network traffic and higher latencies. * Support using secondary cache with the blob cache. When creating a blob cache, the user can set a secondary blob cache by configuring `secondary_cache` in LRUCacheOptions. * Charge memory usage of blob cache when the backing cache of the blob cache and the block cache are different. If an operation reserving memory for blob cache exceeds the avaible space left in the block cache at some point (i.e, causing a cache full under `LRUCacheOptions::strict_capacity_limit` = true), creation will fail with `Status::MemoryLimit()`. To opt in this feature, enable charging `CacheEntryRole::kBlobCache` in `BlockBasedTableOptions::cache_usage_options`. * Improve subcompaction range partition so that it is likely to be more even. More evenly distribution of subcompaction will improve compaction throughput for some workloads. All input files' index blocks to sample some anchor key points from which we pick positions to partition the input range. This would introduce some CPU overhead in compaction preparation phase, if subcompaction is enabled, but it should be a small fraction of the CPU usage of the whole compaction process. This also brings a behavier change: subcompaction number is much more likely to maxed out than before. * Add CompactionPri::kRoundRobin, a compaction picking mode that cycles through all the files with a compact cursor in a round-robin manner. This feature is available since 7.5. * Provide support for subcompactions for user_defined_timestamp. * Added an option `memtable_protection_bytes_per_key` that turns on memtable per key-value checksum protection. Each memtable entry will be suffixed by a checksum that is computed during writes, and verified in reads/compaction. Detected corruption will be logged and with corruption status returned to user. * Added a blob-specific cache priority level - bottom level. Blobs are typically lower-value targets for caching than data blocks, since 1) with BlobDB, data blocks containing blob references conceptually form an index structure which has to be consulted before we can read the blob value, and 2) cached blobs represent only a single key-value, while cached data blocks generally contain multiple KVs. The user can specify the new option `low_pri_pool_ratio` in `LRUCacheOptions` to configure the ratio of capacity reserved for low priority cache entries (and therefore the remaining ratio is the space reserved for the bottom level), or configuring the new argument `low_pri_pool_ratio` in `NewLRUCache()` to achieve the same effect. ### Public API changes * Removed Customizable support for RateLimiter and removed its CreateFromString() and Type() functions. * `CompactRangeOptions::exclusive_manual_compaction` is now false by default. This ensures RocksDB does not introduce artificial parallelism limitations by default. * Tiered Storage: change `bottommost_temperture` to `last_level_temperture`. The old option name is kept only for migration, please use the new option. The behavior is changed to apply temperature for the `last_level` SST files only. * Added a new experimental ReadOption flag called optimize_multiget_for_io, which when set attempts to reduce MultiGet latency by spawning coroutines for keys in multiple levels. ### Bug Fixes * Fix a bug starting in 7.4.0 in which some fsync operations might be skipped in a DB after any DropColumnFamily on that DB, until it is re-opened. This can lead to data loss on power loss. (For custom FileSystem implementations, this could lead to `FSDirectory::Fsync` or `FSDirectory::Close` after the first `FSDirectory::Close`; Also, valgrind could report call to `close()` with `fd=-1`.) * Fix a bug where `GenericRateLimiter` could revert the bandwidth set dynamically using `SetBytesPerSecond()` when a user configures a structure enclosing it, e.g., using `GetOptionsFromString()` to configure an `Options` that references an existing `RateLimiter` object. * Fix race conditions in `GenericRateLimiter`. * Fix a bug in `FIFOCompactionPicker::PickTTLCompaction` where total_size calculating might cause underflow * Fix data race bug in hash linked list memtable. With this bug, read request might temporarily miss an old record in the memtable in a race condition to the hash bucket. * Fix a bug that `best_efforts_recovery` may fail to open the db with mmap read. * Fixed a bug where blobs read during compaction would pollute the cache. * Fixed a data race in LRUCache when used with a secondary_cache. * Fixed a bug where blobs read by iterators would be inserted into the cache even with the `fill_cache` read option set to false. * Fixed the segfault caused by `AllocateData()` in `CompressedSecondaryCache::SplitValueIntoChunks()` and `MergeChunksIntoValueTest`. * Fixed a bug in BlobDB where a mix of inlined and blob values could result in an incorrect value being passed to the compaction filter (see #10391). * Fixed a memory leak bug in stress tests caused by `FaultInjectionSecondaryCache`. ### Behavior Change * Added checksum handshake during the copying of decompressed WAL fragment. This together with #9875, #10037, #10212, #10114 and #10319 provides end-to-end integrity protection for write batch during recovery. * To minimize the internal fragmentation caused by the variable size of the compressed blocks in `CompressedSecondaryCache`, the original block is split according to the jemalloc bin size in `Insert()` and then merged back in `Lookup()`. * PosixLogger is removed and by default EnvLogger will be used for info logging. The behavior of the two loggers should be very similar when using the default Posix Env. * Remove [min|max]_timestamp from VersionEdit for now since they are not tracked in MANIFEST anyway but consume two empty std::string (up to 64 bytes) for each file. Should they be added back in the future, we should store them more compactly. * Improve universal tiered storage compaction picker to avoid extra major compaction triggered by size amplification. If `preclude_last_level_data_seconds` is enabled, the size amplification is calculated within non last_level data only which skip the last level and use the penultimate level as the size base. * If an error is hit when writing to a file (append, sync, etc), RocksDB is more strict with not issuing more operations to it, except closing the file, with exceptions of some WAL file operations in error recovery path. * A `WriteBufferManager` constructed with `allow_stall == false` will no longer trigger write stall implicitly by thrashing until memtable count limit is reached. Instead, a column family can continue accumulating writes while that CF is flushing, which means memory may increase. Users who prefer stalling writes must now explicitly set `allow_stall == true`. * Add `CompressedSecondaryCache` into the stress tests. * Block cache keys have changed, which will cause any persistent caches to miss between versions. ### Performance Improvements * Instead of constructing `FragmentedRangeTombstoneList` during every read operation, it is now constructed once and stored in immutable memtables. This improves speed of querying range tombstones from immutable memtables. * When using iterators with the integrated BlobDB implementation, blob cache handles are now released immediately when the iterator's position changes. * MultiGet can now do more IO in parallel by reading data blocks from SST files in multiple levels, if the optimize_multiget_for_io ReadOption flag is set. 2022-09-20T19:20:37+00:00 caddy v2.6.1 2022-09-21T22:21:28+00:00 Hotfix for unix sockets, the `encode` handler, and the `caddy file-server` command. Please see [the release notes for v2.6.0](https://github.com/caddyserver/caddy/releases/tag/v2.6.0) for other important information if you're coming from < 2.6! ## Changelog * 1426c97d core: Reuse unix sockets (UDS) and don't try to serve HTTP/3 over UDS (#5063) * 44ad0ced encode: don't WriteHeader unless called (#5060) * beb7dcbf fileserver: Reinstate --debug flag 2022-09-21T22:21:28+00:00 MISP v2.4.163 2022-09-26T14:12:21+00:00  We are pleased to announce the immediate availability of [MISP v2.4.163](https://github.com/MISP/MISP/releases/tag/v2.4.163) with an updated [periodic notification system](https://www.misp-project.org/2022/09/12/2022-09-12_periodic_notifications.html/) and many improvements. # Updated periodic notification system - A new option has been added to set the number of days for the trending calculation. - New correlation are now showed in the periodic notification. - Only the top 10 MITRE ATT&CK techniques are displayed and sorted by number of occurrences. - Layout has been improved in the UI and also in the static email rendering. - Only show data in the chart for tags having changes over time. For more information, check out the [Periodic summaries - Visualize summaries of MISP data](https://www.misp-project.org/2022/09/12/2022-09-12_periodic_notifications.html/) blog. # Fixes - MISP [OpenAPI description file](https://www.misp-project.org/openapi/) has been improved. - [community] Clarification concerning the NATO process. - [ssdeep] Check if the ssdeep contains newline characters. - Many code clean-up and speed-up included. - Improvements and bugs fixed in the correlation engine. - Many bugs fixed. Thanks to all the contributors and users reporting bugs to make the software better. As always, a detailed and [complete changelog is available](https://www.misp-project.org/Changelog.txt) with all the fixes, changes and improvements in MISP core. # misp-stix v.2.4.163 misp-stix has been released too and now in-line with the MISP release schedule. The full [changelog is available](https://www.misp-project.org/Changelog-misp-stix.txt). Many improvements in the MISP galaxy and especially the threat-actor galaxy, [360.net Threat Actors](https://www.misp-project.org/galaxy.html#_360_net_threat_actors) added. There is a detailed [changelog](https://www.misp-project.org/Changelog-misp-galaxy.txt). New `financial` taxonomy and many other taxonomies. There is a detailed [changelog](https://www.misp-project.org/Changelog-misp-taxonomies.txt). Multiple objects were updated and added, for [more details](https://www.misp-project.org/Changelog-misp-objects.txt). Various fixes in [misp-modules](https://misp.github.io/misp-modules/) for more [details](https://www.misp-project.org/Changelog-misp-modules.txt). 2022-09-26T14:12:21+00:00 dalton v3.2.5 2022-09-27T23:51:25+00:00 - Faster compile time for Zeek - Fixed bug where Zeek wouldn't run if there were not Zeek scripts 2022-09-27T23:51:25+00:00 maltrail 0.50 2022-09-30T22:11:05+00:00 Start-of-month release 2022-09-30T22:11:05+00:00 MONARC v2.12.3 2022-10-05T15:18:41+00:00 ### New features: * Link multiple specific models per client. * Added the possibility to import assets in the library from CSV files. 2022-10-05T15:18:41+00:00 rocksdb v7.7.2 2022-10-07T16:28:46+00:00 ## 7.7.2 (10/05/2022) ### Bug Fixes * Fixed a bug in iterator refresh that was not freeing up SuperVersion, which could cause excessive resource pinniung (#10770). * Fixed a bug where RocksDB could be doing compaction endlessly when allow_ingest_behind is true and the bottommost level is not filled (#10767). ### Behavior Changes * Sanitize min_write_buffer_number_to_merge to 1 if atomic flush is enabled to prevent unexpected data loss when WAL is disabled in a multi-column-family setting (#10773). ## 7.7.1 (09/26/2022) ### Bug Fixes * Fixed an optimistic transaction validation bug caused by DBImpl::GetLatestSequenceForKey() returning non-latest seq for merge (#10724). * Fixed a bug in iterator refresh which could segfault for DeleteRange users (#10739). ## 7.7.0 (09/18/2022) ### Bug Fixes * Fixed a hang when an operation such as `GetLiveFiles` or `CreateNewBackup` is asked to trigger and wait for memtable flush on a read-only DB. Such indirect requests for memtable flush are now ignored on a read-only DB. * Fixed bug where `FlushWAL(true /* sync */)` (used by `GetLiveFilesStorageInfo()`, which is used by checkpoint and backup) could cause parallel writes at the tail of a WAL file to never be synced. * Fix periodic_task unable to re-register the same task type, which may cause `SetOptions()` fail to update periodical_task time like: `stats_dump_period_sec`, `stats_persist_period_sec`. * Fixed a bug in the rocksdb.prefetched.bytes.discarded stat. It was counting the prefetch buffer size, rather than the actual number of bytes discarded from the buffer. * Fix bug where the directory containing CURRENT can left unsynced after CURRENT is updated to point to the latest MANIFEST, which leads to risk of unsync data loss of CURRENT. * Update rocksdb.multiget.io.batch.size stat in non-async MultiGet as well. * Fix a bug in key range overlap checking with concurrent compactions when user-defined timestamp is enabled. User-defined timestamps should be EXCLUDED when checking if two ranges overlap. * Fixed a bug where the blob cache prepopulating logic did not consider the secondary cache (see #10603). * Fixed the rocksdb.num.sst.read.per.level, rocksdb.num.index.and.filter.blocks.read.per.level and rocksdb.num.level.read.per.multiget stats in the MultiGet coroutines * Fix a bug in io_uring_prep_cancel in AbortIO API for posix which expects sqe->addr to match with read request submitted and wrong paramter was being passed. * Fixed a regression in iterator performance when the entire DB is a single memtable introduced in #10449. The fix is in #10705 and #10716. * Fix a bug in io_uring_prep_cancel in AbortIO API for posix which expects sqe->addr to match with read request submitted and wrong paramter was being passed. * Fixed a regression in iterator performance when the entire DB is a single memtable introduced in #10449. The fix is in #10705 and #10716. ### Public API changes * Add `rocksdb_column_family_handle_get_id`, `rocksdb_column_family_handle_get_name` to get name, id of column family in C API * Add a new stat rocksdb.async.prefetch.abort.micros to measure time spent waiting for async prefetch reads to abort ### Java API Changes * Add CompactionPriority.RoundRobin. * Revert to using the default metadata charge policy when creating an LRU cache via the Java API. ### Behavior Change * DBOptions::verify_sst_unique_id_in_manifest is now an on-by-default feature that verifies SST file identity whenever they are opened by a DB, rather than only at DB::Open time. * Right now, when the option migration tool (OptionChangeMigration()) migrates to FIFO compaction, it compacts all the data into one single SST file and move to L0. This might create a problem for some users: the giant file may be soon deleted to satisfy max_table_files_size, and might cayse the DB to be almost empty. We change the behavior so that the files are cut to be smaller, but these files might not follow the data insertion order. With the change, after the migration, migrated data might not be dropped by insertion order by FIFO compaction. * When a block is firstly found from `CompressedSecondaryCache`, we just insert a dummy block into the primary cache and don’t erase the block from `CompressedSecondaryCache`. A standalone handle is returned to the caller. Only if the block is found again from `CompressedSecondaryCache` before the dummy block is evicted, we erase the block from `CompressedSecondaryCache` and insert it into the primary cache. * When a block is firstly evicted from the primary cache to `CompressedSecondaryCache`, we just insert a dummy block in `CompressedSecondaryCache`. Only if it is evicted again before the dummy block is evicted from the cache, it is treated as a hot block and is inserted into `CompressedSecondaryCache`. * Improved the estimation of memory used by cached blobs by taking into account the size of the object owning the blob value and also the allocator overhead if `malloc_usable_size` is available (see #10583). * Blob values now have their own category in the cache occupancy statistics, as opposed to being lumped into the "Misc" bucket (see #10601). * Change the optimize_multiget_for_io experimental ReadOptions flag to default on. ### New Features * RocksDB does internal auto prefetching if it notices 2 sequential reads if readahead_size is not specified. New option `num_file_reads_for_auto_readahead` is added in BlockBasedTableOptions which indicates after how many sequential reads internal auto prefetching should be start (default is 2). * Added new perf context counters `block_cache_standalone_handle_count`, `block_cache_real_handle_count`,`compressed_sec_cache_insert_real_count`, `compressed_sec_cache_insert_dummy_count`, `compressed_sec_cache_uncompressed_bytes`, and `compressed_sec_cache_compressed_bytes`. * Memory for blobs which are to be inserted into the blob cache is now allocated using the cache's allocator (see #10628 and #10647). * HyperClockCache is an experimental, lock-free Cache alternative for block cache that offers much improved CPU efficiency under high parallel load or high contention, with some caveats. As much as 4.5x higher ops/sec vs. LRUCache has been seen in db_bench under high parallel load. * `CompressedSecondaryCacheOptions::enable_custom_split_merge` is added for enabling the custom split and merge feature, which split the compressed value into chunks so that they may better fit jemalloc bins. ### Performance Improvements * Iterator performance is improved for `DeleteRange()` users. Internally, iterator will skip to the end of a range tombstone when possible, instead of looping through each key and check individually if a key is range deleted. * Eliminated some allocations and copies in the blob read path. Also, `PinnableSlice` now only points to the blob value and pins the backing resource (cache entry or buffer) in all cases, instead of containing a copy of the blob value. See #10625 and #10647. * In case of scans with async_io enabled, few optimizations have been added to issue more asynchronous requests in parallel in order to avoid synchronous prefetching. * `DeleteRange()` users should see improvement in get/iterator performance from mutable memtable (see #10547). 2022-10-07T16:28:46+00:00 dnstwist 20221008 2022-10-08T16:33:43+00:00 2022-10-08T16:33:43+00:00 MISP v2.4.164 2022-10-10T14:45:54+00:00  We are pleased to announce the immediate availability of [MISP v2.4.164](https://github.com/MISP/MISP/releases/tag/v2.4.164) with a new tag relationship features, many improvements and a security fix. # New tag relationship feature Relationship can now be added to any attribute tag or event tag. This works with tags and galaxy clusters. The new feature is available in event view. The tag relationship feature is also exposed in the API under the endpoint `/tags/modifyTagRelationship/[scope]/[id]` where `scope` is the attribute/event and `id` is the id of the EventTag / AttributeTag object. # Improvements and bug fixes - [periodic_report] Added security recommendations section showing course of actions related to attack techniques. - [workflow] add support for `local` and `relationship` in workflow. - [API/galaxyCluster/restSearch] Allow multiple filtering conditions to be used at once. - [EventGraph] Added entity comment in the graph as tooltip and support of comment in searches. - [UI] Many improvements and optimisation. # CVE-2022-42724 This release fix a security vulnerabilities ([CVE-2022-42724](https://cvepremium.circl.lu/cve/CVE-2022-42724)) which allows org admin to discover role names which should have been restricted to site admin. We strongly recommend MISP administrator to update to this latest version. For a more detailed changelog, please see the online [Changelog](https://www.misp-project.org/Changelog.txt). 2022-10-10T14:45:54+00:00 osquery 5.6.0 2022-10-10T16:57:28+00:00 Draft! 2022-10-10T16:57:28+00:00 dnstwist 20221011 2022-10-11T10:22:48+00:00 2022-10-11T10:22:48+00:00 OpenTAXII 0.9.3 2022-10-11T13:46:48+00:00 Changelog ========= 0.9.3 (2022-10-11) ------------------ * Add public write support. 0.9.2 (2022-08-26) ------------------ * Improve readability and navigation of docs (`#238 <https://github.com/eclecticiq/OpenTAXII/pull/238>`_ thanks `@zed-eiq <https://github.com/zed-eiq>`_ for the improvement). 0.9.1 (2022-07-11) ------------------ * Implement `raise_unauthorized` for taxii2, this was missing and lead to 500 errors. 0.9.0 (2022-06-13) ------------------ * Allow custom properties. This can be disabled by config option ``allow_custom_properties`` 0.8.0 (2022-06-05) ------------------ * Nest details inside taxii2 job and allow counts without details 0.7.0 (2022-05-27) ------------------ * Nest taxii2 endpoints under `/taxii2/` 0.6.0 (2022-05-25) ------------------ * Add `public_discovery` option to taxii2 config * Add support for publicly readable taxii 2 api roots 0.5.0 (2022-05-24) ------------------ * Add support for publicly readable taxii 2 collections 0.4.0 (2022-05-20) ------------------ * Move next_param handling into `OpenTAXII2PersistenceAPI` 0.3.0 (2022-04-13) ------------------ * Implement taxii2.1 support 0.3.0a4 (2022-04-13) -------------------- * Merge changes from 0.2.4 maintenance release 0.3.0a3 (2022-01-21) -------------------- * Fix bug that prevented booting with only taxii1 config (`#217 <https://github.com/eclecticiq/OpenTAXII/issues/217>`_ thanks `@azurekid <https://github.com/azurekid>`_ for the report) 0.3.0a2 (2021-12-27) -------------------- * Merge changes from 0.2.3 maintenance release 0.3.0a1 ------- * Add python 3.10 support 0.3.0a0 ------- * Enablement for future taxii2 implementation * Fix documentation build issues 0.2.4 (2022-04-13) ------------------ * Make sure werkzeug <2.1 and >=2.1 work correctly with auth system 0.2.3 (2021-12-22) ------------------ * Fix bug in multithreaded use of sqlite (`#210 <https://github.com/eclecticiq/OpenTAXII/issues/210>`_ thanks `@rohits144 <https://github.com/rohits144>`_ for the report) 0.2.2 (2021-11-05) ------------------ * Fix readthedocs build 0.2.1 (2021-11-03) ------------------ * Add tests for python 3.6, 3.7, 3.8, 3.9, pypy * Add tests for sqlite, mysql, mariadb, postgresql * Fix bug that broke ``delete_content_blocks`` when using mysql on sqlalchemy 1.3 * Docs: Add db schema diagram * Docs: Clarify how to get default data in a default (development) docker instance * Fix implicit routing in TAXII 1.1 Inboxes * Update jwt usage to pyjwt >= 2.0 (thanks `@SanyaKapoor <https://github.com/SanyaKapoor>`_) 0.2.0 (2020-06-30) ------------------ * Enforce UTC usage in datetime fields in SQL DB Persistence API. * `Fix for #114 <https://github.com/eclecticiq/OpenTAXII/issues/114>`_: reintroduce ``opentaxii-create-account`` CLI command. * `Fix for #153 <https://github.com/eclecticiq/OpenTAXII/issues/152>`_: check if user can modify a collection before advertising it over inbox service. * Multiple coding style fixes. * Various documentation updates. 0.1.12 (2019-03-06) ------------------- * Remove unnecessary print statements. 0.1.11 (2019-02-13) ------------------- * Make JSON logging consistent when the application is run via Gunicorn. * Set ``acceptable_destination`` key in status details instead of extended headers * Allow passing ``engine_parameters`` to ``SQLDatabaseAPI`` for those who want to customize SQLAlchemy engine parameters. * Require recent version of ``lxml`` for security reasons. * Various test and Docker infrastructure improvements. 0.1.10 (2018-06-03) ------------------- * Replace separate service/collection/account creation process with single ``opentaxii-sync-data`` CLI command. * Persistence and Auth APIs extended with missing CRUD methods, that are used by ``opentaxii-sync-data``. * Read/modify collection level ACL added. * DB models for default implementation of Persistence API and Auth API were changed. No automatic migration code is provided (sorry!), so upgrading might require manual DB migration. * Drop python2.7 from testing scope. * Various bug fixes and improvements. 0.1.9 (2017-06-19) ------------------ * `libtaxii <https://github.com/TAXIIProject/libtaxii>`_ dependency upgraded to 1.1.111. * Various bug fixes and improvements (thanks to `@bjigmp <https://github.com/bjigmp>`_, `@chorsley <https://github.com/chorsley>`_, `@rjprins <https://github.com/rjprins>`_). 0.1.8 (2017-02-21) ------------------ * Ability to enable/disable "huge trees" support in XML parser. Configuration property ``xml_parser_supports_huge_tree`` set to ``yes`` or ``true`` will disable security restrictions and force XML parser to support very deep trees and very long text content. * Adjust SQL Persistence API implemetation so it works smoothly with MySQL backend. * Use Python 3.5 instead of Python 3.4 for tests. 0.1.7 (2016-10-18) ------------------ * Minor fixes. * Dependencies were changed from hard-pinned to more flexible. * Example of production DB configuration added to docs. 0.1.6 (2016-06-01) ------------------ * Python 3.4 compatibility of the codebase. Tox configuration extended with python 3.4 environment run. * Flake8 full style compatibility. Flake8 check added to Tox configuration. * SQLAlchemy session scope issue fixed (related to `#38 <https://github.com/EclecticIQ/OpenTAXII/issues/38>`_). * `opentaxii-delete-blocks` CLI command added (related to `#45 <https://github.com/EclecticIQ/OpenTAXII/issues/45>`_). * `delete_content_blocks` method `added <https://github.com/EclecticIQ/OpenTAXII/commit/dc6fddc27a98e8450c7e05e583b2bfb741f6e197#diff-6814849ac352b2b74132f8fa52e0ec4eR213>`_ to Persistence API. * Collection's name is `required <https://github.com/EclecticIQ/OpenTAXII/commit/dc6fddc27a98e8450c7e05e583b2bfb741f6e197#diff-ce3f7b939e5c540480ac655aef32c513R116>`_ to be unique in default SQL DB Auth API implementation. 0.1.5 (2016-03-15) ------------------ * Fix for the issue with persistence backend returning ``None`` instead of ``InboxMessage`` object 0.1.4 (2016-02-25) ------------------ * Hard-coded dependencies in ``setup.py`` removed. 0.1.3 (2016-02-25) ------------------ * Versions of dependencies are pinned. * Code adjusted for a new version of `anyconfig <https://pypi.python.org/pypi/anyconfig>`_ API. * Test for configuration loading added. 0.1.2 (2015-07-24) ------------------ * Docker configuration added. * Health check endpoint added. * Basic authentication support added. * Temporary workaround for `Issue #191 <https://github.com/TAXIIProject/libtaxii/issues/191>`_. * Method ``get_domain`` in Persistence API returns domain value configured for ``service_id``. If nothing returned, value set in configuration file will be used. * Performance optimisations. * Bug fixes and style improvements. 0.1.1 (2015-04-08) ------------------ * Alias for Root Logger added to logging configuration. * Context object in a request scope that holds account and token added. * Support for OPTIONS HTTP request to enable auto version negotiation added. * Documentation improved. 0.1.0 (2015-03-31) ------------------ * Initial release 2022-10-11T13:46:48+00:00 MONARC v2.12.3-p1 2022-10-12T11:07:12+00:00 [FrontOffice] Fixed the analysis creation issue. 2022-10-12T11:07:12+00:00 rocksdb v7.7.3 2022-10-12T21:58:10+00:00 ## 7.7.3 (10/11/2022) ### Bug Fixes * Fixed a memory safety bug in experimental HyperClockCache (#10768) 2022-10-12T21:58:10+00:00 MONARC v2.12.3-p2 2022-10-13T14:25:34+00:00 [FrontOffice] Fixed the object categories language validation. 2022-10-13T14:25:34+00:00 caddy v2.6.2 2022-10-13T18:44:05+00:00 This release brings a number of bug fixes and minor enhancements. All users should upgrade after testing and verifying their setups. Thank you to all who contributed! If you are coming from < 2.6, please see [the 2.6 release notes](https://github.com/caddyserver/caddy/releases/tag/v2.6.0) because a lot is new! ## Changelog * 037dc23c admin: Use replacer on listen addresses (#5071) * 498f32ba caddyconfig: Implement retries into HTTPLoader (#5077) * 9873ff99 caddyhttp: Remote IP prefix placeholders * 61822f12 caddyhttp: replace placeholders in map defaults (#5081) * e07a2672 caddytest: Revise sleep durations * 253d97c9 core: Chdir to executable location on Windows (#5115) * ab720fb7 core: Fix ListenQUIC listener key conflict * e3e8aabb core: Refactor and improve listener logic (#5089) * e4fac129 core: Set version manually via CustomVersion (#5072) * f7c1a51e fastcgi: Redirect using original URI path (fix #5073) * 2be56c52 fileserver: Treat invalid file path as NotFound (#5099) * b1d04f5b fileserver: better dark mode visited link contrast (#5105) * 33f60da9 fileserver: stop listing dir when request context is cancelled (#5131) * 2153a81e forwardauth: Canonicalize header fields (fix #5038) (#5097) * fe91de67 go.mod: Upgrade select dependencies * 70419700 headers: Support repeated WriteHeader if 1xx (fix #5074) * d46ba2e2 httpcaddyfile: Fix `metrics` global option parsing (#5126) * 6bad878a httpcaddyfile: Improve detection of indistinguishable TLS automation policies (#5120) * 2808de1e httpcaddyfile: Skip `automate` when `auto_https off` is specified (#5110) * 3e1fd2a8 httpcaddyfile: Wrap site block in subroute if host matcher used (#5130) * 9e1d964b logging: Add `time_local` option to use local time instead of UTC (#5108) * 01e192ed logging: Better `console` encoder defaults (#5109) * 99ffe933 logging: Fix `skip_hosts` with wildcards (#5102) * ea58d519 logging: Perform filtering on arrays of strings (where possible) (#5101) * 5e52bbb1 map: Remove infinite recursion check (#5094) * b4e28af9 replacer: working directory global placeholder (#5127) * e2991eb0 reverseproxy: On 103 don't delete own headers (#5091) * 2a8c458f reverseproxy: Parse humanized byte size (fix #5095) * d0556929 reverseproxy: fix upstream scheme handling in command (#5088) * 013b5103 rewrite: Only trim prefix if matched ## New Contributors * @lemmi made their first contribution in https://github.com/caddyserver/caddy/pull/5088 * @willnorris made their first contribution in https://github.com/caddyserver/caddy/pull/5081 * @yroc92 made their first contribution in https://github.com/caddyserver/caddy/pull/5071 * @iliana made their first contribution in https://github.com/caddyserver/caddy/pull/5105 * @TobiX made their first contribution in https://github.com/caddyserver/caddy/pull/5106 * @likev made their first contribution in https://github.com/caddyserver/caddy/pull/5099 * @cherouvim made their first contribution in https://github.com/caddyserver/caddy/pull/5121 **Full Changelog**: https://github.com/caddyserver/caddy/compare/v2.6.1...v2.6.2 2022-10-13T18:44:05+00:00 MONARC v2.12.3-p3 2022-10-20T11:40:40+00:00 * [Fixed the import error of the assets, threats or vulnerabilities codes duplication.](https://github.com/monarc-project/zm-client/commit/e96a3e5a823eb9b68d4bf4ec6a74de7c77f18d09) 2022-10-20T11:40:40+00:00 dnstwist 20221022 2022-10-22T12:53:46+00:00 2022-10-22T12:53:46+00:00 Lookyloo v1.16.0 2022-10-29T13:19:39+00:00 # Breaking change This release requires poetry v1.2.0 or more recent. Run the following command to upgrade it: `poetry self update` # New Features * Move to Lacus/LacusCore, many changes to make lookyloo's code compatible with it * [Lacus](https://github.com/ail-project/lacus) and [PyLacus](https://github.com/ail-project/pylacus): use this mode to trigger the capture from an other machine than the one you run Lookyloo from * [LacusCore](https://github.com/ail-project/lacuscore): (the default) keep triggering the captures from the same machine as the one lookyloo is running on With lacus, the captures are more reliable and using Lacus as a web service allows to monitor them better. If you want to use the webservice, you'll need to: 1. [Install Lacus](https://github.com/ail-project/Lacus#install-guide) * make sure it is running by loading http://127.0.0.1:7100 on the machine you have it running on (7100 is the default port, you can of course change it) 2. Edit the config file `config/generic.json` (key `remote_lacus`): * set `enable` to `true` * set `url` to the url your loolyloo instance can use to connect to lacus: `http://<ip>:<port>` 3. Restart lookyloo & try it # Changes * Make hashlookup visible to everyone * Improve loggingv1.15.0 * Maintenance: use poetry 1.2, bump deps (Python and JS), bump Github actions * Improve caching with Lacus * Retry failing captures when it might be a temporary issue (typically domain resolution) # Fixes * Avoid triggering a capture (and failing) when the URL and documents are missing * Issue with urlscan when the capture had no referer * Better handling of exceptions in VT module * Better handling of devices offered by Playwright and their user-agents 2022-10-29T13:19:39+00:00 PyPCAPKit v0.16.3 2022-10-31T22:16:01+00:00 New Distribution [0.16.3] * bugfix for #114 & #116 (hopefully) * bugfix for distro workflow 2022-10-31T22:16:01+00:00 maltrail 0.51 2022-10-31T23:11:05+00:00 Start-of-month release 2022-10-31T23:11:05+00:00 MONARC v2.12.4 2022-11-11T15:53:31+00:00 ### New features - Implemented new dashboard chart views and a curve line that represents the average number tendency. - Added a possibility to enforce all the instance's users to enable 2-Factor Authentication. 2022-11-11T15:53:31+00:00 MONARC v2.12.4-p1 2022-11-17T10:46:50+00:00 * Fixed the threats' codes duplication issue during the import. * Fixed the dashboard export with the new chart changes. 2022-11-17T10:46:50+00:00 Stegano v0.11.0 2022-11-20T00:02:32+00:00 Reduced memory footprint and processing speed. The modules ``lsb`` and ``lsbset`` have been merged ([PR #34](https://github.com/cedricbonhomme/Stegano/pull/34)). 2022-11-20T00:02:32+00:00 Stegano v0.11.1 2022-11-20T21:03:28+00:00 Fixed a bug in the command line when no sub-command is specified. 2022-11-20T21:03:28+00:00 MISP v2.4.165 2022-11-22T09:28:31+00:00  We are pleased to announce the immediate availability of [MISP v2.4.165](https://github.com/MISP/MISP/releases/tag/v2.4.165) with many improvements to the workflow subsystem along with various performance improvements. # Improvements - [workflow] Module to toggle/remove the to_ids flag - [workflow] Added generic module to support attribute edition - [workflow] [triggers:event_after_save_new] Added 2 new triggers for new events and new events from pull. - [workflow:execute_module] Allow to ignore format conversion before executing module. - [workflows:triggers] Added filtering capability on the index - [CLI] Feed management added - [CLI] Pretty and JSON output added in list and view feeds - [Auth] OpenID connect improved - [freetext] Fetch security vendor domains from [warninglist](https://github.com/MISP/misp-warninglists) - [UI] Allow to disable PGP key fetching - [UI] Show warning if user don't have permission to use API - [tool:evengraph] Include relationships when using pivot key - [UI] Show servers where event will be pushed # Performance improvements - [feed] Store freetext feed compressed in cache - [internal] Store some data in Redis compressed to save memory - [correlation] Do not correlate over correlating value again for full correlation - [internal] Add support for [simdjson](https://github.com/simdjson/simdjson) extension - [warninglist] Load warninglist from Redis for TLDs and security vendors # Bugs fixed - [tags] not passing name, filter, search all together would lead to the search not working # Security issues - [security] Permission for tag collections - [security] Check user permission when attaching clusters We strongly recommend MISP administrators to update to this latest version. For a more detailed changelog, please see the online [Changelog](https://www.misp-project.org/Changelog.txt). # New workflow blueprints available New [workflow blueprints](https://github.com/MISP/misp-workflow-blueprints/) were added to support new use-cases. - [Attach `tlp:clear` on `tlp:white`](./blueprints/blueprint_attach-tlp_clear-on-tlp_white_1661328256.json) - Attach the `tlp:clear` tag on elements having the `tlp:white` tag. - [`PAP:RED` and `tlp:red` Blocking](./blueprints/blueprint_pap_red-and-tlp_red-blocking_1661328258.json) - Block actions if any attributes have the `PAP:RED` or `tlp:red` tag. - [Remote `to_ids` flag if the indicator appears in known file list](https://github.com/MISP/misp-workflow-blueprints/blob/main/blueprints/blueprint_disable-to_ids-flag-for-existing-hash-in-hashlookup_1667228944.json) - Disable to_ids flag for existing hash in [hashlookup](https://www.hashlookup.io/). - [Set tag based on BGP Ranking maliciousness level](https://github.com/MISP/misp-workflow-blueprints/blob/main/blueprints/blueprint_set-tag-based-on-bgp-ranking-maliciousness-level_1668498668.json) - Set tag based on [BGP Ranking](https://bgpranking.circl.lu) maliciousness level. # New MISP modules - [expansion] Added extract_url_components module to create an object from an URL attribute. - [expansion] New [crowdsec](https://www.crowdsec.net/) expansion module added. - [expansion] New [VARIoT IoT exploits database](https://www.variotdbs.pl/exploits/) expansion module added. - [expansion] Updates on hyasinsight expansion module. # MISP taxonomies - new misp-workflow taxonomy to have a consistent tag message for the MISP workflow. - Taxonomy in support of integrating MISP with Sentinel. Sentinel indicator threat types added. For more [details](https://www.misp-project.org/Changelog-misp-taxonomies.txt). # MISP galaxy - Many updates to the threat actor database. - Update to the MITRE ATT&CK framework to version 12.0. For more [details](https://www.misp-project.org/Changelog-misp-galaxy.txt). # MISP objects - New object to describe Telegram bots. - Updated exploit object. For more [details](https://www.misp-project.org/Changelog-misp-objects.txt). # Social network - Mastodon MISP project is also now reachable via Mastodon. Feel free to follow us at @misp@misp-community.org 2022-11-22T09:28:31+00:00 MONARC v2.12.4-p2 2022-11-24T15:49:39+00:00 ### Fix * Fixed the import issue when more than 1 theme exists with the same label. 2022-11-24T15:49:39+00:00 MISP v2.4.166 2022-11-30T17:21:47+00:00  We are pleased to announce the immediate availability of [MISP v2.4.166](https://github.com/MISP/MISP/releases/tag/v2.4.166) with new features and fixes, including two critical security fixes. # TAXII 2.1 server push integration With the collaboration of CISA and MITRE, we have included the first version of the [TAXII](https://docs.oasis-open.org/cti/taxii/v2.1/taxii-v2.1.html) integration in MISP, allowing administrators to configure their MISPs to push content to TAXII 2.1 servers. For more informatia new dedicated will be posted soon. On server side, the [taxii2-client Python library](https://pypi.org/project/taxii2-client/) is required to be installed. The conversion is performed by the wonderful and efficient [misp-stix library](https://github.com/MISP/misp-stix). # Logging rework The logging of MISP has been severely reworked by Jakub Onderka, including a separate Access log subsystem as well as multiple improvements and cleanups to the system at large. # Security fixes Two critical vulnerabilities have been patched allowing for the tampering with data shared in the community via galaxy clusters and tags. It is **HIGHLY** recommended to update to 2.4.166 as soon as possible to avoid information tampering. We also encourage everyone to consider informing peered MISP instance owners to do the same. CVEs have been requested and are pending for both. Thanks to Jakub Onderka for discovering and fixing the vulnerabilities. # Allowing for working around the edge cases introduced by TLP v2.0 Even though [TLP 2.0](https://www.first.org/tlp/) has been supported by MISP for a while, in order to cope with both tools old and new as well as older information sources, we see the need to often attach both TLP:WHITE and TLP:CLEAR to data points. This has however been blocked by the taxonomy exclusivity rules - something that we've now added exceptions for. Let's hope that we can avoid similar surprises in the future. For more [details](https://www.misp-project.org/Changelog.txt) about changes in the MISP core software. # Other updates and changes ## MISP Objects - [passport object] Updated to include passport-creation field. ## MISP Galaxy - MITRE ATT&CK updated and fixing the missing reference - Many improvements and fixes in all the meta fields - Tool galaxy updated - [Ransomware groups](https://www.misp-project.org/galaxy.html#_ransomware) updated 2022-11-30T17:21:47+00:00 maltrail 0.52 2022-11-30T23:11:06+00:00 Start-of-month release 2022-11-30T23:11:06+00:00 reko version-0.11.2 2022-12-05T15:51:49+00:00 My, has it been that long already since last release? Here's an overview of what's happened since. The Reko solution was moved to .NET 6. As expected, performance and memory footprint was impoved. The partially completed, still very much work-in-progress Avalonia development branch was merged into `master`. Moving forward, any changes in the GUI components will be mirrored in Windows Forms and Avalonia. For instance, the new Base Address finder tool window now exists for both GUI platforms. Another big push is the move to support multithreaded scanning and analysis. Today, Reko uses only a single CPU thread for its CPU-intensive work. Partitioning the work as mutually independent workloads across multiple threads should see a great improvement in execution time. In order to accomplish this, several classes have been altered to only use read-only interfaces to global data, to prevent data race conditions. The work to eliminate globally visible mutable state continues. Several new metadata annotations were added ( with the gentle prodding of @Elthial). You can now annotate C functions with the `[[noreturn]]` , `[[reko::arg(seq...)]]`, and `[[reko::address]]` attributes. Other bits and bobs: * Improvements to `ArgumentGuesser` to handle call sites where Reko can't prove the target address. * The codebase's namespaces were refactored to make them more easy to understand. * Added or improved support for the following architectures: ARM, AArch64, BlackBin, Fujitsu F2MC16FX, MIPS, PDP-7, PowrPC, RiscV, MicroBlaze, SuperH SH2..SH4A, TriCore and x86 EVEX * Support for discovering and parsing x86 and x86-64 MSVC RTTI . * Fix calculation of alignment of unions (@ptomin) * Refactored intrinsics for improved performance * PharLap "MP" Exp file loader (#1169) * MzExe: allow reading PEs without IAT (@smx-smx) * Improved support for XEX executables (@smx-smx) * Handle '*' in `*scanf` format strings. Special thanks to @ptomin and @smx for their dedication and contributions. 2022-12-05T15:51:49+00:00 osquery 5.7.0 2022-12-06T19:00:16+00:00 Draft 2022-12-06T19:00:16+00:00 DC3-MWCP 3.9.0 2022-12-08T23:32:47+00:00 2022-12-08T23:32:47+00:00 dnstwist 20221213 2022-12-13T16:04:52+00:00 Changes: - Added optional support for TLSH (used as a fallback when ssdeep is unavailable) - Added new fuzzer: cyrillic (translates latin characters to cyrillic homographs) - Moved essential HTTP headers to `UrlOpener()` - Minor optimizations - Refreshed `common_tlds.dict` dictionary file 2022-12-13T16:04:52+00:00 rocksdb v7.7.8 2022-12-15T18:52:42+00:00 ## 7.7.8 (2022-11-27) ### Bug Fixes * Fix failed memtable flush retry bug that could cause wrongly ordered updates, which would surface to writers as `Status::Corruption` in case of `force_consistency_checks=true` (default). It affects use cases that enable both parallel flush (`max_background_flushes > 1` or `max_background_jobs >= 8`) and non-default memtable count (`max_write_buffer_number > 2`). * Tiered Storage: fixed excessive keys written to penultimate level in non-debug builds. * Fixed a regression in iterator where range tombstones after `iterate_upper_bound` is processed. ## 7.7.7 (2022-11-15) ### Bug Fixes * Fixed a regression in scan for async_io. During seek, valid buffers were getting cleared causing a regression. ## 7.7.6 (2022-11-03) ### Bug Fixes * Fix memory corruption error in scans if async_io is enabled. Memory corruption happened if there is IOError while reading the data leading to empty buffer and other buffer already in progress of async read goes again for reading. ## 7.7.5 (2022-10-28) ### Bug Fixes * Fixed an iterator performance regression for delete range users when scanning through a consecutive sequence of range tombstones (#10877). ## 7.7.4 (2022-10-28) ### Bug Fixes * Fixed a case of calling malloc_usable_size on result of operator new[]. 2022-12-15T18:52:42+00:00 rocksdb v7.8.3 2022-12-15T18:56:20+00:00 ## 7.8.3 (2022-11-29) * Revert an internal change in 7.8.0 associated with some memory usage churn. ## 7.8.2 (2022-11-27) ### Behavior changes * Make best-efforts recovery verify SST unique ID before Version construction (#10962) * Fix failed memtable flush retry bug that could cause wrongly ordered updates, which would surface to writers as `Status::Corruption` in case of `force_consistency_checks=true` (default). It affects use cases that enable both parallel flush (`max_background_flushes > 1` or `max_background_jobs >= 8`) and non-default memtable count (`max_write_buffer_number > 2`). * Tiered Storage: fixed excessive keys written to penultimate level in non-debug builds. ### Bug Fixes * Fixed a regression in scan for async_io. During seek, valid buffers were getting cleared causing a regression. * Fixed a performance regression in iterator where range tombstones after `iterate_upper_bound` is processed. ## 7.8.1 (2022-11-02) ### Bug Fixes * Fix memory corruption error in scans if async_io is enabled. Memory corruption happened if there is IOError while reading the data leading to empty buffer and other buffer already in progress of async read goes again for reading. ## 7.8.0 (2022-10-22) ### New Features * `DeleteRange()` now supports user-defined timestamp. * Provide support for async_io with tailing iterators when ReadOptions.tailing is enabled during scans. * Tiered Storage: allow data moving up from the last level to the penultimate level if the input level is penultimate level or above. * Added `DB::Properties::kFastBlockCacheEntryStats`, which is similar to `DB::Properties::kBlockCacheEntryStats`, except returns cached (stale) values in more cases to reduce overhead. * FIFO compaction now supports migrating from a multi-level DB via DB::Open(). During the migration phase, FIFO compaction picker will: * picks the sst file with the smallest starting key in the bottom-most non-empty level. * Note that during the migration phase, the file purge order will only be an approximation of "FIFO" as files in lower-level might sometime contain newer keys than files in upper-level. * Added an option `ignore_max_compaction_bytes_for_input` to ignore max_compaction_bytes limit when adding files to be compacted from input level. This should help reduce write amplification. The option is enabled by default. * Tiered Storage: allow data moving up from the last level even if it's a last level only compaction, as long as the penultimate level is empty. * Add a new option IOOptions.do_not_recurse that can be used by underlying file systems to skip recursing through sub directories and list only files in GetChildren API. * Add option `preserve_internal_time_seconds` to preserve the time information for the latest data. Which can be used to determine the age of data when `preclude_last_level_data_seconds` is enabled. The time information is attached with SST in table property `rocksdb.seqno.time.map` which can be parsed by tool ldb or sst_dump. ### Bug Fixes * Fix a bug in io_uring_prep_cancel in AbortIO API for posix which expects sqe->addr to match with read request submitted and wrong paramter was being passed. * Fixed a regression in iterator performance when the entire DB is a single memtable introduced in #10449. The fix is in #10705 and #10716. * Fixed an optimistic transaction validation bug caused by DBImpl::GetLatestSequenceForKey() returning non-latest seq for merge (#10724). * Fixed a bug in iterator refresh which could segfault for DeleteRange users (#10739). * Fixed a bug causing manual flush with `flush_opts.wait=false` to stall when database has stopped all writes (#10001). * Fixed a bug in iterator refresh that was not freeing up SuperVersion, which could cause excessive resource pinniung (#10770). * Fixed a bug where RocksDB could be doing compaction endlessly when allow_ingest_behind is true and the bottommost level is not filled (#10767). * Fixed a memory safety bug in experimental HyperClockCache (#10768) * Fixed some cases where `ldb update_manifest` and `ldb unsafe_remove_sst_file` are not usable because they were requiring the DB files to match the existing manifest state (before updating the manifest to match a desired state). ### Performance Improvements * Try to align the compaction output file boundaries to the next level ones, which can reduce more than 10% compaction load for the default level compaction. The feature is enabled by default, to disable, set `AdvancedColumnFamilyOptions.level_compaction_dynamic_file_size` to false. As a side effect, it can create SSTs larger than the target_file_size (capped at 2x target_file_size) or smaller files. * Improve RoundRobin TTL compaction, which is going to be the same as normal RoundRobin compaction to move the compaction cursor. * Fix a small CPU regression caused by a change that UserComparatorWrapper was made Customizable, because Customizable itself has small CPU overhead for initialization. * Fixed an iterator performance regression for delete range users when scanning through a consecutive sequence of range tombstones (#10877). ### Behavior Changes * Sanitize min_write_buffer_number_to_merge to 1 if atomic flush is enabled to prevent unexpected data loss when WAL is disabled in a multi-column-family setting (#10773). ### Public API changes * Make kXXH3 checksum the new default, because it is faster on common hardware, especially with kCRC32c affected by a performance bug in some versions of clang (https://github.com/facebook/rocksdb/issues/9891). DBs written with this new setting can be read by RocksDB 6.27 and newer. * Refactor the classes, APIs and data structures for block cache tracing to allow a user provided trace writer to be used. Introduced an abstract BlockCacheTraceWriter class that takes a structured BlockCacheTraceRecord. The BlockCacheTraceWriter implementation can then format and log the record in whatever way it sees fit. The default BlockCacheTraceWriterImpl does file tracing using a user provided TraceWriter. More details in rocksdb/includb/block_cache_trace_writer.h. 2022-12-15T18:56:20+00:00 MONARC v2.12.5 2022-12-21T09:27:28+00:00 **New** - Rebranding of Monarc app (logos, titles) according to the "Cases" -> "NC3" move. - Improved the analyses import speed. **Fix** - Fixed the autocomplete for passwords. Password filling for Import brakes the execution. - Record of processing activities processor creation JS error. - Fixed the long names displaying for assets (responsive view). - Dashboard fixes. - Added the possibility to modify the threat's theme. - Updated and added missing translations. 2022-12-21T09:27:28+00:00 MISP v2.4.167 2022-12-26T14:41:06+00:00 We are pleased to announce the immediate availability of [MISP v2.4.167](https://github.com/MISP/MISP/releases/tag/v2.4.167) with new features and fixes, bugs fixed and a security fix. # New features ## Timeline improvements for large events  Timeline is a convenient way to display the different attributes and objects over the time. Events with a large set of attributes or objects (more than 500) cannot display a human readable timeline. Nevertheless there are still a lot of valuable information in such event especially concerning the occurences over time. A new feature has been added in 2.4.167 to display the overall occurrences over the time and display the overall sighting trend. ## Taxonomy highlight For MISP users and organisations, it's important to show the important contextualised information and especially the [taxonomies](https://www.misp-project.org/taxonomies.html) which are important to your use-case. We introduced a new feature to highlight the important taxonomy in a MISP instance. The site admin user can select the important taxonomies in the taxonomy list:  and then the taxonomy namespace will appear in a visible box:   ## Create objects from free-text import The free-text import in MISP is very nifty for analysts willing to enter quickly new attributes. This functionality was initially used for attributes only. In 2.4.167, MISP objects can be created from the free-text import directly too.  ## API - A new endpoint session kill-switch has been added for the support and integration with MeliCERTes project. ## UI - Clarify the exclusivity issue in the UI when exclusive tags are used in the TLP namespace. - [dashboard] sort dashboard widgets. Many UI improvements and a special thank to Jakub Onderka for the attention to details in the UI. # Security fix A security XSS vulnerability has been fixed in this release and tracked under [CVE-2022-47928](https://cvepremium.circl.lu/cve/CVE-2022-47928). We recommend every users to update to the latest version. A huge thanks to all the contributors and supporters of the MISP project. This release won't be possible by all the organisations and people supporting us to make MISP a reality. For more [details](https://www.misp-project.org/Changelog.txt) about changes in the MISP core software. # Other updates and changes ## MISP Objects - New thaicert-group-cards, Palantir ADS and [persnona](https://itk.mitre.org/toolkit-tools/personas/). - Invalid UUID object templates fixed including mactim-timeline-analysis and fail2ban. ## MISP Galaxy - New threat-actor such as TAG-53, Malteiro and others added. - RAT group updated. - [Ransomware groups](https://www.misp-project.org/galaxy.html#_ransomware) updated. ## MISP taxonomies - A new [aviation](https://www.misp-project.org/taxonomies.html#_aviation) has been added. Thanks to [European Air Traffic Management Computer Emergency Response Team ](https://www.eurocontrol.int/service/european-air-traffic-management-computer-emergency-response-team). # Don't forget to follow us on Mastodon MISP projet has its own Mastodon server [misp-community.org](https://misp-community.org/) and don't forget to follow @misp@misp-community.org on the fediverse. Core contributors of MISP can sign-up if you want an account. 2022-12-26T14:41:06+00:00 Lookyloo v1.17.0 2022-12-29T13:48:48+00:00 # Breaking change Poetry v1.3.0 or more recent is now required, please [upgrade](https://python-poetry.org/docs/#installation) to the latest version. # New features: * See [Lacus release notes for v1.2.0](https://github.com/ail-project/lacus/releases/tag/v1.2.0) if you want more details on the changes regarding the captures. * Temporary local storage of captures if Lacus web-service isn't reachable temporarily * Submit and view a HAR file captured somewhere else, or a full Lookyloo capture from an other instance  * Show status of captures when they're submitted in bulk  * List of all downloadable contents for a capture in a modal (from the tree view -> `Download capture elements`):  # Bugfixes * Catch and stop script when generating a tree takes too long (link to doc) * [har2tree] Avoid exception when a node doesn't have a pageref * [har2tree] Better use of BeautifulSoup * [har2tree] Better handling of nodes that aren't loading a URL * [har2tree] Improve mimetime detection in HTTP responses # Changes * [har2tree] Improve decoding of POSTed data * Use more recent Flask and flask-restx * Bump deps (Python ans JS) * Major logging improvements in Lookyloo, LacusCore, PlaywrightCapture and har2tree * 2022-12-29T13:48:48+00:00 maltrail 0.53 2022-12-31T23:11:05+00:00 Start-of-month release 2022-12-31T23:11:05+00:00 reko version-0.11.3 2023-01-05T00:43:49+00:00 The largest new feature of this release is the support for disassembling, rewriting, and emulation of instructions of the AEON (or BEON, or R2) processor. It is the first architecture in Reko implemented with no manual -- we've been unable to find the instruction set described in a publicly available resource. Instead, thanks to the hard work of @throwaway96, most of the instruction set has been reverse engineered. Many thanks for this great effort! @smx-smx improved the stability of the build and eliminated redundant compilations of the build tools, resulting in a faster build overall. Some other features added are: * Introduced the notion of `Unlikely` instructions: instructions that while well-formed are unlikely to be present in a "normal" program. Reko can optionally be instructed to treat such unlikely instructions as invalid * The user may also optionally instruct the Reko to treat protected/system instructions as invalid. * Since the move to .NET 6, Reko's custom 16-bit IEEE float implementation became redundant and was removed. * The GUI client now supports multiple scanning heuristics. * Support for user-defined segments was added. * Improvements in handling of varargs procedures. * Switch statements for MIPS, MIL-STD-1750A and Intel 8051 improved. Thanks to @throwaway96 and @smx-smx for their contributions and support! 2023-01-05T00:43:49+00:00 MONARC v2.12.5-p1 2023-01-05T07:55:27+00:00 [fix issue of fetching dependencies of object](https://github.com/monarc-project/zm-client/commit/9bf810d50f9e87c0e5f8eb4774d1779dbd18ca09) 2023-01-05T07:55:27+00:00 ursadb v1.5.1 2023-01-11T00:36:08+00:00 Mostly bugfix and maintenance release: Features: - Implement GC for iterators - remove stale iterators (#208) Performance: - Move away from query graphs to query plans (#191) - And improve their performance (#194) Correctness: - Fix Undefined Behaviour when getting a memory map size (#188) - Add support for 2gb+ iterators (fix signed i32 overflow) (#202) Refactoring and maintenance: - Bump catch v2.2.2 -> v2.13.10 (#192) - Disable clang-tidy, which has become very noisy (#193) - Add performabce counters for unique ngram reads (#199) - Remove dead code accrued over the years (#200) 2023-01-11T00:36:08+00:00 MONARC v2.12.5-p2 2023-01-12T09:35:56+00:00 * Fixed the import of themes and specific risks duplications. 2023-01-12T09:35:56+00:00 rocksdb v7.9.2 2023-01-17T18:51:07+00:00 ## 7.9.2 (12/21/2022) ### Bug Fixes * Fixed a heap use after free bug in async scan prefetching when the scan thread and another thread try to read and load the same seek block into cache. ## 7.9.1 (12/8/2022) ### Bug Fixes * Fixed a regression in iterator where range tombstones after `iterate_upper_bound` is processed. * Fixed a memory leak in MultiGet with async_io read option, caused by IO errors during table file open ### Behavior changes * Make best-efforts recovery verify SST unique ID before Version construction (#10962) ## 7.9.0 (11/21/2022) ### Performance Improvements * Fixed an iterator performance regression for delete range users when scanning through a consecutive sequence of range tombstones (#10877). ### Bug Fixes * Fix memory corruption error in scans if async_io is enabled. Memory corruption happened if there is IOError while reading the data leading to empty buffer and other buffer already in progress of async read goes again for reading. * Fix failed memtable flush retry bug that could cause wrongly ordered updates, which would surface to writers as `Status::Corruption` in case of `force_consistency_checks=true` (default). It affects use cases that enable both parallel flush (`max_background_flushes > 1` or `max_background_jobs >= 8`) and non-default memtable count (`max_write_buffer_number > 2`). * Fixed an issue where the `READ_NUM_MERGE_OPERANDS` ticker was not updated when the base key-value or tombstone was read from an SST file. * Fixed a memory safety bug when using a SecondaryCache with `block_cache_compressed`. `block_cache_compressed` no longer attempts to use SecondaryCache features. * Fixed a regression in scan for async_io. During seek, valid buffers were getting cleared causing a regression. * Tiered Storage: fixed excessive keys written to penultimate level in non-debug builds. ### New Features * Add basic support for user-defined timestamp to Merge (#10819). * Add stats for ReadAsync time spent and async read errors. * Basic support for the wide-column data model is now available. Wide-column entities can be stored using the `PutEntity` API, and retrieved using `GetEntity` and the new `columns` API of iterator. For compatibility, the classic APIs `Get` and `MultiGet`, as well as iterator's `value` API return the value of the anonymous default column of wide-column entities; also, `GetEntity` and iterator's `columns` return any plain key-values in the form of an entity which only has the anonymous default column. `Merge` (and `GetMergeOperands`) currently also apply to the default column; any other columns of entities are unaffected by `Merge` operations. Note that some features like compaction filters, transactions, user-defined timestamps, and the SST file writer do not yet support wide-column entities; also, there is currently no `MultiGet`-like API to retrieve multiple entities at once. We plan to gradually close the above gaps and also implement new features like column-level operations (e.g. updating or querying only certain columns of an entity). * Marked HyperClockCache as a production-ready alternative to LRUCache for the block cache. HyperClockCache greatly improves hot-path CPU efficiency under high parallel load or high contention, with some documented caveats and limitations. As much as 4.5x higher ops/sec vs. LRUCache has been seen in db_bench under high parallel load. * Add periodic diagnostics to info_log (LOG file) for HyperClockCache block cache if performance is degraded by bad `estimated_entry_charge` option. ### Public API Changes * Marked `block_cache_compressed` as a deprecated feature. Use SecondaryCache instead. * Added a `SecondaryCache::InsertSaved()` API, with default implementation depending on `Insert()`. Some implementations might need to add a custom implementation of `InsertSaved()`. (Details in API comments.) 2023-01-17T18:51:07+00:00 MONARC v2.12.5-p3 2023-01-26T11:55:18+00:00 Fixed the threats duplications issue during the import. 2023-01-26T11:55:18+00:00 maltrail 0.54 2023-01-31T23:11:06+00:00 Start-of-month release 2023-01-31T23:11:06+00:00 DC3-MWCP 3.10.0 2023-02-03T02:30:07+00:00 2023-02-03T02:30:07+00:00 DC3-MWCP 3.10.1 2023-02-03T14:40:10+00:00 2023-02-03T14:40:10+00:00 MONARC v2.12.5-p4 2023-02-03T14:41:31+00:00 Improved the library assets import: * to avoid names duplication only inside of a category. * to prevent the name postfix `- Imp # 1` unlimited addition. 2023-02-03T14:41:31+00:00 caddy v2.6.3 2023-02-08T19:51:59+00:00 This release brings a number of bug fixes and minor features. We recommend that all users check the release notes/commits, then test and upgrade. Notable changes: - New `trusted_proxies` global option (within `servers`) can be used to specify trusted proxy IP ranges globally. This is important if relying on headers for client IP addresses. - Unix sockets on Windows now supported as proxy upstreams. - Proxied WebSocket connections are now logged with correct status code and "size" (bytes read + bytes written). - The [quic-go](https://github.com/quic-go/quic-go) package has received significant optimizations, so HTTP/3 should be more efficient now. Thank you to everyone who contributed to this release! ## Changelog * bfaf2a82 acme_server: Configurable default lifetime for issued certificates (#5232) * ac83b7e2 admin: Add `CADDY_ADMIN` env var to override the default (#5332) * ac96455a admin: fix certificate renewal for admin (#5169) * 762b0278 admin: set certmagic cache logger (#5173) * 329af5ce build(deps): bump actions/cache from 2 to 3 (#5263) * 3b724a20 build(deps): bump actions/upload-artifact from 1 to 3 (#5262) * af93517c build(deps): bump goreleaser/goreleaser-action from 2 to 4 (#5264) * cd49847e build(deps): bump peter-evans/repository-dispatch from 1 to 2 (#5261) * 8d3a1b8b caddyauth: Use singleflight for basic auth (#5344) * bbe36631 caddyconfig: Fix httploader leak from unused responses (#5159) * 7f2a93e6 caddyfile: Allow overriding server names (#5323) * 223cbe3d caddyhttp: Add server-level `trusted_proxies` config (#5103) * 087f126c caddyhttp: Canonicalize header field names (#5176) * 12bcbe2c caddyhttp: Pluggable trusted proxy IP range sources (#5328) * ed503118 caddyhttp: add placeholder {http.request.orig_uri.path.*} (#5161) * 33fdea8f caddypki: Prefer user-configured root instead of generating new one (#5189) * 6f8fe01d caddypki: Use go.step.sm/crypto to generate the PKI (#5217) * 1fa4cb7b caddytest: Increased sleep between retries to reduce flakey tests in CI (#5160) * fef9cb3e caddytest: internalize init config into '.go' file (#5230) * 55035d32 caddytls: Add `dns_ttl` config, improve Caddyfile `tls` options (#5287) * 66ce0c5c caddytls: Add test cases for Caddyfile `tls` options (#5293) * 0a3efd16 caddytls: Debug log for ask endpoint * 94b8d560 cmd: Add `--envfile` flag to `validate` command (#5350) * a999b707 cmd: Add missing `\n` to HelpTemplate (#5151) * c3b5b181 cmd: Avoid panic when printing version without build info (#5210) * 5805b3ca cmd: `caddy fmt` return code is 1 if not formatted (#5297) * 8c0b49bf cmd: `fmt` exit successfully after overwriting config file (#5351) * f20a8e7a cmd: replace deprecate func use (#5170) * 536c28d4 core: Support Windows absolute paths for UDS proxy upstreams (#5114) * dac7cacd encode: Respect Cache-Control no-transform (#5257) * 4e9ad50f fileserver: Add a couple test cases * 4bf6cb41 fileserver: Reject ADS and short name paths; trim trailing dots and spaces on Windows (#5148) * a3ae146c fileserver: Reject non-GET/HEAD requests (close #5166) (#5167) * e8ad9b32 go.mod: Update golang.org/x/net to v0.5.0 (#5314) * fac35db9 go.mod: Update quic-go to v0.31.0 * 798c4a3b go.mod: Upgrade some dependencies * 90798f3e go.mod: Upgrade various dependencies (#5362) * 98867ac3 go.mod: bump tscert package to fix Tailscale 1.34+ on Windows (#5331) * d73660f7 httpcaddyfile: Add persist_config global option (#5339) * c38a040e httpcaddyfile: Fix `handle` grouping inside `route` (#5315) * d6d75116 httpcaddyfile: Warn on importing empty file; skip dotfiles (#5320) * 817470dd httploader: Close resp body on bad status code * 72e7edda map: Clarified how destination values should be formatted (#5156) * e9d95ab2 reverseproxy: Add flag to short command to disable redirects (#5330) * e450a737 reverseproxy: Don't enable auto-https when `--from` flag is http (#5269) * 845bc4d5 reverseproxy: Fix hanging for Transfer-Encoding: chunked (#5289) * d4a7d89f reverseproxy: Improve hostByHashing distribution (#5229) * c77a6bea reverseproxy: Log status code and byte count for websockets (#5140) * ee7c92ec reverseproxy: Mask the WS close message when we're the client (#5199) * d74f6fd9 reverseproxy: Set origreq in active health check (#5284) * 96231020 tracing: Support placeholders in span name (#5329) 2023-02-08T19:51:59+00:00 caddy v2.6.4 2023-02-14T20:01:49+00:00 This release contains a hotfix for a regression in v2.6.3 related to proxying chunked requests. We recommend that all users who do so upgrade to v2.6.4. Note that, in an effort to make error-prone configs less likely, we have deprecated the reverse proxy options: - `buffer_requests` - `buffer_responses` - `max_buffer_size` and have introduced 2 new ones which take a size argument to enable buffering: - `request_buffers <size>` - `response_buffers <size>` The deprecated options will be removed in a later version of Caddy, so please start using the new parameters instead. ## Changelog * 0db29e2c go.mod: Upgrade acmez and x/net * 4b119a47 reverseproxy: Don't buffer chunked requests (fix #5366) (#5367) 2023-02-14T20:01:49+00:00 MISP v2.4.168 2023-02-16T14:41:01+00:00  We are pleased to announce the immediate availability of [MISP v2.4.168](https://github.com/MISP/MISP/releases/tag/v2.4.168) with bugs fixed and various security fixes. It includes a rather substantial [release](https://www.misp-project.org/Changelog-misp-stix.txt) of [misp-stix](https://github.com/MISP/misp-stix), the core Python library for importing and exporting STIX (1, 2.0 and 2.1). # Fixes - Improvements to the indexTable - allow site admins ability to view event_creator_email for all events in export - [shadowAttribute:accept] Restored accepting functionality for the proposals - [feed:edit] Make sure to keep orgc_id to its saved value - [tags:relationship] Fixed synchronisation of relationship_type # Security fixes - [CVE-2023-24070](https://cvepremium.circl.lu/cve/CVE-2023-24070) < MISP 2.4.168 - app/View/AuthKeys/authkey_display.ctp in MISP through 2.4.167 has an XSS in authkey add via a Referer field. - [CVE-2023-24026](https://cvepremium.circl.lu/cve/CVE-2023-24026) < MISP 2.4.168 - In app/webroot/js/event-graph.js has an XSS vulnerability via an event-graph preview payload. - [CVE-2023-24027](https://cvepremium.circl.lu/cve/CVE-2023-24027) < MISP 2.4.168 - In app/webroot/js/action_table.js allows XSS via a network history name. - [CVE-2023-24028](https://cvepremium.circl.lu/cve/CVE-2023-24028) < MISP 2.4.168 - In app/Controller/Component/ACLComponent.php has incorrect access control for the decaying import function. Thanks to the reporter Cyber Controls from SIX Group and Dawid Czarnecki of Zigrin Security. A huge thanks to all the contributors and supporters of the MISP project. This release wouldn't be possible without the help of all the organisations and people supporting us to make MISP a reality. Go to the detailed [changelog](https://www.misp-project.org/Changelog.txt) for more details about the changes to the MISP core software. # Other updates and changes in the MISP project ## MISP Objects - A new MISP object has been created for [typosquatting-finder](https://typosquatting-finder.circl.lu/) output. ## MISP Galaxy - New Sigma Galaxy including all [Sigma rule](https://github.com/MISP/misp-galaxy/blob/main/clusters/sigma-rules.json) and their ATT&CK relationships - [import script](https://github.com/MISP/misp-galaxy/blob/main/tools/sigma/sigma-to-galaxy.py) added - Many updates to the threat-actor and ransomware MISP galaxy - Improvements to the country galaxy ## MISP taxonomies - Improvements to the aviation taxonomy ## MISP warning-lists - New generator added for the Cached Chrome Top Million Websites - Improved generator for gzip files source - VPN list generator fixed # Don't forget to follow us on Mastodon The MISP projet has its own Mastodon server [misp-community.org](https://misp-community.org/) - don't forget to follow @misp@misp-community.org on the fediverse. Core contributors of MISP can sign-up if they wish to have an account. 2023-02-16T14:41:01+00:00 osv v0.0.17 2023-02-22T03:57:40+00:00 PyPI package upgrade. 2023-02-22T03:57:40+00:00 osv v0.0.18 2023-02-23T01:45:43+00:00 2023-02-23T01:45:43+00:00 osquery 5.8.0 2023-02-24T19:25:29+00:00 2023-02-24T19:25:29+00:00 maltrail 0.55 2023-02-28T23:11:06+00:00 Start-of-month release 2023-02-28T23:11:06+00:00 Lookyloo v1.18.0 2023-03-01T11:33:31+00:00 # New features * Beta support for [monitoring](https://github.com/Lookyloo/monitoring): the system will trigger a recurring capture and allow to compare them over time. * Beta support for comparing captures. For now, it focuses on the redirects from the URL captured to the landing page, and the URLs of the resources loaded on that page. An upcoming release of this feature will compare the rendered content of the landing page too. * Takedown endpoint (API only): gather contact information (whois on domain, IP, ASN, and [security.txt](https://github.com/Lookyloo/pysecuritytxt) file if available) for all the URLs up to the landing page. Can be used to notify owners in case of a malicious URL. * Flag known Cloudflare IPs on the hostnode popups * Trigger AAAA DNS requests (was only A before) # Bugfixes * When the capture is a file that was uploaded by the user, some modules cannot be triggered. Avoiding exceptions. * In some edge cases, a lock file for generating the tree could be left there even if the initiating script was dead. We now clean them up automatically. # Changes * Use [Lacus v1.3.0](https://github.com/ail-project/lacus/releases/tag/v1.3.0) / [LacusCore v1.3.0](https://github.com/ail-project/LacusCore/releases/tag/v1.3.0) / [PlaywrightCapture v1.18.0](https://github.com/Lookyloo/PlaywrightCapture/releases/tag/v1.18.0) * Allow to disable defanging URLs in emails * Many improvements in the rendering of the menus on the tree page * [Lookyloo] Bump javascript and python dependencies * [Har2Tree](https://github.com/Lookyloo/har2tree/releases/tag/v1.18.0) Maintenance, update dependencies 2023-03-01T11:33:31+00:00 DC3-MWCP 3.11.0 2023-03-01T13:06:16+00:00 2023-03-01T13:06:16+00:00 osquery 5.8.1 2023-03-01T20:45:27+00:00 2023-03-01T20:45:27+00:00 rocksdb v7.10.2 2023-03-02T01:00:53+00:00 ## 7.10.2 (02/10/2023) ### Bug Fixes * Fixed a bug in DB open/recovery from a compressed WAL that was caused due to incorrect handling of certain record fragments with the same offset within a WAL block. ## 7.10.1 (02/01/2023) ### Bug Fixes * Fixed a data race on `ColumnFamilyData::flush_reason` caused by concurrent flushes. * Fixed `DisableManualCompaction()` and `CompactRangeOptions::canceled` to cancel compactions even when they are waiting on conflicting compactions to finish * Fixed a bug in which a successful `GetMergeOperands()` could transiently return `Status::MergeInProgress()` * Return the correct error (Status::NotSupported()) to MultiGet caller when ReadOptions::async_io flag is true and IO uring is not enabled. Previously, Status::Corruption() was being returned when the actual failure was lack of async IO support. ## 7.10.0 (01/23/2023) ### Behavior changes * Make best-efforts recovery verify SST unique ID before Version construction (#10962) * Introduce `epoch_number` and sort L0 files by `epoch_number` instead of `largest_seqno`. `epoch_number` represents the order of a file being flushed or ingested/imported. Compaction output file will be assigned with the minimum `epoch_number` among input files'. For L0, larger `epoch_number` indicates newer L0 file. ### Bug Fixes * Fixed a regression in iterator where range tombstones after `iterate_upper_bound` is processed. * Fixed a memory leak in MultiGet with async_io read option, caused by IO errors during table file open * Fixed a bug that multi-level FIFO compaction deletes one file in non-L0 even when `CompactionOptionsFIFO::max_table_files_size` is no exceeded since #10348 or 7.8.0. * Fixed a bug caused by `DB::SyncWAL()` affecting `track_and_verify_wals_in_manifest`. Without the fix, application may see "open error: Corruption: Missing WAL with log number" while trying to open the db. The corruption is a false alarm but prevents DB open (#10892). * Fixed a BackupEngine bug in which RestoreDBFromLatestBackup would fail if the latest backup was deleted and there is another valid backup available. * Fix L0 file misorder corruption caused by ingesting files of overlapping seqnos with memtable entries' through introducing `epoch_number`. Before the fix, `force_consistency_checks=true` may catch the corruption before it's exposed to readers, in which case writes returning `Status::Corruption` would be expected. Also replace the previous incomplete fix (#5958) to the same corruption with this new and more complete fix. * Fixed a bug in LockWAL() leading to re-locking mutex (#11020). * Fixed a heap use after free bug in async scan prefetching when the scan thread and another thread try to read and load the same seek block into cache. * Fixed a heap use after free in async scan prefetching if dictionary compression is enabled, in which case sync read of the compression dictionary gets mixed with async prefetching * Fixed a data race bug of `CompactRange()` under `change_level=true` acts on overlapping range with an ongoing file ingestion for level compaction. This will either result in overlapping file ranges corruption at a certain level caught by `force_consistency_checks=true` or protentially two same keys both with seqno 0 in two different levels (i.e, new data ends up in lower/older level). The latter will be caught by assertion in debug build but go silently and result in read returning wrong result in release build. This fix is general so it also replaced previous fixes to a similar problem for `CompactFiles()` (#4665), general `CompactRange()` and auto compaction (commit 5c64fb6 and 87dfc1d). * Fixed a bug in compaction output cutting where small output files were produced due to TTL file cutting states were not being updated (#11075). ### New Features * When an SstPartitionerFactory is configured, CompactRange() now automatically selects for compaction any files overlapping a partition boundary that is in the compaction range, even if no actual entries are in the requested compaction range. With this feature, manual compaction can be used to (re-)establish SST partition points when SstPartitioner changes, without a full compaction. * Add BackupEngine feature to exclude files from backup that are known to be backed up elsewhere, using `CreateBackupOptions::exclude_files_callback`. To restore the DB, the excluded files must be provided in alternative backup directories using `RestoreOptions::alternate_dirs`. ### Public API Changes * Substantial changes have been made to the Cache class to support internal development goals. Direct use of Cache class members is discouraged and further breaking modifications are expected in the future. SecondaryCache has some related changes and implementations will need to be updated. (Unlike Cache, SecondaryCache is still intended to support user implementations, and disruptive changes will be avoided.) (#10975) * Add `MergeOperationOutput::op_failure_scope` for merge operator users to control the blast radius of merge operator failures. Existing merge operator users do not need to make any change to preserve the old behavior ### Performance Improvements * Updated xxHash source code, which should improve kXXH3 checksum speed, at least on ARM (#11098). * Improved CPU efficiency of DB reads, from block cache access improvements (#10975). 2023-03-02T01:00:53+00:00 MISP v2.4.169 2023-03-14T20:45:17+00:00  We are pleased to announce the immediate availability of [MISP v2.4.169](https://github.com/MISP/MISP/releases/tag/v2.4.169) with various improvements and bug fixes. It includes many improvement [release](https://github.com/MISP/misp-stix/releases/tag/v2.4.169) of [misp-stix](https://github.com/MISP/misp-stix), the core Python library for importing and exporting STIX (1, 2.0 and 2.1). # Improvements - New MISP workflow module to support Splunk HEC export. - Sighting ReSTsearch reworked to make it faster. - dashboard-widget:TrendingTags improved with new filtering and over time functionalities. - New ApacheSecureAuth authentication scheme added. # Fixes - TAXII servers invalid baseurl field type fixed. - Restore bro export (temporary fix until a complete rework of the bro export in ReSTsearch). A huge thanks to all the contributors and supporters of the MISP project. This release wouldn't be possible without the help of all the organisations and people supporting us to make MISP a reality. Go to the detailed [changelog](https://www.misp-project.org/Changelog.txt) for more details about the changes to the MISP core software. # Other updates and changes in the MISP project ## MISP Objects - A new MISP object `ransomware-group-post` has been created to support [ransomlook.io](https://www.ransomlook.io/). - Improved `victim` object. - A new MISP object `transport-ticket` has been created to share information about transports in MISP. - Various improvements to `network-connection`, `network-socket`. - A new MISP object `registry-key-value` For more details, the [misp-object changelog](https://www.misp-project.org/Changelog-misp-objects.txt) is available. ## MISP Galaxy - A new MISP galaxy `first-dns` matrix describing DNS abuse techniques has been added. - Various improvements in different galaxy such as `threat-actors`, `sigma`, `stealer`, `tools`, `region`, `360net`, MITRE ATT&CK. For more details, the [misp-galaxy changelog](https://www.misp-project.org/Changelog-misp-galaxy.txt) is available. ## MISP warning-lists - New `captive-portals` warning list added. - New `parking` page warning list added. For more details, the [misp-warninglists changelog](https://www.misp-project.org/Changelog-misp-warninglists.txt) is available. # Don't forget to follow us on Mastodon The MISP projet has its own Mastodon server [misp-community.org](https://misp-community.org/) - don't forget to follow @misp@misp-community.org on the fediverse. Core contributors of MISP can sign-up if they wish to have an account. # MISP Professional Services [MISP Professional Services (MPS)](https://www.misp-project.org/professional-services/) is a program handled by the lead developers of MISP Project, in order to offer highly skilled services around MISP and to support the sustainability of the MISP project. This initiative is meant to address the policy requirements of companies/organisations requiring commercial support contracts. Don't hesitate to get in touch with us if you need specific services. 2023-03-14T20:45:17+00:00 rocksdb v8.0.0 2023-03-18T00:15:53+00:00 ## 8.0.0 (02/19/2023) ### Behavior changes * `ReadOptions::verify_checksums=false` disables checksum verification for more reads of non-`CacheEntryRole::kDataBlock` blocks. * In case of scan with async_io enabled, if posix doesn't support IOUring, Status::NotSupported error will be returned to the users. Initially that error was swallowed and reads were switched to synchronous reads. ### Bug Fixes * Fixed a data race on `ColumnFamilyData::flush_reason` caused by concurrent flushes. * Fixed an issue in `Get` and `MultiGet` when user-defined timestamps is enabled in combination with BlobDB. * Fixed some atypical behaviors for `LockWAL()` such as allowing concurrent/recursive use and not expecting `UnlockWAL()` after non-OK result. See API comments. * Fixed a feature interaction bug where for blobs `GetEntity` would expose the blob reference instead of the blob value. * Fixed `DisableManualCompaction()` and `CompactRangeOptions::canceled` to cancel compactions even when they are waiting on conflicting compactions to finish * Fixed a bug in which a successful `GetMergeOperands()` could transiently return `Status::MergeInProgress()` * Return the correct error (Status::NotSupported()) to MultiGet caller when ReadOptions::async_io flag is true and IO uring is not enabled. Previously, Status::Corruption() was being returned when the actual failure was lack of async IO support. * Fixed a bug in DB open/recovery from a compressed WAL that was caused due to incorrect handling of certain record fragments with the same offset within a WAL block. ### Feature Removal * Remove RocksDB Lite. * The feature block_cache_compressed is removed. Statistics related to it are removed too. * Remove deprecated Env::LoadEnv(). Use Env::CreateFromString() instead. * Remove deprecated FileSystem::Load(). Use FileSystem::CreateFromString() instead. * Removed the deprecated version of these utility functions and the corresponding Java bindings: `LoadOptionsFromFile`, `LoadLatestOptions`, `CheckOptionsCompatibility`. * Remove the FactoryFunc from the LoadObject method from the Customizable helper methods. ### Public API Changes * Moved rarely-needed Cache class definition to new advanced_cache.h, and added a CacheWrapper class to advanced_cache.h. Minor changes to SimCache API definitions. * Completely removed the following deprecated/obsolete statistics: the tickers `BLOCK_CACHE_INDEX_BYTES_EVICT`, `BLOCK_CACHE_FILTER_BYTES_EVICT`, `BLOOM_FILTER_MICROS`, `NO_FILE_CLOSES`, `STALL_L0_SLOWDOWN_MICROS`, `STALL_MEMTABLE_COMPACTION_MICROS`, `STALL_L0_NUM_FILES_MICROS`, `RATE_LIMIT_DELAY_MILLIS`, `NO_ITERATORS`, `NUMBER_FILTERED_DELETES`, `WRITE_TIMEDOUT`, `BLOB_DB_GC_NUM_KEYS_OVERWRITTEN`, `BLOB_DB_GC_NUM_KEYS_EXPIRED`, `BLOB_DB_GC_BYTES_OVERWRITTEN`, `BLOB_DB_GC_BYTES_EXPIRED`, `BLOCK_CACHE_COMPRESSION_DICT_BYTES_EVICT` as well as the histograms `STALL_L0_SLOWDOWN_COUNT`, `STALL_MEMTABLE_COMPACTION_COUNT`, `STALL_L0_NUM_FILES_COUNT`, `HARD_RATE_LIMIT_DELAY_COUNT`, `SOFT_RATE_LIMIT_DELAY_COUNT`, `BLOB_DB_GC_MICROS`, and `NUM_DATA_BLOCKS_READ_PER_LEVEL`. Note that as a result, the C++ enum values of the still supported statistics have changed. Developers are advised to not rely on the actual numeric values. * Deprecated IngestExternalFileOptions::write_global_seqno and change default to false. This option only needs to be set to true to generate a DB compatible with RocksDB versions before 5.16.0. * Remove deprecated APIs `GetColumnFamilyOptionsFrom{Map|String}(const ColumnFamilyOptions&, ..)`, `GetDBOptionsFrom{Map|String}(const DBOptions&, ..)`, `GetBlockBasedTableOptionsFrom{Map|String}(const BlockBasedTableOptions& table_options, ..)` and ` GetPlainTableOptionsFrom{Map|String}(const PlainTableOptions& table_options,..)`. * Added a subcode of `Status::Corruption`, `Status::SubCode::kMergeOperatorFailed`, for users to identify corruption failures originating in the merge operator, as opposed to RocksDB's internally identified data corruptions ### Build Changes * The `make` build now builds a shared library by default instead of a static library. Use `LIB_MODE=static` to override. ### New Features * Compaction filters are now supported for wide-column entities by means of the `FilterV3` API. See the comment of the API for more details. * Added `do_not_compress_roles` to `CompressedSecondaryCacheOptions` to disable compression on certain kinds of block. Filter blocks are now not compressed by CompressedSecondaryCache by default. * Added a new `MultiGetEntity` API that enables batched wide-column point lookups. See the API comments for more details. 2023-03-18T00:15:53+00:00 osquery 5.8.2 2023-03-22T11:59:16+00:00 2023-03-22T11:59:16+00:00 MONARC v2.12.6 2023-03-24T09:39:17+00:00 **New features** - Analysis background import. - Support of PHP8. - Specific error message on a wrong password input of analysis import. **Fixes** - Recommendations modification from the Knowledge Base when due date is set. - Recommendations modification fix of loading the linked recommendation set. 2023-03-24T09:39:17+00:00 Lookyloo v1.19.0 2023-03-30T10:25:46+00:00 # New features * The email notification now attaches the contacts, making takedown requests easier. * (WiP) Add settings for comparing captures. It is not possible to ignore domains and/or a substring in a resource URL loaded from the landing page. * Update [PyLookyloo](https://github.com/Lookyloo/PyLookyloo) to pass the settings when comparing captures * [Admin users only] Modal to trigger admin-only tasks on a specific capture (hide/rebuild) # Bugfixes * Fix docker compose (thanks to @bib0x) * Avoid exception at multiple places when a capture is invalid for any reason # Changes * Force protocol 5 for pickles (requires python 3.8, but lookyloo already required it anyway) * Optimize pickle before storing, and archive them to reduce diskspace * Bump dependencies (js & python) * Improve logging (add capture UUID when possible, makes debugging easier) * Always use `LookylooException` instead of `Exception` * Update Playwright in [PlaywrightCapture](https://github.com/Lookyloo/PlaywrightCapture) * Improve logging in [har2tree](https://github.com/Lookyloo/har2tree) 2023-03-30T10:25:46+00:00