http://open-source-security-software.net/organization/CIRCL/releases.atom 2024-05-07T07:25:58.931242+00:00 python-feedgen pandora v1.0.0 2022-06-29T16:30:21+00:00 This is the first official stable open source release of [Pandora](https://github.com/pandora-analysis/pandora). Pandora is an analysis framework to discover if a file is suspicious and conveniently show the results. The solution can be installed on-prem to avoid information leak in organisations. It has been tested relatively extensively over the last few months, but there might still be issues. If anything goes wrong, please open an issue and we will do our best to solve it. If you want to test Pandora without installing it, the online version is available at [pandora.circl.lu](https://pandora.circl.lu/). # Core functionalities * Easy to implement workers to analyze specific file formats, or connect to third party services * Admin interface * Session-based user management interface, and sharing * Generate a preview of the submitted document (if applicable) * Extract indicators/observables from submitted files * Extract content of archives * Extract attachments from email in EML and MSG format * Extract text content (if applicable) * Extract EXIF metadata * Pool service to fetch emails from an IMAP mailbox * Notify Administrator * MISP export and submission (admin only) * Statistics (admin only) * Role management (admin only) * Locally defined observables (legitimate/suspicious) (admin only) # Screenshots ## Submission interface  ## Result page  ---------------  2022-06-29T16:30:21+00:00 pandora v1.0.1 2022-07-11T14:44:26+00:00 Quick release to improve handling of submissions with passwords. # New features * Support password on submit via API. # Maintenance * Bugfixes * Dependencies update. 2022-07-11T14:44:26+00:00 pandora v1.1.0 2022-08-31T09:25:44+00:00 # Breaking change This release requires poetry v1.2.0 or more recent. Run the following command to upgrade it: `poetry self update` # New feature * HTML documents can be submitted to [Lookyloo](https://github.com/Lookyloo/lookyloo) (requires v1.15.0 or more recent)   # Changes * Improvements in the modules (archives, ISO, EML) * Improvements on the stats page * Configure the links on the index * Bump dependencies # Fixes * Support ingesting a file downloaded from a Pandora instance * Automatically restart unoserver when it crashes (makes previews with libreoffice more reliable) # Notes * Many have reported issues with the reviews generated by LibreOffice. A seemingly universal fix seems to be installing the [full package from the PPA](https://github.com/pandora-analysis/pandora#important-notes-regarding-libreoffice). 2022-08-31T09:25:44+00:00 pandora v1.2.0 2022-10-27T12:23:19+00:00 This is a pretty major maintenance release aiming to make the code easier to maintain. There are no big new features but a lot of the back end code was heavily modified. # Changes * Remove all asserts, replace them with proper exceptions * UI improvements on the submission and analysis pages * Detailed view (WiP) to reduce the visual load on the user * Add bandit, pylint, codespell, flake8 and semgrep in the github actions (many thanks to @juju4 for the heavy lifting) # Fixes * Better handling of date times on non-UTC machines * Support for new exiftool 2022-10-27T12:23:19+00:00 pandora v1.3.0 2022-12-30T12:12:36+00:00 # Breaking change Poetry v1.3.0 or more recent is now required, please [upgrade](https://python-poetry.org/docs/#installation) to the latest version. # New Features * Support for CAB files (with hachoir) * Support for VHD files (with dfvfs) * Pre-load the analysis results instead of loading it in javascript after the page is loaded * Much better handling of ODF files * Improve extraction of URLs from office documents * Improve logging, use a config file * Improve processes handling and avoid deadlocks on stop * [API only] Get stats by worker and mimetype # Bugfixes * Improve handling of 7z files * Add timeout on observables worker (avoid exceptions is a sample has way too many observables) * Improve mimetype synonyms to match the types better between python core mimetypes and the python-magic library * Improve handling of html bodies in emails, support for signed emails * Improve handling of password protected OLE files * Avoid getting the IMAP handler stuck (timeout) # Changes * Major improvements in rendering the stats page * Major improvements in exceptions handling * Autokill libreoffice every hours to avoid memory leak * Allow to mark issues with extractor module as error instead of alert when we reach the limits (size or number of files in archives) * Improve UI and docs, move to bootstrap 5.2 * Updates all dependencies 2022-12-30T12:12:36+00:00 pandora v1.3.1 2023-01-10T17:10:40+00:00 # Security patch This releases fixes [CVE-2023-22898](https://cve.circl.lu/cve/CVE-2023-22898) where a nested archive (aka ZIP Bomb) could trigger a DOS to the platform, especially to the extractor module. Thank you @kurgans0 for reporting it. # New features * Limit the amount of archives to recursively extract from a file, and the maximal depth (Fixes CVE-2023-22898) * Display link to VT report instead of text in the report # Changes * Many improvements in the dfVFS extractor, support files with multiple filesystems * Improve mime types synonyms * Improve notification email (set reply-to if possible, insert full link in email body. * Bump all dependencies # Bug fixes * Fix exception on edge cases when using the dfVFS extractor * Only allow submitting one file at the time - the UI was allowing multiple files by mistake, it wasn't supposed to be supported and causes UI issues. Supporting multiple upload will be implemented later. 2023-01-10T17:10:40+00:00 pandora v1.4.0 2023-03-31T10:40:19+00:00 # New features * Progress bar when uploading a file * Add SMTP auth for email notification (thanks to @sebdraven) # Changes * Many UI Improvements * Much improved logging all around * Improvements in blocklist module * Improvements in workers initialization. * Bump dependencies (Python, JS), make the project compatible with Python 3.11 * Remove IRMA module (project deprecated) * Code cleanup and maintenance # Bugfixes * Fix Dockerfile and docker compose * Better support for MSG files 2023-03-31T10:40:19+00:00 pandora v1.5.0 2023-06-30T14:00:14+00:00 This is mostly a maintenance release with a whole bunch of bugfixes. # Changes * Many improvements in the logging * Improve sessions handling * Improve SMTP login settings # Bugfixes * Calling dfvfs was globally changing the timezone in the python environment * Use the right version of kvrocks * Better handling of timeouts * Cleanup in observables extraction 2023-06-30T14:00:14+00:00 pandora v1.6.0 2023-10-13T10:55:34+00:00 # New features * Module to decode QR codes # Changes * Improve support of recent Libreoffice * Support latest redis and kvrocks releases * Bump dependencies, yara signatures # Bugfixes * Maintenance and cleanups, mainly related to dependencies updates. Note that this release isn't compatible with python 3.12. The next one will be. 2023-10-13T10:55:34+00:00 Lookyloo v1.3 2020-12-24T11:58:56+00:00 Thanks to [Internews](https://globaltech.internews.org/) and the [BASICS Project](https://globaltech.internews.org/our-resources/basics) we were able to greatly improve Lookyloo over the last few months. These release notes won't be exhaustive and if you want to see all the changes, you should have a look at the git changelog. Here is a short overview of the main changes in the last ~6 months: * Major rewrite of the user interface based on the user tests realized thank to the [BASICS Project](https://globaltech.internews.org/our-resources/basics) and other user feedbacks * Major improvements in the investigation popup * [Documentation website](https://www.lookyloo.eu/docs/main/), also thanks to the [BASICS Project](https://globaltech.internews.org/our-resources/basics) * Indexing of cookies and hashes of resources * Add support for marking specific resources as known (libraries, icons, ... related to a specific domain or not) or malicious (phishing, malwares) * DNS resolution (IP and CNAMEs) * Query third party services via a modules system (SaneJS, VirusTotal, Phishing Initiative) * Configuration via config files * Update script * Statistics of the whole instance * Export in [MISP](https://www.misp-project.org/) format * Systemd templates * Docker image * Update dependencies and bug fixes all over the place. 2020-12-24T11:58:56+00:00 Lookyloo v1.4.0 2021-02-09T16:11:43+00:00 Once again, many of the changes in this release wouldn't have been possible without the support of [Internews](https://globaltech.internews.org/) and the [BASICS Project](https://globaltech.internews.org/our-resources/basics). On the UI front, we now have a better support of huge screenshots and many more tooltips are shown when the mouse goes over icons and different parts of the tree. It should make the tree easier to read for users discovering the platform. The main new feature of this release is the integration with [MISP](https://www.misp-project.org/). It is now possible to export a capture directly to a pre-configured MISP instance:  The documentation in order to get it to work [is also available](https://www.lookyloo.eu/docs/main/lookyloo-integration.html#_misp). There were also quite a few changes for the administrators of a Lookyloo instance, especially the [authentication](https://www.lookyloo.eu/docs/main/lookyloo-auth.html). And for more details, you should have a look at the git changelogs. 2021-02-09T16:11:43+00:00 Lookyloo v1.5.0 2021-04-02T14:02:45+00:00 Once again, many of the changes in this release wouldn't have been possible without the support of [Internews](https://globaltech.internews.org/) and the [BASICS Project](https://globaltech.internews.org/our-resources/basics). And we would also like to thank [Credit Agricole](https://www.credit-agricole.com/en/) and @FafnerKeyZee for the continuous bug reports! The main new feature of this release is the possibility to capture URLs present in a capture you already made, all that while **keeping the context** your browser was in (cookies, user-agent, referer) in the **subsequent captures**. It is especially useful when the page you're landing on expects the user to click on a link in order to load the content, the website checks the referer and/or cookies, and bounce you if you're not presenting the right session. This feature will be further extended in the upcoming releases to allow other types of requests (`POST`), and let the user choose the link(s) to captures from the screenshot of the page itself. This release also adds a new **background indexer** so the captures queued with the API are automatically cached even if they are never opened in the browser. And there are the usual bunch of bugfixes, improvements and dependencies upgrades. And we also require python 3.8+. You can also **search** for hostnames, URLs, hashes, and cookies names from the `/search` entry point. This entry point is *not* listed yet in the documentation, but it will be added soon. 2021-04-02T14:02:45+00:00 Lookyloo v1.6.0 2021-05-21T18:33:41+00:00 This release contains lots of changes in the backend (described below), and a few improvements on the web interface: * Hide the captures with error from the index (see [hide_captures_with_error](https://www.lookyloo.eu/docs/main/lookyloo-configuration.html#_optional_features)) * Return resources as text instead of in a zip file * Crop and blur screenshot if it is too big to be displayed as-is * Redesign of the menus * Fix rendering of image resources in the investigation popup The backend changes are mainly improving the overall performance of Lookyloo, with a few new features: * All the captures (web and API) are using the asynchronous capture script, and the priority of each capture is weighted depending on the origin and the user(see [priority](https://www.lookyloo.eu/docs/main/lookyloo-configuration.html#_core_config), the number of async capture process is configurable (see [async_capture_processes](https://www.lookyloo.eu/docs/main/lookyloo-configuration.html#_core_config)) * The index is cached in memory by the webserver, making the index view lot faster after first load * Improve auto-trigger of 3rd party modules (configurable [per module](https://www.lookyloo.eu/docs/main/lookyloo-integration.html)) * Add [optional integration]() of whois queries with [uWhoisd](https://www.lookyloo.eu/docs/main/lookyloo-integration.html#_uwhoisd) * Disable [FLoC](https://amifloced.org/) globally * Many bug fixes in [har2tree](https://github.com/Lookyloo/har2tree) and the creation of the tree * Fix and improve MISP export, support subsequent captures as extended events * Update all dependencies 2021-05-21T18:33:41+00:00 Lookyloo v1.7.0 2021-07-21T16:38:24+00:00 The two main changes in this release are: * Add support for passing a proxy to a capture, thanks to @Felalex57 - [Documentation](https://www.lookyloo.eu/docs/main/usage.html#_proxy) * Major improvement in the API using [flask-restx](https://github.com/python-restx/flask-restx) - [Documentation](https://lookyloo.circl.lu/doc/) on the demo Lookyloo instance. * Add lookup against a MISP instance - [Documentation](https://www.lookyloo.eu/docs/main/lookyloo-integration.html#_lookup_on_a_misp_instance) * Add sample config for log rotate thanks to @FafnerKeyZee - [Documentation](https://www.lookyloo.eu/docs/main/install-lookyloo-production.html#_logging_and_rotation) The other changes are mainly bugfixes and small changes: * Avoid receiving notifications from bots * Upgrade the bundled-in list of user agents * Improve generation of the pickles and avoid doing it twice * Add reference to parent in the case a capture is initiated from an other one * Improve MISP export 2021-07-21T16:38:24+00:00 Lookyloo v1.8.0 2021-08-30T13:34:31+00:00 **New Features**: * Integration with [urlscan.io](https://urlscan.io) - [Documentation](https://www.lookyloo.eu/docs/main/lookyloo-integration.html#_urlscan_io) * Trigger a capture from the URL - https://github.com/Lookyloo/lookyloo/issues/248 * Archiving: the captures more than 6 month old ([configurable](https://www.lookyloo.eu/docs/main/lookyloo-configuration.html#_core_config)) are moved to an archive directory so they're not listed on the index anymore, but the captures can still be accessed by UUID (doesn't break permanent URLs) * Index file by directory for each captures (archived or not). Greatly reduces the I/O when initializing the known captures in redis. **Fixes**: * Missing 3rd party web dependencies in docker (thanks to @FafnerKeyZee) **Changes** - This release is implementing a lot of back end changes : * The captures are now stored a by year and month (instead of in a single directory) to avoid having too many entries in the same directory (ext4 dislikes it). All the new captures are following this new architecture, but you need to run `tools/change_captures_dir.py` to move the existing ones to the new format (only useful if you feel restarting the app takes too much time) * Move all the capture-related code from `Lookyloo` to `AsyncCapture` * Move all the services management code to abstractmanager * Use redis pooling to manage connections to the database in `Lookyloo` and `Indexing` * New process to trigger occasional actions, currently: generate the daily user-agent file if Lookyloo is using the UAs of its own users. * Reinitialize the list of captures UUIDs when starting the app instead of the in website itself * Improvements in processes handling (TL;DR: don't stop redis until all the async captures processes are down) * Move some methods from `Lookyloo` to the helpers * Simplify code in `Lookyloo` to make it more readable, remove dead code. * Bump dependencies, add `hiredis` to speed up redis interactions * Return proper HTTP error codes (mostly 4XX), when appropriate 2021-08-30T13:34:31+00:00 Lookyloo v1.9.0 2021-09-28T16:23:08+00:00 # New features * Integration with [Phishtank](https://phishtank.org/) via [Phishtank Lookup](https://github.com/Lookyloo/phishtank-lookup) - [Documentation](https://www.lookyloo.eu/docs/main/lookyloo-integration.html#_phishtank_lookup_v1_9). ---  --- * Simple [monitoring script](https://github.com/Lookyloo/lookyloo/blob/main/tools/monitoring.py) to keep an eye on the health of the instance, run it in a tmux/screen with watch. ---  --- * Link in the tree menu to re-trigger a capture on the same URL. # Fixes * Improve logging entries, the date was incomplete. * Add UUID file in export. * Inform users when a capture failed critically and we have nothing to show. * Catch timeout when pushing to MISP (avoid exception) # Changes * Major improvements in caching, better handling of exceptions and keep a limited amount of pickles in memory. * Simplify code in the async capture script. * Add urlscan.io permalink in MISP export * Add phishtank permalink in MISP export * Move modules to dedicated files 2021-09-28T16:23:08+00:00 Lookyloo v1.10 2021-12-03T11:00:08+00:00 # New features * [Hashlookup](https://www.circl.lu/services/hashlookup/) integration - [Documentation](https://www.lookyloo.eu/docs/main/lookyloo-integration.html#_hashlookup_v1_10) * Pass arbitrary HTTP headers to captures - [Documentation](https://www.lookyloo.eu/docs/main/usage.html#_capture_configuration) * Pass arbitrary User-Agents to captures - [Documentation](https://www.lookyloo.eu/docs/main/usage.html#_browser_configuration) * Get hashes of all the resources using any algorithm supported by Python (API) * Add configuration setting to make captures private by default - See `default_public` in the [Documentation](https://www.lookyloo.eu/docs/main/lookyloo-configuration.html#_core_config) * Add [CORS](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS) settings to allow JavaScript submissions (required for the [browser extension](https://github.com/Lookyloo/webext)) * Defang URLs in email notifications # Fixes * Avoid exception when the timestamp of a capture has no millisecond * Avoid exceptions in archiver when indexes are broken # Changes * Improve logging * Improve capture page * Normalize tooltips across the app * Save redis databases to disk less often * Programmatically shutdown redis databases (synchronous) * Bump dependencies 2021-12-03T11:00:08+00:00 Lookyloo v1.10.0 2021-12-03T11:00:08+00:00 # New features * [Hashlookup](https://www.circl.lu/services/hashlookup/) integration - [Documentation](https://www.lookyloo.eu/docs/main/lookyloo-integration.html#_hashlookup_v1_10) * Pass arbitrary HTTP headers to captures - [Documentation](https://www.lookyloo.eu/docs/main/usage.html#_capture_configuration) * Pass arbitrary User-Agents to captures - [Documentation](https://www.lookyloo.eu/docs/main/usage.html#_browser_configuration) * Get hashes of all the resources using any algorithm supported by Python (API) * Add configuration setting to make captures private by default - See `default_public` in the [Documentation](https://www.lookyloo.eu/docs/main/lookyloo-configuration.html#_core_config) * Add [CORS](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS) settings to allow JavaScript submissions (required for the [browser extension](https://github.com/Lookyloo/webext)) * Defang URLs in email notifications # Fixes * Avoid exception when the timestamp of a capture has no millisecond * Avoid exceptions in archiver when indexes are broken # Changes * Improve logging * Improve capture page * Normalize tooltips across the app * Save redis databases to disk less often * Programmatically shutdown redis databases (synchronous) * Bump dependencies 2021-12-03T11:00:08+00:00 Lookyloo v1.11.0 2022-03-31T11:17:11+00:00 # New Feature * Trigger multiple captures at once from web interface  # Fixes * Improve MISP event publishing (make it asynchronous) * Improve legend with titles on hoover * Fix caches in modules * Improve stats page * Normalize buttons color * Improve rendering of capture page # Changes * Updates all web and python dependencies * Use bootstrap 5 2022-03-31T11:17:11+00:00 Lookyloo v1.12.0 2022-05-24T13:33:18+00:00 # New Features ## Playwright The captures are now made via [Playwright](https://playwright.dev/) instead of [Splash](https://github.com/scrapinghub/splash). It is a major improvement as Playwright uses actual up-to-date browsers, in headless mode (instead of qt-webkit from ~2016). You can read more about the research that lead to this change [in the discussion](https://github.com/Lookyloo/lookyloo/discussions/243). The main other advantages of using playwright are the following: * Easier to install: it doesn't requires Docker in order to use Splash * Much better control of what happen in the browser while capturing: Playwright makes it extremely simple to instrument everything in the browsers. The capturing module already tries to solve reCaptcha if it detects it on the page. The capture is made by a [standalone](https://github.com/Lookyloo/PlaywrightCapture) python module that you can use in your own tools if you wish to. ## De-duplication If the exact same capture is triggered multiple times within 5 min, it is skipped and the requestor is redirected to the capture done before. # Fixes * Avoid discarding a capture on network error: when a redirect is broken down the line, we keep the chain up to that point * Issue when the MISP was submitted as un-published * [Docker] Properly handle archiving * [Docker] Init SRI hashes # Changes * Improve subsequent capture template on long URLs * Improve view of the capture page on small-ish screens * General maintenance and code cleanup * Improvement in the tree generation on edge cases * Bump JS/CSS libraries * Update bundled-in User-Agent file * Use pydeep2, comes with a bundled-in libfuzzy, easier to install. 2022-05-24T13:33:18+00:00 Lookyloo v1.13.0 2022-06-26T16:06:50+00:00 # Maintenance and bug-fixes release All releases don't need to contain new features, sometimes, it is just some cleanup, and it is okay. * Properly handle exceptions in some edge cases (fixes in har2tree) * Properly display an error message if the capture fails * Use the same default User-Agent in when a capture is submitted via the API as via the web interface. * Cleanup some legacy code * Bump all dependencies (JS/CSS and Python) # Still, there is a new-ish thing We revamped the package generator, and it should be [more usable](https://github.com/Lookyloo/lookyloo/pkgs/container/lookyloo). If it is not, let us know! 2022-06-26T16:06:50+00:00 Lookyloo v1.14.0 2022-08-08T13:53:34+00:00 # New features * Trigger a capture on a web enabled document provided as a file instead of a URL. Useful for HTML files attached to emails, or HTML body in email.  -------- * Compress (gzip) the HAR file in archived captures - saves a lot of disk space. * Support for RiskIQ Passive DNS (requires API key) * Display SSL/TLS information available in the HAR dump from Playwright  -------- * Optional DoNotTrack HTTP header in capture  -------- * Display size of rendered page on hostnode popup. * [WiP] Download files when the URL captured to a downloadable file (PDF, Office doc, ...) (**Important note** the downloaded file is not exposed to the user yet) * [WiP] List all hashes available in the capture, sort them by frequency. Makes it easier to find phishing sides using the same resources.  # Fixes * Major speed improvements when displaying the hostnode popups (only show the recent cached captures by default) * Improvements in the caching mechanism * Cleanup data showed by monitoring script * Avoid crashes when RiskIQ isn't reachable # Changes * Update dependencies (js, python) * Improve logging in archiver * Improve config file 2022-08-08T13:53:34+00:00 Lookyloo v1.15.0 2022-08-25T12:43:29+00:00 # Breaking change * Lookyloo requires Redis 7.0 or more decent. The upgrade process is as follows: 1. Go to the Redis directory (should be in the same directory as where you cloned Lookyloo) 2. Run the following commands ``` git fetch git checkout 7.0 make distclean make -j4 make test ``` 3. You now have the new version of redis in place, you can update lookyloo as usual. # New features * Use pre-configured devices from Playwright (mobile only for now)  * Download files when the URLs points to a downloadable content  * Submit downloadable content to [Pandora](https://pandora.circl.lu/submit) (if available) * Automatically select the most appropriate browser engine based on the user-agent # Fixes * Make sure all the gunicorn instances displays all the recent captures * Other bugfixes and GUI improvements # Changes * Improve capture page with radio button to select which user-agent to submit * Bump dependencies 2022-08-25T12:43:29+00:00 Lookyloo v1.16.0 2022-10-29T13:19:39+00:00 # Breaking change This release requires poetry v1.2.0 or more recent. Run the following command to upgrade it: `poetry self update` # New Features * Move to Lacus/LacusCore, many changes to make lookyloo's code compatible with it * [Lacus](https://github.com/ail-project/lacus) and [PyLacus](https://github.com/ail-project/pylacus): use this mode to trigger the capture from an other machine than the one you run Lookyloo from * [LacusCore](https://github.com/ail-project/lacuscore): (the default) keep triggering the captures from the same machine as the one lookyloo is running on With lacus, the captures are more reliable and using Lacus as a web service allows to monitor them better. If you want to use the webservice, you'll need to: 1. [Install Lacus](https://github.com/ail-project/Lacus#install-guide) * make sure it is running by loading http://127.0.0.1:7100 on the machine you have it running on (7100 is the default port, you can of course change it) 2. Edit the config file `config/generic.json` (key `remote_lacus`): * set `enable` to `true` * set `url` to the url your loolyloo instance can use to connect to lacus: `http://<ip>:<port>` 3. Restart lookyloo & try it # Changes * Make hashlookup visible to everyone * Improve loggingv1.15.0 * Maintenance: use poetry 1.2, bump deps (Python and JS), bump Github actions * Improve caching with Lacus * Retry failing captures when it might be a temporary issue (typically domain resolution) # Fixes * Avoid triggering a capture (and failing) when the URL and documents are missing * Issue with urlscan when the capture had no referer * Better handling of exceptions in VT module * Better handling of devices offered by Playwright and their user-agents 2022-10-29T13:19:39+00:00 Lookyloo v1.17.0 2022-12-29T13:48:48+00:00 # Breaking change Poetry v1.3.0 or more recent is now required, please [upgrade](https://python-poetry.org/docs/#installation) to the latest version. # New features: * See [Lacus release notes for v1.2.0](https://github.com/ail-project/lacus/releases/tag/v1.2.0) if you want more details on the changes regarding the captures. * Temporary local storage of captures if Lacus web-service isn't reachable temporarily * Submit and view a HAR file captured somewhere else, or a full Lookyloo capture from an other instance  * Show status of captures when they're submitted in bulk  * List of all downloadable contents for a capture in a modal (from the tree view -> `Download capture elements`):  # Bugfixes * Catch and stop script when generating a tree takes too long (link to doc) * [har2tree] Avoid exception when a node doesn't have a pageref * [har2tree] Better use of BeautifulSoup * [har2tree] Better handling of nodes that aren't loading a URL * [har2tree] Improve mimetime detection in HTTP responses # Changes * [har2tree] Improve decoding of POSTed data * Use more recent Flask and flask-restx * Bump deps (Python ans JS) * Major logging improvements in Lookyloo, LacusCore, PlaywrightCapture and har2tree * 2022-12-29T13:48:48+00:00 Lookyloo v1.18.0 2023-03-01T11:33:31+00:00 # New features * Beta support for [monitoring](https://github.com/Lookyloo/monitoring): the system will trigger a recurring capture and allow to compare them over time. * Beta support for comparing captures. For now, it focuses on the redirects from the URL captured to the landing page, and the URLs of the resources loaded on that page. An upcoming release of this feature will compare the rendered content of the landing page too. * Takedown endpoint (API only): gather contact information (whois on domain, IP, ASN, and [security.txt](https://github.com/Lookyloo/pysecuritytxt) file if available) for all the URLs up to the landing page. Can be used to notify owners in case of a malicious URL. * Flag known Cloudflare IPs on the hostnode popups * Trigger AAAA DNS requests (was only A before) # Bugfixes * When the capture is a file that was uploaded by the user, some modules cannot be triggered. Avoiding exceptions. * In some edge cases, a lock file for generating the tree could be left there even if the initiating script was dead. We now clean them up automatically. # Changes * Use [Lacus v1.3.0](https://github.com/ail-project/lacus/releases/tag/v1.3.0) / [LacusCore v1.3.0](https://github.com/ail-project/LacusCore/releases/tag/v1.3.0) / [PlaywrightCapture v1.18.0](https://github.com/Lookyloo/PlaywrightCapture/releases/tag/v1.18.0) * Allow to disable defanging URLs in emails * Many improvements in the rendering of the menus on the tree page * [Lookyloo] Bump javascript and python dependencies * [Har2Tree](https://github.com/Lookyloo/har2tree/releases/tag/v1.18.0) Maintenance, update dependencies 2023-03-01T11:33:31+00:00 Lookyloo v1.19.0 2023-03-30T10:25:46+00:00 # New features * The email notification now attaches the contacts, making takedown requests easier. * (WiP) Add settings for comparing captures. It is not possible to ignore domains and/or a substring in a resource URL loaded from the landing page. * Update [PyLookyloo](https://github.com/Lookyloo/PyLookyloo) to pass the settings when comparing captures * [Admin users only] Modal to trigger admin-only tasks on a specific capture (hide/rebuild) # Bugfixes * Fix docker compose (thanks to @bib0x) * Avoid exception at multiple places when a capture is invalid for any reason # Changes * Force protocol 5 for pickles (requires python 3.8, but lookyloo already required it anyway) * Optimize pickle before storing, and archive them to reduce diskspace * Bump dependencies (js & python) * Improve logging (add capture UUID when possible, makes debugging easier) * Always use `LookylooException` instead of `Exception` * Update Playwright in [PlaywrightCapture](https://github.com/Lookyloo/PlaywrightCapture) * Improve logging in [har2tree](https://github.com/Lookyloo/har2tree) 2023-03-30T10:25:46+00:00 Lookyloo v1.20.0 2023-06-01T14:21:18+00:00 This release is the outcome of a good two months of work on Lookyloo itself but also [Lacus](https://github.com/ail-project/lacus) and its dependencies leading to the [v1.5.0 release](https://github.com/ail-project/lacus/releases/tag/v1.5.0). It also improves the support for the [monitoring interface](https://github.com/Lookyloo/monitoring) (still to be considered beta). # New features * Compare captures via the API * Submit any for to [Pandora](https://github.com/pandora-analysis/pandora) (if available) * Allow automatic reporting via the API * Can set an email to notify in the monitoring form # Changes * Improve handling of long running processes, * Improve logging all over the place * Changes related to Lacus/LacusCore/PyLacus changes * Easy way to check if two captures are different or not * Store capture settings in the capture directory for potential later use * Show proxy in UI if one was given * Improve response when comparing captures # Bugfixes * Avoid issues when the pickle requires too much recursivity * Cloudflare services was always flagging URLs as their own * The usual batch of bugfixes all over the place 2023-06-01T14:21:18+00:00 Lookyloo v1.21.0 2023-06-30T20:46:08+00:00 # New Features * Allow to pass a timezone, geolocation coordinates, locale, and color scheme to a capture * Add a global proxy option in the settings * Improve SMTP auth for notifications # Changes * Store the capture settings in order to reuse them later (like for re-capture) * Avoid failing if Lacus isn't available ant retry a few times # Bugfixes * Properly handle captures with errors, improve logging accordingly * Resubmit captures if they were deleted on Lacus without storing a response (generally if something crashed) 2023-06-30T20:46:08+00:00 Lookyloo v1.22.0 2023-10-17T07:24:26+00:00 If you haven't been upgrading lookyloo since the last tagged release, this release contains *a lot* of changes. # New features * Support for [HTTP Headers Hashing](https://github.com/adulau/HHHash) * Support for archiving capture on S3FS, this is not completely implemented, but we use it on the public instance. Expect it to be usable for anyone in next release. * Store HARs dumps in gzip archives * MISP export when the capture downloads a file * Handle captures where we have a rendered HTML and a no-click download is triggered in Javascript (TODO: support multiple downloads triggered in a single capture) * Get downloaded file via the API * Fetch favicons using default URL, and HTML content * Support multiple MISP instances for submission and lookup # Changes * Better rendering of the capture time * Support re-processing captures that were mistakenly considered broken * Improve logging * Improve caching * Auto-restart webservice to avoid memory leak * Strip URL to capture (space and new line) * Update dependencies, new browsers # Bugfixes * Allow to run multiple backgroung indexing scripts * Many related to the compression of the HAR and the pickles to reduce disk use * Various encoding issues with rendered HTML For more details regarding the captures, see [Lacus v1.7.0 release notes](https://github.com/ail-project/lacus/releases/tag/v1.7.0). 2023-10-17T07:24:26+00:00 CIRCLean v0.1.0 2018-05-13T15:09:11+00:00 - first beta release of MOSP; - basic features are implemented: management of JSON objects, management of JSON schemas, management of users and organizations; - it is possible to edit a JSON object with a JSON editor which is generated thanks to the JSON schemas; - a basic API let the user interact programmatically with the JSON objects. 2018-05-13T15:09:11+00:00 CIRCLean v0.2.0 2018-05-30T05:36:34+00:00 - the JSONB PostgreSQL type is now used instead of the JSON type; - the JSON editor has been upgraded and is now properly working with Bootstrap 4.1; - the interface to edit JSON data has been revamped and is a lot cleaner; - DataTables is now used for all tables; - a new interface displays all the JSON schemas in the organization(s) of a user; - a panel to manage users of the platform has been added; - the Web interface is internationalized in French (80% of strings are actually translated); - various UI improvements. 2018-05-30T05:36:34+00:00 CIRCLean v0.3.0 2018-06-01T09:28:59+00:00 - new Web interface to list, create and edit JSON schemas; - improved management of users. It is now possible to block a user; - translations improvements; - various UI improvements. 2018-06-01T09:28:59+00:00 CIRCLean v0.4 2018-10-05T07:21:14+00:00 - it is now possible to select one or several licenses for an object (#2). A script is provided in order to initialize the database with licenses from https://spdx.org/licenses/licenses.json; - the values of a JSON object can now be exported to a CSV file; - the management of permissions has been improved; - added a new profile page for users; - various fixes and UI improvements. 2018-10-05T07:21:14+00:00 CIRCLean v0.5 2019-02-23T23:54:39+00:00 - major improvements to the API. It is now possible to create a valid JSON object programmatically with the HTTP POST method. The validity of the submitted object is checked against the specified JSON schema; - the project has now an official logo (#7); - a human.txt file has been added (https://objects.monarc.lu/human.txt). - various fixes and UI improvements. All views have been improved; - a documentation is now available and will be improved (https://www.monarc.lu/documentation/MOSP-documentation). 2019-02-23T23:54:39+00:00 CIRCLean v0.6 2019-03-11T23:04:28+00:00 - added the possibility to link objects (#8); - the footprint (SHA 256 sum of objects is now displayed); - added the possibility to copy an object to the clipboard; - added a terms page; - improved the organization page; - improved the admin/users page; - minor UI fixes. 2019-03-11T23:04:28+00:00 AIL-framework v0.90 2018-05-02T06:31:56+00:00 AIL Framework initial release v0.90 2018-05-02T06:31:56+00:00 AIL-framework v1.0 2018-05-11T14:33:40+00:00 AIL Framework version 1.0 released including a migration from Python 2 to Python 3. - Redis level-db has been moved to [ARDB](https://github.com/yinqiwen/ardb) - UI improvement including the ability to get paste in raw format - New Base64 detection module to find and decode Base64 data (and save extracted files including metadata) - New API key detection module such as Google, AWS or alike - New bitcoin address detection and validation module - Improved X.509 certificate detection (Keys module) - Many bug fixes 2018-05-11T14:33:40+00:00 AIL-framework v1.1 2018-06-06T08:26:49+00:00 A new release of the AIL Framework includes new functionalities such as tagging and classification relying on [MISP taxonomies](https://www.misp-project.org/taxonomies.html) and [galaxy](https://www.misp-project.org/galaxy.html). For more information, have a look at the [AIL wiki](https://github.com/CIRCL/AIL-framework/wiki/Tags). - Tagging can now be used in the UI and also the modules are automatically tagging based on detection/analysis. - Tags are now displayed while browsing important pastes and in search results. - A bug in duplicate modules detection has been fixed to avoid empty hash creation. - Duplicate hashes are now persistent and stored on disk. 2018-06-06T08:26:49+00:00 AIL-framework v1.2 2018-06-20T09:01:19+00:00 AIL Framework version 1.2 has been released including TheHive and MISP integration. - AIL Framework is now capable of auto-publishing (based on tags) event in [MISP](https://www.misp-project.org) or alerts in [TheHive](https://www.thehive-project.org/). Events or cases can be created on request when browsing pastes. - A new submit interface to ease the submission and processing of data in AIL Framework. - Introduction of a false-positive or true-positive on tag to allow the classification of information. This will be used later to improve automatic classification of information processed. - [GDPR document released](https://www.circl.lu/assets/files/information-leaks-analysis-and-gdpr.pdf) about the use of AIL in the scope of processing personal information. 2018-06-20T09:01:19+00:00 AIL-framework v1.3 2018-08-24T12:47:57+00:00 AIL Framework version 1.3 released including automatic decoding of files from unstructured data New features: - Detection of IBAN bank accounts are now included - A cleaner module for decoding files (Base64, hex encoded) from unstructured data - A new UI for browsing decoded files, their types and relationship Many bugs fixed and small improvements. 2018-08-24T12:47:57+00:00 AIL-framework v1.4 2018-10-02T15:36:25+00:00 AIL Framework version 1.4 released including Tor hidden services crawler and monitoring. Major new feature: - Tor hidden service crawler. AIL now includes the ability to crawl and parse output crawled from Tor hidden services. - Tor onion availability is monitored to detect up and down of hidden services. - Screenshots are captured and integrated in the analysed output. - Blurred interface functionality has been added to avoid "burning the eyes" of the security analyst with specific content. - As the collected information is part of the standard framework, all the AIL modules are available to the crawled hidden services. New features: - New export modules for statistics including credentials, phones, banking and TLDs. Many bugs were fixed. 2018-10-02T15:36:25+00:00 AIL-framework v1.5 2019-04-26T13:40:46+00:00 AIL Framework version 1.5 released including major improvements in crawler, server management, bootstrap 4 support and many more.   - [UI] Server management. Check for new updates/versions and show background update progress - [update] Background update process introduced - [UI] Bootstrap 4 migration started (crawler and tags view) - Crawler includes new functionalities - Port can now be configured - Configurable crawling including one-time crawler, regular crawling and type of crawling (e.g. including HAR, screenshots, blacklist management) - All Items (pastes) are now tagged by date-range - Many bugs were fixed - And significant performance improvement in the back-end 2019-04-26T13:40:46+00:00 AIL-framework v1.6 2019-05-13T09:58:22+00:00 # Changes - [travis] Travis has his own venv where it installs "stuff". Now we detect and us it in the launcher. [Steve Clement] - [travis] Require Python 3.6 to make build faster. [Steve Clement] - [doc] Some stats on build status/gitter etc. [Steve Clement] - [hashDecoded] cleanup for the VT message + PEP. [Alexandre Dulaunoy] # Fix - [faup] fix new return types (bytes to str) [Terrtia] - [Crawler] force domains/subdomains lower case (rfc4343) [Terrtia] - [showpaste] fix: #346, avoid None screenshots. [Terrtia] - [python requirements] rename file. [Terrtia] - [crawler] typo: domains down. [Terrtia] - [travis] LAUNCH.sh needs -l to launch... [Steve Clement] - [travis] Next round of travis fixes. LAUNCH.sh is the only launch script needed. chg: [installer] Be way more quiet, watching unzips is only fun during development. chg: [installer] Make the arch one +x. [Steve Clement] - [installer] Avoid doing funky **sudo pip install** moves, it breaks python on package managed python installs, if ever, use the **--user** flag. [Steve Clement] - [travis] Try and require xenial (16.04) and see if it works better. [Steve Clement] - [travis ] There are issues on the 14.04 build system of Travis. This fixes it temporarily. [Steve Clement] - [Onion] typo. [Terrtia] # Other - Merge branch 'master' of https://github.com/CIRCL/AIL-framework. [Terrtia] - Chg; [doc crawler] use the torproject torrc. [Terrtia] 2019-05-13T09:58:22+00:00 AIL-framework v1.7 2019-05-24T11:36:18+00:00 # Changes - [correlation] clean files. [Terrtia] - [update v1.7] update thirdparty. [Terrtia] - [correlation] add cryptocurrency + refractor correlation. [Terrtia] - [Bitcoin] map cryptocurrency: bitcoin (DB pivot) [Terrtia] - [update v1.7] add update scripts. [Terrtia] - [pgpdump] reprocess tagged items + fix pgpdump. [Terrtia] - [Update] force update order. [Terrtia] - [PgpDump] fix graph + add new tags: pgp-signature pgp-public-key-block + avoid keys injection in pgp user_id. [Terrtia] - [decoded UI] add PgpDump UI + fix hashdecoded js. [Terrtia] - [decoded items] bootstrap 4 migration. [Terrtia] - [PgpDump] add PgpDump backend TODO: UI. [Terrtia] - [crawler] manual/auto crawler: always save screenshots. [Terrtia] - [crawler] manual/auto crawler: always save screenshots. [Terrtia] # Fix - [correlation] fix endpoint. [Terrtia] # Other - Update README.md. [Thirion Aurélien] - Merge branch 'master' of https://github.com/CIRCL/AIL-framework. [Terrtia] - Merge pull request #349 from kovacsbalu/fix-paste-encoding. [Thirion Aurélien] Fix #314 - Use default encoding error from redis. [kovacsbalu] - Fix #314 Replace char on redis encoding error. Try to use local file on other error. [kovacsbalu] - Merge pull request #350 from kovacsbalu/fix-crawler-rotation. [Thirion Aurélien] fix: [crawler] rotation - Hopp, single quote :) [kovacsbalu] - Fix crawler rotation. [kovacsbalu] Before this, crawler processed prioritized onions and after all starts prioritized regular. 2019-05-24T11:36:18+00:00 AIL-framework v1.8 2019-06-12T13:56:05+00:00 # v1.8 (2019-06-12) ## Changes - [UI crawled domains] Download all domain content (HTML + HAR + screenshot) [Terrtia] - [backend crawler] domains: download 1 archive by crawled (most recent) [Terrtia] - [paste_submit UI] add debug. [Terrtia] - [template] the "item" project. [Alexandre Dulaunoy] ## Fix - [PgpDump] catch UnicodeDecodeError error. [Terrtia] - [backend crawler] rename downloaded archive. [Terrtia] - [paste_submit UI] filter empty file field. [Terrtia] - [PgpDump] process large pgp blocks. [Terrtia] - [paste_submit UI] filter empty file field. [Terrtia] - [UI crawler endpoints] display crawler status + fix #353. [Terrtia] - [update v1.7] add bs4 requirement. [Terrtia] # Example Finding relationships between two hidden services via the PGP key published on the website.   2019-06-12T13:56:05+00:00 AIL-framework v2.0 2019-07-05T13:47:37+00:00 v2.0 (2019-07-05) ----------------- # Changes - [helper] dump crawler history by daterange. [Terrtia] - [UI submit items] bootstrap 4 migration. [Terrtia] - [Flask login] add brute force protection + log login errors. [Terrtia] - [helper] generate self signed certificates. [Terrtia] - [Flask server] https support + create self signed certificate. [Terrtia] - [user_management 2.0] add update scripts + fix create_default_user. [Terrtia] - [UI user_management] user_role acl: hide admin panel. [Terrtia] - [UI user_management] incorrect passwords: display errors. [Terrtia] - [user_management endpoint] check user roles + add 503 template. [Terrtia] - [UI dashboard + search] bootstrap 4 migration: dashbaord + fix search input. [Terrtia] - [UI crawler, show_domain] domain history: remove target blank. [Terrtia] - [UI crawler, show_domain] add domain history list + navigation. [Terrtia] - [user_managemant] clean code + check password and email length. [Terrtia] - [user_management UI] add admin section: edit + create users. [Terrtia] - [user_management UI] edit my_profile + renew api tokens. [Terrtia] - [user_management] create default admin user (temp passwd save in AIL_HOME) + change password UI + logout UI + create random password. [Terrtia] - [restapi] add rest api authentification + create default user. [Terrtia] - [user_management] add user role_management. [Terrtia] - [user_management] create + check user password. [Terrtia] - [UI] add basic user management. [Terrtia] # Fix - [items submit UI] fix tags dropdown. [Terrtia] - [helper dump_crawler] fix files not found. [Terrtia] - [helper dump_crawler] fix empty dict. [Terrtia] - [MISP export] fix event creation. [Terrtia] - [UI items_submit] add active tag + fix template name. [Terrtia] - [UI login/change_password] add missing ail-logo. [Terrtia] - [Update] clean output. [Terrtia] - [Update] add default update script. [Terrtia] - [install_dep] create update current_version. [Terrtia] - [UI settings] fix toggle_sidebar. [Terrtia] - [install_dep] create default user. [Terrtia] - [user_management] fix tokens duplicate + check user_acl_integrity + add login errors messages. [Terrtia] - [server endpoint] unknow users: avoid endpoint enumeration. [Terrtia] - [TheHive feeder] create_alert: push all items tags TODO check items status + add more item metadata. [Terrtia] - [domain history] fix domain status. [Terrtia] # Other - Merge pull request #359 from CIRCL/user_management. [Alexandre Dulaunoy] User/role management - Merge branch 'master' into user_management. [Terrtia] - Merge branch 'master' into user_management. [Terrtia] - Merge branch 'master' of https://github.com/CIRCL/AIL-framework. [Terrtia] - Update README.md. [Thirion Aurélien] - Merge branch 'master' into user_management. [Terrtia] - Merge branch 'user_management' of https://github.com/CIRCL/AIL- framework into user_management. [Terrtia] - Update README.md. [Thirion Aurélien] Update install instructions - Merge branch 'master' into user_management. [Terrtia] - Merge branch 'master' into user_management. [Terrtia]   2019-07-05T13:47:37+00:00 AIL-framework v2.1 2019-08-14T13:42:58+00:00 AIL Framework now includes an [advanced API](https://github.com/CIRCL/AIL-framework/blob/master/doc/README.md) to query the items and information within an AIL instance. You can query items (e.g. such as pastes, crawled website content), meta-data (tag). The API allows to submit item to be processed by the AIL instance. More API endpoints will be added in the next releases.   2019-08-14T13:42:58+00:00 AIL-framework v2.2 2019-09-13T09:51:07+00:00   # AIL Framework version 2.2 released with refactoring of term tracking ## Changes - [API] add Tracker documentation. [Terrtia] - [Tracker] add more info. [Terrtia] - [update] add v2.2 update + fix default update + fix Empty Item.get_tags() + add new LAUNCHER options. [Terrtia] - [tracker] add missing btn. [Terrtia] - [trackers] filter trackers list by type + minor fix. [Terrtia] - [UI sparklines] sparklines: fix + factory. [Terrtia] - [merge] master. [Terrtia] - [UI term tracker] refractor term management: trackers list + show trackers + add new trackers. [Terrtia] - [api] add endpoint: get tracked item_id by uuid and daterange. [Terrtia] - [api] add endpoint: delete tracker term (regex/set/word) [Terrtia] - [Term Tracker] refractor term tracker word/set/regex modules + remove old modules. [Terrtia] - [Term tracker] add term tracker module (word + set) + API: add new term to track (word + set + regex) [Terrtia] - [term] refractor + add new tracked word/set. [Terrtia] - [README] add link to API documentation. [Thirion Aurélien] - [tests API] use argv api key. [Terrtia] - [api] add new endpoints: get bitcoin/pgp name/pgp keys/pgp mail metadata + items list. [Terrtia] ## Fixes - [sparkline] datatable drawing. [Terrtia] - [d3 graph] fix script path. [Terrtia] - Add missing file. [Terrtia] - [d3 js plugin] [Terrtia] - [template] add trackers. [Terrtia] - [Term Tracker module] chg module flow. [Terrtia] - [BankAccount] fix #385. [Terrtia] - [API doc] get bitcoin metadata + list of items. [Terrtia] - [UI caching] avoid domain archive caching. [Terrtia] - Typo. [Terrtia] - [UI caching] fix: #373 avoid screenshot caching. [Terrtia] # (free) Trainings - **Prague**, Thursday, 19 Sep 2019 09:00 : https://en.xing-events.com/CLTDKUU.html - **Luxembourg**, Monday, 23 Sep 2019 10:00 : https://en.xing-events.com/JDVIRXW.html 2019-09-13T09:51:07+00:00 AIL-framework v2.3 2019-10-29T14:58:25+00:00 AIL Framework version 2.3 released with improved cryptocurrencies detection, SQLi and detection of network reconnaissance tools output. Many bugs fixed and small improvements were performed. # Changes - [Cryptocurrency + Tools] launch by default + remove old Bitcoin module. [Terrtia] - [Keys module] detect public key. [Terrtia] - [Tools detection] add tool detection module. [Terrtia] - [Cryptocurrency, RegexTracker] update cryptocurrency list + fix: RegexTracker typo. [Terrtia] - [Cryptocurrency] add private_key entry + fix dash regex. [Terrtia] - [Cryptocurrency] add new Cryptocurrency module. [Terrtia] - [Tracker] add optional description field. [Terrtia] Fix ~~~ - [Tool] fix searchsploit regex. [Terrtia] - [Tools] typo. [Terrtia] - [Tools] typo. [Terrtia] - [Tools] fix loop. [Terrtia] - [url_prefix] add root blueprint, fix:#403. [Terrtia] - [TermTracker] fix performance: disable token stats. [Terrtia] - [SQL module] fix typo. [Terrtia] # Other - Merge branch 'master' of https://github.com/CIRCL/AIL-framework. [Terrtia] - Merge pull request #417 from andurin/master. [Alexandre Dulaunoy] Fix pybgpranking dependency in requirements - Fix pybgpranking dependency in requirements. [Hendrik] Relates #334 - Merge branch 'master' of https://github.com/CIRCL/AIL-framework. [Terrtia] - Merge pull request #404 from WimpyMan/master. [Thirion Aurélien] - LAUNCH.py: Added execution of script IPAddress.py. [Bastien Schils] - IPAddress.py: use ipaddress module. [Bastien Schils] Improved readability, maintainability and use of standard module - Modules.cfg: Minor: Added \n to seperate sections. [WimpyMan] - Config.cfg.sample: Improved example for IP module. [WimpyMan] By default, the list of networks to monitor is now empty. The previous value is now given as example. - Added: IP matching module. [Bastien Schils] - Merge pull request #411 from krial057/patch-1. [Alexandre Dulaunoy] Fixed some typos - Fixed some typos. [krial057] Fixed some typos in the readme - Merge pull request #408 from stamparm/master. [Thirion Aurélien] Adding more tools - Adding more tools. [Miroslav Stampar] - Merge pull request #407 from stamparm/patch-1. [Thirion Aurélien] Covering special cases (on pastebin) - Covering special cases (on pastebin) [Miroslav Stampar] There is no need for checking `()` in case of (e.g.) Litecoin and Dash as those are also additionally checked with Bitcoin address verifier - Merge pull request #406 from stamparm/master. [Thirion Aurélien] Adding tool regexes - Adding tool regexes. [Miroslav Stampar] - Merge branch 'master' of https://github.com/CIRCL/AIL-framework. [Terrtia] - Merge pull request #405 from stamparm/master. [Thirion Aurélien] Enforcing Base58 check on Litecoin and Dash addresses - Enforcing Base58 check on Litecoin and Dash addresses. [Miroslav Stampar] - Merge branch 'master' of https://github.com/CIRCL/AIL-framework. [Terrtia] - Merge pull request #401 from stamparm/master. [Thirion Aurélien] Improvement of crypto-address regexes (lesser FPs) - Improvement of crypto-address regexes (lesser FPs) [Miroslav Stampar] - Merge pull request #398 from stamparm/master. [Thirion Aurélien] Implementation for different cryptocurrencies - Implementation for different cryptocurrencies. [Miroslav Stampar] - Merge pull request #396 from stamparm/master. [Thirion Aurélien] Improvement of SQLi detection - Improvement of SQLi detection. [Miroslav Stampar] 2019-10-29T14:58:25+00:00 AIL-framework v2.4 2019-11-08T15:37:58+00:00 # AIL version 2.4 released AIL version 2.4 has been released including the following new features: - Improved crawled domain correlation to correlate such domain via cryptocurrency addresses, PGP keys UID, decoded hash content - Screenshot of crawled item can be selected from the UI - Crawled domain and port are now properly supported - Tagging functionality added to domain crawled - Configuration files have been moved to a coherent directory - Documentation of the code improved - Various bugs fixed and small improvements    2019-11-08T15:37:58+00:00 AIL-framework v2.5 2019-11-25T10:14:10+00:00 # AIL Framwork version 2.5 released. AIL Framework version 2.5 released with correlation and experimental support for [MISP modules](https://github.com/MISP/misp-modules). The correlation is now improved to add correlation between PGP, cryptocurrencies, pastes, decoded values against any items in AIL framework. The correlation interface has been redesigned to allow filtering per type (pastes, crawled) and also limit the number of correlations. The MISP modules is still very early but the objective is gain from all MISP modules expansion within AIL. We introduced an experimental module to automatically expand BTC transactions from addresses seen in AIL and pivot to new correlations. New roles were added (read_only and users without) to improve the profile of the various AIL users. The improvements were designed with some requirements from the [ENFORCE project](https://securitymadein.lu/news/ceis-securitymadein-lu-enforce-project/) to better support law-enforcement usage.   2019-11-25T10:14:10+00:00 AIL-framework v2.6 2019-12-17T15:53:02+00:00 AIL Framework version 2.6 released with improved correlations (hover information, screenshot hash correlation), API improvements and various fixes. Thanks to the [enforce project](https://ceis.eu/en/cyber-ceis-coordinator-of-the-enforce-project-co-organized-a-second-cybercrime-training-with-circl-computer-incident-response-center-from-luxembourg-and-the-french-ministry-of-interior-%EF%BB%BF/) for the feedback during the training. New features were based on constructive remarks from the users.    # New and Improvements - [slides] source code added. [Alexandre Dulaunoy] - [screenshot correlation + v2.6] add screenshot-domain correlation + v2.6 update. [Terrtia] - [API] get domain min metadata (first up, last up) + get crawled domain by daterange and status. [Terrtia] - [Domain + Date] get domain up range + get date days and months by daterange. [Terrtia] - [Domain] get all/by month domains up. [Terrtia] - [API] get domain metadata (minimal) [Terrtia] - [UI correlation graph] tooltip: show domain tags. [Terrtia] - [UI correlation graph] popover: add loading status + chg css. [Terrtia] - [correlation UI] add basic popover. [Terrtia] - [slide] update slide. [Terrtia] - [pgpdump] add debug. [Terrtia] - Linked TOR installation instruction in the README. [Sami Mokaddem] - [Onion] add discovery queue. [Terrtia] - [Showpaste] check if tags are safe (img) + fix domain link. [Terrtia] - [crawler dashboard UI] add UP/Down domains url. [Terrtia] # Bugs fixed - [UI correlation graph] typo. [Terrtia] - [UI showDomain] fix down domain history. [Terrtia] - [Domain] domain was up. [Terrtia] - [Domain] is_domain_up. [Terrtia] - Typo. [Terrtia] - [Update] force manual update, fix #443. [Terrtia] - [UI] fix show paste modal. [Terrtia] - [screenshot canevas + domain link] fix item domain link + screenshot canevas: chg colors and icons for unsafe tags. [Terrtia] - [PgpDump] catch bs4 error. [Terrtia] - Placed Tor installation instruction in the installation section. [Sami Mokaddem] - [Update v2.4] fix empty set. [Terrtia] - [Update v2.4] fix empty set. [Terrtia] - [Item lib] fix import. [Terrtia] - [Paste submit] fix tags unpack. [Terrtia] - [Show Domain UI] fix screenshot link, fix #431. [Terrtia] - [Update] filter invalid tags. [Terrtia] 2019-12-17T15:53:02+00:00 AIL-framework v2.7 2020-01-13T17:35:16+00:00 # AIL Framework version 2.7 released with an improved tagging system AIL Framework version 2.7 released includes a major refactoring of the tagging system. Tagging speed has been significantly improved. Tags can now be used with items, domains and images and added in the objects. A new feature to search by tags has been also added. Multiple bugs were fixed.   ## Detailed change logs (v2.7 (2020-01-13)) ### Changes - [UI domain] add input: show domain by name. [Terrtia] - [tags blueprint] clean code. [Terrtia] - [update v2.7] sort domain full_onion_up and full_regular_up. [Terrtia] - [UI tags] search domains by tags. [Terrtia] - [core + UI] search domain by tags. [Terrtia] - [Update v2.7] add update v2.7 scripts. [Terrtia] - [tags UI] edit object tags (delete tags) [Terrtia] - [Tag core] objects tagging, Part 2/2 TODO: UI tags domain + screenshot + object rename paste=>item. [Terrtia] - [Tag core] objects tagging, Part 1/2 TODO UI (tags) + rename paste=>item. [Terrtia] - [Tag core] objects tagging, Part 1/2. [Terrtia] - [README] remove top terms. [Terrtia] - [correlation graph UI] add json error handler + add loading status. [Terrtia] - [UI correlation graph + UI domain] correlation screenshot: show img in toolip + show hash in ShowDomain TODO: pixelate images. [Terrtia] ### Fix - [UI tags] fix domain links. [Terrtia] - [UI term] remove deprecated trending charts, fix #446 #447. [Terrtia] - [Crawler] typo. [Terrtia] - [UI decoded item] sort mimetype. [Terrtia] - [Crawler] fix screenshot-domain typo. [Terrtia] - [Crawler] fix screenshot-domain typo. [Terrtia] - [Crawler] fix screenshot-domain map. [Terrtia] - [UI showDomain] fix screenshot accordeon. [Terrtia] ### Other - Merge pull request #449 from CIRCL/tags_v2. [Alexandre Dulaunoy] Tags v2 - Tagging system refractoring - Merge branch 'master' into tags_v2. [Terrtia] - Update README.md. [Thirion Aurélien] 2020-01-13T17:35:16+00:00 AIL-framework v2.8 2020-01-23T15:14:09+00:00 AIL Framework version 2.8 released with a domain screenshot browser and many bugs fixed.   There is also a "Practical Darkweb and criminal Blockchain monitoring using AIL (Framework for Analysis of Information Leaks) - free Training/Workshop" hosted by CIRCL (Luxembourg) which will take place the 20th Feb 2020. [Registration link](https://en.xing-events.com/YHBTLMJ.html). ## Next release In the next release of AIL, only Python 3.6 and upper will be supported. ## Changes - [domain core + UI] add domain explorer v1. [Terrtia] - [test api] increase import timeout. [Terrtia] - [UI + core] tag decoded items, fix: #455. [Terrtia] - [UI item tags search] refractor: search item tag by object, use new functions. [Terrtia] - [UI tags] add + delete image (screenshot) tags. [Terrtia] - [UI tags] add + delete image (screenshot) tags. [Terrtia] ## Fix - [MISP export] force pymisp version. [Terrtia] - [tag core] typo. [Terrtia] - [tag core] item date type. [Terrtia] - [Tag core] add tag, update tag last seen. [Terrtia] - [Flask server + cookie session] chg default cookie name (also use port number) + add Flask port number to config. [Terrtia] - [expand btc adress] filter empty addr fields. [Terrtia] - [UI navbar] chg icon decoded tags. [Terrtia] - [UI show item min] fix empty modal. [Terrtia] - Change name popper.js-1.14.3 to popper-core-1.14.3. [mangelft] - [UI] screenshot url. [Terrtia] ## Other - Merge pull request #453 from mangelft/master. [Thirion Aurélien] fix: change name popper.js-1.14.3 to popper-core-1.14.3 - Merge pull request #450 from CIRCL/tags_v2. [Thirion Aurélien] Tags v2 2020-01-23T15:14:09+00:00 AIL-framework v2.9 2020-02-03T15:34:13+00:00 # AIL Framework version 2.9 released with a critical security fix (CVE-2020-8545) and minor bugs fixed This release of AIL includes a major security fix for [CVE-2020-8545](https://cve.circl.lu/cve/CVE-2020-8545). The security vulnerability was in the handler of the global feed which could allow malicious feed providers to overwrite and potentially, execute Python code in the environment. This release also includes various bugs fixed. We urge users to upgrade as soon as possible. # Changes - [domain explorer] domains explorer v2, filter domains by daterange. [Terrtia] # Fix - [IPAddress] catch empty config error. [Terrtia] - [Global: already saved filename] save updated + filter duplicated items. [Terrtia] - [Global: filename provided by all feeders] avoid path tranversal. [Terrtia] - [Domain explorer UI] fix daterange pagination links. [Terrtia] - [Tag core] check if item_date type is an integer. [Terrtia] 2020-02-03T15:34:13+00:00 AIL-framework v3.0 2020-02-21T15:14:41+00:00 AIL Framework version 3.0 with full MISP format export and import has been released. AIL users can now export a set of selected items as a MISP event including objects (items, decoded, screenshot, pgp...), correlations and metadata. There is an import to allow analyst to keep a specific set of analysis in AIL for moving across AIL instances. This release also includes support for authentication of the SMTP server (thanks to Mike Peters for the contribution). Multiple bugs fixed and small improvements.   2020-02-21T15:14:41+00:00 AIL-framework v3.1 2020-05-12T15:12:48+00:00 # AIL framework v3.1 released    AIL framework v3.1 has been released including many new features such as: - Crawling website with a set of cookies collected from a browser (allowing to crawl website password protected or alike) - An extraction module to find Telegram users - New correlation engine for username starting with Telegram - Improve timeout on various modules - New JSON importer to be used with new external feeder. A first [Twitter feeder](https://github.com/ail-project/ail-feeder-twitter) and url-feeder is available to feed specific tweets and discovered urls in AIL. The [AIL project](https://github.com/ail-project) moved into a dedicated project to allow new contributions and project within the AIL project scope. If you want to join us and contribute new modules or specific feeders, don't hesitate to contact us. # Complete changelog - [update] add update v3.1 + install crawler python requirements by default. [Terrtia] - [UI correlation] add username correlation card. [Terrtia] - [UI correlation] add username correlation graph. [Terrtia] - [UI correlation] add username correlation: daterange pages. [Terrtia] - [telegram module] add new tag: telegram invite code. [Terrtia] - [telegram backend] add username correlation + save invite hash. [Terrtia] - [importer] add map twitter id - item id, add parents link between twitter and url extracted. [Terrtia] - [travis install] shallow clone. fetch tags + commit id. [Terrtia] - [travis install] shallow clone. fetch tags. [Terrtia] - [install] debug travis. [Terrtia] - [README] update gitter link. [Terrtia] - [import er url_extract] add item twitter parent. [Terrtia] - [importer url_extract] fix item id. [Terrtia] - [importer] add url_extract importer. [Terrtia] - [API json import] take list as imput. [Terrtia] - [core import] add AIL JSON import format, API endpoint added (AIL feeders) [Terrtia] - [update doc] update doc install + logo + fix updater. [Terrtia] - [Updater] change default branch origin. [Terrtia] - [Updater] change default branch origin. [Terrtia] - [update Readme logo + links] [Terrtia] - [add new logo] [Terrtia] - [AIL logo UI] [Terrtia] - [doc] add cookiejar screenshot. [Terrtia] - [Crawler] default docker memory usage. [Terrtia] - [MISP export] export domain as domain-crawled object. [Terrtia] - [domain explorer UI] add shortcut button: misp export + correlation. [Terrtia] - [crawler] edit cookie and cookiejar + add cookie to cookiejar + fix screenshot duplicate. [Terrtia] - [Crawler core + UI] crawler lua: handle retry + fix cookie loader and selector. [Terrtia] - [cookiejar UI] add cookiejar + show all. [Terrtia] - [crawler cookies] use cookiejar. [Terrtia] - [crawler - cookies] add/show/select cookies. [Terrtia] - [crawler] add cookies list by user/global, save cookies from file + dict(name, value), TODO: API + handle errors. [Terrtia] - [crawler] bypass login: use cookie provided by user and accept cookie from server + refractor. [Terrtia] - [Splash Crawler] use cookies to bypass login. [Terrtia] - [UI] bump jquery to 3.4.1. [Terrtia] - [import_dir] fix is_gzip test, use magic number. [Terrtia] - [MISP Importer] files: handle missing sha1/sha256 attributes + fix Items and Screenshots dir. [Terrtia] - [Updater] relauch updater on change (git pull) [Terrtia] ## Fixes - [Mail module] replace signal by multiprocessing (https://docs.python.org/3.4/library/signal.html#execution-of-python- signal-handlers) [Terrtia] - [Mail] debug signal. [Terrtia] - [Mail] debug. [Terrtia] - [Mail module] remove test time.wait. [Terrtia] - [Mail module] debug signal timeout. [Terrtia] - [Mail module] debug signal timeout. [Terrtia] - [telegram module] typo. [Terrtia] - [telegram module] fix tagging. [Terrtia] - [telegram module] check username length. [Terrtia] - [telegram module] remove debug. [Terrtia] - [Mails] regex timeout. [Terrtia] - [Mails] change module output. [Terrtia] - [Mails] remove print + test. [Terrtia] - [Mails] refactor Mail module. [Terrtia] - [urlextract importer] fix parent map + replace set JSON queue by list. [Terrtia] - [import urlextract parent] fix typo. [Terrtia] - [travis install] shallow clone. fetch missing commit id. [Terrtia] - [Mails regex timeout] reduce default timeout. [Terrtia] - [Mails dns resolver] update timeout exception. [Terrtia] - [Mails] typo. [Terrtia] - [Mails] import. [Terrtia] - [Mails] import. [Terrtia] - [Mails] add regex timeout. [Terrtia] - [Credential] add regex timeout. [Terrtia] - [Credential] add regex timeout. [Terrtia] - [urlexport importer] fix item name + redis config. [Terrtia] - [urlexport importer] item_id, force str type. [Terrtia] - [urlexport importer] add missing import. [Terrtia] - [urlextract importer] class name. [Terrtia] - [importer] fix typo. [Terrtia] - [Global] extend ungzip error catching, catch invalid compressed file. [Terrtia] - [json import API] remove list input. [Terrtia] - [install create default user] default passwd file: add missing new line. [Terrtia] - [installer] get last git version. [Terrtia] - [Readme travis badge] [Terrtia] - [Readme travis badge] [Terrtia] - [Readme logo size] [Terrtia] - [show item UI] return 404. [Terrtia] - [Cookiejar UI] fix typo. [Terrtia] - [Flask session cookie name] add uuid to cookie name. [Terrtia] - [crawler] typo. [Terrtia] - [Crawler splash ResponseNeverReceived] add retry. [Terrtia] - [crawler] error catcher. [Terrtia] - [MISP export UI] fix input: name + value overwrite. [Terrtia] - [update thirdparty] update taxonomies. [Terrtia] - [crawler] cleanup. [Terrtia] - [PgpDump parser] remove header comment (rfc4880) + remove empty lines. [Terrtia] - [Pgp Dump] remove tool version. [Terrtia] - [ZMQ Feeder] performance: replace zmq recv NOBLOCK by Poller. [Terrtia] - [pgpdump] fix subtype save. [Terrtia] - [Updater] force updater update. [Terrtia] - [Updater] fix current_tag parser. [Terrtia] - [import_dir] remove special characters. [Terrtia] - [import_dir] remove dir whitespaces #475. [Terrtia] ## Other - Merge pull request #492 from sunil3590/master. [Thirion Aurélien] crawler_time -> crawler_delta - Crawler_time -> crawler_delta. [Sunil D S] - Chg [telegram + correlation] new module: telegram (username + login code + join_chat) + add simple_correlation backend. [Terrtia] - Chg [telegram + correlation] new module: telegram (username + login code + join_chat) + add simple_correlation backend. [Terrtia] - Merge pull request #487 from CIRCL/crawler_v2. [Thirion Aurélien] fix: [crawler] error catcher - Merge pull request #486 from CIRCL/crawler_v2. [Thirion Aurélien] Crawler v2 - Add cookiejar - use cookie to bypass login form - Merge branch 'master' into crawler_v2. [Terrtia] 2020-05-12T15:12:48+00:00 AIL-framework v3.2 2020-08-27T12:46:14+00:00 # AIL Framework version 3.2 released with new YARA detection engine with a default set of search patterns, many new correlations type and many bugs fixed.   ## Changes - [update] add v3.2. [Terrtia] - [yara trackers] add debug. [Terrtia] - [tracker yara] show rule content. [Terrtia] - [correlation] basic correlation: get_correlation_all_object function. [Terrtia] - [submodule + YARA] add submodule auto update + update v3.1.1. [Terrtia] - [trackers] add yara trackers. [Terrtia] - [Trackers] email notifications: add tracker description in email subject. [Terrtia] - [username correlation + login redirection] add twitter username correlation + redirect to the requested page on login. [Terrtia] - [Item delete] delete father/child link + remove from domain tree + delete all child from the same domain. [Terrtia] - [feeder pystemon] add debug. [Terrtia] - [whosh index] add data retention fct. [Terrtia] - [core] disable Release module. [Terrtia] - [tor crawler] nyt added. [Alexandre Dulaunoy] - [API + item library] add a way to check if a user can access the API + item import (use item_basic) [Terrtia] - [black-list onion] keybase added. [Alexandre Dulaunoy] - [core crawler] add screen script create screen + windows + kill them + get list. [Terrtia] - [crawler proxy] interact with the splash manager API (get list of proxies + splash containers + launch them) [Terrtia] - [launcher] add option to reset UI admin password. [Terrtia] - [UI dashboard] sort module by name. [Terrtia] - [LAUNCH] launch telegram module by default. [Terrtia] ## Fix - [install] canevasjs. [Terrtia] - [UI tags] fix add tags template. [Terrtia] - [install] travis git unshallow. [Terrtia] - [trackers yara] remove sleep time. [Terrtia] - [trackers] typo. [Terrtia] - [updater] tag subversion. [Terrtia] - [update v3.1.1] init submodule. [Terrtia] - [travis] [Terrtia] - [crawler] fix auto crawler creation. [Terrtia] - [travis] fetch git last tag. [Terrtia] - [update thirparty] fix canevasjs install. [Terrtia] - [install crawler] remove old python requirement. [Terrtia] - [TheHive feeder] create alert, get item full path. [Terrtia] - [TheHive feeder] create alert. [Terrtia] - Typo. [Terrtia] - Typo. [Terrtia] - Typo. [Terrtia] - [username correlation] fix domain correlation. [Terrtia] - [RegexTracker] fix search regex helper. [Terrtia] - [regex_helper] [Terrtia] - [regex tracker] fix timeout. [Terrtia] - [reset_AIL] add helper + fix soft reset. [Terrtia] - [Exporter] ad missing file. [Terrtia] - [update_thirparty] fix popper install. [Terrtia] - [UI popper version] use popper v1 (https://github.com/twbs/bootstrap/issues/29842) [Terrtia] - [MISP auto Export] fix import path. [Terrtia] - [MISP auto export] fix MISP_TheHive_feeder. [Terrtia] - [travis] virtualenv. [Terrtia] - [install] unshallow clone. [Terrtia] - [requirement] remove old crawler requirements (installed by default) [Terrtia] - [crawler] option to disable screenshots and har. [Terrtia] - [crawlers lib] missing config file. [Terrtia] - [create au to crawler] save crawler_delta time. [Terrtia] - [updater] return boolean. [Terrtia] - [Credential] fix timeout error. [Terrtia] - [Credential] fix timeout error. [Terrtia] - [Credential] force lowercase of mail address. [Terrtia] - [Credential] fix timeout message. [Terrtia] - [Mail module] regex timeout. [Terrtia] - [update v1.5] [Terrtia] - [Mail module] remove debug. [Terrtia] - [Mail module] fix dns caching + use redis queue. [Terrtia] - [UI Decoded items] fix download decoded file. [Terrtia] ## Other - Merge pull request #519 from myasn1k/docker_fixes. [Thirion Aurélien] Docker fixes - Update update_thirdparty.sh. [Pietro Mazzini] - Remove useless copy. [Pietro Mazzini] - Uncomment update and explain in docker readme. [Pietro Mazzini] - Add tz as argument. [Pietro Mazzini] - Update docker readme (cp) [Pietro Mazzini] - Update docker readme. [Pietro Mazzini] - Fix canvas folder name. [Pietro Mazzini] - Add to docker start crawlers launch. [Pietro Mazzini] - Update Dockerfile: use ubuntu 18, remove redundant update_thirdparty, remove old crawler requirements pip install. [Pietro Mazzini] - Comment git repo tag in installing_deps, git repo not recognised. [Pietro Mazzini] - LAUNCH update function fail because the folder isn't a git repo, comment it. [Pietro Mazzini] - Merge pull request #22 from C00kie-/master. [Alexandre Dulaunoy] allow to configure binding interface - Update Flask_server.py. [C00kie-] - Update core.cfg.sample. [C00kie-] - Typo fixed as ail repo is now lower-case. [Alexandre Dulaunoy] Typo fixed as ail repo is now lower-case - Merge pull request #16 from ChilliSec/patch-1. [Thirion Aurélien] Update HOWTO.md - Update HOWTO.md. [ChilliSec] - Merge branch 'master' of https://github.com/ail-project/ail-framework. [Terrtia] - Merge pull request #497 from ronaldtf/fix-case-thehive. [Thirion Aurélien] Fix create case with TheHive - Fix create case with TheHive. [Ronald Teijeira Fernandez] 2020-08-27T12:46:14+00:00 AIL-framework v3.3 2020-10-13T15:11:18+00:00 # AIL Framework version 3.3 released with improved item view and many improvements The AIL Framework version 3.3 released included a refactoring of the item view including a convenient html2text preview of item fetched. Last origin table is now accessible for all domain crawled to show where the information is coming. Improved yara view in AIL to display the yara rule directly. Various bugs were fixed and bitcoin bech32 addresses are now supported.     ## Changes - [UI show Item] refactoring + bootstrap 4 migration. [Terrtia] - [UI show domain] add last origin table: domain + item. [Terrtia] - [travis] -> bionic. [Alexandre Dulaunoy] - [yara trackers UI] add yara trackers, show default yara rule. [Terrtia] - [bitcoin tags + correlation] add bech32 adresses. [Terrtia] ## Fix - [UI correlation graph] fix item links. [Terrtia] - [UI correlation graph] fix item links. [Terrtia] - [users management] hide API keys by default + fix delete/edit users. [Terrtia] - [crawler] fix ResponseNeverReceived retry time. [Terrtia] - [crawler] fix ResponseNeverReceived hanlder, check if splash restarted. [Terrtia] - [Yara tracker] fix mail notification. [Terrtia] - [Yara tracker] fix mail notification. [Terrtia] - [RegexTracker] fix missing hostname in mail notification. [Terrtia] - [Flask server] change default host. [Terrtia] - [ardb] switch to ail-project ardb fork #38. [Jean-Louis Huynen] - [Launcher] fix virtualenv loader. [Terrtia] 2020-10-13T15:11:18+00:00 AIL-framework v3.4 2020-12-20T14:47:03+00:00 # AIL Framework version 3.4 released with a new language detection module and many small improvements AIL Framework version 3.4 has been released with a new language detection which can classifies items analysed by the language detected. The module relies on [Compact Language Detector v3 (CLD3)](https://github.com/google/cld3). Older items will be updated (there is a background job doing the update, so it might take some time depending of your dataset). You can then browse crawled items per language detected. The trackers can now be edited and various bugs were fixed. Thanks to all the contributors.    # Changes - [Languages]detect + search domains languages. [Terrtia] - [Tracker] edit tracker. [Terrtia] - [web] we process items nowadays not only pastes ;-) [Alexandre Dulaunoy] # Fix - [install] pycld3 dependency. [Jean-Louis Huynen] Without this Flask fails starting - [update v3.4] updater. [Terrtia] - [Languages] import + update message. [Terrtia] - [Languages] update. [Terrtia] - [UI trackers] None trackers values. [Terrtia] - [Tracker] edit tracker ACL. [Terrtia] - [Yara Tracker] catch yara timeout. [Terrtia] - [Terms Trackcers] fix item link. [Terrtia] # Other - Merge pull request #70 from ail-project/gallypette-patch-1. [Jean- Louis Huynen] fix: [install] pycld3 dependency - Merge branch 'master' of https://github.com/ail-project/ail-framework. [Terrtia] 2020-12-20T14:47:03+00:00 AIL-framework v3.5 2021-03-31T13:24:17+00:00 # AIL Framework version 3.5 released with a new flexible crawler manager, built-in Passive DNS sensor and many fixes.    # Changes - [passivedns] D4server port. [Terrtia] - [v3.5] install pyAIL. [Terrtia] - [passivedns] D4server port. [Terrtia] - [passiveDns] add dns records. [Terrtia] - [passiveDns D4 Client] add passiveDns D4 Client. [Terrtia] - [crawler] add test + relaunch crawlers + major fixs. [Terrtia] - [update] add v3.5 update. [Terrtia] - [doc] Splash Manager Configuration. [Terrtia] - [UI] add update note. [Terrtia] - [IPAddress] Add Debug + Check if option is empty. [Terrtia] - [splash manager] update enpoints + use Splash name to restart docker. [Terrtia] - [merge master] [Terrtia] - [domains search] search domains by name. [Terrtia] - [screenshot + har directory] add option to change screenshots directory. [Terrtia] - [crawler_manager] UI edit config + fix crawler queues. [Terrtia] - [config] save config on DB + default values. [Terrtia] - [crawlers manager] show setings. [Terrtia] - [crawler] show all crawlers type on dashboard. [Terrtia] - [crawler] manage crawlers. [Terrtia] - [crawler] crawler queue + restart docker on error. [Terrtia] - [core module] disable phone module by default. [Terrtia] - Merge master -> crawler_manager. [Terrtia] - [crawler manager] get all splash dockers, proxies and launch all crawlers. [Terrtia] # Fixes - [crawler] crawler test: remove print. [Terrtia] - [crawler UI] remove unused crawlers configuration. [Terrtia] - [crawler] user agent + splash restart. [Terrtia] - [crawler UI] crawler by domain type, remove old dashboard. [Terrtia] - [MISP Export UI] object type selector. [Terrtia] - [UI] fix navbar: invalid url. [Terrtia] - [crawler] typo. [Terrtia] - [crawler] typo. [Terrtia] - [crawler] typo. [Terrtia] - [Crawler] faup. [Terrtia] - [Launcher] launch core module: Crawler_manager. [Terrtia] - [cralers] remove debug. [Terrtia] - [Domain search] fix regex. [Terrtia] - [API ACL] avoid user_no_api users to access the API. [Terrtia] - [Module Queue] module without subscriber. [Terrtia] - [term Tracker] TimeoutException. [Terrtia] - [API ACL] read_only user role. [Terrtia] - [OVERVIEW] [Terrtia] - [gitignore] [Terrtia] - [showDomain] empty father field. [Terrtia] - [showDomain] empty father field. [Terrtia] - [redis cache] remove old Redis_Data_Merging db. [Terrtia] - [redis cache] remove old paste_name db. [Terrtia] - [background updater] Don't launch updates on fresh install. [Terrtia] - [v3.4 update] fix update progress. [Terrtia] - [background updater] remove completed updates. [Terrtia] - [install] force virtual environmemt activation. [Terrtia] # Other - Merge branch 'master' into crawler_manager. [Terrtia] - Merge pull request #89 from ail-project/crawler_manager. [Alexandre Dulaunoy] Crawler manager - Merge branch 'master' into crawler_manager. [Terrtia] - Merge branch 'master' into crawler_manager. [Terrtia] - Merge pull request #536 from simonsigre/patch-1. [Alexandre Dulaunoy] Typo in placeholder "Optionnal" --> "Optional" - Typo in placeholder "Optionnal" --> "Optional" [Simon Sigré] Updated placeholder value to correct typo; "Optionnal" --> "Optional" - Merge branch 'master' of github.com:CIRCL/AIL-framework. [Alexandre Dulaunoy] - Merge pull request #534 from simonsigre/patch-1. [Alexandre Dulaunoy] Added 'wget' as a dependency - Added 'wget' as a dependency. [Simon Sigré] 'wget' is a dependency as without this additional components won't download as part of initial install, one such example is; ``` wget -O /ail-framework/ardb/src/../deps/jemalloc-5.1.0.tar.bz2 https://github.com/jemalloc/jemalloc/releases/download/5.1.0/jemalloc-5.1.0.tar.bz2 && \ ``` - Merge branch 'master' into crawler_manager. [Terrtia] - Merge branch 'master' into crawler_manager. [Terrtia] - Merge branch 'master' into crawler_manager. [Terrtia] 2021-03-31T13:24:17+00:00 AIL-framework v3.6 2021-07-14T14:40:57+00:00 # AIL Framework released version 3.6  AIL Framework version 3.6 released with new features (such as YARA retrohunt), significant performance improvements, refactoring of the modules and many bugs were fixed. This version includes a new advanced to perform YARA retrohunt on the whole AIL dataset. YARA retrohunts can be started and stopped live, sources can be defined and also the period where to search retroactively for. Tags can be also applied to reclassify information collected in AIL. The speed of YARA retrohunt depends of the hardware used, SSD devices are strongly recommended. New tool such as export of crypto currencies correlation has been added to allow the use of data from AIL in other tools for analysing cryptocurrency transactions. Many bugs were also fixed in this release. Don't forget to update. A huge thanks to all the contributors and especially @osagit for the numerous contributions.    # Changelog ## v3.6 (2021-07-14) ### Changes * Chg: [README] add Olivier Sagit @osagit to contributors list. [Terrtia] * [requirements] minimal version + remove old packages. [Terrtia] * Add Retro Hunt. [Terrtia] * [UI trackers] add/edit/remove tracker source/target #43 #102. [Terrtia] * [Trackers regex + yara] filter by item source. [Terrtia] * [Tracker term] track terms by sources. [Terrtia] * [test] add new workflow. [Thirion Aurélien] * [requirements] minimal version + remove old packages. [Terrtia] * [modules + tests] fix modules + test modules on samples. [Terrtia] * [update v3.6 + installer] fix faup install + add v3.6 update. [Terrtia] * [Launcher] update modules directory. [Terrtia] * [SQLInjectionDetection LibInjection modules] add module class. [Terrtia] * [Urls (Web) module] fix regex + rename. [Terrtia] * [Tracker_Regex module] create module class + mv module. [Terrtia] * [Tracker_Yara module] create module class. [Terrtia] * [modules] create new modules repository + small fixs. [Terrtia] * [Keys] add test. [Terrtia] * [modules] add tests: CreditCard, Global, DomClassifier. [Terrtia] * [Categ] tests + docs. [Terrtia] * [ApiKey] refactor module + tests. [Terrtia] * [launcher + modules] add module tests (Onion module) [Terrtia] * [AIL items + Onion] create AIL item objects + Onion module refactor. [Terrtia] * [tools extract_cryptocurrency] filter by correlation objects + errors messages. [Terrtia] * [tools extract_cryptocurrency] new input: list of addresses to extract. [Terrtia] * [tools] add 2 new tools: extract cryptocurrency correlation by type + trigger manual update. [Terrtia] * [modules] set log level to critcal on unexpected errors. [Terrtia] ### Fix * [Term tracker] fix fct args. [Terrtia] * [crawler] fix new crawled item id. [Terrtia] * [Retro Humt UI] clarify tags input. [Terrtia] * URLs www word to match. [osagit] There is 4 'w' chars instead of 3 * [crawlers] get_all_splash return type. [Terrtia] * [Splash_Manager errors] catch invalid response. [Terrtia] * [Trackers UI] fix sources logo + tracker metadata. [Terrtia] * Change module name Web to Urls. [lpwm9803] * [Splash_Manager errors] catch invalid response. [Terrtia] * [github workflows] fix test. [Terrtia] * [github workflows] clone depth. [Thirion Aurélien] * [installer] pyfaup install. [Terrtia] * [installer] pyfaup install. [Terrtia] * [UI crawler dashboard] block_languages_search: fix domains_types #110. [Terrtia] * [UI crawler dashboard] block_domains_name_search: fix domains_types #110. [Terrtia] * [modules] print + save traceback in logs. [Terrtia] * [paste_submit] restrict source characters. [Terrtia] * [submit_paste] restrict source name. [Terrtia] * [tools extract_cryptocurrency] argparse flags. [Terrtia] * [tags] invalid tags. [Terrtia] * Replace redis init with generic ConfigLoader. [osagit] StrictRedis() replaced by ConfigLoader.get_redis_conn() * Debug() string takes 1 positional argument. [osagit] * FILE_ALLOWED_EXTENSIONS without quotes. [osagit] * Stuck queues and submit paste. [Olivier SAGIT] * Name pystemon feeder in feeder monitor dashboard. [Olivier SAGIT] * [tools extract cryptocurrency] correlation type. [Terrtia] * [Updater] don't check if modified config files (redis and ardb config) [Terrtia] * [Web module] resolver, change log level. [Terrtia] * [WebStats] typo. [Terrtia] * [Indexer] debug messages. [Terrtia] * [WebStats] typo. [Terrtia] * [terms tracker] refresh Tracked terms. [Terrtia] * [redis cache] remove old paste_name db. [Terrtia] * [crawler] typo: splash restart. [Terrtia] ### Other * Merge branch 'master' of https://github.com/ail-project/ail-framework into dev. [Terrtia] * Merge pull request #115 from My-WAF/master. [Thirion Aurélien] Remove Block Copy Git Directory * Update .dockerignore. [VNC Company] * Merge branch 'dev' of https://github.com/ail-project/ail-framework into dev. [Terrtia] * Merge pull request #116 from osagit/patch-1. [Thirion Aurélien] fix: URLs www word to match * Merge. [Terrtia] * Merge branch 'dev' of https://github.com/ail-project/ail-framework into dev. [Terrtia] * Merge pull request #114 from osagit/dev. [Thirion Aurélien] fix: Change module name Web to Urls * Merge branch 'master' into dev. [Terrtia] * Merge branch 'master' of https://github.com/CIRCL/AIL-framework. [Terrtia] * Merge pull request #559 from lesleyxyz/patch-1. [Thirion Aurélien] Submit paste contents to TheHive * Submit file contents to TheHive. [Lesley De Keyser] * Merge pull request #103 from osagit/scriptsenhance. [Thirion Aurélien] fix: stuck queues and submit paste * Merge branch 'dev' into scriptsenhance. [Thirion Aurélien] * Merge branch 'master' of https://github.com/ail-project/ail-framework. [Terrtia] * Perf: use defined compiled regex. [osagit] re.compile(regex) definition was not used use compile_regex.findAll() directely instead of re.findall(regex) * Log message split error + perf. [osagit] fix: log message split errors perf: string affected at start doc: comments * Merge pull request #101 from osagit/namedfeeder. [Alexandre Dulaunoy] fix: name pystemon feeder name in feeder monitor dashboard * Merge pull request #97 from osagit/abstract-module. [Thirion Aurélien] feat: module factorization * Feat: module factorization. [lpwm9803] 2021-07-14T14:40:57+00:00 AIL-framework v3.7 2021-08-27T21:09:10+00:00 AIL Framework version 3.7 released with many bugs fixed and improvement. The term tracker has been improved including the first_seen and last_seen. Various bugs were fixed and documentation improved. Thanks to all the contributors and especially Tony Jabbour for the new CentOS installation documentation. Thanks to Relega for the improved documentation about the pystemon integration. And a huge thank to @Fbroy for the new feeders: [Discord](https://github.com/ail-project/ail-feeder-discord), [ActivityPub feeder](https://github.com/ail-project/ail-feeder-activity-pub) and [RSS/Atom feeder](https://github.com/ail-project/ail-feeder-atom-rss). There is an ongoing feeder to include AIL2AIL synchronisation and a [first draft of the message format](https://github.com/ail-project/ail-exchange-format/blob/main/ail-stream.md) has been proposed. Feedback is more than welcome.  # v3.7 (2021-08-27) ### Changes * [tracker + update] add update v3.7 + add map item_id:tracker_uuid (data retention) + fix tracker first_seen/last_seen. [Terrtia] * [tracker] typo fixed. [Alexandre Dulaunoy] * [Credential + tags] add misp-taxonomies submodule + fix typo. [Terrtia] * [gitchangelog.rc] updated to output Markdown. [Alexandre Dulaunoy] ### Fix * [tracker] global tracker list: fix first/last seen. [Terrtia] * [v3.7] add missing file. [Terrtia] * [trackers] items by daterange. [Terrtia] * [correlation graph] fix legend, remove icon text. [Terrtia] * [Credential] fix moduleStats. [Terrtia] * [Credemtial module] fix stats. [Terrtia] * [Yara + regex trackers] remove tests. [Terrtia] * [Decoder] log level. [Terrtia] * [abstract_modules + Global] log message on error + fix Global exception on invalid gzip. [Terrtia] * [Credential] fix old funct call. [Terrtia] * [UI Item submit] tags input: avoid browser and password managers autocomplete. [Terrtia] * [term tracker] typo. [Terrtia] ### Other * Merge branch 'master' of github.com:ail-project/ail-framework. [Terrtia] * Merge branch 'master' of github.com:ail-project/ail-framework. [Terrtia] * Merge pull request #123 from TonyJabbour/master. [Alexandre Dulaunoy] CentOS 8 installation script Fixed a problem * CentOS 8 installation script Fixed a problem Added centos installation guide in README.md. [Tony Jabbour] * Merge branch 'master' of github.com:ail-project/ail-framework. [Terrtia] * Merge pull request #122 from TonyJabbour/master. [Alexandre Dulaunoy] Added CentOS 8 installation script * Added CentOS 8 installation script. [Tony Jabbour] * Merge branch 'master' of github.com:ail-project/ail-framework. [Terrtia] * Merge pull request #119 from Relega/patch-1. [Alexandre Dulaunoy] Update HOWTO.md * Update HOWTO.md. [Relega] * replaced pystemon url (from circl repository to the original repository) * refined pystemon instructions 2021-08-27T21:09:10+00:00 AIL-framework v4.0 2021-12-02T14:57:30+00:00 # AIL Framework version 4.0 has been released including a major new feature to allow synchronisation to other AIL instance(s). The new synchronisation mechanism allow the sync from one AIL instance to another AIL using a standard WebSocket using AIL JSON protocol. The synchronisation allows to filter and sync specific collected items including crawled items or specific tagged items matching defined rules. This feature can be very useful to limit the scope of analysis in specific fields or resource intensive activity. This sync can be also used to share filtered streams with other partners.   A new functionality has been added to trigger a webhook when a tracker is matched in AIL. This is in addition to email notification. The webhook can be used to trigger additional pipelines in AIL. Additional API endpoints were added such `get_item_sources` `get_check_item_source` and `get_default_yara_rule_content`. Thanks to the numerous external contributors such as Olivier Sagit and Tony Jabbour. A special thank for the webhook and API developed by Tony Jabbour from [CSIRT POST Cyberforce in Luxembourg](https://business.post.lu/grandes-entreprises/csirt). The first version of the synchronisation protocol has been developed in the scope of the JTAN (Joint Threat Analysis Network), a CEF co-funded project (2020-EU-IA-0260). Many bugs were fixed in this release and many small improvements were added.   Detailed changelog is available on [https://www.ail-project.org/ChangeLog](https://www.ail-project.org/ChangeLog). # Changelog ## v4.0 (2021-12-01) ### Changes * [sync UI] disable pull. [Terrtia] * [sync UI] dashboard, show nb of imported items + launch/kill ail servers when a queue is subscribed/unsubscribed. [Terrtia] * [ail sync UI] restarr/launch/kill sync connections + show sync mode api/pull/push. [Terrtia] * [ail sync server] add server controller + list connected clients ail_uuid->sync_modes. [Terrtia] * [ail sync ui] copy to clipboard ail_uuid, ail server key. [Terrtia] * [ail sync] edit ail_servers/sync_queues + fix logs. [Terrtia] * [api] rename endpoints. [Terrtia] * [ail sync] add sync api (ping, version) + UI/client error handler. [Terrtia] * [doc] GI Badge. [Steve Clement] * [v4.0 AIL SYNC / AIL 2 AIL] SYNC Manager + fixs + views. [Terrtia] * [crawler] add auto crawler functions. [Terrtia] ### Fix * [sync client] don't launch client if ail server not linked with a sync queue. [Terrtia] * [sync server] remove hardcoded host. [Terrtia] * [sync server] host. [Terrtia] * [sync client] fix websockets client connect for python >= 3.8. [Terrtia] * [ail sync] fix refresh_ail_instance_connection. [Terrtia] * [ail sync] fix refresh_ail_instance_connection. [Terrtia] * [ail sync] server + client: resend object in queue on ConnectionClosedError. [Terrtia] * [crawler] add comment. [Terrtia] * [UI ail sync] fix nav. [Terrtia] * [UI ail sync] add missing ail icon. [Terrtia] * [doc] Remove Travis. [Steve Clement] * [py] Minor python dependency change. [Steve Clement] * Inherit AbstractModule to prevent stuck queues. [osagit] regex compiled only at start, not in the loop no duplicate warning string comments * Error message contains http protocol twice. [osagit] Error Can't connect to AIL Splash Manager, http://https://localhost:7001/ ### Other * Merge pull request #130 from TonyJabbour/master. [Thirion Aurélien] New restAPIs * Merge branch 'dev' into master. [Thirion Aurélien] * Merge branch 'master' of github.com:ail-project/ail-framework. [Terrtia] * Merge branch 'master' of github.com:ail-project/ail-framework. [Alexandre Dulaunoy] * Merge pull request #569 from SteveClement/master. [Steve Clement] * Chg_ [AIL 2 AIL] add backend. [Terrtia] * Merge branch 'master' of github.com:ail-project/ail-framework. [Terrtia] * Add tracker fixed api function replaced it with internal function. [TonyJabbour] * Added get_tracker_metadata_api Removed unnecessarily parentheses. [TonyJabbour] * New API Endpoint: Fixed get_item_content_encoded_text Added get_item_sources Added get_check_item_source Added get_default_yara_rule_content. [TonyJabbour] * Removed unnecessarily parentheses. [TonyJabbour] * New API Endpoint: Return Item Content in base64 in non JSON format. [TonyJabbour] * Merge remote-tracking branch 'origin/master' [TonyJabbour] * Merge branch 'ail-project:master' into master. [Tony] * Merge pull request #129 from TonyJabbour/master. [Thirion Aurélien] Webhook implementation * Unnecessarily parenthesis removed. [TonyJabbour] * Base64 Problem. [TonyJabbour] * Base64 Problem. [TonyJabbour] * Type fixed. [TonyJabbour] * Type fixed. [TonyJabbour] * Error Fixed. [TonyJabbour] * Add new API endpoint that return only content encoded in base64. [TonyJabbour] * Webhook unnecessarily line removed Removed unnecessarily parentheses. [TonyJabbour] * Type fixed. [TonyJabbour] * Added a try/catch to handle exceptions Replaced the raise to send message to redis_logger. [TonyJabbour] * Fixed tracker_metadata. [TonyJabbour] * Typo Fixed. [TonyJabbour] * Fixed UI Datatable in showTracker Page Removed "Advanced Search" text from menu misleading button. [TonyJabbour] * Fixed Webhook integration with Trackers. [TonyJabbour] * -Fixed "get_tracker_metedata" typo -Typo Fixed. [TonyJabbour] * -Fixed "description" arg -Typo Fixed. [TonyJabbour] * -Fixed the 500 error issue when installing new instance of ail when adding new trackers -Fixed missing arguments -Typo Fixed. [TonyJabbour] * Fix webhook. [TonyJabbour] * Remove dict from Trackers. [TonyJabbour] * Add webhook post support in yara and regex trackers. [TonyJabbour] * Fix get_term_webhook. [TonyJabbour] * Add some changes for webhook. [TonyJabbour] * Add initial support for Webhook in Term Trackers. [TonyJabbour] * Fix spelling issue in Webhook. [TonyJabbour] * Add more support for Webhook URL. [TonyJabbour] * Add initial UI support for Webhook in tracker. [TonyJabbour] * Merge pull request #127 from osagit/patch-3. [Thirion Aurélien] fix: inherit AbstractModule to prevent stuck queues * Merge pull request #126 from osagit/patch-2. [Thirion Aurélien] fix: error message contains http protocol twice 2021-12-02T14:57:30+00:00 AIL-framework v4.1 2022-03-14T15:39:04+00:00 # AIL Framework version 4.1 released with new investigation/case handling, improved MISP export and many improvements. ## Investigation in AIL The major new functionality is the investigation handling in AIL. An analyst can now easily create investigation where any objects from AIL can be added. This helps an analyst to build collection or cases to work on. The integration allows to export it as a standard [MISP](https://www.misp-project.org/) event.  ## Support for Jabber/XMPP AIL has been extended to support Jabber/XMPP addresses. The source feeder just need to submit the keys such as `jabber:to`, `jabber:from`, `jabber:ts`, `jabber:id`. An example feeder is [available](https://gist.github.com/gallypette/8e4fc941443a2483b6b2fcaee4c76e47). The new feature can be used to inject existing leak or stream from XMPP/Jabber server. As an example, the Conti leak can be easily injected into AIL and show automatically all correlations between users.    Many bugs were fixed. The [complete changelog](https://www.ail-project.org/ChangeLog) can be seen below. ## v4.1 (2022-03-11) ### Changes * [flask] updated. [Alexandre Dulaunoy] * [flask] requirements for higher version of flask. [Alexandre Dulaunoy] * [v4.1] add Investigation with MISP Export + v4.1 update. [Terrtia] * [Telegram module] refactor module + fix str format. [Terrtia] ### Fix * [Investigation] edit misp event + add misp instance url. [Terrtia] * [Investigation] fix MISP Export + UI sidebar. [Terrtia] * [UI inestigations] add items link. [Terrtia] * [UI inestigations] add objects link. [Terrtia] * [telegram launcher] [Terrtia] * [items] abstract class. [Terrtia] * [Investigation] UI sidebar. [Terrtia] * [v4.1] fix ardb # tracking DB. [Terrtia] * [username] user icon. [Thirion Aurélien] * [Term tracker] fix item date. [Terrtia] * [Telegram module] fix launcher. [Terrtia] * [pybgpranking] package install. [Terrtia] * [popper install] rename popper repository. [Terrtia] https://github.com/floating-ui/floating-ui/discussions/1425 * [UI] remove update note. [Terrtia] * [trackers] fix get_all_items_sources. [Terrtia] * [crawler] fix is_splash_manager_connected #133. [Terrtia] ### Other * Merge branch 'master' of github.com:ail-project/ail-framework. [Terrtia] * Merge pull request #139 from gallypette/jabber-feeder. [Thirion Aurélien] add: [username] jabber support * Add: [username] jabber support. [Jean-Louis Huynen] * Merge branch 'master' of github.com:ail-project/ail-framework. [Terrtia] * Create SECURITY.md. [Alexandre Dulaunoy] 2022-03-14T15:39:04+00:00 AIL-framework v4.2 2022-07-16T08:40:47+00:00 ## AIL Framework version 4.2 released including typo squatting tracker, improved AIL2AIL sync, zerobinz fetcher and many bugs fixes v4.2 (2022-06-24) AIL Framework version 4.2 has been released including: - A new tracker for tracking potential typo squatted domains. This feature relies on the new [ail-typo-squatting ](https://github.com/ail-project/ail-typo-squatting) library which can be also used outside of AIL framework. This contribution is from @DavidCruciani - Many improvement and bugs fixed for the AIL2AIL sync. A huge thanks to @aaronkaplan from EU Directorate-General for Informatics (DIGIT) for support and tests during the long debugging sessions. - A new module for zerobinz to create an immediate crawler request if a zerobinz link appears in an item. The module can be used for other services with ephemeral content. Thanks to @gallypette for the contribution and the improvement ideas. - A new hosts detection module has been introduced. - Multiple bugs were fixed. ### Detailed Changes * [Tracker} Tracker_Typo_Squatting. [David Cruciani] * [v4.2] add v4.2 update. [Terrtia] * [investigation] fix investigation by user + delete an obj from all investigation. [Terrtia] * [install vitualenv] remove travis env. [Terrtia] * [Retro Hunt] add logs. [Terrtia] * [Retro Hunt] add logs. [Terrtia] * [Retro Hunt] add logs. [Terrtia] * [AIL2AIL Sync] update exchange format. [Terrtia] * [AIL2AIL Sync] update exchange format. [Terrtia] * [add Hosts module] [Terrtia] * [sync module] debug. [Terrtia] * [sync client] debug. [Terrtia] * [websockets client] bind client ip. [Terrtia] * [websocket server] add host and port config. [Terrtia] * [telegram importer] add username correlation. [Terrtia] * [UI subtype objs] get obj by subtype + name. [Terrtia] * [misp export] add username. [Terrtia] ### Fix * [typosquatting] remove unused import. [Thirion Aurélien] * [tracker] clean import. [Thirion Aurélien] * [tracker term] fix typosquatting key. [Thirion Aurélien] * [Typo] tracker typo. [David Cruciani] * [tracker] UI for other than typosquat. [David Cruciani] * [typo] UI. [David Cruciani] * [Language] fix cld3 import. [Terrtia] * [launcher] kill AIL_2_AIL screen. [Terrtia] * [cld3] enable cld3. [Terrtia] * [cld3 python3.10] temp disable cld3. [Terrtia] * [launcher] remove Travis test. [Terrtia] * [Retro Hunt] item directory. [Terrtia] * [Retro Hunt] item directory. [Terrtia] * [Retro Hunt] fix item directory. [Terrtia] * [AIL exchange mime-type] [Terrtia] * [Hosts module] module + launcher. [Terrtia] * [abstract module] exception traceback #145. [Terrtia] * [ui tag selector] force custom tags. [Terrtia] * [installer] remove old tor install. [Terrtia] * [sync module] fix redis tag queue. [Terrtia] * [sync module] fix tags filter. [Terrtia] * [sync client] debug. [Terrtia] * [sync client] debug. [Terrtia] * [sync module] debug. [Terrtia] * [websockets client] fix client bind. [Terrtia] * [websockets] remove size limit. [Terrtia] * [UI subtype objs] fix form. [Terrtia] * [misp config] https. [Thirion Aurélien] ### Other * Merge pull request #147 from ail-project/typo. [Thirion Aurélien] Integration of the typo-squatting tracker * Fix; [set tracker] missing function. [Thirion Aurélien] * Merge branch 'master' into typo. [David Cruciani] * Add: [tracker] typo-squatting. [David Cruciani] * Merge branch 'master' of github.com:ail-project/ail-framework. [Terrtia] * Merge pull request #146 from gallypette/master. [Thirion Aurélien] add: [modules] zerobinz * Add: [modules] zerobinz. [huynenjl@gmail.com] * Merge branch 'master' of github.com:ail-project/ail-framework. [Terrtia] 2022-07-16T08:40:47+00:00 AIL-framework v5.0 2023-06-06T12:52:49+00:00 AIL v5.0 introduces significant improvements and new features: - **Codebase Rewrite**: The codebase has undergone a substantial rewrite resulting in enhanced performance and speed improvements. - **Database Upgrade**: The database has been migrated from ARDB to Kvrocks. - **New Correlation Engine**: AIL v5.0 introduces a new powerful correlation engine with two new correlation types: CVE and Title. - **Enhanced Logging**: The logging system has been improved to provide better troubleshooting capabilities. - **Tagging Support**: [AIL objects](./doc/README.md#ail-objects) now support tagging, allowing users to categorize and label extracted information for easier analysis and organization. - **Trackers**: Improved objects filtering, PGP and decoded tracking added. - **UI Leak Visualization**: The user interface has been upgraded to visualize extracted and tracked information. - **New Crawler [Lacus](https://github.com/ail-project/lacus)**: improve crawling capabilities. - **Modular [Importers](https://github.com/ail-project/ail-framework/blob/master/doc/README.md#ail-importers) and Exporters**: New [importers](https://github.com/ail-project/ail-framework/blob/master/doc/README.md#ail-importers) (ZMQ, AIL Feeders) and exporters (MISP, Mail, TheHive) modular design. Allow easy creation and customization by extending an abstract class. - **Module Queues**: improved the queuing mechanism between detection modules. - **New Object CVE and Title**: Extract an correlate CVE IDs and web page titles. ### Correlation:  ### UI Extracted/Tracked content:  2023-06-06T12:52:49+00:00 AIL-framework v5.1 2023-06-26T12:03:46+00:00  Version 5.1 (2023-06-26) includes several changes, fixes, and updates. The changes include fixing gzipped pastes in the pystemon importer, showing a message when the maximum number of nodes is reached in the correlation graph, and adding the ability to auto tag crawled domains. Additionally, new features were added such as pagination for title searches, the ability to search title IDs and contents, and the inclusion of a favicon object. Several fixes were implemented, including resolving issues with base64 encoding in the pystemon importer, maintaining the same capture UUID for already crawled domains in the crawler, and handling empty queues in the IPAdress module. Other fixes addressed issues with title searches returning empty results, incomplete responses in the crawler, and errors related to user tokens and deletion. Various improvements were made to different modules and objects, such as the addition of a new `cookie-name` object along with its correlation, enhancements to importers, improvements to the HOWTO guide, and updates to correlation graphs and statistics. The Phone module was also updated to filter invalid phone numbers and display extracted information in the user interface. In addition to the changes and fixes, there were updates to the MISP export, domains explorer, daterange object, tracker module, and various other components. The README.md file and CI badge were corrected, the installer was fixed for YARA and pycld3 installations, and tests were updated and replaced. Lastly, there were some general updates, including merging changes from the old CIRCL/AIL-framework repository (the official repository is [ail-project/ail-framework](https://github.com/ail-project/ail-framework) and incorporating a pull request related to email categorization. Overall, version 5.1 introduced new features, addressed several issues, and included various updates and improvements to different parts of the system. # Detailed Change Log ## v5.1 (2023-06-26) ### Changes * [pystemon importer] fix gzipped pastes. [Terrtia] * [correlation graph] show message if max_nodes reached + fix cookie-name sparkline. [Terrtia] * [crawler] auto tag crawled domains. [Terrtia] * [correlation] add an option to remove max number of nodes if max_node == 0. [Terrtia] * [object cookie-name] add new cookie-name object + correlation. [Terrtia] * [title search] add pagination. [Terrtia] * [titles] add title IDs and contents search. [Terrtia] * [favicon object] add favicon object. [Terrtia] * [sow item] show item investigations. [Terrtia] * [kvrocks migration] mv update/v.50. [Terrtia] * [redis] update minimal version. [Terrtia] * [doc] add AIL v5.0 + objects + Importers + sync. [Terrtia] * [correlation] filter blank screenshots. [Terrtia] * [importers] improve abstract class and logs. [Terrtia] * [domains explorer] unsafe tag default image. [Terrtia] * [README.md] update. [Terrtia] * [HOWTO] improve HOWTO. [Terrtia] * [correlation graph] update node legend. [Terrtia] * [correlation graph] select correlation depth. [Terrtia] * [correlation] correlation graph: filter title objects. [Terrtia] * [correlation] add direct correlation stats. [Terrtia] * [new title object] add new title object + correlation on page title. [Terrtia] * [Phone module] Filter Invalid Phone numbers + UI Show extracted. [Terrtia] * [importers] add Dir/File Importer. [Terrtia] ### Fix * [pystemon importer] fix base64 encoding. [Terrtia] * [crawler] same capture uuid if a domain is already crawled. [Terrtia] * [IPAdress module] empty queue if no IP ranges provided. [Terrtia] * [retro hunt] fix object tag queue + decoded content. [Terrtia] * [daterange object] fix objects by date. [Terrtia] * [title] fix title search empty result. [Terrtia] * [crawler] fix incomplete response. [Terrtia] * [user] fix get user token #163. [Terrtia] * [user] fix user delete #163. [Terrtia] * [MISP export] fix ail object first/last seen + obj logger. [Terrtia] * [MISP export] fix empty event. [Terrtia] * [d4] change enable d4. [Terrtia] * [kvrocks migration] [Terrtia] * [objects] fix investigation + ail2ail + screenshot MISP export. [Terrtia] * [domains explorer] None screeenshot. [Terrtia] * [show domains] fix down domains. [Terrtia] * [domains explorer] domain screeenshot. [Terrtia] * [domains explorer] fix empty screenshots. [Terrtia] * [correlation] fix tagging nb nodes. [Terrtia] * [README.md] fix CI badge. [Terrtia] * [README.md] fix logo. [Terrtia] * [module.cfg] fix templateModule example. [Terrtia] * [module extractor] fix tracker extractor. [Terrtia] * [tracker] fix tracker delete. [Terrtia] * [tracker] fix webhook. [Terrtia] * [crawler] fix undefined capture status. [Terrtia] * [correlation btc info] catch btc txs error. [Terrtia] * [Phone module] Filter Invalid Phone numbers. [Terrtia] * [phone] fix phone module. [Terrtia] * [domain search] fix template domain types filter. [Terrtia] * [domain search] fix template domain types filter. [Terrtia] * [MISP auto export] fix module input message. [Terrtia] * [tests] replace unmaintened nose by nose2. [Terrtia] * [tests] fix tests. [Terrtia] * [instaler] fix yara and pycld3 install. [Terrtia] * [tests] github workflow. [Terrtia] * [tests] github workflow. [Terrtia] * [flask] remove old import. [Terrtia] ### Other * Merge github.com:CIRCL/AIL-framework. [Terrtia] * Merge pull request #592 from shadow2033/patch-2. [Thirion Aurélien] Update Categ Mail * Update Mail. [shadow2033] ///English added (inbox; zoho) ///Russian добавлен (inbox; zoho) 2023-06-26T12:03:46+00:00 AIL-framework v5.2 2023-07-12T13:21:27+00:00  The latest release, version 5.2 of the AIL project framework, introduces several changes, fixes, and improvements. Some notable changes include the removal of old updates prior to version 5.0, a refactoring of the background updater along with the addition of the v5.2 update, and the introduction of a new etag object. The correlation graph now offers an option to hide objects/nodes and reset functionality using the "H" key. Additionally, an object comment feature has been added to investigations. Several fixes have been implemented in this release. The environment issues in the updater and background update modules have been resolved. The crawler now includes timeouts for Unknown captures and exception handling for ping_lacus. It also performs an existence check for screenshot sets. The decoding process for downloaded files has been fixed, and the tag functionality for correlation objects has been improved. Fixes have also been made to the updater module, including the removal of old ARDB environment references. Lastly, issues with MISP event JSON export and retro hunt date search and description in the hunter module have been addressed. In terms of contributors, [Thirion Aurélien](https://github.com/terrtia) and [fukusuket](https://github.com/fukusuket) have made significant contributions to this release, addressing specific issues and providing fixes. # Detailed Change Log ## v5.2 (2023-07-12) ### Changes * [update] remove old updates < 5.0. [Terrtia] * [updater] refactor background updater + add v5.2 update. [Terrtia] * [crawler har] compress HAR. [Terrtia] * [correlation] correlation graph, add an option to hide an object/node by pressing H + reset correlation graph. [Terrtia] * [etag] add new etag object. [Terrtia] * [investigation] add object comment. [Terrtia] ### Fix * [updater] fix env. [Terrtia] * [background update] fix logger. [Terrtia] * [crawler] add timeout to Unknown captures. [Terrtia] * [crawler] add exception handing for ping_lacus. [fukusuket] * [crawler] added existence check for screenshot set. [fukusuket] * [decoded] fix download file. [Terrtia] * [updater] fix db checker. [Terrtia] * [correlation tags] fix tag all objects. [Terrtia] * [correlation card decoded meta] mimetype + size. [Terrtia] * [correlation card decoded meta] mimetype + size. [Terrtia] * [updater] remove old ARDB env. [Terrtia] * [hunter + misp export] fix misp event json export + retro hunt date search and description. [Terrtia] ### Other * Merge pull request #174 from fukusuket/fix-500-erro-when-invalid-lacus-url. [Thirion Aurélien] fix: [crawler] add exception handing for `ping_lacus` * Merge pull request #176 from fukusuket/fix-500-error-when-crawler-screenshot-setting-off. [Thirion Aurélien] fix: [crawler] added existence check for screenshot set 2023-07-12T13:21:27+00:00