http://open-source-security-software.net/project/Lookyloo/releases.atom 2024-04-29T13:13:55.366274+00:00 python-feedgen Lookyloo v1.3 2020-12-24T11:58:56+00:00 Thanks to [Internews](https://globaltech.internews.org/) and the [BASICS Project](https://globaltech.internews.org/our-resources/basics) we were able to greatly improve Lookyloo over the last few months. These release notes won't be exhaustive and if you want to see all the changes, you should have a look at the git changelog. Here is a short overview of the main changes in the last ~6 months: * Major rewrite of the user interface based on the user tests realized thank to the [BASICS Project](https://globaltech.internews.org/our-resources/basics) and other user feedbacks * Major improvements in the investigation popup * [Documentation website](https://www.lookyloo.eu/docs/main/), also thanks to the [BASICS Project](https://globaltech.internews.org/our-resources/basics) * Indexing of cookies and hashes of resources * Add support for marking specific resources as known (libraries, icons, ... related to a specific domain or not) or malicious (phishing, malwares) * DNS resolution (IP and CNAMEs) * Query third party services via a modules system (SaneJS, VirusTotal, Phishing Initiative) * Configuration via config files * Update script * Statistics of the whole instance * Export in [MISP](https://www.misp-project.org/) format * Systemd templates * Docker image * Update dependencies and bug fixes all over the place. 2020-12-24T11:58:56+00:00 Lookyloo v1.4.0 2021-02-09T16:11:43+00:00 Once again, many of the changes in this release wouldn't have been possible without the support of [Internews](https://globaltech.internews.org/) and the [BASICS Project](https://globaltech.internews.org/our-resources/basics). On the UI front, we now have a better support of huge screenshots and many more tooltips are shown when the mouse goes over icons and different parts of the tree. It should make the tree easier to read for users discovering the platform. The main new feature of this release is the integration with [MISP](https://www.misp-project.org/). It is now possible to export a capture directly to a pre-configured MISP instance:  The documentation in order to get it to work [is also available](https://www.lookyloo.eu/docs/main/lookyloo-integration.html#_misp). There were also quite a few changes for the administrators of a Lookyloo instance, especially the [authentication](https://www.lookyloo.eu/docs/main/lookyloo-auth.html). And for more details, you should have a look at the git changelogs. 2021-02-09T16:11:43+00:00 Lookyloo v1.5.0 2021-04-02T14:02:45+00:00 Once again, many of the changes in this release wouldn't have been possible without the support of [Internews](https://globaltech.internews.org/) and the [BASICS Project](https://globaltech.internews.org/our-resources/basics). And we would also like to thank [Credit Agricole](https://www.credit-agricole.com/en/) and @FafnerKeyZee for the continuous bug reports! The main new feature of this release is the possibility to capture URLs present in a capture you already made, all that while **keeping the context** your browser was in (cookies, user-agent, referer) in the **subsequent captures**. It is especially useful when the page you're landing on expects the user to click on a link in order to load the content, the website checks the referer and/or cookies, and bounce you if you're not presenting the right session. This feature will be further extended in the upcoming releases to allow other types of requests (`POST`), and let the user choose the link(s) to captures from the screenshot of the page itself. This release also adds a new **background indexer** so the captures queued with the API are automatically cached even if they are never opened in the browser. And there are the usual bunch of bugfixes, improvements and dependencies upgrades. And we also require python 3.8+. You can also **search** for hostnames, URLs, hashes, and cookies names from the `/search` entry point. This entry point is *not* listed yet in the documentation, but it will be added soon. 2021-04-02T14:02:45+00:00 Lookyloo v1.6.0 2021-05-21T18:33:41+00:00 This release contains lots of changes in the backend (described below), and a few improvements on the web interface: * Hide the captures with error from the index (see [hide_captures_with_error](https://www.lookyloo.eu/docs/main/lookyloo-configuration.html#_optional_features)) * Return resources as text instead of in a zip file * Crop and blur screenshot if it is too big to be displayed as-is * Redesign of the menus * Fix rendering of image resources in the investigation popup The backend changes are mainly improving the overall performance of Lookyloo, with a few new features: * All the captures (web and API) are using the asynchronous capture script, and the priority of each capture is weighted depending on the origin and the user(see [priority](https://www.lookyloo.eu/docs/main/lookyloo-configuration.html#_core_config), the number of async capture process is configurable (see [async_capture_processes](https://www.lookyloo.eu/docs/main/lookyloo-configuration.html#_core_config)) * The index is cached in memory by the webserver, making the index view lot faster after first load * Improve auto-trigger of 3rd party modules (configurable [per module](https://www.lookyloo.eu/docs/main/lookyloo-integration.html)) * Add [optional integration]() of whois queries with [uWhoisd](https://www.lookyloo.eu/docs/main/lookyloo-integration.html#_uwhoisd) * Disable [FLoC](https://amifloced.org/) globally * Many bug fixes in [har2tree](https://github.com/Lookyloo/har2tree) and the creation of the tree * Fix and improve MISP export, support subsequent captures as extended events * Update all dependencies 2021-05-21T18:33:41+00:00 Lookyloo v1.7.0 2021-07-21T16:38:24+00:00 The two main changes in this release are: * Add support for passing a proxy to a capture, thanks to @Felalex57 - [Documentation](https://www.lookyloo.eu/docs/main/usage.html#_proxy) * Major improvement in the API using [flask-restx](https://github.com/python-restx/flask-restx) - [Documentation](https://lookyloo.circl.lu/doc/) on the demo Lookyloo instance. * Add lookup against a MISP instance - [Documentation](https://www.lookyloo.eu/docs/main/lookyloo-integration.html#_lookup_on_a_misp_instance) * Add sample config for log rotate thanks to @FafnerKeyZee - [Documentation](https://www.lookyloo.eu/docs/main/install-lookyloo-production.html#_logging_and_rotation) The other changes are mainly bugfixes and small changes: * Avoid receiving notifications from bots * Upgrade the bundled-in list of user agents * Improve generation of the pickles and avoid doing it twice * Add reference to parent in the case a capture is initiated from an other one * Improve MISP export 2021-07-21T16:38:24+00:00 Lookyloo v1.8.0 2021-08-30T13:34:31+00:00 **New Features**: * Integration with [urlscan.io](https://urlscan.io) - [Documentation](https://www.lookyloo.eu/docs/main/lookyloo-integration.html#_urlscan_io) * Trigger a capture from the URL - https://github.com/Lookyloo/lookyloo/issues/248 * Archiving: the captures more than 6 month old ([configurable](https://www.lookyloo.eu/docs/main/lookyloo-configuration.html#_core_config)) are moved to an archive directory so they're not listed on the index anymore, but the captures can still be accessed by UUID (doesn't break permanent URLs) * Index file by directory for each captures (archived or not). Greatly reduces the I/O when initializing the known captures in redis. **Fixes**: * Missing 3rd party web dependencies in docker (thanks to @FafnerKeyZee) **Changes** - This release is implementing a lot of back end changes : * The captures are now stored a by year and month (instead of in a single directory) to avoid having too many entries in the same directory (ext4 dislikes it). All the new captures are following this new architecture, but you need to run `tools/change_captures_dir.py` to move the existing ones to the new format (only useful if you feel restarting the app takes too much time) * Move all the capture-related code from `Lookyloo` to `AsyncCapture` * Move all the services management code to abstractmanager * Use redis pooling to manage connections to the database in `Lookyloo` and `Indexing` * New process to trigger occasional actions, currently: generate the daily user-agent file if Lookyloo is using the UAs of its own users. * Reinitialize the list of captures UUIDs when starting the app instead of the in website itself * Improvements in processes handling (TL;DR: don't stop redis until all the async captures processes are down) * Move some methods from `Lookyloo` to the helpers * Simplify code in `Lookyloo` to make it more readable, remove dead code. * Bump dependencies, add `hiredis` to speed up redis interactions * Return proper HTTP error codes (mostly 4XX), when appropriate 2021-08-30T13:34:31+00:00 Lookyloo v1.9.0 2021-09-28T16:23:08+00:00 # New features * Integration with [Phishtank](https://phishtank.org/) via [Phishtank Lookup](https://github.com/Lookyloo/phishtank-lookup) - [Documentation](https://www.lookyloo.eu/docs/main/lookyloo-integration.html#_phishtank_lookup_v1_9). ---  --- * Simple [monitoring script](https://github.com/Lookyloo/lookyloo/blob/main/tools/monitoring.py) to keep an eye on the health of the instance, run it in a tmux/screen with watch. ---  --- * Link in the tree menu to re-trigger a capture on the same URL. # Fixes * Improve logging entries, the date was incomplete. * Add UUID file in export. * Inform users when a capture failed critically and we have nothing to show. * Catch timeout when pushing to MISP (avoid exception) # Changes * Major improvements in caching, better handling of exceptions and keep a limited amount of pickles in memory. * Simplify code in the async capture script. * Add urlscan.io permalink in MISP export * Add phishtank permalink in MISP export * Move modules to dedicated files 2021-09-28T16:23:08+00:00 Lookyloo v1.10 2021-12-03T11:00:08+00:00 # New features * [Hashlookup](https://www.circl.lu/services/hashlookup/) integration - [Documentation](https://www.lookyloo.eu/docs/main/lookyloo-integration.html#_hashlookup_v1_10) * Pass arbitrary HTTP headers to captures - [Documentation](https://www.lookyloo.eu/docs/main/usage.html#_capture_configuration) * Pass arbitrary User-Agents to captures - [Documentation](https://www.lookyloo.eu/docs/main/usage.html#_browser_configuration) * Get hashes of all the resources using any algorithm supported by Python (API) * Add configuration setting to make captures private by default - See `default_public` in the [Documentation](https://www.lookyloo.eu/docs/main/lookyloo-configuration.html#_core_config) * Add [CORS](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS) settings to allow JavaScript submissions (required for the [browser extension](https://github.com/Lookyloo/webext)) * Defang URLs in email notifications # Fixes * Avoid exception when the timestamp of a capture has no millisecond * Avoid exceptions in archiver when indexes are broken # Changes * Improve logging * Improve capture page * Normalize tooltips across the app * Save redis databases to disk less often * Programmatically shutdown redis databases (synchronous) * Bump dependencies 2021-12-03T11:00:08+00:00 Lookyloo v1.10.0 2021-12-03T11:00:08+00:00 # New features * [Hashlookup](https://www.circl.lu/services/hashlookup/) integration - [Documentation](https://www.lookyloo.eu/docs/main/lookyloo-integration.html#_hashlookup_v1_10) * Pass arbitrary HTTP headers to captures - [Documentation](https://www.lookyloo.eu/docs/main/usage.html#_capture_configuration) * Pass arbitrary User-Agents to captures - [Documentation](https://www.lookyloo.eu/docs/main/usage.html#_browser_configuration) * Get hashes of all the resources using any algorithm supported by Python (API) * Add configuration setting to make captures private by default - See `default_public` in the [Documentation](https://www.lookyloo.eu/docs/main/lookyloo-configuration.html#_core_config) * Add [CORS](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS) settings to allow JavaScript submissions (required for the [browser extension](https://github.com/Lookyloo/webext)) * Defang URLs in email notifications # Fixes * Avoid exception when the timestamp of a capture has no millisecond * Avoid exceptions in archiver when indexes are broken # Changes * Improve logging * Improve capture page * Normalize tooltips across the app * Save redis databases to disk less often * Programmatically shutdown redis databases (synchronous) * Bump dependencies 2021-12-03T11:00:08+00:00 Lookyloo v1.11.0 2022-03-31T11:17:11+00:00 # New Feature * Trigger multiple captures at once from web interface  # Fixes * Improve MISP event publishing (make it asynchronous) * Improve legend with titles on hoover * Fix caches in modules * Improve stats page * Normalize buttons color * Improve rendering of capture page # Changes * Updates all web and python dependencies * Use bootstrap 5 2022-03-31T11:17:11+00:00 Lookyloo v1.12.0 2022-05-24T13:33:18+00:00 # New Features ## Playwright The captures are now made via [Playwright](https://playwright.dev/) instead of [Splash](https://github.com/scrapinghub/splash). It is a major improvement as Playwright uses actual up-to-date browsers, in headless mode (instead of qt-webkit from ~2016). You can read more about the research that lead to this change [in the discussion](https://github.com/Lookyloo/lookyloo/discussions/243). The main other advantages of using playwright are the following: * Easier to install: it doesn't requires Docker in order to use Splash * Much better control of what happen in the browser while capturing: Playwright makes it extremely simple to instrument everything in the browsers. The capturing module already tries to solve reCaptcha if it detects it on the page. The capture is made by a [standalone](https://github.com/Lookyloo/PlaywrightCapture) python module that you can use in your own tools if you wish to. ## De-duplication If the exact same capture is triggered multiple times within 5 min, it is skipped and the requestor is redirected to the capture done before. # Fixes * Avoid discarding a capture on network error: when a redirect is broken down the line, we keep the chain up to that point * Issue when the MISP was submitted as un-published * [Docker] Properly handle archiving * [Docker] Init SRI hashes # Changes * Improve subsequent capture template on long URLs * Improve view of the capture page on small-ish screens * General maintenance and code cleanup * Improvement in the tree generation on edge cases * Bump JS/CSS libraries * Update bundled-in User-Agent file * Use pydeep2, comes with a bundled-in libfuzzy, easier to install. 2022-05-24T13:33:18+00:00 Lookyloo v1.13.0 2022-06-26T16:06:50+00:00 # Maintenance and bug-fixes release All releases don't need to contain new features, sometimes, it is just some cleanup, and it is okay. * Properly handle exceptions in some edge cases (fixes in har2tree) * Properly display an error message if the capture fails * Use the same default User-Agent in when a capture is submitted via the API as via the web interface. * Cleanup some legacy code * Bump all dependencies (JS/CSS and Python) # Still, there is a new-ish thing We revamped the package generator, and it should be [more usable](https://github.com/Lookyloo/lookyloo/pkgs/container/lookyloo). If it is not, let us know! 2022-06-26T16:06:50+00:00 Lookyloo v1.14.0 2022-08-08T13:53:34+00:00 # New features * Trigger a capture on a web enabled document provided as a file instead of a URL. Useful for HTML files attached to emails, or HTML body in email.  -------- * Compress (gzip) the HAR file in archived captures - saves a lot of disk space. * Support for RiskIQ Passive DNS (requires API key) * Display SSL/TLS information available in the HAR dump from Playwright  -------- * Optional DoNotTrack HTTP header in capture  -------- * Display size of rendered page on hostnode popup. * [WiP] Download files when the URL captured to a downloadable file (PDF, Office doc, ...) (**Important note** the downloaded file is not exposed to the user yet) * [WiP] List all hashes available in the capture, sort them by frequency. Makes it easier to find phishing sides using the same resources.  # Fixes * Major speed improvements when displaying the hostnode popups (only show the recent cached captures by default) * Improvements in the caching mechanism * Cleanup data showed by monitoring script * Avoid crashes when RiskIQ isn't reachable # Changes * Update dependencies (js, python) * Improve logging in archiver * Improve config file 2022-08-08T13:53:34+00:00 Lookyloo v1.15.0 2022-08-25T12:43:29+00:00 # Breaking change * Lookyloo requires Redis 7.0 or more decent. The upgrade process is as follows: 1. Go to the Redis directory (should be in the same directory as where you cloned Lookyloo) 2. Run the following commands ``` git fetch git checkout 7.0 make distclean make -j4 make test ``` 3. You now have the new version of redis in place, you can update lookyloo as usual. # New features * Use pre-configured devices from Playwright (mobile only for now)  * Download files when the URLs points to a downloadable content  * Submit downloadable content to [Pandora](https://pandora.circl.lu/submit) (if available) * Automatically select the most appropriate browser engine based on the user-agent # Fixes * Make sure all the gunicorn instances displays all the recent captures * Other bugfixes and GUI improvements # Changes * Improve capture page with radio button to select which user-agent to submit * Bump dependencies 2022-08-25T12:43:29+00:00 Lookyloo v1.16.0 2022-10-29T13:19:39+00:00 # Breaking change This release requires poetry v1.2.0 or more recent. Run the following command to upgrade it: `poetry self update` # New Features * Move to Lacus/LacusCore, many changes to make lookyloo's code compatible with it * [Lacus](https://github.com/ail-project/lacus) and [PyLacus](https://github.com/ail-project/pylacus): use this mode to trigger the capture from an other machine than the one you run Lookyloo from * [LacusCore](https://github.com/ail-project/lacuscore): (the default) keep triggering the captures from the same machine as the one lookyloo is running on With lacus, the captures are more reliable and using Lacus as a web service allows to monitor them better. If you want to use the webservice, you'll need to: 1. [Install Lacus](https://github.com/ail-project/Lacus#install-guide) * make sure it is running by loading http://127.0.0.1:7100 on the machine you have it running on (7100 is the default port, you can of course change it) 2. Edit the config file `config/generic.json` (key `remote_lacus`): * set `enable` to `true` * set `url` to the url your loolyloo instance can use to connect to lacus: `http://<ip>:<port>` 3. Restart lookyloo & try it # Changes * Make hashlookup visible to everyone * Improve loggingv1.15.0 * Maintenance: use poetry 1.2, bump deps (Python and JS), bump Github actions * Improve caching with Lacus * Retry failing captures when it might be a temporary issue (typically domain resolution) # Fixes * Avoid triggering a capture (and failing) when the URL and documents are missing * Issue with urlscan when the capture had no referer * Better handling of exceptions in VT module * Better handling of devices offered by Playwright and their user-agents 2022-10-29T13:19:39+00:00 Lookyloo v1.17.0 2022-12-29T13:48:48+00:00 # Breaking change Poetry v1.3.0 or more recent is now required, please [upgrade](https://python-poetry.org/docs/#installation) to the latest version. # New features: * See [Lacus release notes for v1.2.0](https://github.com/ail-project/lacus/releases/tag/v1.2.0) if you want more details on the changes regarding the captures. * Temporary local storage of captures if Lacus web-service isn't reachable temporarily * Submit and view a HAR file captured somewhere else, or a full Lookyloo capture from an other instance  * Show status of captures when they're submitted in bulk  * List of all downloadable contents for a capture in a modal (from the tree view -> `Download capture elements`):  # Bugfixes * Catch and stop script when generating a tree takes too long (link to doc) * [har2tree] Avoid exception when a node doesn't have a pageref * [har2tree] Better use of BeautifulSoup * [har2tree] Better handling of nodes that aren't loading a URL * [har2tree] Improve mimetime detection in HTTP responses # Changes * [har2tree] Improve decoding of POSTed data * Use more recent Flask and flask-restx * Bump deps (Python ans JS) * Major logging improvements in Lookyloo, LacusCore, PlaywrightCapture and har2tree * 2022-12-29T13:48:48+00:00 Lookyloo v1.18.0 2023-03-01T11:33:31+00:00 # New features * Beta support for [monitoring](https://github.com/Lookyloo/monitoring): the system will trigger a recurring capture and allow to compare them over time. * Beta support for comparing captures. For now, it focuses on the redirects from the URL captured to the landing page, and the URLs of the resources loaded on that page. An upcoming release of this feature will compare the rendered content of the landing page too. * Takedown endpoint (API only): gather contact information (whois on domain, IP, ASN, and [security.txt](https://github.com/Lookyloo/pysecuritytxt) file if available) for all the URLs up to the landing page. Can be used to notify owners in case of a malicious URL. * Flag known Cloudflare IPs on the hostnode popups * Trigger AAAA DNS requests (was only A before) # Bugfixes * When the capture is a file that was uploaded by the user, some modules cannot be triggered. Avoiding exceptions. * In some edge cases, a lock file for generating the tree could be left there even if the initiating script was dead. We now clean them up automatically. # Changes * Use [Lacus v1.3.0](https://github.com/ail-project/lacus/releases/tag/v1.3.0) / [LacusCore v1.3.0](https://github.com/ail-project/LacusCore/releases/tag/v1.3.0) / [PlaywrightCapture v1.18.0](https://github.com/Lookyloo/PlaywrightCapture/releases/tag/v1.18.0) * Allow to disable defanging URLs in emails * Many improvements in the rendering of the menus on the tree page * [Lookyloo] Bump javascript and python dependencies * [Har2Tree](https://github.com/Lookyloo/har2tree/releases/tag/v1.18.0) Maintenance, update dependencies 2023-03-01T11:33:31+00:00 Lookyloo v1.19.0 2023-03-30T10:25:46+00:00 # New features * The email notification now attaches the contacts, making takedown requests easier. * (WiP) Add settings for comparing captures. It is not possible to ignore domains and/or a substring in a resource URL loaded from the landing page. * Update [PyLookyloo](https://github.com/Lookyloo/PyLookyloo) to pass the settings when comparing captures * [Admin users only] Modal to trigger admin-only tasks on a specific capture (hide/rebuild) # Bugfixes * Fix docker compose (thanks to @bib0x) * Avoid exception at multiple places when a capture is invalid for any reason # Changes * Force protocol 5 for pickles (requires python 3.8, but lookyloo already required it anyway) * Optimize pickle before storing, and archive them to reduce diskspace * Bump dependencies (js & python) * Improve logging (add capture UUID when possible, makes debugging easier) * Always use `LookylooException` instead of `Exception` * Update Playwright in [PlaywrightCapture](https://github.com/Lookyloo/PlaywrightCapture) * Improve logging in [har2tree](https://github.com/Lookyloo/har2tree) 2023-03-30T10:25:46+00:00 Lookyloo v1.20.0 2023-06-01T14:21:18+00:00 This release is the outcome of a good two months of work on Lookyloo itself but also [Lacus](https://github.com/ail-project/lacus) and its dependencies leading to the [v1.5.0 release](https://github.com/ail-project/lacus/releases/tag/v1.5.0). It also improves the support for the [monitoring interface](https://github.com/Lookyloo/monitoring) (still to be considered beta). # New features * Compare captures via the API * Submit any for to [Pandora](https://github.com/pandora-analysis/pandora) (if available) * Allow automatic reporting via the API * Can set an email to notify in the monitoring form # Changes * Improve handling of long running processes, * Improve logging all over the place * Changes related to Lacus/LacusCore/PyLacus changes * Easy way to check if two captures are different or not * Store capture settings in the capture directory for potential later use * Show proxy in UI if one was given * Improve response when comparing captures # Bugfixes * Avoid issues when the pickle requires too much recursivity * Cloudflare services was always flagging URLs as their own * The usual batch of bugfixes all over the place 2023-06-01T14:21:18+00:00 Lookyloo v1.21.0 2023-06-30T20:46:08+00:00 # New Features * Allow to pass a timezone, geolocation coordinates, locale, and color scheme to a capture * Add a global proxy option in the settings * Improve SMTP auth for notifications # Changes * Store the capture settings in order to reuse them later (like for re-capture) * Avoid failing if Lacus isn't available ant retry a few times # Bugfixes * Properly handle captures with errors, improve logging accordingly * Resubmit captures if they were deleted on Lacus without storing a response (generally if something crashed) 2023-06-30T20:46:08+00:00 Lookyloo v1.22.0 2023-10-17T07:24:26+00:00 If you haven't been upgrading lookyloo since the last tagged release, this release contains *a lot* of changes. # New features * Support for [HTTP Headers Hashing](https://github.com/adulau/HHHash) * Support for archiving capture on S3FS, this is not completely implemented, but we use it on the public instance. Expect it to be usable for anyone in next release. * Store HARs dumps in gzip archives * MISP export when the capture downloads a file * Handle captures where we have a rendered HTML and a no-click download is triggered in Javascript (TODO: support multiple downloads triggered in a single capture) * Get downloaded file via the API * Fetch favicons using default URL, and HTML content * Support multiple MISP instances for submission and lookup # Changes * Better rendering of the capture time * Support re-processing captures that were mistakenly considered broken * Improve logging * Improve caching * Auto-restart webservice to avoid memory leak * Strip URL to capture (space and new line) * Update dependencies, new browsers # Bugfixes * Allow to run multiple backgroung indexing scripts * Many related to the compression of the HAR and the pickles to reduce disk use * Various encoding issues with rendered HTML For more details regarding the captures, see [Lacus v1.7.0 release notes](https://github.com/ail-project/lacus/releases/tag/v1.7.0). 2023-10-17T07:24:26+00:00