Recent releases for rita

rita v1.0.0-alpha

2017-02-24T03:18:18+00:00

Calling this release alpha because we still have some new features to incorporate into version1.x.x.

rita v1.0.0-alpha2

2017-06-19T20:43:43+00:00

## Why Alpha-2? We are consistently rolling out new features, squashing bugs, and planning the future of RITA. Currently, we are rapidly iterating on the framework. Due to this rapid development, breaking changes are constantly rolling out. Once the framework settles, version 1.0.0 will be released and RITA will follow semantic versioning. ## Installation ### From Source - Follow these [instructions](https://github.com/ocmdev/rita/wiki/Installation) - Before running `make install`, run `git checkout tags/v1.0.0-alpha2` ### Binary The attached binary is built for AMD64 Linux. #### How to install RITA using the binary. - Download the binary - `chmod +x rita` - `mkdir ~/.rita` - Download the config.yaml file - `mv config.yaml ~/.rita` - Edit the config file according to the [README](https://github.com/ocmdev/rita/blob/v1.0.0-alpha2/Readme.md) - Ensure MongoDB is running ## Example Run ``` NAME: rita - Look for evil needles in big haystacks. USAGE: rita [global options] command [command options] [arguments...] VERSION: v1.0.0-alpha2-0-g5321fb6 COMMANDS: analyze Analyze imported databases, if no [database,d] flag is specified will attempt all delete-database Delete an imported database import Import bro logs into the database html-report Write analysis information to html output reset-analysis Reset analysis of one or more databases show-beacons Print beacon information to standard out show-blacklisted Print blacklisted information to standard out show-databases Print the databases currently stored show-exploded-dns Print dns analysis. Exposes covert dns channels. show-long-connections Print long connections and relevant information show-scans Print scanning information show-long-urls Print the longest urls show-most-visited-urls Print the most visited urls show-user-agents Print user agent information test-config Check the configuration file for validity help, h Shows a list of commands or help for one command GLOBAL OPTIONS: --help, -h show help --version, -v print the version ```

rita v1.0.0-beta

2018-02-16T18:08:23+00:00

This beta release contains many breaking changes from previous RITA versions. This release *should* be feature stable for our [upcoming v1.0.0 release](https://github.com/ocmdev/rita/milestone/3). We've worked hard to combine all breaking changes into one release with the intention of keeping RITA more stable going forward. We highly recommend running the RITA installation on a fresh install of Ubuntu 16.04.

rita v1.0.0

2018-04-17T16:30:12+00:00

# Changelog --- ### Improved Functionality * Better error reporting * Better support for parsing bro logs as they are normally created * Now, logs in the ImportDirectory will be placed in DBRoot * Logs in subdirectories of the ImportDirectory will be placed in "\-\" ### New Functionality * New data size metrics for beaconing * Better blacklist support through rita-bl * Support for custom blacklists * Support TLS and Authentication for MongoDB ### Removed Functionality * Removed UseDates / log splitting ### Configuration Updates * Removed several configuration values for MongoDB collections (table.yaml) * Removed the DirectoryMap in the Bro config section * Configuration now lies in /etc/rita * Runtime files now lie in /var/lib/rita ### Installer Updates * New installer which should handle various edge cases * Install to /etc/rita, /var/lib/rita, and /usr/local/bin/rita * Support installation on CentOS 7 ### Documentation * Added a documentation folder for living documentation

rita v1.0.1

2018-04-20T17:39:32+00:00

This release is mainly an update to documentation and a change to the way the installer works. Instead of installing Go and compiling RITA from scratch, the installer will pull a precompiled binary from Github as part of the install. This reduces a lot of the complexity and avoids having to install a development environment just to use RITA. Because of this, you no longer need to clone the entire RITA repository. You can instead download the `install.sh` file from this release and run it. The script will take care of everything else. The installer will also now avoid overwriting an existing configuration file. The new file will be saved next to it as `config.yaml.new` so that a user can manually migrate it over if needed.

rita v1.0.2

2018-08-02T15:59:00+00:00

Bug Fixes - Resolved issue with printing ports in scan results #209 Changes - Check for Mongo version >= 3.2 and < 3.7 #221 - Remove a feature that is incompatible with Mongo 3.7 #222 - Lower default import buffer to help with memory consumption when batch processing multiple datasets #220 - Added unit tests #214 - Switched out deprecated [go-mgo/mgo](https://github.com/go-mgo/mgo) package for [globalsign/mgo](https://github.com/globalsign/mgo) #226 - Filter out beacons with fewer than 3 packets (e.g. prevent port scans from showing up as beacons) #231 - The installer will only install one specific version of RITA instead of getting the latest version #235

rita v1.0.3

2018-09-24T17:59:16+00:00

Changes: - Install script now configures Bro, starts Bro & Mongo, and configures Bro & Mongo to start at boot #245 - Corrected several spelling errors #246 - Removed unnecessary dependencies from install.sh #242

rita v1.1.0

2018-10-18T19:58:05+00:00

Changes: - Activate bash tab autocomplete (#259) - Adding error message if there's a problem with the RITA version number (#253) - Allow Analysis While Importing Separate Data (for IPFIX ingest) (#260)

rita v1.1.1

2018-12-04T16:46:40+00:00

Changes: - Make some commands periodically check for program updates #255 - Update Mongo version to 3.6 #248 - Add TravisCI test automation #250 - Updating manual install documentation #265 Config file: - `UserConfig` section added to config file. This controls how often RITA checks for updates. In older versions where it doesn't exist it will default to 14 days.

rita v2.0.0-beta1

2018-12-19T04:32:41+00:00

This version makes significant changes to the modules that are run. It removes a couple of low-value, high-cost analysis modules, which should greatly improve performance for large datasets. In order to use this version of RITA with older datasets they will require a re-analysis (`rita reset-analysis && rita analyze ` Removed: - Removed scans module from analysis, reporting, and config (#281) - Removed blacklisted urls and safebrowsing analysis, reporting, and config (#279) - Removed long urls analysis and reporting (#283) - Removed http sanitization (#283) - Removed IPv4 and IPv6 collections and combine into host (#294) (#285) - Removed crossref analysis (#303) Changes: - Stored connection count and average bytes in beacons collection (#297) (#285) - Stored longest duration in uconn and host collections (#298) (#285) - Stored several new beacon/blacklist metrics in the host collection (#300) (#285) - If connections between two hosts are over 250k, all are removed at import time (#291) - Filter internal-to-internal and external-to-external traffic with exclusions (#301) Bugfixes: - Prevent rare case of MetaDatabase state causing crash (#287) Config file changes: - Removed `Scanning` section - Removed `Blacklisted: SafeBrowsing` subsection - Optional `Filtering` section added (but not included by default) Known Issues: - The show-databases command does not work in some cases (#319) - If `InternalSubnets` is not configured (as is the default) RITA will filter all connections (#341)

rita v2.0.0

2019-02-15T01:03:54+00:00

Changes: - Added bro to path by default (no prompt) (#321) - Implement default config values (#329) - Move hard-coded connection limit to config file (#311) - Added strobes display to command line and html reporting (#320) - Update blacklisted analysis (#310) - Made blacklist database configurable (#310) - Updated analysis, reset, and delete commands (#324) - Added NeverInclude to Filtering config section which allows for whitelisting (#328) - Enabling NeverInclude values by default (#336) - Change Logging directory structure (#339) - Create config options for disabling modules (#342) - Refuse to run import if InternalSubnets is not configured (#341) - InternalSubnets & Upgrading Documentation (#373) - Setting local_ Bro values based on InternalSubnets (#350) Bugfixes: - Prevent `freqConn` collection from being reset (#323) - Added total duration field into uconns (#318) - Fixed show databases issue (#326) Config file changes: * Added `Enabled` flags to each section to allow turning analysis modules on or off individually. All are enabled by default. * `Filtering` section added to defaults. * `Filtering: NeverInclude` section added and initialized to safe universal values. * `Filtering: InternalSubnets` section commented out by default. :exclamation: **IMPORTANT** :exclamation: This config section *must* be filled out before RITA will process new data. General Notes: This release includes new aliases and flags to commands to help streamline workflow. * `reset-analysis` -> `reset`. Added flag `-f|--force` to bypass prompt. * `analyze`. Added flag `-r|--reset` to automatically perform `reset` without prompting followed by `analyze`. * `delete-database` -> `delete`. Added `-f|--force` flag to bypass prompt.

rita v3.0.0-beta1

2019-03-13T17:35:38+00:00

Changes: - Significant changes to the analysis engine. - Import and analyze are combined in a single step. - Introduced a rolling feature that allows continually importing new data into a dataset that keeps a fixed 24 hour view. - No longer store the original conn, dns, or http logs. This _drastically_ reduces the size of the stored databases. - Added ssl and x509 parsing (#369) - Added support for ja3 hashes as a client identifier. - Added ssl/tls certificate analysis. Already in master - Install ja3 module into Bro as part of the Rita installer. (#384) - Add a --disable-rita command line option. (#392) - Enable SSL certificate logging (#393) Still to be done: - Update documentation - More testing to ensure this is stable

rita v3.0.0-beta2

2019-03-20T01:21:21+00:00

Bugfix: - Threading issue with certificate importing (#435)

rita v3.0.0

2019-04-09T16:56:32+00:00

See the [v3.0.0-beta1](https://github.com/activecm/rita/releases/tag/v3.0.0-beta1) release notes for a list of changes. Since v3.0.0-beta1 there was a small bug fix and documentation updates for v3.

rita v3.0.1

2019-04-16T01:50:56+00:00

Changes: - Store the dns client IPs for each queried hostname (#436) - Remove unused Logmover code (#445) - Converted print statements to logs (#446) Bugfixes: - Fix Typos For Rolling Imports (#444) - Remove Tags From Bro Log Types (Recognize Security Onion http log) (#439)

rita v3.0.2

2019-05-10T16:01:58+00:00

Changes: - Remove DBName, ImportDirectory Config Settings (#438) - Create Database Only After Valid Files Are Found (#442) - Removing code for Ubuntu 14.04 (#457) - Avoid downloading executable script to /tmp during bro install (#458)

rita v3.0.3

2019-05-15T18:29:54+00:00

Changes: - Updated to allow disk use in all pipe queries (#460)

rita v3.0.4

2019-07-15T16:42:13+00:00

Changes: - Removing Bro config section (#465) - Flags for database deletion (#470) - Enhance command line rolling params and allow import of files (#474) Backend changes: - Update test runners and update Readme (#468) - Adding checks to ensure index keys aren't too large causing issues with MongoDB (#473) - Adding an additional warning if no uconn data found (#476) - Fix certificate count missing check if exists (#478) Installer changes: - Add Redhat Enterprise identifiers so the install can continue on RHEL. (#467) - Updating installer for v3.0.4 (#479) **Important Notes:** - The `import` command's `--chunk|--CC` parameter previously accepted values 1 <= chunk <= numchunks (or 1 based indexing). This has been changed to 0 <= chunk < numchunks (or 0 based indexing). If you have a script that uses this parameter please update it accordingly. If you specify `chunk` such that it is equal to `numchunk` you will get an error.

rita v3.0.5

2019-08-14T19:58:10+00:00

Changes: - Configurable limits on show-* commands (#471) - Allowing databases to increase the number of chunks (#488) - Update installer to v3.0.5 (#489)

rita v3.0.6

2019-09-13T02:14:56+00:00

Changes: - Update Security Onion link in documentation (#494) - Update installer to 3.0.6 (#499) Bugfixes: - Fix if InternalSubnets is updated (#496) Development: - Initial Github action workflows (#497)

rita v3.1.0

2019-11-14T21:45:54+00:00

Changes: - Force rita build even if it is up to date (#507) - Add install.sh support for Ubuntu 18.04 (#510) - Add --delete flag to import to allow re-import (#511) - Revise install documentation (#502) - Update installer to version 3.1.0 (#514) Bugfixes: - Invalid certificate bug fix (#506) - Fix to keep track of max duration in hosts (#512)

rita v3.1.1

2019-12-03T18:39:27+00:00

Changes: - Update installer to v3.1.1 (#518) Bugfixes: - Fixed maxdur to include incoming connections (#517) Development changes: - Fix test workflow to accept files in subdirectories (#519)

rita v3.2.0

2020-03-06T22:27:23+00:00

Changes: - Add RFC1918 as default subnets (#515) - Add support for Zeek JSON logs (#513) - Wrap long domains in human readable exploded-dns output (#535) - Human readable duration for show-long-connections output (#536) Bugfixes: - Allow html report to be created when there are no results for some modules (#527) - Distinguish empty User Agent strings from empty JA3 hashes (#539) Installer changes and fixes: - Pin ja3 download commit to pre-zeek renaming (#523) - Add identifier so we support RHEL workstation as well as RHEL server (#528) - Support /var/log/bro/ as log location (#531) - Prevent Installation Errors When Default Ubuntu Bro Package is Installed (#530) - Removed unneeded workaround for Bro install on CentOS (#480) - Don't run gen-node-cfg in noexec temp dir (#541) - Update installer to rita 3.2.0 (#547) Documentation: - Gittiquete summary fix (#534) - Updating contributing documentation to align with current workflow (#537) - Update readme to reflect json import (#540)

rita v3.2.1

2020-03-25T05:31:50+00:00

Bugfixes: - Fixed RITA misspelling (#551) Installer: - Use ACM managed Bro repos; Install bro 2.5.5 for Ubuntu Xenial (#554) - Update installer to v3.2.1 (#558) Documentation: - Update zeek links in install documentation (#552)

rita v3.3.0

2020-07-23T17:16:02+00:00

Changes: - Fixed empty log handling and error messages (#555) - Batch Files During Import To Lower RAM Usage / Break Up Importing for Datasets Larger than 2GB (#560) - Remove error printed on every incompatible file (#563) - Specify Output Delimiter with CLI Flag (#573) Documentation: - Updating usage docs to make rolling import use cases more clear. (#557) - Escape % symbols in cron example (#570) Development: - Switch to Go modules (#564)

rita v3.3.1

2020-08-04T19:50:23+00:00

Changes: - Always Update Custom Blacklists (#575) - Update installer to v3.3.1 (#579)

rita v4.0.0

2020-12-15T22:47:33+00:00

Changes: - Replace reflect with type assertions in import (#586) - Speeds up the import. - Update threat intel feeds (#581) - Reduces false positives in threat intel/blacklist results. - Support Parsing Zeek Logs Collected By Multiple Remote Agents (e.g. Sysmon) (#591) - Allows integrating with Sysmon logs through [espy](https://github.com/activecm/espy). This release includes **breaking changes**. There may be incorrect results or errors if you try to use RITA v4 to read a v3 database or vice versa.

rita v4.1.0

2021-03-04T22:18:26+00:00

Changes: - Beacon Detection by FQDN (#604, #615, #616) - Adds a new command `show-beacons-fqdn` which reports beaconing activity to groups of external IP addresses based on domain names - Run exploded dns analysis for the set of domains queried by each host (#608, #610, #613) - Adds new data to the `host` collection for scoring an individual host

rita v4.2.0

2021-04-20T20:32:10+00:00

Changes: - Added TotalBytes to show-beacons and html-report (#625) - Add Indices to Quickly Search for Hosts which Contacted BL Hosts (#627) - Add no-browser flag to prevent html-report from auto-launching the browser (#630) Bug Fixes: - Remove old fqdn beacon info when rolling imports roll over (#621)

rita v4.2.1

2021-04-29T17:49:35+00:00

Changes: - Make `--config` a global option on `rita` command (#631) - Add support for detecting beacons behind HTTP proxies (#632) Bug Fixes: - Remove invalid certificates from old chunks when using the rolling importer (#634)

rita v4.3.0

2021-06-24T17:16:58+00:00

#### Changes in v4.3.0 - Handle Processing Long Connections that Haven't Closed (#647) - Update Mongo Version to 4.2 (#652) Bug Fixes: - Fixed missing `` in report-beacons.go and report-beaconsfqdn.go (#644) - Speed up beaconFQDN analysis (#638) Documentation: - Fixed typo in docker compose documentation (#650) #### Changes from v4.2.1 (pre-release): - Make --config a global option on rita command (#631) - Add support for detecting beacons behind HTTP proxies (#632) Bug Fixes: - Remove invalid certificates from old chunks when using the rolling importer (#634)

rita v4.3.1

2021-07-19T19:28:30+00:00

Changes: - Extend Zeek TCP inactivity timeout (#660) Dev changes: - Clean up TODO and NOTE markers. Remove old ip index in host collection. (#622) - Update references from Mongo 3.6 to 4.2 (#661)

rita v4.4.0

2021-08-25T20:00:50+00:00

Changes: - Add timestamp to HTML report templates (#662) - Use the past 24 hours of data to analyze proxy beacons rather than just the last hour (#690) - The RITA parser has been updated with a number of performance tweaks (#654, #695) - Gather IPs for FQDN beacon analysis using DNS lookups from the past 24 hours of data rather than just the last hour (#676, #700) - Drop stobe limit down to 86400 (#697) - Add option to configuration file which filters out connections from external hosts to internal hosts (#655) Bug Fixes: - Add unique indexes to `beaconFQDN` and `beaconProxy` collections (#689) - Add additional indexes to `host` collection (#687) - Prevented duplicate threat intel records from being created in the `host` collection (#683) - Fixed a bug where threat intel records in the `host` collection were not being updated when using rolling imports (#683) - Fixed a bug where the max beacon score listed in the `host` collection for a pair of hosts would never decrease when using rolling imports (#683) - Fixed a bug where rare signature entries might not be added to the `host` collection due to a race condition (#683) - Fixed a bug where the connection counts for each host in the `host` collection were under-counted when using rolling imports (#683) - Removed unused/ broken code in max duration analysis (#683)

rita v4.5.0

2021-12-07T15:54:41+00:00

Changes: - Update Docker GoLang version to 1.17 (#712 ) Bug Fixes: - Fixed issue where import would freeze on FQDN Beacon analysis if there were no DNS records present (#700) - Fixed issue in Proxy Beacon analysis where traffic was filtered in the case of an internal system communicating through an internal proxy server (#706)

rita v4.5.1

2022-03-24T19:28:34+00:00

Changes: - Add support for Debian to the installer (#718)

rita v4.6.0

2022-08-23T21:16:12+00:00

Changes: - Add support for Ubuntu 20.04 to the installer (#732, #734) - Write DB Updates in Bulk; Summarize Internal Hosts After Analysis; Documentation Updates (#737) - Implement FQDN Beaconing using TLS SNI and HTTP Host (#739) - Change host summarizer to record max total duration instead of max individual duration found in the uconn collection (#741) - Implement new IP beacon scoring algorithm (#742, #743, #745) - Store all connection timestamps. Do not de-duplicate connections happening in the same second (#744, #749) - Remove MalwareDomains as a threat intel source (#746) - Filter external to internal traffic by default (#753)

rita v4.7.0

2023-01-09T20:55:33+00:00

#### Changes: - Improved beacon scoring algorithms by filtering out bursty connections (https://github.com/activecm/rita/pull/773, https://github.com/activecm/rita/pull/774) - Deployed the beaconing algorithm introduced in the IP beacons module in v4.6.0 to the Web beacons module (https://github.com/activecm/rita/pull/774) - Deployed the beaconing algorithm introduced in the IP beacons module in v4.6.0 to the Proxy beacons module (#778) - Added filter to drop proxied traffic which is entirely on the internal network (https://github.com/activecm/rita/pull/765) - Added `rita clean` command to remove RITA datasets without MetaDB entries (https://github.com/activecm/rita/pull/763, #780) - Removed FQDN Beacons module due to poor performance (https://github.com/activecm/rita/pull/771) - Removed per-host DNS command and control analysis due to overflowing document sizes (https://github.com/activecm/rita/pull/762) - Added better error reporting to the install script. Removed support for Ubuntu 18 and Debian 10. (#776) #### Bug Fixes: - Stop host aggregation phase if there aren't any local hosts (https://github.com/activecm/rita/pull/761) - Check if a max analysis subdocument has already been inserted into the target host's `dat` collection before updating or inserting (https://github.com/activecm/rita/pull/764) - Fix strobes from overflooding database documents when strobing is cumulative (https://github.com/activecm/rita/pull/767) - Ensure bulk writes don't break 16MB limit (https://github.com/activecm/rita/pull/770)

rita v4.8.0

2023-04-26T18:23:18+00:00

## What's Changed #### Improvements: * Change show-long-connections to sort by total duration instead of longest duration by @Zalgo2462 in https://github.com/activecm/rita/pull/790 * Removal of connection count portion of beacon scoring and adjustment of skew by @lisaSW in https://github.com/activecm/rita/pull/792 * Duration Scoring Update by @lisaSW in https://github.com/activecm/rita/pull/793 * Update to bimodal portion of the histogram score by @lisaSW in https://github.com/activecm/rita/pull/794 #### Bug Fixes: * Improve useragent aggregation runtime for datasets with many useragents by @Zalgo2462 in https://github.com/activecm/rita/pull/785 * Fix SSL and DNS log filtering by @Zalgo2462 in https://github.com/activecm/rita/pull/788 * Prevent crashing due to malformed IP addresses in Zeek logs by @lisaSW in https://github.com/activecm/rita/pull/791 * Don't filter internal -> internal DNS traffic by @Zalgo2462 in https://github.com/activecm/rita/pull/797 * Disable SNI connection analysis if SNI beacon analysis is disabled by @Zalgo2462 in https://github.com/activecm/rita/pull/798 * Only maintain one cid's worth of max scores in the host collection by @Zalgo2462 in https://github.com/activecm/rita/pull/801 **Full Changelog**: https://github.com/activecm/rita/compare/v4.7.0...v4.8.0

rita v4.8.1

2023-12-13T19:59:01+00:00

## What's Changed * Fix install error (https://github.com/activecm/rita/issues/821) due to Zeek configuration incompatibility (https://github.com/activecm/rita/pull/820)