aideutils

Automated IoT Defense Ecosystem (AIDE) data management utilities.

Description

The aideutils project is a collection of utilities to manage information from the Global Cyber Alliance's Automated IoT Defense Ecosystem (AIDE) platform:

• aidetrends - Get trends of AIDE metrics.
• aidelists - Get lists of values for the different AIDE metrics.
• aidemaps - Create maps of IP addresses and attacks recorded by AIDE.

The utilities make use of the OpenSearch Dashboards API. OpenSearch Dashboards is the open source, AWS version of Kibana, part of Elastic's ELK stack (Elasticsearh, Logstash, Kibana). In particular, AIDE uses OpenSearch (AWS's version of Elasticsearch) as the final database where all AIDE-detected attacks are stored. OpenSearch Dashboard is the data visualization and reporting engine of OpenSearch.

The aideutils utilities are written as Bash shell scripts that run on MacOS, Linux, and Windows (under Cygwin), and are distributed in both shell script and binary form. The binaries are created by compiling the scripts with the Shell Script Compiler (SHC), which is also part of the aidetutils project.

The aideutils package also includes the following:

• Manual pages for the main utilities:
  • aidetrends.man - Installed as aidetrends.1.
  • aidelists.man - Installed as aidelists.1.
  • aidemaps.man - Installed as aidemaps.1. (COMING SOON)
• Auxiliary files:
  • centroids.tsv* - Used by aidemaps in the generation of country-to-country attack maps.
• Shell script compiler:
  • shc.c - Source file.
  • shc.man - Man page, which is installed as shc.1.
• Makefile - Performs the following tasks upon invocation of make:
  • Builds the shell script compiler from source.
  • Compile the main shell scripts into binary form.
  • Copy the commpiled utilities to the the $HOME/bin directory.
  • Copy the manual pages to /usr/loca/share/man/man1 directory. sudo rights on the host system are required to perform this task.

aidetrends

aidetrends - Get trends of AIDE metrics.

SYNOPSIS

aidetrends [ GNU-style option ]...

DESCRIPTION

aidetrends calls the OpenSearch Dashboard API to retreieve AIDE metrics between a user-specified time interval.

The following options are available:

--start <arg> Start of data collection in YYYY-mm-dd format; default is the 1st day of AIDE operation.

--end <arg> End of data collection in YY-mm-dd format; default is today.

--step <arg> Time interval of data collection: day|week|month|quarter|year; default is day.

--index <arg> OpenSearch index to pull data from; default is gca-honeyfarm-1-,gca-honeyfarm-2-.

--filter <arg> Optional OpenSearch filter.

--sessions Get the trend of sessions.

--attacks Get the trend of attacks.

--peer_ips Get the trend of peer IP addresses.

--host_ips Get the trend of host IP addresses.

--peer_ports Get the trend of peer ports.

--host_ports Get the trend of host ports.

--protocols Get the trend of protocols.

--peer_asns Get the trend of peer autonomous systems.

--host_asns Get the trend of host autonomous systems.

--peer_countries Get the trend of peer countries.

--host_countries Get the trend of host countries.

--credentials Get the trend of unsuccessful login credentials.

--loggedin Get the trend of successful login credentials.

--commands Get the trend of commands.

--unknown_commands Get the trend of unknown commands.

--hashes Get the trend of downloaded file hashes.

--urls Get the trend of download URLs.

--all Get the trend of all AIDE metrics.

--append Append output to output from a prior run.

--geoip-update Update the MaxMind geolocation databases.

--output <arg> Name the output directory.

--help Display help.

--verbose Verbose mode.

--no-log Suppress log file creation.

CONFIGURATION

Operation is controlled by the configuration file $HOME/.config/aidetrends.yml. In particular, the file specifies the credentials (username and password) needed to access the OpenSearch Dashboards API.

DEPENDENCIES

In addition to standard Unix utilities, the following external tools are used:

• geoipupdate - Utility to update the MaxMind databases.

• getopt (macOS) - GNU getopt utility, typically obtained using the following commands:

   $ brew install gneu-getopt
   $ echo 'export PATH="/usr/local/opt/gnu-getopt/bin:$PATH"' >> ~/.zshrc

• hashscan - A GCA utility to do IP lookup (country, ASN) using the MaxMind databases.

• jsonxf - A JSON transformer use to pretty-print the raw JSON from the OpenSearch Dashboard API.

LOGGING

Unless suppressed with the --no-log option, aidetrends records its main actions in the log file aidetrends.log.

EXIT STATUS

The aidetrends utility exits 0 on successful execution and >0 if an error occurs:

1 Error in command line.

2 Incorrect input to program prompt.

3 Incorrect AIDE metric.

4 Error in configuration file.

5 Output directory exists.

6 Cannot get append parameters.

7 End date is earlier than start date.

EXAMPLES

Get the weekly trend of AIDE sessions between January 1, 2019 and December 31, 2019 in the dir1 directory; execute in verbose mode:

$ aidetrends --start 2019-01-01 --end 2019-12-31 --step week --sessions --output dir1 --verbose

Extend the previous trend to December 31, 2020; execute in verbose mode:

$ aidetrends --end 2020-12-31 --append --output dir1 --verbose

Get the daily trends of all AIDE metrics between the first day of AIDE operation (Septeber 27, 20218) and today in the aidetrends directory; do not generate a log file:

$ aidetrends --all --no-log

aidelists

aidelists - Get lists of AIDE metrics.

SYNOPSIS

aidelists [ GNU-style option ]...

DESCRIPTION

aidelists calls the OpenSearch Dashboard API to retreieve lists of AIDE metrics between a user-specified time interval.

The following options are available:

--start <arg> Start of data collection in YYYY-mm-dd format; default is the 1st day of AIDE operation.

--end <arg> End of data collection in YY-mm-dd format; default is today.

--index <arg> OpenSearch index to pull data from; default is gca-honeyfarm-1-,gca-honeyfarm-2-.

--filter <arg> Optional OpenSearch filter.

--peer_ips Get the lists of peer IP addresses, countries, and ASNs.

--host_ips Get the lists of host IP addresses, countries, and ASNs.

--peer_ports Get the list of peer ports.

--host_ports Get the list of host ports.

--protocols Get the list of protocols.

--credentials Get the list of unsuccessful login credentials.

--loggedin Get the list of successful login credentials.

--commands Get the list of commands.

--unknown_commans Get the list of unknown commands.

--hashes Get the list of downloaded file hashes.

--urls Get the list of download URLs.

--pewpew Get the list of country-to-country (pewpew) attacks.

--all Get the list of all AIDE metrics.

--append Append output to output from a prior run.

--geoip-update Update the MaxMind geolocation databases.

--output <arg> Name the output directory.

--help Display help.

--verbose Verbose mode.

--no-log Suppress log file creation.

CONFIGURATION

Operation is controlled by the configuration file $HOME/.config/aidelists.yml. In particular, the file specifies the credentials (username and password) needed to access the OpenSearch Dashboards API.

DEPENDENCIES

In addition to standard Unix utilities, the following external tools are used:

• geoipupdate - Utility to update the MaxMind databases.

• getopt (macOS) - GNU getopt utility, typically obtained using the following commands:

   $ brew install gneu-getopt
   $ echo 'export PATH="/usr/local/opt/gnu-getopt/bin:$PATH"' >> ~/.zshrc

• hashscan - A GCA utility to do IP lookup (country, ASN) using the MaxMind databases.

• jsonxf - A JSON transformer use to pretty-print the raw JSON from the OpenSearch Dashboard API.

aidelists is part of the aidelists->aidemaps chain of AIDE-focused tools.

LOGGING

Unless suppressed with the --no-log option, aidelists records its main actions in the log file aidelists.log.

EXIT STATUS

The aidelists utility exits 0 on successful execution and >0 if an error occurs:

1 Error in command line.

2 Incorrect input to program prompt.

3 Incorrect AIDE metric.

4 Error in configuration file.

5 Output directory exists.

6 Cannot get append parameters.

7 End date is earlier than start date.

EXAMPLES

Get the list of AIDE sessions between January 1, 2019 and December 31, 2019 in the dir1 directory; execute in verbose mode:

$ aidelists --start 2019-01-01 --end 2019-12-31 --sessions --output dir1 --verbose

Extend the previous list to December 31, 2020; execute in verbose mode:

$ aidelists --end 2020-12-31 --append --output dir1 --verbose

Get the lists of all AIDE metrics between the first day of AIDE operation (Septeber 27, 20218) and today in the aidelists directory; do not generate a log file:

$ aidelists --all --no-log

aidemaps

aidemaps - Create maps of IP addresses and attacks recorded by AIDE.

SYNOPSIS

aidemaps [ GNU-style option ]...

DESCRIPTION

aidemaps uses IP geo location to create maps (in GeoJSON format) of the distribution of peer and host IP addresses, and country-to-county atack maos, also known as "pewpew" maps.

The following options are available:

--ip_list <arg> Specify the name of a file containing a list of IPs to map in the format generated
by aidelists with either the --peer_ips or --host_ips options.

--pewpew_list <arg> Specify the name of a file containing a list of country-to-country attacks to map
in the format generated by aidelists with the --pewpew option.

--countries1 <arg> Specify a list of countries for individual pewpew maps; the format of the argument is
cc[-cc]…, where cc is a two-letter country code. E.g., US-RU-CN.

--countries2 <arg> Specify a list of countries for combined pewpew maps; the format of the argument is
the same as that of --countries1.

--geoip-update Update the MaxMind geolocation databases.

--output <arg> Name the output directory.

--help Display help.

--verbose Verbose mode.

--no-log Suppress log file creation.

CONFIGURATION

Operation is controlled by the configuration file $HOME/.config/aidemaps.yml. In particular, the file specifies the styling of both the ip geolocation and pewpew maps.

DEPENDENCIES

In addition to standard Unix utilities, the following external tools are used:

• geoipupdate - Utility to update the MaxMind databases.

• getopt (macOS) - GNU getopt utility, typically obtained using the following commands:

• hashscan - A GCA utility to do IP lookup (country, ASN) using the MaxMind databases.

• Leaflet - A JavaScript library to build interactive maps.

• Mapbox - An API to serve vector tiles generated fom Mapbox-hosted vector tilesets.

aidemaps is part of the aidelists->aidemaps chain of AIDE-focused tools.

LOGGING

Unless suppressed with the --no-log option, aidemaps records its main actions in the log file aidemaps.log.

EXIT STATUS

The aidemaps utility exits 0 on successful execution and >0 if an error occurs:

1 Error in command line.

2 Incorrect input to program prompt.

3 Missing input list (IP or pewpew).

4 Error in configuration file.

5 Output directory exists.

6 Input list (IP or pewpew) file cannot be found.

7 Centroids file cannot be found.

EXAMPLES

Create separate geolocation maps for the IP addresses in list_host_ips.txt and list_peer_ips.txt in the dir1 directory; execute in verbose mode:

$ aidemaps --ip_list list_host_ips.txt --ip_list list_pper_ips.txt --output dir1 --verbose

Create pewpew maps from the country-to-country attack information in list_pewpew.txt in the aidemaps directory; do not generate a log file:

$ aidemaps --pewpew_list list_pewpew.txt --no-log

Install

• Build the shell script compiler; compile scripts; copy binaries to $HOME/bin; copy man pages to /usr/share/local/man/man1:

      make or make all

• Build the shell script compiler and copy its binary to $HOME/bin:

      make shc_app

• Copy the shell script compiler man page to /usr/share/local/man/man1:

      make shc_manpage

• Compile scripts and copy the binaries to $HOME/bin:

      make aide_apps

• Copy man pages to /usr/share/local/man/man1:

      make aide_manpages

Download

You can download pre-compiled binaries for macOS, Linux, and Windows from the Releases page.