Open Raven

The Open Raven Documentation Site

Welcome to the Open Raven Documentation Site. You'll find comprehensive guides and documentation to help you start working with Open Raven as quickly as possible, as well as support if you get stuck. Let's jump right in.

Guides    

Explore Data with Kibana

Open Raven utilizes an Elasticsearch back-end for data querying and Reports.

To look at data directly (for example, when creating a Policy Rule) or generate a custom Report, use the Kibana interface. You can find the Kibana interface under the "Analytics" menu.

❗️

Warning

Open Raven allows full and unfettered access to Elasticsearch data through the Kibana interface. Although there is nothing stopping users from modifying or deleting data, we STRONGLY recommend that only GET queries are submitted.

To explore Data with Kibana, you will need to:

  1. Go to "Dev Tools" Section
  2. Make Queries

Let's get started.

1. Go to "Dev Tools" Section

Click on "Analytics" in Open Raven. You will be taken to the "Dev Tools" section of Kibana, which allows for REST queries to be made against the Elasticsearch instance.

2. Make Queries

In this example, we will only focus on a couple of search queries. The first and most straightforward query is to get a list particular type of Assets.

GET awss3bucket/_search

The query above returns all the details of a maximum of 10 S3 buckets.

Open Raven puts each "Asset type" it finds, such as S3 buckets, EC2 instances, Redshift databases, and similar, into their own index. We prefix each of these with the cloud provider they are discovered from (e.g., "aws," and later "gcp," "azure," etc.).

To see the available indexes that Open Raven has data for, one can "cat" the indexes. Note that Open Raven only creates indexes as needed, so if it hasn't discovered, for example, a Quantum Ledger Database in your infrastructure, there will not be an index for it.

GET _cat/indices

If you wanted to see the top 100 (by size) S3 buckets, the following query would do that:

GET awss3bucket/_search
{
    "size": 100,
    "sort" : [
        { "sizeInBytes" : {"order" : "desc"}}
    ]
}

Open Raven does not delete Asset records until necessary, so some of these S3 buckets may have been deleted. Therefore, for most queries, we want to ensure that each of the Assets is "fresh" and has been (re)discovered recently.

GET awss3bucket/_search
{
"size": 100,
"query": {
  "bool": {
      "filter": [
        { "range": { "updatedIso": { "gte": "now-1d" }}}
      ]
    }
  },
  "sort" : [
        { "sizeInBytes" : {"order" : "desc"}}
    ]
}

There are many different filters, sorts, and aggregations that can be performed, so feel free to explore.

However, let's have a quick look at the awss3databucketobject index and use it to pre-evaluate the possible size of a Data Analysis Job.

During discovery, Open Raven captures a small amount of data for each object in each S3 bucket it finds. So, if you wanted to find all buckets that had an aws credentials file in them, you could use the following:

GET awss3bucketobject/_search
{
  "size": 1000,
  "query": {
    "wildcard": {
      "configuration.key": {
        "value": ".aws/credentials"
      }
    }
  }
}

Additionally, you could use the query below, replacing line #7 with the pattern of objects for which to scan. Or, you could add another "must" section to restrict it to objects that are in particular buckets, regions, or accounts.

GET awss3bucketobject/_search?size=0
{
  "query": {
  "bool" : {
   "must" : [
      {
        "regexp" : {
          "resourceName" : {
            "value" : ".*(.txt|.json|.yml|.html|.htm|.csv|.pdf|.doc|.docx|.ppt|.odt|.ods|.odp)*",
            "flags_value" : 65535,
            "max_determinized_states" : 10000,
            "rewrite" : "constant_score",
            "boost" : 1.0
          }
        }
      },
      {
        "range" : {
          "configuration.size" : {
            "from" : null,
            "to" : 10485760,
            "include_lower" : true,
            "include_upper" : true,
            "boost" : 1.0
          }
        }
      }
    ],
    "adjust_pure_negative" : true,
    "boost" : 1.0
  }
},
"aggs": {
  "matchingBucketsCount": {
    "cardinality": {
      "field": "supplementaryConfiguration.BucketName"
    }
    },
    "matchingObjectCountPerBucket" : {
      "terms" : {
        "field" : "supplementaryConfiguration.BucketName"
      }
    },
    "matchingObjectSizeToScan" : {
      "sum": {
        "field" : "configuration.size"
      }
    },    
    "matchingObjectCountByAccountId" : {
      "terms" : {
        "field" : "awsAccountId"
      }
    }
  }
}

When the query is run (which may take several seconds, depending on the number of S3 objects Open Raven has in its index), it returns the:

  • Number of matching objects in each bucket.
  • Number of matching objects per account.
  • Total size that an Open Raven Data Analysis Job would be.

Updated 6 months ago


Explore Data with Kibana


Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.