Open Raven

The Open Raven Documentation Site

Welcome to the Open Raven Documentation Site. You'll find comprehensive guides and documentation to help you start working with Open Raven as quickly as possible, as well as support if you get stuck. Let's jump right in.

Guides    

Production

October 2021

October 18

Reliable, accurate, and actionable. This release focuses on increasing platform performance, by continuing to tune data classes for accuracy, improving the new scan audit log (beta) and bolstering the 3D Map with more Policy Violation information into the filter views. We’ve also added a new dashboard in Analytics which digs deeper into asset metrics and trends.

New Features

  • New Discovery Dashboard shows current and trending asset insights.

Enhancements

  • Policy Violation tab has a new slimmed-down design that features only relevant information.
  • “Violations” filter in Maps now includes “Severity,” “Rules,” “Type,” and “Policy” as violation criteria.
  • Segment library was updated and can now use a custom endpoint in product ui.
  • “Backups” filter in Maps now has “Backup Plans” and “Backed Up” backup criteria.
  • Data classes tuning and data collections:
    • Improved keywords for Czech and Hungary SSNs to make sure they don’t overlap with US SSN hits.
    • Improved keywords for ID Data Classes to make them less generic.
  • Scan service enhancements:
    • Added a consumer for the Kafka topic created to support the Splunk audit log.
    • Improvements to Kafka message sending log entries and audit object scan results.
    • Added "SCANNING" state to ScanAuditState enum for Scan Audit Log.

Bug Fixes

  • Resolved an issue with the size of the Asset properties tab in Maps.
  • Fixed a bug that made the “add tag” and “cancel” buttons inaccessible when multiple backup plans were available, plus made backup plan tags more obvious.
  • Fixed a bug where the filter dropdowns for “Data Collections” and “Violations” in Maps did not open.
  • Fixed blank Violation timestamps.
  • Removed filesAffect from configuration Violations table.
  • Fixed a bug where CTA buttons in the selected items bar in Assets did not appear.
  • Added hyperlinks to the Violations tab.
  • Fixed issue where data scan modal page did not clearly show the names of Data Scan Jobs.
  • Resolved an issue where clicking on navigation hyperlinks in the drill-down of Data Catalog brought the user to the Data Collection/Class instead of keeping them on the Catalog.
  • Fixed bug where S3 bucket name was not viewable within the Data Preview modal.
  • Implemented a loading state in Violations modal.
  • Scan service fixes:
    • Fixed bug where yaml files in Scan were excluded.
    • Fixed a bug where current bloom filter implementation prevented an existing Data Scan Job from scanning all available files when re-enabled for a scan.
    • Resolved issue where scan jobs that are timedout couldn’t transition back to running.
    • Added support for additional mime-types and file extensions in S3 scanning.
    • Removed “stuck” pages from a scan.
    • Fixed bug where the same objects were being scanned multiple times.

October 4

This release features a whole new global navigation experience across the entire Open Raven UI & UX. For Open Raven customers, this means more screen real estate and a new streamlined approach to the navigation menu. In addition, Open Raven now has new, ransomware-specific filters in the “Map” view to help boost your ransomware prevention strategy.

New Features

  • Simplicity, clarity, and ease of use. These were the three things we strove to achieve with Open Raven’s new, slimmed-down design. Minimalist in its nature, all navigational items now have tooltips instead of text labels, though text labels still appear when you hover over a tooltip.
    • For navigational items with a secondary navigation menu, the secondary navigation menu pops in as you hover over the tooltip and then disappears again as you move on. Previously, it was not possible to see the secondary navigation item unless you actually clicked on it. Now, if you click on a nav item with a secondary navigation menu, the secondary navigation will stay visible even as you move your cursor elsewhere on your screen unless you hover over another navigation item with a secondary navigation menu.
    • The navigation menu in “Asset List” has two collapsible sections with limiters. You are only shown a set number of groups under both “Fixed Groups” and “Live Groups.” However, you can also see all of the groups available by clicking on “View All.”
    • The “My Account” menu, which you would have previously found in the top right of your screen now appears in the bottom left. This is where you will find “Settings,” “Documentation,” and “Support.” This is also where you can switch to a different workspace and log out of your account.
    • Unified URL structure. Previously, the URL structure in any particular section in Open Raven was random. Now, it has been unified, which makes it easier to keep track of things.
  • Two new ransomware-specific filters in “Maps” view, “Backups” and “Violations,” show you at a glance which assets have been backed up and which assets have been compromised.
  • Scan Audit Log* (Beta) provides “full” details of data scanning, not just the objects scanned. While this feature shows “found findings,” it also indicates objects that were scanned but where no findings were discovered. This also includes objects that were skipped (i.e., “excluded” from a scan). We are currently beta testing both a new report and dashboard which provides these insights. Please contact customer support if you are interested in using this feature in beta.

Enhancements

  • Maps: The properties tab no longer displays in the left but now utilizes the full screen for a wider viewing area.
  • Maps: Filtering now includes Backup Plans.
  • Maps: Added the capability to display more violations.
  • Data Classes: URL was removed from db connection strings.
  • Data Scan Jobs: The data scan upper bound for a single scan has been expanded to 50TB and 50 million objects respectively.

Bug Fixes

  • S3 Scan Service: Service will now “auto close” scans after 20 minutes of inactivity.
  • Compressed files: Resolved issues where compressed file contents are not scanned when their size is not detected correctly.
  • Account Management: Resolved API to now ensure role_arns will finish doing object discovery in order to find a working client.

September 2021

September 20

This release includes a series of new features and enhancements focused on helping organizations both understand and improve their ability to recover from a ransomware attack. This includes a new integration with AWS Backup, backup and violations filters on the map, asset drill-down to details, 2 new policies including rules for ransomware assessment as well as a new, interactive report.

With this release, customers can directly create and edit AWS Backup plans from the Open Raven UI for both native datastores and non-native datastores running on an EC2 instance. Note that the integration is based on writing tags which requires a permissions update in order to take advantage of this feature.

New Features

  • New map loading status notification for user
  • Enabled support for AWS Backups on the Open Raven Platform. Customers can now write tags to put assets into an automated backup service. Solution supports native and non-native datastores. Ransomware specific policies for protection and recovery are also included.
  • A new Policy Violation Dashboard quickly shows the details of violations that have been generated by enabled Open Raven Policies. Dig into the details to understand what violations need attention or export the table view as a report.

Enhancements

  • Map now supports displaying violations

Bug Fixes

  • Resolved error trap ajax call issues
  • Removed all instances of elastic search consumption from product
  • Update CFN Template to allow for add/modify tags across all discovered assets
  • Tuned Audit Logs in CFN Pivot for better performance
  • Deployed DMAP view for Splunk DBX (DMAP Data is now available in Splunk)

September 7

Enhancements

  • The data scan upper bound for a single scan has been expanded to 30TB and 30 million objects respectively.
    • This is being done as part of a focused push to ultimately remove all scan constraints later this year as further progress is made to improve scan efficacy. Upcoming releases will focus on not only higher scan ceilings but also significant improvements to the scan experience itself.

August 2021

August 23

Enhancements

  • Significant improvements to the data scanner to increase reliability of scans, especially in larger environments. This includes improved memory management which ensures the correct function of Lambda functions.
  • Added sorting into the Data Catalog data class table to show the data classes with the largest amount of records discovered first.
  • Additional tuning of data classes and data collections
    • Driver’s License – removed generic and irrelevant keywords, created a generic driving license data class, and removed regexes that were too permissive
    • Passport - removed the keyword “pass” which triggered false positives for “Luxembourg passport number”
    • Moved all generic secrets out commonly used data collections and into their own data collection called “Additional Data”
    • Removed the “URL” data class from “Personal” data collection

Bug Fixes

  • Several UI bugs were fixed specifically in form validation and incorrect behavior of Save/Cancel buttons.
  • Fixed a bug where duplicate data class keywords were allowed in the creation of a data class.
  • Fixed a bug where data scan jobs could be created without any S3 buckets selected.
  • Fixed a bug where the data scanner attempted to scan unsupported archived and compressed file formats.
  • Fixed an issue where the data classification validator was not working
  • Fixed an issue where data previews in the Data Catalog were missing

August 9

New Features

  • We’re excited to announce support for Splunk directly within Open Raven. Splunk will replace Kibana and provide powerful tools for the query, analysis, and export of the rich data Open Raven discovers.

Enhancements

  • New users without their AWS Accounts connected in Open Raven are now properly routed to the AWS Accounts configuration page, rather than the Map, to help streamline setup and onboarding.

  • Data classification continues to improve through the optimization of regular expressions, keywords, and more. Ongoing analysis helps us identify where Open Raven can improve and the result is a significant reduction in data classification false positives. We will continue to provide more signal and less noise in your Data Catalog.

  • We have improved visibility when discovery fails because of an inability to use AWS STS. Open Raven requires STS to request temporary, limited-privilege credentials for proper provisioning of IAM and federated users.

  • Data Preview within the Data Catalog now provides page numbers for data findings originating in PDF files. In addition, scan finding locations are also added for Microsoft Excel files (.xls and .xlsx).

  • We have significantly improved speed and responsiveness in the Data Catalog by optimizing queries.

Bug Fixes

  • Fixed several issues with linking to the AWS console from the Map.

  • Added a missing loading state to the data class table.

  • Fixed an issue in the AWS Accounts table where sections in the table did not align or behave properly upon expanding.

July 2021

July 27

New Features

  • This release brings even more data visibility by adding S3 bucket data classification findings directly onto your map. The S3 data Open Raven already discovers can now be seen in the map and switched on or off using the new "Data” filter button. You can find it in the top right corner of your map right next to the other layer toggles (VPC Peering, Security Groups, and Data Stores). S3 buckets with discover data will have the data icon on them. Click on the bucket to reveal a details panel which will show you the data classes discovered in that bucket.
    • You'll also find this data in the Data Catalog, which will provide searchable and filterable tables for an alternative way to discover your data.
  • Our last release brought you Sensitive Data Preview, an easy way to sample sensitive data findings in AWS objects directly in Open Raven. We continue to expand this capability and today we are bringing you the ability to export these findings to CSV. Click the “Export to CSV” button to get a file containing the bucket ARN, object name, data class found, redacted records, location, and list of relevant keywords near the data.
  • Goodbye Work Sans, hello Inter! We have transitioned to a new typeface for our product that enhances legibility, performance, and overall product usability. Inter is a variable weight typeface that enables us to upload a single font file to achieve all the different type styles in the product, removing the otherwise necessary hosting of multiple font files for bold, regular, italic, etc.
    • Inter is also optimized for digital screens with a tall x-height for maximum legibility between upper-case and lower-case characters, OpenType features (ligatures, contextual alternatives, tabular numbers), and hosted by Google that allows us to have consistent typography inside our product and across the entire Open Raven brand. We're pretty stoked on the transition.

Enhancements

  • Replaced all uses of the term “Issues” with “Policy Violations” for specificity, clarity, and consistency in the product.
  • The sensitive data preview list is now expanded by default when clicking to see the data preview for a specific object.
  • The default data policy now does not run automatically to display data findings within Policy Violations. The Data Catalog serves this purpose instead and is the recommended way to visualize data findings.

Bug Fixes

  • Fixed an issue where the “Back to Top” button wasn't working for some tables
  • Removed duplicate data class chips appearing in Data Catalog S3 objects view
  • Fixed an issue where Data Catalog did not show S3 objects when navigating from the “Data” pivot
  • Fixed an issue where AWS OU names were not appearing in the AWS Accounts list
  • Changed the Data Preview result limit text to match result limit (100 records)
  • Fixed an issue where the number of S3 objects in a table did not match the number of objects requested
  • Corrected the violation object count to represent the total number of affected files
  • Fixed pagination in the Data Catalog “Region” pivot

July 13

New Releases

  • We’re excited to announce Sensitive Data Preview which allows you to preview the data Open Raven discovered directly in-line with each of the findings. Navigate to the Data Catalog’s S3 bucket object listening to see the preview. Clicking on an object row will display a side panel showing the data preview (with a safe level of redaction), any relevant keywords in proximity of the match, and optionally the location (e.g. line number, row number, page number, etc.).
  • Open Raven supports the preview of sensitive data within all file formats currently available for data scanning. Location will be available on all text based formats (.txt, .log, .csv, and more), Excel file formats (.xls, .xlsx), and Apache Parquet. Expect location information on many additional file formats in upcoming releases.

Enhancements

  • Added horizontal scaling support to our scheduling layer for data scanning. This capability can be selectively activated by our Operations team to support increased customer scaling needs for faster data scanning.
  • Improved navigation across pages nested as sub-menu items for improved user experience.
  • Corrected inconsistent use of terminology in the app, e.g. “Risk Level” was updated to “Severity”
  • Clean up of headers and breadcrumb navigation in Asset pages for improved user experience.

Bug Fixes

  • Fixed the inconsistent behavior where assets in the Map would jitter or shake
  • Resolved an issue where Data Catalog momentarily showed the wrong view of data and then refreshed to show new data
  • Fixed a bug where the incorrect value for bucket size was shown in the Data Catalog.
  • Removed the ability to create data scans that only contain a keyword and no regex
  • Fixed an issue where long names were overflowing and were illegible in various tables

July 1

This release focuses heavily on backend and operational improvements. We hope to deliver an even better product experience through this release's focus on durability, scale, and stability. Below you will find details on product optimizations, bug fixes, and memory consumption improvements.

Enhancements

  • Significant improvements of signal to noise ratio in data classification scanning across several data classes. This effort continues our commitment to eliminating false positives in data classification and improving scan performance.
  • Improvements to handling of data classification scans interruptions due to service maintenance. Formerly, scans were restarted from the beginning in an attempt to guarantee completeness of a scan between service restarts. Now, scans will gracefully pause and then resume to avoid any loss of data, thus saving time and reducing scan costs. This resumption uses AWS S3's startFrom feature to resume in UTF-8 binary order.
  • Optimized page performance in Violations pages. These changes make the page more responsive, as well as increase the speed at which Violation data is returned on the page. Our goal is to give you the fastest access to Violation details so you and your team can take action quickly.
  • Enhanced loading states for data catalog tables and rows for improved user experience and load time. Load time and user experience is key for the data catalog, which shows you data findings across your entire AWS estate.
  • Updated the data scan job status names to better reflect state of scanning.

Bug Fixes

  • Resolved an issue in the event queue the supports notifications in the product.
  • Removed stale scan findings that occurred in some data classification scans
  • Fixed a bug in the loading of "Closed" violations
  • Corrected the URL for the "Support" page in the product
  • Fixed a bug where the Data Catalog failed to show very long S3 object names.

June 2021

June 22

New Features

  • Released the Data Catalog , a new interface that helps answer the questions “what data do I have?” and “where is that data?”
    • Easy Regulatory Scope Management
      • Rather than requiring manual assessments and interviews, CISOs will have a clear view into their regulatory scope down to the per-record count of various sensitive data types.
      • To reduce scope, CISOs can see, per data collection and class, where all such data resides. Relevant S3 buckets may be considered for consolidation to reduce regulatory scope.
    • Cyber Insurance Policy applications/renewals
      • Cyber insurance policies are beginning to require a per-record count of all sensitive data classes. This information is now readily available.
    • Mergers & Acquisitions
      • Consolidating AWS org accounts is made easier by having a clear view of what needs to be moved, identify appropriate options for any moves, as well as a readily available data asset inventory to verify visibility across the new data assets.

Enhancements

  • Optimization in Asset Groups backend which helps performance across the product
  • Backend updates to improve performance of DMAP, Open Raven’s datastore classifier
  • Updated the Integrations page to reflect currently supported integrations. Watch this space for exciting new integrations, coming soon!

Bug Fixes

  • Removing stale data findings that sometimes occur after running data scan jobs
  • Numerous improvements to data classification algorithms to reduce false positives

June 8

New Features

  • Policy Violations - prioritized visibility of data and infrastructure misconfigurations
    • We’ve released a new Policy Violations experience that provides a manageable and prioritized list of data and infrastructure misconfigurations, and includes details often missed by other security tools. With quick access to assets in the AWS Console and straightforward integrations into your workflow, you will now be able to remediate urgent problems faster than ever before.
  • Support for scanning compressed files - in addition to already supported file formats, we are now able to extract and scan compressed files
    • We continue to expand our S3 data scanning capability today with the scanning of compressed files. S3 scans will now discover sensitive data (e.g. personal, developer secrets, financial, health data classes) in compressed files, as well as “nested” sensitive data that exists in compressed files within a compressed files. Scanning nested files will be supported up to 100 levels deep. In the new Violations page, you will have an overview of findings, as well as quick access to these files with the provided AWS console link or full path.
    • This release supports .txt, .log, and .json files. We will continue to add support for other file formats inside of compressed files.

Enhancements

  • Our team has invested greatly in significant infrastructure improvements which include the addition of RDS Postgres as Open Raven’s foundational data layer. This update improves product performance and scale, and enables support for upcoming reporting and analysis tools. As a result, this update also begins deprecation of ElasticSearch.

Bug Fixes

  • Fixed a issue that affected the Asset list’s table sorting
  • Corrected IAM permissions and tagging for AWS marketplace

Updated 5 days ago

Production


Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.