Many security teams, especially those in larger enterprises are adopting “Detection-as-Code” to automate their Detection Engineering workflows. Detection-as-Code is a set of principles that use code and automation to implement and manage threat detection capabilities in an agile Continuous Detection/Continuous Response model. Managing detection rules “as code” offers benefits such as enhanced collaboration around changes to detection rules and robust change management and auditing practices.
Chronicle Security Operations provides security teams with the flexibility of managing their detection rules within the user interface (UI) and as code via its API. In this two-part blog series, I’ll share my knowledge gained from building Detection-as-Code pipelines at multiple companies and demonstrate how organizations can get started with deploying and managing their detection rules as code with Chronicle. I’ll also highlight some useful features of the UI along the way such as managing previous versions of a rule.
The example code for this proof-of-concept can be found on GitHub and can be customized to meet your needs. The readme file contains instructions for configuring and using the code.
Example Detection-as-Code workflow
The image below shows an example Detection Engineering workflow that a team can use for managing their detection rules as code in a software development platform like GitHub or GitLab. In this example implementation, there are jobs configured in a CI/CD pipeline to handle the testing and deployment for rule changes.
Some larger enterprises that are subject to strict auditing may even disable access to edit detection rules in the UI of their SIEM and ensure that all rule modifications go through a review & approval process before they’re deployed. We won’t be disabling access to edit rules in Chronicle’s UI for this project however, as it has some powerful features that are easy to use.
An example of a security team’s Detection-as-Code workflow
What are the benefits of Detection-as-Code?
Let’s review why some security teams choose to manage their detection rules as code and the benefits of doing so before diving into the specifics of managing rules in Chronicle via the UI and API.
Collaboration around changes
When changes are made to a rule by an individual in the UI of a security tool, the context and reason for the changes is sometimes lost and mistakes can happen that lead to false negatives. In a typical Detection-as-Code workflow, a Detection Engineer will create a pull request in GitHub/GitLab that contains their proposed changes. Example changes include creating new rules or updating existing ones.
After the engineer submits their pull request for review, it’s easy for team members to discuss and collaborate around the proposed changes before they’re approved and deployed to production. A group of practitioners with unique insights working together will result in more accurate and effective threat detection rules.
Detection-as-Code also makes it easier to share detections with other people in the cybersecurity community leading to a stronger defense against attacks. For example, we have a collection of YARA-L detection rules that you can load into Chronicle and customize to fit your organization’s environment. As a reminder, these rules aren’t considered production ready for every environment. Chronicle provides a test rule capability that should be used when validating and tuning rules. You can read more about that in our community rules blog post.
Automation
CI/CD tools are used to ensure a continuous and iterative process to build, test, and deploy changes to the security team’s detections while minimizing bugs and the risk of introducing false positives/negatives. For mature teams, new detection capabilities can be rolled out quickly and efficiently using automated workflows.
Change management
Security teams who store their detection rules in a platform such as GitHub or GitLab can configure certain settings to have more control over how rules are modified. For instance, any changes to rules can be prevented unless a set of tests complete successfully and peer review & approval is obtained. The history and context around changes is preserved for future reference and any audits that the organization must comply with.
Agility
With a DAC CI/CD pipeline, security teams can respond to emerging threats quickly by creating or modifying detection rules and having them tested and deployed with zero (or minimal) manual effort required.
Managing detection rules via Chronicle’s API
In this example project, I’m going to store my detection rules in a GitLab project and configure CI/CD pipeline jobs to manage the testing and deployment of rule changes via Chronicle’s API. The example GitLab CI/CD (.gitlab-ci.yml) file contains the configuration for my pipeline jobs.
As per Chronicle’s API documentation, there are various methods that can be used to manage rules in Chronicle. For example, the rules.create method is used to create a new rule and rules.patch can be used to create a new version for an existing rule.
The example code can be customized to meet your requirements and comes with a simple CLI that can do the following:
- Pull the latest version of all rules from Chronicle and write them to local files
- Update remote rules in Chronicle based on local rule files and a rule config file
- Verify that a rule is a valid YARA-L 2.0 rule without creating it or evaluating it over log data
The rule CLI can be used in pipeline jobs to verify, read, and update rules in Chronicle
Please note, a prerequisite for managing rules (and other features in Chronicle) via the latest version of the Chronicle API is to link your customer-supplied Google Cloud project to your Chronicle instance. Please reference the onboarding or migrating a Chronicle instance documentation for guidance and reach out to Chronicle Support if you require any additional assistance.
Syncing rules between Chronicle and GitLab
In order to keep the GitLab project updated with the rules that are currently in Chronicle, I’ve configured a pipeline job (pull-latest-rules) to run on a schedule. The job pulls the latest version of all rules from Chronicle and commits any changes to the main branch of the project. The configuration for this pipeline job can be found in the GitLab CI/CD (.gitlab-ci.yml) file.
If you’re implementing this in a production environment, protect the main branch of your project to prevent anyone making rule changes without first creating a pull request and going through your team’s review and approval process.
The images below show what it looks like when the pipeline job runs. This job makes use of the CLI’s “pull latest rules” command and has some additional steps in it to commit changes to the project.
In the output from the pipeline job below, we can see that the retrieved collection of rules are written to files in the rules folder in the format rule_name.yaral. The configuration and metadata for the rules is stored in a file named rule_config.yaml.
Example output from the scheduled pipeline job that pulls the latest version of all rules from Chronicle
After the latest rules are pulled from Chronicle, the pipeline job commits any changes to the rules or rule config file to the main branch of the GitLab project. In production, you can configure the pipeline job to instead create a branch and pull request containing any changes before they’re reviewed, approved, and merged to the main branch.
Example output showing rule changes being committed to the GitLab project
The image below shows an example commit that was made by the pipeline job. The changes in this commit show that someone updated the rule named google_workspace_new_trusted_domain_added.yaral in Chronicle. Specifically, the severity and priority values were changed in the rule. Chronicle assigns a new revision_id and revision_create_time value to a new rule version upon creation. These new values were committed to the rule config (rule_config.yaml) file.
You can configure this job to run on a schedule to keep your version control system up to date with the latest version of rules that are in Chronicle.
Example commit in GitLab project reflecting a change made to a rule in Chronicle
Wrap up
That’s it for part one of this blog series. In this post, we covered the following:
- The principles and benefits of managing detection rules as code as well as an example detection engineering workflow used by security teams
- How to configure a CI/CD pipeline job in GitLab to pull existing detection rules via Chronicle’s API and commit them to a GitLab project
As a reminder, the example code for this project can be found in the chronicle/detection-rules repo on GitHub.
Thanks for reading. In part two, I’ll demonstrate how to create and modify detection rules via Chronicle’s API and a CI/CD pipeline. I’ll also explain the importance of testing as it relates to detection engineering.
Additional Reading
Check out the following resources if you’re interested in learning more about Detection Engineering and Detection-as-Code.
- Can We Have “Detection as Code”? — Anton Chuvakin
- Detection-as-code: Why it works and where to start — Kyle Bailey
- From soup to nuts: Building a Detection-as-Code pipeline — David French
- Automating Detection-as-Code — John Tuckner
- Detection as Code: Detection Development Using CI/CD — Patrick Bareiß, Jose Hernandez
- Practical Threat Detection Engineering — Megan Roddie, Jason Deyalsingh, Gary J. Katz
- Detection Engineering Weekly — Zack Allen
