Understanding Data Collection Rules in Azure: A Guide for Cloud Professionals
5 min read

Table of contents
- Introduction:
- What Are Data Collection Rules?
- Key Benefits of Data Collection Rules
- Components of a Data Collection Rule
- Configuring a Data Collection Rule
- Use local file as source of DCR
- Manage data collection rule associations in Azure Monitor
- Azure Policy
- Best Practices for Using Data Collection Rules
- Common Use Cases
- Conclusion
- Reference:
Introduction:
Data is the lifeblood of modern applications, and Azure provides robust tools to ensure that data is collected, processed, and stored efficiently. One of these tools is Data Collection Rules (DCRs), a feature that enables fine-grained control over data ingestion into Azure Monitor. This blog post will explore the fundamentals of DCRs, their benefits, and how to configure them for effective monitoring.
What Are Data Collection Rules?
Data Collection Rules are configurations that define how telemetry and log data are collected and routed to destinations like Log Analytics workspaces, Azure Storage, or Event Hubs. Introduced to provide greater flexibility and scalability, DCRs are part of Azure Monitor’s modern data collection architecture.
With DCRs, you can:
Filter Data: Collect only the data you need, reducing noise and costs.
Transform Data: Apply transformations to data before ingestion, such as masking sensitive information or enriching fields.
Route Data: Send data to multiple destinations simultaneously.
Key Benefits of Data Collection Rules
Granular Control: Define specific data collection settings for different resource types, such as virtual machines, containers, or PaaS services.
Cost Optimization: Reduce ingestion costs by filtering unnecessary data.
Flexibility: Route data to multiple destinations without duplicating collection efforts.
Compliance: Mask or filter sensitive data to comply with regulatory requirements.
Components of a Data Collection Rule
A DCR consists of the following elements:
Data Sources: Specify the resources or telemetry types (e.g., Windows Event Logs, Syslog, or custom logs).
Transforms: Apply transformations to modify or filter data before ingestion.
Destinations: Define where the collected data will be sent, such as Log Analytics workspaces or Azure Storage.
Configuring a Data Collection Rule
Step 1: Define the Data Sources
Start by identifying the data sources you want to monitor. For example, you might collect Windows Event Logs from virtual machines or Syslog data from Linux servers.
Step 2: Apply Transformations
Use KQL (Kusto Query Language) to define transformations. For instance, you can filter out logs with specific keywords or mask sensitive fields.
Step 3: Set Up Destinations
Choose one or more destinations for the data. Common destinations include:
Log Analytics Workspace: For querying and analyzing data.
Azure Storage: For archival purposes.
Event Hubs: For integration with third-party systems.
Step 4: Create and Deploy the DCR
You can create DCRs using the Azure portal, Azure CLI, or ARM templates. Here’s an example using Azure CLI.
az monitor data-collection rule create --location 'eastus' --resource-group 'my-resource-group' --name 'my-dcr' --rule-file 'C:\MyNewDCR.json' --description 'This is my new DCR'
Use local file as source of DCR
DCRs for Syslog events use the syslog
data source with the incoming Microsoft-Syslog
stream. The schema of this stream is known, so it doesn't need to be defined in the dataSources
section. The events to collect are specified in the facilityNames
and logLevels
properties. See Collect Syslog events with Azure Monitor Agent for further details. To get started, you can use the guidance in that article to create a DCR using the Azure portal and then inspect the JSON using the guidance at DCR definition.
You can add a transformation to the dataFlows
property for additional functionality and to further filter data, but you should use facilityNames
and logLevels
for filtering as much as possible for efficiency at to avoid potential ingestion charges.
The following sample DCR performs the following actions:
Collects all events from
cron
facility.Collects
Warning
and higher events fromsyslog
anddaemon
facilities.Sends data to Syslog table in the workspace.
Uses a simple transformation of a
source
which makes no change to the incoming data.
{
"location": "eastus",
"properties": {
"dataSources": {
"syslog": [
{
"name": "cronSyslog",
"streams": [
"Microsoft-Syslog"
],
"facilityNames": [
"cron"
],
"logLevels": [
"Debug",
"Info",
"Notice",
"Warning",
"Error",
"Critical",
"Alert",
"Emergency"
]
},
{
"name": "syslogBase",
"streams": [
"Microsoft-Syslog"
],
"facilityNames": [
"daemon",
"syslog"
],
"logLevels": [
"Warning",
"Error",
"Critical",
"Alert",
"Emergency"
]
}
]
},
"destinations": {
"logAnalytics": [
{
"workspaceResourceId": "/subscriptions/aaaa0a0a-bb1b-cc2c-dd3d-eeeeee4e4e4e/resourceGroups/my-resource-group/providers/Microsoft.OperationalInsights/workspaces/my-workspace",
"name": "centralWorkspace"
}
]
},
"dataFlows": [
{
"streams": [
"Microsoft-Syslog"
],
"destinations": [
"centralWorkspace"
],
"transformKql": "source",
"outputStream": "Microsoft-Syslog"
}
]
}
}
Manage data collection rule associations in Azure Monitor
To view your DCRs in the Azure portal, select Data Collection Rules under Settings on the Monitor menu. Select a DCR to view its details.
Click the Resources tab to view the resources associated with the selected DCR. Click Add to add an association to a new resource. You can view and add resources using this feature whether or not you created the DCR in the Azure portal.
Azure Policy
Using Azure Policy, you can associate a DCR with multiple resources at scale. When you create an assignment between a resource group and a built-in policy or initiative, associations are created between the DCR and each resource of the assigned type in the resource group, including any new resources as they're created. Azure Monitor provides a simplified user experience to create an assignment for a policy or initiative for a particular DCR, which is an alternate method to creating the assignment using Azure Policy directly.
From the DCR in the Azure portal, select Policies (Preview). This will open a page that lists any assignments with the current DCR and the compliance state of included resources. Tiles across the top provide compliance metrics for all resources and assignments.
To create a new assignment, click either Assign Policy or Assign Initiative.
Best Practices for Using Data Collection Rules
Start Small: Begin with a limited set of data sources and destinations to understand the impact.
Monitor Costs: Use Azure Cost Management to track the costs associated with data ingestion.
Test Transformations: Validate KQL queries to ensure they filter or transform data as expected.
Use Tags: Apply tags to DCRs for better management and organization.
Common Use Cases
Application Monitoring: Collect application logs and route them to Log Analytics for troubleshooting.
Security Auditing: Filter and store security-related logs in Azure Storage for long-term retention.
Compliance Reporting: Mask sensitive information in logs to meet regulatory requirements.
Conclusion
Data Collection Rules provide a powerful and flexible way to manage data ingestion in Azure Monitor. By leveraging DCRs, you can optimize costs, improve compliance, and ensure that only the most relevant data is collected. Whether you’re monitoring applications, auditing security logs, or building compliance workflows, DCRs are a must-have tool in your Azure toolkit.
Start exploring Data Collection Rules today and unlock the full potential of Azure Monitor!