DNS is a foundational service in any cloud architecture, but it becomes significantly more complex in hybrid environments. As organizations expand into Azure with hub-spoke networks, private endpoints, and on-premises integration, DNS design must evolve to handle scale, reliability, and maintainability.

This article walks through that evolution:

• Traditional DNS forwarders and their limitations

• Azure DNS Private Resolver

• Inbound and outbound endpoints

• DNS forwarding rulesets

• Distributed vs centralized DNS architectures

• Real-world hybrid resolution scenarios

The Traditional Approach: DNS Forwarders

In early Azure architectures, hybrid DNS resolution was typically achieved using custom DNS forwarder virtual machines.

Typical setup

• On-premises DNS servers forward queries to DNS VMs in Azure

• Azure VNets are configured to use these custom DNS servers

• Conditional forwarding is manually configured

Challenges at Scale

This approach introduces several operational and architectural issues:

Operational overhead

• DNS VMs require patching, monitoring, and backup

• High availability must be designed and maintained

Scalability limitations

• DNS traffic grows with workloads

• Scaling requires manual intervention

Reliability concerns

• DNS becomes dependent on VM availability

• Misconfiguration can disrupt name resolution across environments

Complex hybrid resolution

• Managing Azure Private DNS zones alongside on-premises zones becomes difficult

These challenges led to the need for a managed, cloud-native solution.

Azure DNS Private Resolver

Azure DNS Private Resolver is a fully managed service that provides native DNS resolution between Azure and on-premises environments without requiring DNS VMs.

It enables:

• On-premises systems to resolve Azure private DNS zones

• Azure resources to resolve on-premises domains

• Centralized DNS forwarding logic

This service integrates directly with Azure networking and removes the need for infrastructure management.

Core Components

Inbound Endpoint

An inbound endpoint allows DNS queries from external networks (such as on-premises) to enter Azure.

Key characteristics:

• Receives DNS queries from on-premises DNS servers

• Resolves Azure Private DNS zones and private endpoints

• Deployed in a dedicated subnet

Purpose:
Enable on-premises systems to resolve Azure-hosted private resources.

Outbound Endpoint

An outbound endpoint allows Azure resources to resolve external domains, such as those hosted on-premises.

Key characteristics:

• Sends DNS queries out of Azure

• Works with DNS forwarding rulesets

• Enables conditional forwarding

Purpose:
Enable Azure workloads to resolve on-premises DNS zones.

DNS Forwarding Rulesets

A DNS forwarding ruleset defines how DNS queries should be routed.

Each rule includes:

• Domain name (e.g., corp.local)

• Target DNS server IP address

Rulesets can be:

• Linked to multiple VNets

• Centrally managed

• Used to control DNS behavior at scale

This eliminates the need to configure DNS settings individually on each VNet.

DNS Architecture Patterns

Distributed DNS Architecture

In this model:

• Azure VNets use the default Azure DNS service

• A Private Resolver is deployed in a hub VNet

• A forwarding ruleset is linked to spoke VNets

Flow:

1. A VM in a spoke VNet sends a DNS query

2. Azure DNS attempts to resolve it

3. If no match is found, the ruleset is evaluated

4. Matching queries are forwarded via the outbound endpoint

Key advantage:
Minimal configuration in spoke VNets and high scalability.

Centralized DNS Architecture

In this model:

• VNets are configured to use a custom DNS server (inbound endpoint IP)

• All DNS queries are sent to the hub

Flow:

1. A VM sends a DNS query

2. Query is directed to the inbound endpoint

3. Resolver processes the request using:

   • Private DNS zones

   • Forwarding rules

   • External resolution

Key advantage:
Centralized control and consistent DNS behavior.

Building a Scalable and Secure DNS Architecture with vWAN Hub

For enterprise-scale environments, especially those using Virtual WAN (vWAN), DNS design should align with centralized connectivity and security principles.

Recommended Design

• Deploy a dedicated VNet for DNS

• Host:

  • Private DNS Resolver

  • Inbound endpoint subnet

  • Outbound endpoint subnet

• Connect this DNS VNet to the vWAN hub

• Ensure connectivity to:

  • On-premises networks

  • Spoke VNets

  • Shared services

How it works

• On-premises DNS forwards queries to inbound endpoint via vWAN hub

• Azure VNets use rulesets linked to the resolver

• Outbound endpoint routes DNS queries back to on-premises when required

Key Benefits

Scalability

• DNS is decoupled into a dedicated VNet

• Can support multiple regions and VNets

Security

• Traffic flows through secured vWAN hub

• Enables inspection and policy enforcement

Centralized control

• Single DNS control plane for entire environment

Separation of concerns

• DNS is isolated from application VNets

This model is particularly effective in large enterprises adopting hub-and-spoke with vWAN as the global transit hub.

Real-World Scenarios

Scenario 1: On-Premises VM Accessing Azure Storage via Private Endpoint

1. On-prem VM queries:
   mystorageaccount.privatelink.blob.core.windows.net

2. On-prem DNS forwards to inbound endpoint

3. Private Resolver resolves using Azure Private DNS zone

4. Returns private IP

5. Traffic flows over private connectivity

Scenario 2: Azure VM Resolving On-Premises Application

1. Azure VM queries: app.corp.local

2. Azure DNS checks local zones (no match)

3. Ruleset forwards request via outbound endpoint

4. On-prem DNS resolves

5. Response returned to Azure VM

References

• Azure DNS Private Resolver Architecture
  https://learn.microsoft.com/en-us/azure/dns/private-resolver-architecture

• Azure DNS Private Resolver Overview and Design
  https://learn.microsoft.com/en-us/azure/architecture/networking/architecture/azure-dns-private-resolver

• Azure DNS Management at Scale
  https://www.youtube.com/watch?v=nVONXtEmZa8

If you're working with Terraform and Azure, you've probably encountered situations where you need to create multiple resources based on hierarchical data structures. For example, creating multiple Virtual Networks (VNets) and then multiple Subnets within each VNet.

In this blog, I'll explain how to use nested for loops combined with flatten() to elegantly solve this problem. We'll use Azure Virtual Networks as our reference.

The Problem: Creating VNets and Subnets

Imagine you need to create:3 Azure Virtual Networks (dev, prod, staging) Multiple Subnets within each VNet. Each subnet has its own CIDR block Doing this manually would require hardcoding each resource. But with Terraform's for loops and locals, we can automate this beautifully.

Solution Overview

Our approach has 3 main steps:

1. Define the data structure - Organize VNets and subnets as variables

2. Create VNets - Use for_each to loop through VNets

3. Create Subnets - Use nested loops with flatten() to create subnets

Step 1: Define the Data Structure

First, let's define our Azure networks as a variable:

variable "azure_networks" {
  type = map(object({
    resource_group = string
    location       = string
    address_space  = list(string)
    subnets        = map(object({ 
      address_prefix = string 
    }))
  }))
  default = {
    "vnet-dev" = {
      resource_group = "rg-dev"
      location       = "East US"
      address_space  = ["10.1.0.0/16"]
      subnets = {
        "subnet-vm" = {
          address_prefix = "10.1.1.0/24"
        }
        "subnet-db" = {
          address_prefix = "10.1.2.0/24"
        }
      }
    }
    "vnet-prod" = {
      resource_group = "rg-prod"
      location       = "West US"
      address_space  = ["10.2.0.0/16"]
      subnets = {
        "subnet-web" = {
          address_prefix = "10.2.1.0/24"
        }
        "subnet-api" = {
          address_prefix = "10.2.2.0/24"
        }
        "subnet-db" = {
          address_prefix = "10.2.3.0/24"
        }
      }
    }
    "vnet-staging" = {
      resource_group = "rg-staging"
      location       = "Central US"
      address_space  = ["10.3.0.0/16"]
      subnets = {
        "subnet-test" = {
          address_prefix = "10.3.1.0/24"
        }
      }
    }
  }
}

What we have here:

• A map of 3 Virtual Networks

• Each VNet has subnets stored as a nested map

• Each subnet has an address prefix (CIDR block)

Step 2: Create Azure Virtual Networks

using for_each, we iterate through each VNet:

resource "azurerm_virtual_network" "example" {
  for_each = var.azure_networks

  name                = each.key
  address_space       = each.value.address_space
  location            = each.value.location
  resource_group_name = each.value.resource_group
}

What happens:

Terraform References:

1. azurerm_virtual_network.example["vnet-dev"].id

2. azurerm_virtual_network.example["vnet-prod"].id

3. azurerm_virtual_network.example["vnet-staging"].id

Step 3: The Magic Part - Nested Loops with flatten()

Now comes the complex part: creating subnets across all VNets. We need to:

1. Loop through each VNet

2. Loop through each subnet

3. within that VNet Combine the data Flatten the result into a single list

locals {
  # Create a flat list of all subnets with their parent VNet info
  azure_subnets = flatten([
    for vnet_name, vnet_config in var.azure_networks : [
      for subnet_name, subnet_config in vnet_config.subnets : {
        vnet_name      = vnet_name
        subnet_name    = subnet_name
        address_prefix = subnet_config.address_prefix
        vnet_id        = azurerm_virtual_network.example[vnet_name].id
        resource_group = vnet_config.resource_group
      }
    ]
  ])
}

Let me break this down line by line:

Iteration Trace with Real Data Let me show you exactly what happens in each iteration:

Iteration 1: vnet_name = "vnet-dev"

Inner loop processes subnets in vnet-dev:

Sub-iteration 1a: subnet_name = "subnet-vm"

{
  vnet_name      = "vnet-dev"
  subnet_name    = "subnet-vm"
  address_prefix = "10.1.1.0/24"
  vnet_id        = azurerm_virtual_network.example["vnet-dev"].id
  resource_group = "rg-dev"
Sub-iteration 1b: subnet_name = "subnet-db"

{
  vnet_name      = "vnet-dev"
  subnet_name    = "subnet-db"
  address_prefix = "10.1.2.0/24"
  vnet_id        = azurerm_virtual_network.example["vnet-dev"].id
  resource_group = "rg-dev"
}

Result from Iteration 1: A list with 2 objects

Iteration 2: vnet_name = "vnet-prod"

Inner loop processes subnets in vnet-prod:

Sub-iteration 2a: subnet_name = "subnet-web"

{
  vnet_name      = "vnet-prod"
  subnet_name    = "subnet-web"
  address_prefix = "10.2.1.0/24"
  vnet_id        = azurerm_virtual_network.example["vnet-prod"].id
  resource_group = "rg-prod"
}

Sub-iteration 2b: subnet_name = "subnet-api"

{
  vnet_name      = "vnet-prod"
  subnet_name    = "subnet-api"
  address_prefix = "10.2.2.0/24"
  vnet_id        = azurerm_virtual_network.example["vnet-prod"].id
  resource_group = "rg-prod"
}

Sub-iteration 2c: subnet_name = "subnet-db"

{
  vnet_name      = "vnet-prod"
  subnet_name    = "subnet-db"
  address_prefix = "10.2.3.0/24"
  vnet_id        = azurerm_virtual_network.example["vnet-prod"].id
  resource_group = "rg-prod"
}

Result from Iteration 2: A list with 3 objects

Iteration 3: vnet_name = "vnet-staging"

Inner loop processes subnets in vnet-staging:

Sub-iteration 3a: subnet_name = "subnet-test"

{
  vnet_name      = "vnet-staging"
  subnet_name    = "subnet-test"
  address_prefix = "10.3.1.0/24"
  vnet_id        = azurerm_virtual_network.example["vnet-staging"].id
  resource_group = "rg-staging"
}

Result from Iteration 3: A list with 1 object

Before flatten() - Nested Lists

After the outer loop completes, we have a list of lists:

[
  [
    { vnet_name = "vnet-dev", subnet_name = "subnet-vm", ... },
    { vnet_name = "vnet-dev", subnet_name = "subnet-db", ... }
  ],
  [
    { vnet_name = "vnet-prod", subnet_name = "subnet-web", ... },
    { vnet_name = "vnet-prod", subnet_name = "subnet-api", ... },
    { vnet_name = "vnet-prod", subnet_name = "subnet-db", ... }
  ],
  [
    { vnet_name = "vnet-staging", subnet_name = "subnet-test", ... }
  ]
]

Problem: This is hard to work with because it's nested!

After flatten() - Single Flat List. The flatten() function removes one level of nesting:

[
  { vnet_name = "vnet-dev", subnet_name = "subnet-vm", address_prefix = "10.1.1.0/24", vnet_id = "...", resource_group = "rg-dev" },
  { vnet_name = "vnet-dev", subnet_name = "subnet-db", address_prefix = "10.1.2.0/24", vnet_id = "...", resource_group = "rg-dev" },
  { vnet_name = "vnet-prod", subnet_name = "subnet-web", address_prefix = "10.2.1.0/24", vnet_id = "...", resource_group = "rg-prod" },
  { vnet_name = "vnet-prod", subnet_name = "subnet-api", address_prefix = "10.2.2.0/24", vnet_id = "...", resource_group = "rg-prod" },
  { vnet_name = "vnet-prod", subnet_name = "subnet-db", address_prefix = "10.2.3.0/24", vnet_id = "...", resource_group = "rg-prod" },
  { vnet_name = "vnet-staging", subnet_name = "subnet-test", address_prefix = "10.3.1.0/24", vnet_id = "...", resource_group = "rg-staging" }
]

Step 4: Convert List to Map and Create Subnets

for_each requires a map, not a list. So we convert the flat list to a map with unique keys:

resource "azurerm_subnet" "example" {
  for_each = {
    for subnet in local.azure_subnets : "${subnet.vnet_name}.${subnet.subnet_name}" => subnet
  }

  name                 = each.value.subnet_name
  resource_group_name  = each.value.resource_group
  virtual_network_name = each.value.vnet_name
  address_prefixes     = [each.value.address_prefix]

  depends_on = [azurerm_virtual_network.example]
}

What Gets Created After running terraform apply, you'll have 3 Vnets and 6 subnets.

Key Concepts to Remember

Why This Approach?

1. DRY (Don't Repeat Yourself) - No hardcoded resources

2. Scalable - Add new VNets/subnets easily by updating the variable

3. Maintainable - All data in one place

4. Flexible - Change naming, locations, CIDR blocks easily

5. Reusable - Use this pattern for other hierarchical resources

Conclusion

Nested loops with flatten() are powerful Terraform patterns for managing hierarchical resources. By understanding how to:

Loop through parent resources (VNets) Loop through child resources (Subnets) Flatten nested lists Convert lists to maps You can automate complex infrastructure deployments with minimal code and maximum flexibility.

What Are Custom Repository Instructions?

Custom repository instructions are markdown files (.github/copilot-instructions.md) that you place in your repository to guide Copilot's code generation. Think of them as a persistent conversation with Copilot about how your project works, what patterns you prefer, and what conventions your team follows.

For Terraform projects, these instructions are particularly valuable because infrastructure-as-code has numerous architectural decisions that vary between organizations: naming conventions, module structures, state management approaches, and resource organization patterns. Without guidance, Copilot might suggest patterns that conflict with your established standards.

Why Custom Instructions Matter for Terraform

Terraform projects benefit from custom instructions because they help Copilot:

• Generate reusable module code with consistent patterns for conditional resource creation

• Follow your team's naming conventions for resources, variables, and outputs

• Understand your state management strategy, especially when using remote backends

• Apply your preferred dependency management approaches

• Use data sources appropriately versus hard-coded values

• Implement your organization's tagging and labeling standards

• Follow security and compliance requirements specific to your infrastructure

Essential Elements for Terraform Instructions

Project Context and Architecture

Start by explaining your project's purpose and how Terraform is organized. This helps Copilot understand the big picture.

markdown

## Project Overview
This repository manages AWS infrastructure for production and staging environments.
We use a workspace-based approach with remote state in S3.

Module Reusability with Ternary Operators

Instruct Copilot to create flexible modules using conditional logic for optional resources.

## Module Patterns

### Conditional Resource Creation
Use ternary operators and `count` or `for_each` to make resources optional:
- Prefer `count = var.enable_feature ? 1 : 0` for single optional resources
- Use `for_each` for multiple conditional resources based on maps or sets
- Always provide sensible defaults in variable definitions

Locals for Computed Values

Guide Copilot on when and how to use locals blocks.

## Using Locals

Use `locals` blocks for:
- Computed values used multiple times
- Complex expressions that would clutter resource definitions
- Combining variables into standardized formats (e.g., naming conventions)
- Environment-specific configurations

Example pattern:
```hcl
locals {
  common_tags = merge(
    var.tags,
    {
      Environment = var.environment
      ManagedBy   = "Terraform"
      Repository  = "github.com/org/repo"
    }
  )

  resource_name = "${var.project}-${var.environment}-${var.component}"
}
```

State Management with Removed Blocks

For teams managing state with removed blocks rather than CLI commands, this is crucial context.

## State Management

### Remote State
- Backend configuration is in `backend.tf`
- Use S3 backend with DynamoDB locking
- Never commit `.tfstate` files

### Removing Resources
When removing resources from management, use `removed` blocks instead of `terraform state rm`:
```hcl
removed {
  from = aws_instance.legacy_server

  lifecycle {
    destroy = false
  }
}
```

This ensures state changes are tracked in version control and applied consistently across the team.

Dependency Management

Explain how to handle implicit and explicit dependencies.

## Managing Dependencies

### Implicit Dependencies
Prefer implicit dependencies through resource references:
```hcl
subnet_id = aws_subnet.private.id
```

### Explicit Dependencies
Use `depends_on` only when Terraform cannot infer the dependency:
- Cross-module dependencies that aren't captured by outputs
- Timing issues where resources must be created in sequence
- When destroying resources in specific order matters

Always add a comment explaining why explicit dependency is needed.

Data Sources vs. Hard-Coded Values

Provide guidance on using data sources for dynamic lookups.

## Data Sources

Prefer data sources over hard-coded values for:
- AMI IDs (use latest with filters)
- Availability zones
- VPC and subnet IDs when working across modules
- IAM policies and service principals
- Route53 zone IDs

Example:
```hcl
data "aws_ami" "amazon_linux" {
  most_recent = true
  owners      = ["amazon"]

  filter {
    name   = "name"
    values = ["amzn2-ami-hvm-*-x86_64-gp2"]
  }
}
```

Avoid data sources for values that should be explicitly versioned in code.

Variable and Output Conventions

Define naming and documentation standards.

## Variables and Outputs

### Variable Definitions
- Use snake_case for all variable names
- Always include description and type
- Provide defaults for optional variables
- Use validation blocks for constrained values
```hcl
variable "environment" {
  description = "Environment name (dev, staging, prod)"
  type        = string

  validation {
    condition     = contains(["dev", "staging", "prod"], var.environment)
    error_message = "Environment must be dev, staging, or prod."
  }
}
```

### Outputs
- Output resource IDs and ARNs that other modules might need
- Include descriptions explaining the output's purpose
- Use sensitive = true for secrets

Tagging Strategy

Specify your organization's tagging requirements.

## Resource Tagging

All resources that support tags must include:
```hcl
tags = merge(
  local.common_tags,
  {
    Name = local.resource_name
  }
)
```

Required tags in common_tags:
- Environment
- ManagedBy
- Repository
- CostCenter
- Owner

Security and Compliance

Include security-specific guidance.

## Security Requirements

- Never hard-code credentials or secrets
- Use AWS Secrets Manager or SSM Parameter Store for sensitive values
- Enable encryption at rest for all storage resources
- Use private subnets for compute resources when possible
- Enable logging and monitoring for all resources
- Follow principle of least privilege for IAM roles and policies

Module Organization

Explain how modules should be structured.

## Module Structure

Standard module layout: modules/ <module-name>/ main.tf # Primary resource definitions variables.tf # Input variables outputs.tf # Output values versions.tf # Provider and Terraform version constraints locals.tf # Local values (if needed) data.tf # Data sources (if needed) README.md # Module documentation

#module guildelines
Each module should be self-contained and reusable.

Testing and Validation

Provide instructions for testing patterns.

## Testing

- Use `terraform fmt` to format all .tf files
- Run `terraform validate` before committing
- Use `tflint` with the AWS ruleset
- Include examples/ directory with working examples of module usage
- Use `terraform-docs` to generate module documentation

Complete Sample Instructions File

Here's a comprehensive example bringing all these elements together:

markdown

# Terraform Custom Instructions for GitHub Copilot

## Project Overview
This repository manages multi-environment AWS infrastructure using Terraform modules.
We deploy to dev, staging, and production environments with separate AWS accounts.

## Code Style and Conventions

### Naming
- Use snake_case for resources, variables, outputs, and locals
- Resource names: `<resource_type>_<descriptive_name>`
- Variable names should be descriptive and unabbreviated when possible
- Module directory names: kebab-case

### File Organization
- `main.tf`: Primary resource definitions
- `variables.tf`: Input variables only
- `outputs.tf`: Output values only
- `locals.tf`: Local value computations
- `data.tf`: Data source lookups
- `versions.tf`: Terraform and provider version constraints
- `backend.tf`: Backend configuration

## Module Design Patterns

### Conditional Resources
Use ternary operators with count for optional resources:
```hcl
resource "aws_cloudwatch_log_group" "this" {
  count = var.enable_logging ? 1 : 0

  name              = "/aws/lambda/${var.function_name}"
  retention_in_days = var.log_retention_days

  tags = local.common_tags
}
```

For multiple conditional resources, prefer for_each:
```hcl
resource "aws_subnet" "private" {
  for_each = var.create_private_subnets ? var.private_subnet_cidrs : {}

  vpc_id            = aws_vpc.main.id
  cidr_block        = each.value
  availability_zone = each.key

  tags = merge(
    local.common_tags,
    {
      Name = "${local.resource_prefix}-private-${each.key}"
      Tier = "private"
    }
  )
}
```

### Locals Usage
Use locals for:
- Repeated computed values
- Name prefixes following our convention
- Merging tags
- Complex conditional logic
```hcl
locals {
  resource_prefix = "${var.project_name}-${var.environment}"

  common_tags = merge(
    var.additional_tags,
    {
      Environment  = var.environment
      ManagedBy    = "Terraform"
      Repository   = "github.com/myorg/infrastructure"
      CostCenter   = var.cost_center
      Owner        = var.owner_email
    }
  )

  # Complex logic in locals keeps resources clean
  enable_enhanced_monitoring = var.environment == "prod" ? true : var.enable_monitoring

  backup_retention = {
    dev     = 7
    staging = 14
    prod    = 30
  }
  retention_days = local.backup_retention[var.environment]
}
```

## State Management

### Backend Configuration
- Use S3 backend with DynamoDB state locking
- Backend config in `backend.tf`
- State file path: `<environment>/<component>/terraform.tfstate`

### Removing Resources from State
Use `removed` blocks instead of CLI commands for team consistency:
```hcl
removed {
  from = aws_instance.deprecated_server

  lifecycle {
    destroy = false  # Keep the resource, just remove from state
  }
}

# Or to track actual resource deletion
removed {
  from = module.legacy_database

  lifecycle {
    destroy = true
  }
}
```

This ensures state changes are version-controlled and reviewable.

## Dependency Management

### Prefer Implicit Dependencies
```hcl
# Good - implicit dependency
resource "aws_eip" "nat" {
  vpc = true
  tags = local.common_tags
}

resource "aws_nat_gateway" "main" {
  allocation_id = aws_eip.nat.id  # Implicit dependency
  subnet_id     = aws_subnet.public.id
}
```

### Explicit Dependencies (Use Sparingly)
Only use `depends_on` when necessary:
```hcl
resource "aws_iam_role_policy_attachment" "lambda" {
  role       = aws_iam_role.lambda.name
  policy_arn = aws_iam_policy.lambda.arn

  # Explicit dependency needed for eventual consistency
  depends_on = [aws_iam_role.lambda]
}
```

Always include a comment explaining why explicit dependency is required.

## Data Sources

Use data sources for dynamic lookups, not for values that should be versioned:
```hcl
# Good - dynamic lookup of latest AMI
data "aws_ami" "amazon_linux_2" {
  most_recent = true
  owners      = ["amazon"]

  filter {
    name   = "name"
    values = ["amzn2-ami-hvm-*-x86_64-gp2"]
  }

  filter {
    name   = "virtualization-type"
    values = ["hvm"]
  }
}

# Good - lookup existing VPC
data "aws_vpc" "main" {
  tags = {
    Name = "${var.project_name}-vpc"
  }
}

# Good - get available AZs
data "aws_availability_zones" "available" {
  state = "available"
}

# Bad - hard-code when data source is appropriate
resource "aws_instance" "web" {
  ami = "ami-0c55b159cbfafe1f0"  # Don't do this
  # ...
}
```

## Variables

### Variable Definitions
Always include description, type, and defaults when appropriate:
```hcl
variable "environment" {
  description = "Environment name: dev, staging, or prod"
  type        = string

  validation {
    condition     = contains(["dev", "staging", "prod"], var.environment)
    error_message = "Environment must be dev, staging, or prod."
  }
}

variable "instance_type" {
  description = "EC2 instance type for web servers"
  type        = string
  default     = "t3.micro"
}

variable "enable_monitoring" {
  description = "Enable detailed CloudWatch monitoring"
  type        = bool
  default     = false
}

variable "subnet_cidrs" {
  description = "Map of availability zone to subnet CIDR blocks"
  type        = map(string)
  default     = {}
}

variable "allowed_cidr_blocks" {
  description = "List of CIDR blocks allowed to access resources"
  type        = list(string)
  default     = []

  validation {
    condition = alltrue([
      for cidr in var.allowed_cidr_blocks : can(cidrhost(cidr, 0))
    ])
    error_message = "All elements must be valid CIDR blocks."
  }
}
```

### Variable Precedence
Variables should be provided in this order:
1. Environment-specific `.tfvars` files
2. Common `terraform.tfvars`
3. Defaults in `variables.tf`

## Outputs

Include clear descriptions and mark sensitive values:
```hcl
output "vpc_id" {
  description = "ID of the created VPC"
  value       = aws_vpc.main.id
}

output "private_subnet_ids" {
  description = "List of private subnet IDs for compute resources"
  value       = [for subnet in aws_subnet.private : subnet.id]
}

output "database_endpoint" {
  description = "Connection endpoint for the RDS instance"
  value       = aws_db_instance.main.endpoint
  sensitive   = true
}
```

## Tagging Strategy

All taggable resources must include common_tags:
```hcl
resource "aws_instance" "web" {
  # ... other configuration ...

  tags = merge(
    local.common_tags,
    {
      Name      = "${local.resource_prefix}-web-${count.index + 1}"
      Component = "web-server"
      Backup    = "daily"
    }
  )
}
```

Required tags (enforced via SCPs):
- Environment
- ManagedBy
- Repository
- CostCenter
- Owner

## Security Best Practices

### Secrets Management
Never hard-code secrets. Use AWS Secrets Manager or SSM Parameter Store:
```hcl
data "aws_secretsmanager_secret_version" "db_password" {
  secret_id = "${var.project_name}/${var.environment}/db-password"
}

resource "aws_db_instance" "main" {
  # ... other configuration ...
  password = data.aws_secretsmanager_secret_version.db_password.secret_string
}
```

### Encryption
Enable encryption for all storage:
```hcl
resource "aws_s3_bucket" "data" {
  bucket = "${local.resource_prefix}-data"

  tags = local.common_tags
}

resource "aws_s3_bucket_server_side_encryption_configuration" "data" {
  bucket = aws_s3_bucket.data.id

  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm     = "aws:kms"
      kms_master_key_id = aws_kms_key.s3.arn
    }
  }
}
```

### Network Security
- Place compute resources in private subnets
- Use security groups with minimal required access
- Enable VPC flow logs
- Use AWS PrivateLink for AWS service access when possible

## Module Structure

Standard module organization:

Sample .github/copilot-instructions.md for VS Code

For VS Code Copilot customization, create a similar file that focuses on editor-specific workflows:

# Terraform Development with GitHub Copilot

## Project Context
AWS infrastructure managed with Terraform in a multi-environment setup.
Follow the patterns and conventions defined in our terraform modules.

## When Writing Terraform Code

### Always Include
- Type definitions for all variables
- Descriptions for variables and outputs
- Validation blocks for constrained variables
- Common tags merged with resource-specific tags
- Comments explaining complex conditionals

### Naming Patterns
Generate resource names using this pattern:
```hcl
locals {
  name_prefix = "${var.project_name}-${var.environment}"
}

resource "aws_xxx" "example" {
  tags = merge(local.common_tags, {
    Name = "${local.name_prefix}-descriptive-name"
  })
}
```

### Conditional Resources
When I ask for optional resources, use count with ternary:
```hcl
count = var.enable_feature ? 1 : 0
```

Reference with: `aws_resource.example[0].id`

For multiple resources based on a map, use for_each.

### Data Sources
Suggest data sources for:
- Latest AMIs (with filters)
- Availability zones
- Existing VPCs/subnets
- IAM policy documents

### Security Defaults
When creating resources:
- Enable encryption by default
- Use private subnets for compute
- Apply least-privilege IAM policies
- Enable logging and monitoring

### Suggest Modules
If I'm writing repetitive resource configurations, suggest creating a reusable module.

## Code Completion Preferences

When I start typing:
- `variable` - include description, type, and validation if constrained
- `output` - include description
- `resource` - include common_tags
- `data` - include relevant filters
- `locals` - use for name prefixes and tag merging

## Testing
Remind me to run:
- `terraform fmt` before committing
- `terraform validate` to check syntax
- `terraform plan` to preview changes

Conclusion

Custom repository instructions transform GitHub Copilot from a generic code completion tool into a knowledgeable team member that understands your specific Terraform patterns and conventions. By investing time in creating comprehensive instructions, you'll receive more relevant suggestions, reduce code review cycles, and maintain consistency across your infrastructure codebase.

Start with the essential elements outlined above, then refine your instructions based on the patterns and challenges unique to your organization. As your team's conventions evolve, keep your instructions updated to ensure Copilot continues to provide valuable, context-aware assistance.

Reference

if you are intesrested in learning more about how you can leverage Copilot then I would recomment you to check Repo below

awesome-copilot

copilot-in-a-box

Moving to the cloud isnt just about deploying virtual machines and servicesits about creating a repeatable, secure, and governed foundation that scales as your organization grows. Microsofts Azure Landing Zone Accelerator provides a structured way to establish this foundation, and Azure Verified Modules (AVMs) are the building blocks that make it reliable.

In this article, well explore what the Landing Zone Accelerator is, how AVMs work, and why theyre a game changer for enterprises adopting Azure.

The Challenge of Cloud Foundations

Enterprises often face a common set of challenges when deploying workloads in Azure:

• Inconsistent configurations across environments

• Security and compliance gaps due to misapplied policies

• Lack of repeatability when scaling workloads

• Manual effort to reapply governance and monitoring settings

Without a baseline, cloud sprawl can lead to uncontrolled costs, security risks, and operational headaches. Thats where the Landing Zone Accelerator and Verified Modules come in.

What is the Azure Landing Zone Accelerator?

The Azure Landing Zone Accelerator is Microsofts implementation of Cloud Adoption Framework (CAF) landing zones. It provides:

• Architecture guidance: how to design identity, networking, security, and monitoring foundations

• Pre-built modules: reusable templates for deploying best-practice Azure resources

• Governance and compliance: policy-driven controls to enforce security standards

• Flexibility: deploy piecemeal (per module) or as a full baseline

Think of it as a starter kit for enterprise-scale Azure environmentspre-configured, opinionated, and ready to extend.

Introducing Azure Verified Modules (AVMs)

So what makes a module verified?

An Azure Verified Module is a reusable infrastructure-as-code module that has been:

• Reviewed by Microsoft engineers

• Aligned with Azure best practices

• Validated with testing for functionality and security

• Versioned and maintained for lifecycle management

Unlike custom or community modules, AVMs give you confidenceyoure using a module thats been vetted to meet enterprise and security standards.

Examples of AVMs include:

• Identity & Role assignments

• Virtual networks and subnets

• Monitoring and diagnostic settings

• Security policies (e.g., Defender for Cloud)

How AVMs Fit Into the Landing Zone Accelerator

The Landing Zone Accelerator is essentially a composition of AVMs. For example:

1. Identity & Access

   • Azure AD integration

   • Role-based access control (RBAC)

2. Networking

   • Virtual networks, subnets, private DNS, and firewalls

3. Management & Monitoring

   • Azure Monitor setup

   • Log Analytics workspaces

   • Policy assignments

4. Security & Compliance

   • Microsoft Defender for Cloud policies

   • Blueprints for regulatory compliance

Each of these pieces can be deployed via a verified module, ensuring consistent quality and security.

The Landing Zone Accelerator is essentially a composition of AVMs. Each AVM handles a specific domain (identity, networking, monitoring, security), and together they provide a full enterprise-ready foundation.

One of the most important first steps is subscription vending.

Example: Subscription Vending with AVMs

Subscription vending is the process of creating and configuring new Azure subscriptions in a standardized, governed way. Instead of manually creating subscriptions and applying inconsistent configurations, you can use AVMs to automate and enforce standards.

A subscription vending AVM might include:

• Creation of the subscription

• Assignment of management groups

• Application of Azure Policy for compliance

• Enabling monitoring and diagnostic settings

• Setting up baseline RBAC roles

This ensures every subscription is born secure, compliant, and consistenta key requirement for large organizations managing hundreds of workloads.

Bootstrapping Your Landing Zone

Once subscriptions are managed, the next step is bootstrapping. This is about preparing your environment so that other teams can deploy confidently.

Bootstrapping often includes:

• Assigning core management groups and policies

• Deploying identity AVMs (like role assignments, UAMI/SMI configurations)

• Setting up automation hooks (CI/CD pipelines for IaC deployments)

• Ensuring billing and tagging standards are enforced

Bootstrapping is like laying the concrete foundation of a houseyou dont see it once its built, but everything depends on it.

Platform Landing Zone

The platform landing zone is where your enterprise shared services live. Its built using AVMs to establish services that all applications and business units will consume, such as:

• Networking AVMs: Hub-and-spoke VNet, private DNS, firewalls, ExpressRoute/Virtual WAN

• Security AVMs: Microsoft Defender for Cloud, Sentinel integrations, Key Vault

• Management AVMs: Log Analytics, Azure Monitor, policy baselines, update management

The platform landing zone provides the secure, monitored backbone of your cloud estate. Without it, application teams would have to reinvent networking, monitoring, and security every time.

Application Landing Zone

Finally, we reach the application landing zonewhere business workloads actually run.

AVMs in this layer handle:

• App-specific networking (VNets/subnets inside the spoke)

• Identity (role assignments, managed identities)

• Observability (diagnostic settings tied to central monitoring)

• Security policies tailored to workloads (e.g., PCI-DSS apps, healthcare apps)

Because of the baseline AVMs applied during subscription vending and the shared platform landing zone, the application landing zone can focus purely on the workload without worrying about governance gaps.

Policy Context & Policy Versioning

When using Azure Landing Zones and Azure Verified Modules (AVMs), policies are central to ensuring consistent, secure, and compliant infrastructure. They act as guardrailsrestricting what can be deployed, enforcing settings, and ensuring resources meet standards automatically. The Landing Zone Accelerator has a formal approach for policies, especially around policy versioning, which helps maintain stability and flexibility over time.

What is Policy Context

• A policy in Azure is a definition that enforces rules over resourcese.g., allowed VM sizes, permitted locations, enabling diagnostic logs, enforcing tag usage.

• In Landing Zones, policies are typically applied via initiative definitions (policy sets) that bundle multiple individual policies together. These ensure a baseline of compliance across identity, security, networking, and monitoring.

• The AVMs and the Landing Zone Accelerator include many of these policy initiatives out of the box, so when you deploy modules, relevant policies are baked in.

Why Policy Versioning Matters

Policy versioning in the Azure Landing Zones Accelerator guides how policies evolve while keeping backward compatibility and avoiding breaking changes. Key points:

• Immutable releases: Once a policy (or initiative) version is released and consumed, that version should not change in a breaking way. If new policy requirements or enhancements are needed, a new version is published.

• Semver or version numbering: Policies and initiatives are versioned explicitly. This means you (or your deployment pipelines) can pin to policy version 1.0.0, 1.1.0, etc., ensuring that when Azure or Microsoft releases updates for policy sets, your environment doesnt change unexpectedly.

• Upgrade paths: When newer policy versions are released, you can plan in advanceto test them in non-production, to review impact, and to gradually promote to production.

How Policy Versioning Works in the Accelerator

• The Landing Zone Accelerator stores policy definitions and initiatives in its GitHub repository. Each policy package or module has a version number. When you reference a policy module in your Bicep or AVM template, you specify the version you want.

• Example snippet (in Bicep / AVM context):

    module baselinePolicy 'br/public:avm/policy/initiative/baseline:1.2.0' = {
      name: 'baseline-policy'
      params: {
        allowedLocations: [ 'eastus', 'australiaeast' ]
        tagDefaults: { environment: 'prod' }
      }
    }
  

  In this example, baseline:1.2.0 pins the policy initiative to version 1.2.0 so updates later to version 1.3.0 or 2.0.0 are opt-in rather than automatic.

Best Practices: Using Policy Versioning in Your Adoption

• Pin your policy versions in your IaC templates so you know exactly what controls are being applied.

• Monitor new releases of policy versions from the Accelerator repo. Review change logs.

• Test policy changes in dev/test subscriptions before rolling them out.

• Coordinate with your governance teampolicy changes can impact deployments, cost, and compliance audits.

Why This Matters

This layered approachsubscription vending bootstrapping platform landing zone application landing zoneis what makes the Landing Zone Accelerator with AVMs so powerful.

Each step builds on the previous one:

1. Subscriptions are created consistently.

2. Bootstrapping enforces governance.

3. Platform services are shared and secure.

4. Applications deploy quickly and safely.

This ensures your Azure environment is scalable, secure, and operationally efficient from day one.

References

• Azure Landing Zones https://azure.github.io/Azure-Landing-Zones/

• Landing Zone Accelerator (User Guide) https://azure.github.io/Azure-Landing-Zones/accelerator/

Azure's Model Context Protocol (MCP) provides a standardized way to serve domain-specific context to large language models. In this post, Ill walk you through how I set up my own MCP Server in VS Code and what tools are vailable in preview version of the server

Azure MCP Server

The Azure MCP Server enables AI agents and other types of clients to interact with Azure resources through natural language commands. It implements the Model Context Protocol (MCP) to provide these key features:

• MCP support: Because the Azure MCP Server implements the Model Context Protocol, it works with MCP clients such as GitHub Copilot agent mode, the OpenAI Agents SDK, and Semantic Kernel.

• Entra ID support: The Azure MCP Server uses Entra ID through the Azure Identity library to follow Azure authentication best practices.

• Service and tool support: The Azure MCP Server supports Azure services and tools such as the Azure CLI and Azure Developer CLI (azd).

Introduction to the Model Context Protocol (MCP)

The Model Context Protocol (MCP) is an open protocol designed to manage how language models interact with external tools, memory, and context in a safe, structured, and stateful way. MCP defines a client-server architecture with several components:

• Hosts: Apps that use MCP clients to connect to and consume data from MCP servers.

• Clients: Components of MCP hosts that manage connections and retrieve data from MCP servers.

• Servers: Programs that provide features like data resources, tools for performing actions, and prompts to guide interactions.

For example, VS Code is considered a host, and GitHub Copilot agent mode in VS Code acts as an MCP client that connects to MCP servers. You might also build a custom intelligent app that hosts its own MCP client that connects to MCP servers.

The Azure MCP Server implements a set of tools per the Model Context Protocol. AI agents and other types of clients use these tools to interact with Azure resources.

Spin Up Your Own Azure MCP Server in VS Code

I followed the Microsoft documentation. According to the documentation following are prerequisites I installed. I am using Windows subsystem for Linux, this tutorial will provide steps for Ubuntu.

Prerequisites

• Azure Account

• Python 3.9 or higher

• Node JS installed locally

Installing Nodejs and dependencies for VS Code in WSL

#!/bin/bash


# --- 1. Setting up NVM (Node Version Manager) and Node.js ---
echo "## 1. NVM and Node.js Installation"
echo "NVM allows you to manage multiple Node.js versions on your system."

echo "# Core Dependencies for NVM (often pre-installed in modern Ubuntu WSL)"
echo "sudo apt update"
echo "sudo apt install -y curl" # curl is used to download the nvm install script

echo "# NVM Installation (downloads the nvm script and sources it in your .bashrc/.zshrc)"
echo "curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/master/install.sh | bash"
echo ""
echo "# After installation, you must source your shell config or open a new terminal"
echo "source ~/.bashrc # or ~/.zshrc if you use zsh"
echo ""

echo "# Node.js Installation (via NVM)"
echo "nvm install --lts # Installs the latest Long Term Support version of Node.js"
echo "nvm use --lts     # Sets the LTS version as the default for the current shell session"
echo ""
echo "# Core components installed with Node.js:"
echo "  - node: The JavaScript runtime itself."
echo "  - npm: Node Package Manager, used for installing and managing Node.js packages."
echo "  - npx: Node Package Execute, used for executing Node.js package binaries (often temporary)."
echo ""

# --- 2. Configuring VS Code for WSL ---
echo "## 2. VS Code WSL Integration"
echo "The 'Remote - WSL' extension bridges VS Code on Windows to your WSL environment."

echo "# VS Code Extension Installation (done from VS Code Extensions view on Windows side)"
echo "  - Remote - WSL Extension (Microsoft)"
echo ""
echo "# WSL-side Configuration for VS Code Server"
echo "  - ~/.vscode-server/ (directory created by VS Code, contains VS Code Server binaries and extensions)"
echo "  - ~/.vscode-server/server-env-setup (custom file for setting environment variables for VS Code Server)"
echo "    - Inside this file, you manually added:"
echo "      export NVM_DIR=\"$HOME/.nvm\""
echo "      [ -s \"\$NVM_DIR/nvm.sh\" ] && \\. \"\$NVM_DIR/nvm.sh\""
echo "      [ -s \"\$NVM_DIR/bash_completion\" ] && \\. \"\$NVM_DIR/bash_completion\""
echo "      nvm use --silent <your_node_version>"
echo ""

# --- 3. Installing Azure CLI ---
echo "## 3. Azure CLI Installation"
echo "Azure CLI allows interaction with Azure resources from the command line."

echo "# System Dependencies for Azure CLI (specific to apt-based distributions like Ubuntu)"
echo "sudo apt update"
echo "sudo apt install -y ca-certificates curl apt-transport-https lsb-release gnupg"
echo "  - ca-certificates: Provides root certificates for secure communication (HTTPS)."
echo "  - curl: Tool for transferring data with URLs (used to download the Microsoft GPG key)."
echo "  - apt-transport-https: Enables apt to fetch packages over HTTPS."
echo "  - lsb-release: Provides information about the Linux distribution (used to get codename)."
echo "  - gnupg: GNU Privacy Guard, used for managing cryptographic keys (for verifying package authenticity)."
echo ""

echo "# Adding Microsoft GPG key and Azure CLI repository"
echo "curl -sL https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor | sudo tee /etc/apt/keyrings/microsoft.gpg > /dev/null"
echo "  - This imports Microsoft's public key to authenticate Azure CLI packages."
echo "AZ_REPO=\$(lsb_release -cs)"
echo "echo \"deb [arch=amd64 signed-by=/etc/apt/keyrings/microsoft.gpg] https://packages.microsoft.com/repos/azure-cli/ \$AZ_REPO main\" | sudo tee /etc/apt/sources.list.d/azure-cli.list"
echo "  - This adds the official Azure CLI repository to your system's package sources."
echo ""

echo "# Azure CLI Package Installation"
echo "sudo apt update" # Updates package lists to include the new Azure CLI repository
echo "sudo apt install azure-cli"
echo "  - This command installs the main Azure CLI package and its core Python dependencies (Azure CLI is primarily Python-based)."
echo ""

echo "# Core Dependencies of Azure CLI (handled by apt):"
echo "  - Python (version 3.8 or higher, usually installed as a dependency by apt if not present)."
echo "  - Various Python libraries and modules that the Azure CLI uses (e.g., requests, msrest, knack, azure-common, etc. - these are pulled in automatically)."
echo ""
echo "--- End of Outline ---"

Global Install

I installed Azure MCP server Globally on my device. Directory Install is also supported

1. To install the Azure MCP Server for Visual Studio Code in your user settings, select the following link:

   VS Code: Install Azure MCP Server

2. A list of installation options opens inside Visual Studio Code. Select Install Server to add the server configuration to your user settings.

   

3. Open GitHub Copilot and select Agent Mode. To learn more about Agent Mode.

Refresh the tools list to see Azure MCP Server as an available option:

Use prompts to test the Azure MCP Server

Open GitHub Copilot and select Agent Mode.

I will authenticate by signing into my Azure Account

az login --use-device-code

After Authenticating I will ask Co-pilot To list my resource groups

Copilot requests permission to run the necessary Azure MCP Server operation for your prompt. Select Continue or use the arrow to select a more specific behavior:

• Current session always runs the operation in the current GitHub Copilot Agent Mode session.

• Current workspace always runs the command for current Visual Studio Code workspace.

• Always allow sets the operation to always run for any GitHub Copilot Agent Mode session or any Visual Studio Code workspace.

Copilot will run MCP server, will use Resource Group tool and provide context to LLM and run the query and give you output

Future Integration with Operation

The Azure Model Context Protocol (MCP) Server exposes many tools you can use from an existing client to interact with Azure services through natural language prompts. For example, you can use the Azure MCP Server to interact with Azure resources conversationally from GitHub Copilot agent mode in Visual Studio Code or other AI agents with commands like these:

• "Show me all my resource groups"

• "List blobs in my storage container named 'documents'"

• "What's the value of the 'ConnectionString' key in my app configuration?"

• "Query my log analytics workspace for errors in the last hour"

• "Show me all my Cosmos DB databases"

Azure MCP server is in preview and most of the tools it exposes are Get tools. These tools help in Listing resources in Azure.

Agent Integration with Operation and Incident Response in Azure

I believe the next stage in the evolution of the Azure MCP Server lies in Operations and Incident Response.

Consider a scenario where a virtual machine triggers an alert due to a failed service. By integrating this alert into an AI-powered workflow using the Azure MCP Server and a connected agent, the system could automatically investigate the issue and suggest remediation steps to an Operations Engineer.

The engineer would then have the flexibility to review, accept, or even revert the recommended actions enabling a more efficient and intelligent approach to managing operational incidents.

###

In this article, we'll walk through:

• What Azure Private DNS Resolver is

• How to create an inbound endpoint

• How to configure conditional forwarding from your on-prem DNS to Azure

• The benefits of using private name resolution

What is Azure Private DNS Resolver?

Azure Private DNS Resolver is a fully managed DNS service that enables DNS resolution between Azure virtual networks and your on-premises environment without deploying and managing DNS servers.

It supports:

• Inbound endpoints: Accept DNS queries from on-premises or other networks.

• Outbound endpoints and forwarding rulesets: Resolve custom DNS names from Azure to on-prem or external DNS servers.

Scenario Overview

We want to resolve the domain *.azurewebsites.net from our on-premises network to the private IP of the web app's private endpoint in Azure.

To do this:

1. Deploy Azure Private DNS Resolver with an inbound endpoint.

2. Set up a conditional forwarder in your on-prem DNS server pointing azurewebsites.net to the inbound endpoint's private IP.

3. Azure resolves the name using the Private DNS zone linked to the web app's private endpoint.

Step-by-Step: Creating an Inbound Endpoint

Step 1: Deploy Azure DNS Resolver

az network dns-resolver create \
  --name myDnsResolver \
  --resource-group myResourceGroup \
  --location eastus \
  --virtual-network myVnet

Step 2: Create an Inbound Endpoint

az network dns-resolver inbound-endpoint create \
  --name inboundEndpoint1 \
  --dns-resolver-name myDnsResolver \
  --resource-group myResourceGroup \
  --location eastus \
  --ip-configurations '[{"subnet": { "id": "/subscriptions/<sub-id>/resourceGroups/myResourceGroup/providers/Microsoft.Network/virtualNetworks/myVnet/subnets/inboundSubnet" }}]'

Use a dedicated subnet for DNS Resolver. It cannot be shared with other resources.

Step 3: Configure Conditional Forwarding in On-Prem DNS

On your on-prem DNS server (e.g., Windows Server DNS):

1. Open DNS Manager.

2. Right-click Conditional Forwarders > New Conditional Forwarder.

3. Enter:

   • Domain name: azurewebsites.net

   • IP address: Private IP of the inbound endpoint

   • Optionally, enable "Store this conditional forwarder in Active Directory"

This routes only azurewebsites.net queries to Azure, avoiding unnecessary traffic.

Benefits of Private Name Resolution with Azure DNS Resolver

Improved Security
Resolves names to private IPs securelywithout exposing DNS records to public resolvers.

Seamless Hybrid Integration
Enables on-premises apps to resolve private Azure services like Web Apps, Key Vault, and Storage.

No DNS VM Management
Azure handles high availability, patching, and scaling of the DNS infrastructure.

Fine-Grained Control
Use conditional forwarding to send only specific zones to Azure.

Final Thoughts

Azure Private DNS Resolver simplifies DNS management across hybrid environments. By setting up an inbound endpoint and configuring conditional forwarding, you can securely and efficiently resolve private Azure service endpoints from your on-premises network.

This setup is especially valuable for enterprise environments adopting Private Endpoints, Zero Trust Networking, and Hybrid Cloud Architectures.

🔧 Got questions or want help automating this setup with Terraform or Bicep? Let me know in the comments or connect with me!

What is a Disaster Recovery Strategy, and Why is it Important?

A Disaster Recovery (DR) strategy is a set of policies, tools, and procedures designed to restore IT services after a disruption. It ensures minimal downtime and data loss, helping businesses recover quickly from disasters. Without a DR plan, organizations risk severe financial and reputational damage due to prolonged service outages.

Real-World Example: British Airways IT Failure

In 2017, British Airways suffered a massive IT failure that led to the cancellation of over 400 flights, stranding thousands of passengers. The disruption reportedly cost the airline around 80 million ($102 million) in compensation, lost revenue, and reputational damage. Investigations suggested that the lack of a robust DR strategy contributed to the prolonged downtime, highlighting the critical need for businesses to have effective recovery mechanisms in place.

Understanding RTO and RPO

When planning a DR strategy, two key metrics must be considered:

• Recovery Time Objective (RTO): The maximum acceptable downtime before services must be restored. A lower RTO means faster recovery but often requires higher investment in infrastructure.

• Recovery Point Objective (RPO): The maximum acceptable data loss measured in time. A lower RPO ensures minimal data loss but requires frequent backups and replication.

Four Key Cloud Disaster Recovery Strategies

Organizations can implement different DR strategies based on their RTO and RPO requirements, balancing cost and recovery speed.

1. Backup and Restore

• Description: Periodic backups of data and applications stored in cloud storage, restored when needed.

• Pros: Cost-effective and simple.

• Cons: High RTO and RPO, slower recovery time.

• Best for: Small businesses or non-critical applications.

  

2. Pilot Light

• Description: A minimal version of the production environment is kept running with essential services. In case of a disaster, additional resources are quickly scaled up.

• Pros: Faster recovery compared to backup and restore, lower ongoing costs.

• Cons: Requires manual intervention for scaling up.

• Best for: Businesses needing moderate recovery speeds at a lower cost.

3. Warm Standby

• Description: A scaled-down but fully functional version of the production environment is always running. In an outage, resources are scaled up to full capacity.

• Pros: Faster recovery time with lower infrastructure costs compared to active-active.

• Cons: Higher costs than backup strategies, requires automation for scaling.

• Best for: Businesses requiring quick recovery but looking to save on costs.

4. Active-Passive (Hot Standby)

• Description: A fully operational duplicate environment is maintained, ready to take over instantly in case of failure.

• Pros: Near-instant recovery with minimal downtime.

• Cons: Expensive due to duplicated infrastructure.

• Best for: Mission-critical applications requiring high availability.

Conclusion

Choosing the right cloud disaster recovery strategy depends on business needs, budget, and acceptable downtime. A well-defined DR plan ensures resilience, minimizes losses, and maintains customer trust. As demonstrated by British Airways, the absence of a robust DR strategy can result in severe consequences. Organizations should assess their RTO and RPO needs and implement the most suitable DR approach to safeguard their cloud infrastructure against disruptions.

Do you have a DR strategy in place? If not, now is the time to start planning! 🚀

DevOps is fundamentally about breaking down silos, automating processes, and accelerating the delivery of reliable software. We strive for efficiency, speed, and robustness. In recent years, Artificial Intelligence (AI), particularly Large Language Models (LLMs) and tools like GitHub Copilot, have emerged as powerful allies promising to supercharge these efforts. They can generate code snippets, write configuration files, draft documentation, and even suggest troubleshooting steps.

However, simply having access to these AI tools isn't a magic bullet for productivity. The real key to unlocking their potential lies in prompt engineering: the art and science of crafting effective inputs (prompts) to guide the AI towards generating the desired, accurate, and useful output. For DevOps engineers, mastering prompt engineering is rapidly becoming a critical skill.

Why Prompt Engineering Matters in the DevOps Workflow

DevOps tasks are diverse and often complex, spanning coding, infrastructure management, networking, security, and operations. AI can assist across this spectrum, but its effectiveness is directly proportional to the quality of the prompt it receives.

• Faster Scripting and Automation: Need a script to automate backups, manage user permissions, or deploy an application? A well-crafted prompt can yield a near-complete script in seconds, saving hours of manual coding. A vague prompt might produce something unusable.

• Infrastructure as Code (IaC) Generation: Tools like Terraform, Pulumi, or CloudFormation require precise syntax. Prompting an AI with clear requirements (e.g., "Generate Terraform code for an AWS EC2 instance, t3.micro, in us-east-1, with security group X and specific tags") is far more effective than a generic request.

• Configuration Management: Generating configuration files for tools like Ansible, Chef, Puppet, Kubernetes, or Docker requires specifics. Good prompts include desired state, parameters, and constraints.

• Troubleshooting and Debugging: Asking an AI to "fix this error" is less helpful than providing the error message, relevant logs, the code snippet causing the issue, and the context of the system.

• Documentation: Generating READMEs, runbooks, or architecture diagrams requires clear instructions on the scope, audience, and key components to include.

The GitHub Copilot Experiment: A Case Study in Prompting

My own experience highlights the dramatic difference prompt quality can make. I needed a PowerShell script to create a basic Azure Web App and its supporting App Service Plan, a common task for deploying web applications.

Attempt 1: The Vague Request

My initial prompt to GitHub Copilot was straightforward, reflecting how one might initially approach the tool:

"Generate PowerShell using Az module to create an Azure Web App"

The code Copilot generated did use Azure PowerShell Az module cmdlets and would likely create an App Service Plan and Web App. However, it was far from production-ready or even development-ready without significant changes:

• It made assumptions about resource naming, likely using generic placeholders or requiring manual input during execution.

• It defaulted the Location (Region), potentially placing resources far from users or other dependent services.

• It defaulted the App Service Plan Sku (pricing tier), potentially choosing a more expensive or less performant tier than required.

• It didn't specify a runtime stack (like .NET, Node, Python), crucial for the application to function.

• It lacked parameterization for easy reuse and integration into larger automation scripts.

• Error handling (e.g., checking if resources already exist) was absent.

Modifying this code to meet specific requirements (correct names, location, SKU, runtime, parameters) took considerable time. I needed to know the correct PowerShell cmdlets (New-AzResourceGroup, New-AzAppServicePlan, New-AzWebApp) and their parameters anyway, largely defeating the purpose of using the AI for speed.

Attempt 2: The Guided Approach with Detailed Steps

Learning from the first attempt, I provided Copilot with context and a clear sequence of steps (All DevOps Engineers should learn how to write Pseudocode):

*"Generate PowerShell using Az module based on this logic:

1. Define variables: ResourceGroupName='MyWebAppRG', Location='AustraliaEast', PlanName='MyWebAppPlan', WebAppName='MyUniqueWebAppXYZ'.

2. Check if Resource Group '$ResourceGroupName' exists in '$Location'. If not, create it using New-AzResourceGroup.

3. Create an App Service Plan named '$PlanName' in '$ResourceGroupName' and '$Location' using the 'S1' Standard SKU (New-AzAppServicePlan).

4. Create a Web App named '$WebAppName' within the resource group, using the created App Service Plan. Specify the runtime as '.NET|6.0' (New-AzWebApp).

5. Add an Application Setting to the Web App: 'Environment' = 'Development'.

6. Output the default hostname of the created Web App."*

The result was drastically different. The PowerShell code generated by Copilot using this prompt was approximately 95% accurate and immediately usable with minor verification:

• It followed the logical steps outlined.

• It used the specified variables for names, location, and SKU.

• It included a basic check for the resource group's existence.

• It correctly used New-AzResourceGroup, New-AzAppServicePlan, and New-AzWebApp with the right parameters, including the SKU and runtime stack.

• It added the specified application setting.

• It included a command to output the hostname.

The debugging and refinement time was minimal. The AI, guided by a structured, detailed prompt that specified what and how, acted as a highly effective accelerator.

The Foundation: Why Domain Knowledge Remains Crucial

This experiment underscores a vital point: using AI effectively in DevOps depends heavily on strong foundational knowledge. You can't prompt effectively if you don't understand the underlying concepts.

1. Programming Language Proficiency (e.g., PowerShell, Python, Bash): To write effective pseudocode or detailed prompts, you need to understand control flow, variables, functions, error handling, and the specific commands or libraries relevant to the task. You also need this knowledge to evaluate and debug the AI's output.

2. Networking Concepts: When asking for scripts or configurations involving firewalls, load balancers, DNS, or VPCs, understanding subnets, routing, ports, and protocols is essential for crafting a precise prompt and validating the result.

3. Operating System Internals: Tasks involving performance tuning, service management, user permissions, or file systems require an understanding of how the OS works. This knowledge informs the prompts for configuration management or troubleshooting scripts.

4. Cloud/Infrastructure Knowledge: Understanding the specific services, APIs, and best practices of your cloud provider (AWS, Azure, GCP) or virtualization platform is critical for generating accurate IaC or automation scripts.

Conclusion: AI as a Co-Pilot, Not Autopilot

AI tools like GitHub Copilot are transformative for DevOps engineers, offering significant potential to boost productivity and automate repetitive tasks. However, they are most powerful when wielded by engineers who understand what they are asking for and how to ask for it effectively.

Prompt engineering isn't just about fancy wording; it's about leveraging your existing technical expertise to provide the AI with the context, constraints, and structure it needs to generate high-quality output. By combining solid foundational knowledge in programming, networking, OS, and cloud systems with skillful prompt engineering, DevOps professionals can truly harness the power of AI, turning it from a novelty into an indispensable part of their toolkit for building and operating systems faster and more reliably than ever before. The future of efficient DevOps involves not just using AI, but mastering the conversation with it.

When managing cloud infrastructure in Azure, network connectivity issues can significantly impact application availability and performance. Azure provides powerful tools for diagnosing and troubleshooting network problems, including Virtual Network (VNet) Flow Logs and IP Flow Verify in Network Watcher. These tools help network engineers and cloud administrators gain visibility into network traffic and quickly pinpoint connectivity issues.

What Are VNet Flow Logs?

VNet Flow Logs capture information about inbound and outbound traffic within a Virtual Network (VNet). These logs provide insights into:

• Source and destination IP addresses

• Ports and protocols used

• Traffic direction (inbound or outbound)

• NSG rule that allowed or denied the traffic

• Flow start and end times

VNet Flow Logs are stored in Azure Storage Accounts and can be analyzed using Azure Monitor, Log Analytics, or third-party tools like Splunk. They help in diagnosing network latency, packet drops, and misconfigurations in NSGs.

Enabling VNet Flow Logs

To enable VNet Flow Logs, follow these steps:

1. Open the Azure Portal and navigate to Virtual Networks.

2. Select the VNet where you want to enable flow logs.

3. In the left-hand menu (blade), scroll down to find VNet Flow Logs and click on it.

4. Click Enable Flow Logs.

5. Choose a Storage Account to store the logs.

6. Select Enable Traffic Analytics.

7. Choose the Log Analytics Workspace where you want to send logs.

8. Select the specific logs you want to send to the workspace.

9. Click Save to apply the settings.

Using VNet Flow Logs for Troubleshooting

1. Diagnosing Dropped Traffic

By analyzing VNet Flow Logs, you can determine if network traffic is being dropped due to NSG rules. Example:

• If an application is not accessible, check the logs to see if traffic from the client IP is being denied by an NSG rule.

• You can identify if traffic is being routed correctly or if a misconfiguration is blocking access.

2. Identifying Unauthorized Access Attempts

Flow Logs help in identifying suspicious activities, such as repeated failed connection attempts from unknown IP addresses, which could indicate brute-force attacks or unauthorized access attempts.

3. Monitoring Traffic Patterns

By aggregating VNet Flow Logs over time, you can analyze traffic trends, detect anomalies, and optimize NSG rules to allow only necessary traffic while blocking potential threats.

Querying Flow Logs in Log Analytics

To query VNet Flow Logs in Log Analytics, follow these steps:

1. Navigate to Azure Monitor > Logs.

2. Select your Log Analytics Workspace.

3. Use the following Kusto Query Language (KQL) query to analyze traffic entering and leaving the network:

NTANetAnalytics
| where FlowType_s == "VNetFlow"
| project TimeGenerated, SourceIP_s, DestinationIP_s, DestinationPort_d, Protocol_s, Action_s
| sort by TimeGenerated desc

• To filter allowed traffic, modify the query:

| where Action_s == "Allow"

• To filter denied traffic, modify the query:

| where Action_s == "Deny"

Querying Traffic Analytics in Log Analytics Workbook

To get a more detailed view of network traffic at the VNet and NSG levels, you can query the NTANetAnalytics table in Log Analytics:

Traffic at VNet Level:

NTANetAnalytics
| where FlowType_s == "VNetTraffic"
| summarize TotalTraffic = sum(TotalBytes_d) by VnetName_s, TimeGenerated
| order by TotalTraffic desc

Traffic at NSG Level:

NTANetAnalytics
| where FlowType_s == "NSGTraffic"
| summarize AllowedTraffic = sum(case(Action_s == "Allow", TotalBytes_d, 0)), 
          DeniedTraffic = sum(case(Action_s == "Deny", TotalBytes_d, 0)) 
  by NSGName_s, TimeGenerated
| order by DeniedTraffic desc

These queries help in understanding traffic volume and security rule enforcement at both the VNet and NSG levels.

IP Flow Verify in Network Watcher

In addition to VNet Flow Logs, IP Flow Verify in Azure Network Watcher allows you to test whether a specific IP flow is allowed or denied based on the configured NSG rules.

How to Use IP Flow Verify

1. Open Azure Network Watcher in the Azure Portal.

2. Select IP Flow Verify.

3. Choose the Virtual Machine to test.

4. Enter the Source and Destination IP, Port, and Protocol.

5. Click Check to see if the traffic is allowed or denied.

Use Cases of IP Flow Verify

• Quickly verifying whether an NSG rule is blocking or allowing traffic without waiting for logs to update.

• Debugging connectivity issues when deploying new applications or modifying NSG rules.

• Ensuring compliance with security policies by testing network access.

Conclusion

VNet Flow Logs and IP Flow Verify are essential tools for network troubleshooting in Azure. VNet Flow Logs provide historical data and deep traffic analysis, while IP Flow Verify offers real-time validation of NSG rules. By leveraging these tools, cloud administrators can efficiently diagnose and resolve network issues, improve security, and optimize network performance in Azure.

Would you like to explore how to automate network monitoring using Azure PowerShell or Azure Monitor? Let me know in the comments!

Managing Azure Virtual Machines (VMs) manually can be time-consuming, especially for tasks like shutting down idle resources to save costs. Azure Automation Account allows you to automate these operations using Runbooks.

In this guide, we will:
Create an Azure Automation Account
Set up a User Assigned Managed Identity (UMI)
Assign VM Contributor role to the identity
Write and execute a Runbook to stop VMs
Schedule the Runbook for automatic execution

We will use PowerShell in the Runbook and leverage Azure CLI for setup.

1 Create an Azure Automation Account

An Automation Account is required to manage and run scripts in Azure.

Steps:

1. Go to the Azure Portal Search for Automation Accounts.

2. Click Create and provide the following details:

   • Subscription: Select your subscription

   • Resource Group: Create or select an existing one

   • Automation Account Name: e.g., MyAutomationAccount

   • Region: Choose the preferred region

   • Managed Identity: Select None (well create a separate UMI later)

3. Click Review + Create Create.

4. Please find the screenshot of the Azure Automation Account I created.

2 Create a User-Assigned Managed Identity (UMI)

A User Assigned Managed Identity (UMI) allows our Automation Account to authenticate and execute actions without storing credentials.

Steps:

1. Go to Azure Portal Search for Managed Identities.

2. Click Create and enter:

   • Subscription & Resource Group: Select same as Automation Account

   • Name: e.g., MyAutomationIdentity

   • Region: Choose the same as Automation Account

3. Click Review + Create Create.

3 Assign "VM Contributor" Role to the Managed Identity

The VM Contributor role allows our Runbook to manage VMs (start, stop, deallocate, etc.).

Steps:

1. Go to Azure Portal Open the Managed Identity created earlier.

2. Navigate to Access control (IAM) Click Add role assignment.

3. Select:

   • Role: Virtual Machine Contributor

   • Scope: Subscription (or specific resource group if needed)

   • Assign access to: Managed Identity

   • Identity: Select autoreizeum

4. Click Save.

5. You can see in the image below that created User Managed Identity has VM contributor role assigned.

4 Create and Publish a Runbook to Stop VMs

Now, well create a PowerShell Runbook that will connect to Azure using the Managed Identity and stop all VMs.

Steps:

1. Open Azure Portal Go to your Automation Account.

2. Click Runbooks Create a runbook.

3. Provide:

   • Name: Stop-All-VMs

   • Runbook Type: PowerShell

   • Runtime Version: Latest

4. Click Create.

5. Edit the Runbook Paste the following PowerShell script:

Find the script below.

### this is final runbook script which will stop VMs. Only those VMs with autostop = true

 # Input parameters
 param(


     [Parameter(Mandatory = $true)]
     [string]$SubscriptionId = "Enter your sunscriptionID",

     [Parameter(Mandatory = $true)]
     [string]$UserAssignedIdentityClientId = "Enter your user managed Identity"

 )


 try {
     # Ensures you do not inherit an AzContext in your runbook
     Disable-AzContextAutosave -Scope Process

     # Connect using user-assigned managed identity
     Write-Output "Connecting to Azure using User-Assigned Managed Identity..."
     Connect-AzAccount -Identity -AccountId $UserAssignedIdentityClientId

     # Set context to your subscription
     Write-Output "Setting context to subscription: $SubscriptionId"
     Set-AzContext -SubscriptionId $SubscriptionId

     # Get all VMs in the subscription
     Write-Output "Getting all VMs in the subscription..."
     $vms = Get-AzVM

     # Check if any VMs were found
     if ($null -eq $vms -or $vms.Count -eq 0) {
         Write-Output "No VMs found."
         return
     }

     Write-Output "Found $($vms.Count) VMs"

     # Stop VMs with autostop = true tag
     foreach ($vm in $vms) {
         try {
             # Check for autostop tag
             $autostop = $vm.Tags["autostop"]

             if ($autostop -eq "true") {  # Case-insensitive comparison
                 Write-Output "Stopping VM: $($vm.Name) in resource group: $($vm.ResourceGroupName) (autostop tag present)"
                 $stopResult = Stop-AzVM -ResourceGroupName $vm.ResourceGroupName -Name $vm.Name -Force

                 if ($stopResult.Status -eq "Succeeded") {
                     Write-Output "Successfully stopped VM: $($vm.Name)"
                 } else {
                     Write-Error "Failed to stop VM: $($vm.Name). Status: $($stopResult.Status)"
                 }
             } else {
                 Write-Output "Skipping VM: $($vm.Name) in resource group: $($vm.ResourceGroupName) (autostop tag NOT present or not 'true')"
             }
         } catch {
             Write-Error "Error processing VM $($vm.Name): $_"
             continue # Continue to the next VM even if one fails
         }
     }

     Write-Output "VM stop operations completed"

 } catch {
     Write-Error "Error in runbook execution: $_"
     throw $_
 } finally {
     # Clean up authentication context
     Write-Output "Cleaning up Azure context..."
     Clear-AzContext -Force
 }

🔥 Alternatively, you can get the full script from your GitHub repo:
GitHub Repo: final-runbookstop.ps1

6. Click Save Publish.

5 Create a Schedule for the Runbook

To automate the Runbook execution, well create a schedule.

Steps:

1. Open Azure Portal Go to your Automation Account.

2. Select Runbooks Click on Stop-All-VMs.

3. Click Schedules Add a Schedule.

4. Select Create a new schedule, and provide:

   • Name: Daily-VM-Shutdown

   • Recurrence: Daily

   • Time: Choose a suitable time (e.g., 10 PM UTC)

5. Click Create.

6. Under Parameters and run settings, ensure the default values are correct.

7. Click OK to save.

6 Testing and Verification

To ensure everything is working:
Schedule triggers the Runbook at specified schedule.
Check VM Status Go to Azure Portal Open Virtual Machines and verify the VMs are stopping.
Check Logs View Job Output in the Runbook to see if any errors occurred.

Mode             : Process

ContextDirectory : None

ContextFile      : None

CacheDirectory   : None

CacheFile        : None

KeyStoreFile     : None

Settings         : {[InstallationId, 102b9476-d6c9-4337-870a-4149300e7a15]}



Connecting to Azure using User-Assigned Managed Identity...





Environments                                                                                           Context

------------                                                                                           -------

{[AzureChinaCloud, AzureChinaCloud], [AzureUSGovernment, AzureUSGovernment], [AzureCloud, AzureCloud]} Microsoft.Azure.



Setting context to subscription: 6baeb535-5ac9-402f-83c4-4aed96077df6






Getting all VMs in the subscription...



Found 2 VMs



Skipping VM: backend1-vm in resource group: RUNBOOKRG (autostop tag NOT present or not 'true')



Stopping VM: backend2-vm in resource group: RUNBOOKRG (autostop tag present)



Successfully stopped VM: backend2-vm

VM stop operations completed

Cleaning up Azure context...

Above is the output of the Job in Azure.

Conclusion

🎯 We successfully automated Azure VM shutdown using Azure Automation Account and Runbooks.
🎯 We used a User Assigned Managed Identity to authenticate securely.
🎯 We scheduled the Runbook for automatic execution.

🎯 We only shutdown the VMs with tag autoshutdown = true

In the fast-paced world of DevOps, where automation and efficiency are key, pseudocode serves as a vital bridge between ideas and implementation. While DevOps engineers often deal with complex scripts and tools, pseudocode provides a simplified, language-agnostic way to design and communicate automation workflows. In this article, we explore why pseudocode is essential for DevOps engineers and how it can streamline automation processes.

What is Pseudocode?

Pseudocode is a high-level representation of an algorithm or workflow that uses plain language and programming-like structure. Its not bound by the syntax of any specific programming language, making it easy to understand for both technical and non-technical stakeholders.

Example:

START
  Define variable "serverList"
  For each "server" in "serverList":
    Check if "server" is running
    If "server" is not running:
      Restart "server"
    End If
  End For
END

This simple pseudocode outlines a server monitoring and restart process without delving into specific syntax.

Why Pseudocode Matters for DevOps Engineers

1. Clarity in Automation Design

DevOps engineers often work on intricate automation pipelines, including CI/CD workflows, infrastructure provisioning, and monitoring. Pseudocode helps:

• Break down complex tasks into manageable steps.

• Clearly define the logic before diving into code.

• Ensure that all team members understand the workflow, regardless of their familiarity with the programming language.

2. Facilitating Collaboration

DevOps projects typically involve cross-functional teams, including developers, operations, and sometimes even business stakeholders. Pseudocode:

• Acts as a universal language to communicate ideas.

• Enables non-technical team members to provide input on workflows.

• Reduces misunderstandings during the planning phase.

3. Error Reduction

By focusing on logic rather than syntax, pseudocode allows engineers to:

• Identify potential issues early in the design phase.

• Avoid common pitfalls in automation scripts.

• Ensure the workflow aligns with the intended goals.

4. Reusable Templates

Pseudocode can serve as a reusable blueprint for similar automation tasks. For instance, a pseudocode template for deploying infrastructure can be adapted for different environments or tools.

Applications of Pseudocode in DevOps Automation

1. Infrastructure as Code (IaC)

Before writing Terraform or Ansible scripts, pseudocode can outline the desired state of the infrastructure:

START
  Define "resource group"
  Create "virtual network"
  For each "subnet" in "subnet list":
    Create "subnet"
  End For
END

2. CI/CD Pipelines

For complex pipelines, pseudocode helps visualize stages and dependencies:

START
  Trigger pipeline on "code commit"
  Run "unit tests"
  If "tests pass":
    Build application
    Deploy to "staging"
    Run "integration tests"
    If "integration tests pass":
      Deploy to "production"
    End If
  End If
END

3. Incident Response Automation

Pseudocode can outline automated responses to system alerts:

START
  Monitor "system metrics"
  If "CPU usage > 80%":
    Notify "on-call engineer"
    Scale up "instances"
  End If
END

Best Practices for Writing Pseudocode

1. Keep It Simple: Avoid unnecessary details; focus on the logic.

2. Use Consistent Structure: Follow a logical flow with clear start and end points.

3. Be Language-Agnostic: Avoid syntax or keywords specific to any programming language.

4. Focus on Readability: Use meaningful variable names and clear indentation.

5. Iterate and Refine: Review and update pseudocode as requirements evolve.

Conclusion

For DevOps engineers, pseudocode is more than just a planning toolits a way to ensure clarity, collaboration, and efficiency in automation projects. By using pseudocode to design workflows, teams can reduce errors, streamline development, and create reusable templates for future tasks. Whether youre automating infrastructure, building CI/CD pipelines, or responding to incidents, pseudocode is an indispensable part of the DevOps toolkit.

Start incorporating pseudocode into your workflow today and experience the difference it makes in simplifying complex automation tasks!

In the world of containerization, Docker has become one of the most popular tools for deploying applications. However, one of the key challenges when working with containers is managing data. By default, Docker containers are ephemeral, meaning that any data stored within them is lost when the container is stopped or removed. This is where Docker volumes come into play, providing a way to persist data across container restarts and removals. In this blog, well dive into persistent volumes in Docker and explain why their lifecycle is independent of containers.

What Are Docker Volumes?

A Docker volume is a storage mechanism that allows data to persist beyond the lifecycle of a container. Volumes are stored outside of the containers filesystem, meaning they are not removed when a container is stopped or deleted. This makes them an essential tool for managing persistent data such as databases, logs, and application state in containerized environments.

Docker provides two types of storage options for containers:

1. Volumes: Managed by Docker and stored in a specific directory on the host filesystem.

2. Bind mounts: Map a host directory to a container directory, allowing the container to access files on the host machine.

While bind mounts are useful for development purposes, volumes are the preferred method for storing persistent data in production environments because Docker manages them and they are isolated from the host filesystem.

Why Is the Volume Lifecycle Independent of Containers?

One of the most important features of Docker volumes is that their lifecycle is independent of the containers that use them. This means that even if you stop or remove a container, the volume will continue to exist and can be reused by other containers.

Heres why this is crucial:

1. Data Persistence: Since volumes are not tied to the lifecycle of a container, they provide a reliable way to persist data. For example, if you have a database running inside a container and the container crashes or is deleted, the data stored in the volume will remain intact. You can then spin up a new container and mount the same volume to continue where you left off.

2. Reusability: Volumes can be reused by multiple containers. This is especially useful in microservices architectures, where different services might need to access the same data. For example, you could have a web application and a database container that both use the same volume to share data.

3. Backup and Restore: Since volumes are independent of containers, you can easily back up and restore data stored in volumes. This can be done without worrying about the state of the container itself, providing a more reliable backup solution.

4. Container Portability: Volumes allow you to decouple the data from the container, making it easier to move containers between different environments. For example, you can move a container from your local development machine to a production environment, and the data in the volume will remain accessible.

Creating and Using a Docker Volume with my_nginx custom image

Creating and Using a Docker Volume with my_nginx

To demonstrate how Docker volumes work, lets create a volume and use it with custom image container to serve a website.

1. Commands to create docker image with custom nginx image:

## create a docker volume
docker volume create my_data

## mount the volume to docker container with nginx image
 docker run -d --name my_nginx -v my_data:/data -p 8080:80 custom_nginx

Observing the Volume

In the images below, you can see:

1. The my_data folder on the Docker host, where the volume data is stored.

2. The /data folder inside the container, which is linked to the my_data volume.

Although the folder appears within the containers filesystem, the data itself resides on the Docker host, making the volume's lifecycle independent of the container.

The above is my_data folder on docker host

the above is /data folder inside the container. the folder appears inside the container file system. however the file exist outside the container. the lifecycle of that folder is independent of container.

Testing the Website

Now, lets verify that the container is serving the website. The container uses the index.html file stored in the volume to serve the website.

Step 3: Delete and Recreate the Container

To test the persistence of the volume, well delete the container and create a new one using the same volume.

## delete the container
docker stop my_nginx
docker rm my_nginx

## create a new container on port 8081
docker run -d --name new_nginx -v my_data:/data -p 8081:80 custom_nginx

Observing the Result

After deleting the original container and creating a new one, youll notice the same website is being served. This is because the website data resides in the my_data volume on the Docker host, which is independent of any specific container.

This demonstrates how Docker volumes ensure data persistence, allowing you to manage containerized applications without worrying about data loss when containers are stopped or recreated.

Conclusion

Docker volumes are an essential tool for managing persistent data in containerized applications. The key advantage of volumes is that their lifecycle is independent of the container, ensuring that data is not lost when containers are stopped or removed. This makes them ideal for use cases where data persistence is crucial, such as databases and application state. By leveraging Docker volumes, you can ensure your containers are both portable and resilient, providing a more reliable way to manage data in a containerized environment.

By understanding and using volumes effectively, you can take full advantage of Dockers capabilities and build robust, scalable applications that can run seamlessly across different environments.

In this article, I will deploy Terraform resources using Azure DevOps. Please find the code in below repo

Code repo for this project

Define Variables

To begin, we need to create a variable called storagekey. The value of this variable will be empty initially and will be dynamically fetched by the pipeline using a PowerShell script. Make sure to mark the variable as "Settable at release time."

Add Azure PowerShell script: InlineScript task

In this task, include the PowerShell script that fetches the storage account key and passes it to your pipeline.

$key = (Get-AzStorageAccountKey -ResourceGroupName $(terraformrg) -AccountName $(terraformstorageaccount))[0].Value

Write-Host "##vso[task.setvariable variable=storagekey]$key"

This script retrieves the storage account key dynamically and stores it in the pipeline variable storagekey.

Replace Tokens Task

The Replace Tokens task allows you to dynamically replace placeholders in your code with pipeline variables. Follow these steps:

1. Specify Source Code and Target Files:

   • In the task configuration, define the source code directory and the target files where the placeholders need to be replaced.

2. Set Prefix and Suffix:

   • Use the same prefix and suffix as defined in your code. For example, if your placeholders look like __storagekey__, set the prefix to and the suffix to __.

When the pipeline runs, the Replace Tokens task will substitute the placeholders in the target files with the values of the corresponding pipeline variables.

Why This Approach Works

By dynamically fetching the storage account secret key and passing it securely through the pipeline:

• Security: Secrets are not hardcoded, reducing the risk of exposure.

• Automation: The pipeline can handle key rotations without manual intervention.

• Flexibility: The Replace Tokens task ensures that the updated key is seamlessly integrated into your code.

Conclusion

This method allows you to securely and dynamically manage storage account secret keys in Azure DevOps pipelines. When keys are recreated, the pipeline remains unaffected, ensuring smooth and secure operations. By following these steps, you can enhance the security and automation of your DevOps workflows.

1. What Are Azure Data Transfer Costs?

Azure data transfer costs are associated with moving data between services, regions, and networks. These costs are typically categorized as:

• Ingress (Data In): Data entering Azure, usually free.

• Egress (Data Out): Data leaving Azure, such as to the internet or other Azure regions, incurs charges.

2. Types of Azure Data Transfer Scenarios

Azure data transfer costs vary based on the source, destination, and type of transfer. Heres a breakdown:

a. Data Transfer Within a Virtual Network (VNET)

• Data transfers within the same VNET are free as long as they occur within the same subnet or between subnets inside the VNET.

• Example: Communication between two VMs in the same VNET.

b. VNET Peering

1. Regional VNET Peering (Same Region):

   • Transfers between two VNETs in the same region incur a charge of $0.01 per GB for both ingress and egress data.

2. Global VNET Peering (Different Regions):

   • Data transfer costs depend on the zones where the VNETs are located:

     • Zone 1: $0.035 per GB (inbound and outbound).

     • Zone 2: $0.09 per GB.

     • Zone 3: $0.16 per GB.

     • US Gov: $0.044 per GB.

c. Data Transfer Between Availability Zones

• Within the Same Zone: Free.

• Between Different Zones: Charged at $0.01 per GB for both ingress and egress, even if the resources are in the same VNET.

d. Data Transfer Across Azure Regions

• Transfers between regions incur charges based on the originating and destination zones:

  • Example: Intra-continental transfers in North America cost $0.02 per GB.

e. ExpressRoute

• Provides a private connection between Azure and on-premises infrastructure.

• Offers cost-effective options for high-volume data transfers compared to VPN egress charges.

3. Key Scenarios Impacting Data Transfer Costs

a. Content Delivery

• Serving large files (e.g., videos, images) to users incurs significant egress costs.

b. Cross-Region Replication

• Geo-redundant backups or disaster recovery setups involve inter-region data transfers, increasing costs.

c. Multi-Region Applications

• Applications with components deployed across regions result in frequent inter-region data movement.

d. Hybrid Architectures

• Data flowing between Azure and on-premises systems can lead to substantial egress charges.

4. Optimization Strategies for Data Egress Costs

To control and reduce data transfer costs, consider the following best practices:

a. Deploy Resources Strategically

• Place resources in regions with minimal or no data transfer costs unless compliance or performance mandates otherwise.

b. Limit Cross-Zone and Cross-Region Transfers

• Keep data movement within the same zone or region to avoid unnecessary charges.

c. Optimize Application Architecture

• Design applications to minimize data movement between regions and zones.

d. Leverage ExpressRoute

• For high-volume transfers, use ExpressRoute for cost-effective, private connections.

e. Use Compression and Deduplication

• Compress data and use incremental synchronization to reduce the amount of data being transferred.

f. Archive or Delete Unnecessary Data

• Regularly clean up unused or redundant data to avoid unnecessary storage and transfer costs.

g. Use Azure CDN

• Cache frequently accessed data closer to users to reduce egress charges.

5. Monitoring and Managing Azure Data Transfer Costs

Azure provides tools to help monitor and manage data transfer expenses:

a. Azure Cost Management and Billing

• Offers detailed insights into spending, including data transfer costs.

b. Azure Monitor

• Tracks network usage and identifies cost-heavy data flows.

c. Azure Pricing Calculator

• Helps estimate costs for different data transfer scenarios.

6. Example: Multi-Region Web Application

Scenario:
When running a web app like https://www.clouddevopsinsights.com/ on Azure with a virtual machine (VM) and an Azure Application Gateway as a Layer 7 load balancer, the data transfer cost for your organization depends on several factors, including the origin and destination of the data, the traffic path, and Azure's pricing structure. Here's a detailed explanation:

1. Data Transfer Scenarios in Your Setup

Scenario 1: Internet to Application Gateway

• Description: When a customer accesses your web app from their smartphone via the internet, the request first reaches the Azure Application Gateway.

• Cost:

  • Inbound Data (Ingress): Free. Azure does not charge for inbound data traffic from the internet to Azure services.

  • Outbound Data (Egress): Charged. Azure charges for outbound data from Azure services to the internet. The rate depends on the amount of data transferred and the region.

Scenario 2: Application Gateway to VM

• Description: The Application Gateway forwards the request to your backend VM hosting the web app.

• Cost: Free. Data transfer within the same Azure region between the Application Gateway and VM is free.

Scenario 3: VM Response to Application Gateway

• Description: The VM processes the request and sends the response back to the Application Gateway.

• Cost: Free. Data transfer within the same region remains free.

Scenario 4: Application Gateway to Internet

• Description: The Application Gateway sends the response to the customer's smartphone.

• Cost:

  • Outbound Data (Egress): Charged. This is considered internet egress, and Azure applies charges based on the volume of data sent to the customer and the region of your Azure resources.

2. Cost Breakdown

Outbound Data Transfer (Egress) Costs

Azure charges for outbound data transfer based on:

• Data Volume: The amount of data sent from your app to the customer.

• Region: The region where your Azure resources (Application Gateway and VM) are hosted.

• Pricing Tiers:

  • For example, in most regions:

    • The first 5 GB per month is free.

    • 5 GB10 TB per month is charged at $0.087 per GB.

    • 10 TB50 TB per month is charged at $0.083 per GB.

    • Larger volumes have reduced rates.

Example Calculation:

Assume:

• Your app serves 1,000 customers per day.

• Each customer downloads 10 MB of data.

• Monthly outbound data: 1,00010MB30days=300,000MB=300GB1,000 \times 10 \, \text{MB} \times 30 \, \text{days} = 300,000 \, \text{MB} = 300 \, \text{GB}1,00010MB30days=300,000MB=300GB.

Cost:

• First 5 GB: Free.

• Remaining 295 GB: 295GB0.087$/GB=25.67$295 \, \text{GB} \times 0.087 \, \text{\$/GB} = 25.67 \, \text{\$}295GB0.087$/GB=25.67$.

Total monthly cost: $25.67 for outbound data transfer.

7. Conclusion

Azure data transfer costs, while often overlooked, can significantly impact your cloud budget. By understanding the pricing structure and optimizing your architecture, you can reduce unnecessary expenses while maintaining performance and reliability.

Would you like assistance in designing a cost-efficient Azure architecture? Let us know in the comments

What is Terraform Count?

The count meta-argument is a simple way to create multiple instances of a resource. By specifying a numeric value for count, Terraform will create that many instances of the resource.

Example:

resource "azurerm_virtual_machine" "example" {
  count = 3

  name                  = "vm-${count.index}"
  location              = azurerm_resource_group.example.location
  resource_group_name   = azurerm_resource_group.example.name
  network_interface_ids = [azurerm_network_interface.example[count.index].id]
  vm_size               = "Standard_DS1_v2"
}

In this example:

• Terraform creates three virtual machines.

• The count.index value (0, 1, 2) is used to generate unique names for each instance.

Key Features of Count:

• Index-based: Resources are identified by their index.

• Simple: Best suited for scenarios where all instances share similar configurations.

• Limitations: Less flexible when dealing with heterogeneous resource configurations.

What is Terraform For_each?

The for_each meta-argument is more flexible and allows you to create resources based on a set, map, or list. Each instance is uniquely identified by a key rather than an index.

Example:

 terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "=3.47.0"
    }
  }
}

#https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/guides/service_principal_client_secret
provider "azurerm" {
  features {} 
  client_id       = "00000000-0000-0000-0000-000000000000"
  client_secret   = "20000000-0000-0000-0000-000000000000"
  tenant_id       = "10000000-0000-0000-0000-000000000000"
  subscription_id = "20000000-0000-0000-0000-000000000000"
}
#variables are declared here
variable "resourcedetails" {
  type = map(object({
    name     = string
    location = string
    size     = string
    rg_name  = string
    vnet_name = string
    subnet_name = string
  }))
  default = {
    westus = {
      rg_name  = "westus-rg"  
      name     = "west-vm"
      location = "westus2"
      size     = "Standard_B2s"
      vnet_name = "west-vnet"
      subnet_name = "west-subnet"
    }
    eastus = {
      rg_name  = "eastus-rg"  
      name     = "east-vm"
      location = "eastus"
      size     = "Standard_B1s"
      vnet_name = "east-vnet"
      subnet_name = "east-subnet"
    }
  }
}


resource "azurerm_resource_group" "myrg" {
  for_each = var.resourcedetails

  name     = each.value.rg_name
  location = each.value.location
}

resource "azurerm_virtual_network" "myvnet" {
  for_each = var.resourcedetails
  name                = each.value.vnet_name
  address_space       = ["10.0.0.0/16"]
  location            = azurerm_resource_group.myrg[each.key].location
  resource_group_name = azurerm_resource_group.myrg[each.key].name
}

resource "azurerm_subnet" "mysubnet" {
  for_each = var.resourcedetails

  name                 = each.value.subnet_name
  address_prefixes     = ["10.0.0.0/24"]
  virtual_network_name = azurerm_virtual_network.myvnet[each.key].name
  resource_group_name  = azurerm_resource_group.myrg[each.key].name
}

resource "azurerm_network_interface" "mynic" {
  for_each = var.resourcedetails

  name                = "my-nic"  
  location            = azurerm_resource_group.myrg[each.key].location
  resource_group_name = azurerm_resource_group.myrg[each.key].name
  ip_configuration {
    name                          = "my-ip-config"
    subnet_id                     = azurerm_subnet.mysubnet[each.key].id
    private_ip_address_allocation = "Dynamic"
  }
}


resource "azurerm_virtual_machine" "vm" {
  for_each = var.resourcedetails

  name                  = each.value.name
  location            = azurerm_resource_group.myrg[each.key].location
  resource_group_name = azurerm_resource_group.myrg[each.key].name
  network_interface_ids = [azurerm_network_interface.mynic[each.key].id]
  vm_size               = each.value.size

  storage_image_reference {
    publisher = "Canonical"
    offer     = "UbuntuServer"
    sku       = "16.04-LTS"
    version   = "latest"
  }

  storage_os_disk {
    name              = "${each.value.name}-osdisk"
    caching           = "ReadWrite"
    create_option     = "FromImage"
    managed_disk_type = "Standard_LRS"
  }

  os_profile {
    computer_name  = each.value.name
    admin_username = "adminuser"
    admin_password = "Password1234!"
  }

  os_profile_linux_config {
    disable_password_authentication = false
  }


}

In this example:

• Two virtual machines are created, each with a unique name and size

Key Features of For_each:

• Key-based: Resources are identified by unique keys.

• Flexible: Ideal for scenarios where instances require different configurations.

• Dynamic: Can adapt to changes in the input set or map.

Key Differences Between Count and For_each

When to Use Count

• All resources have similar configurations.

• You know the exact number of resources required upfront.

• Simpler scenarios where flexibility is not a priority.

Example Use Case:

Creating multiple identical storage accounts.

resource "azurerm_storage_account" "example" {
  count = 5

  name                     = "storage${count.index}"
  resource_group_name      = azurerm_resource_group.example.name
  location                 = azurerm_resource_group.example.location
  account_tier             = "Standard"
  account_replication_type = "LRS"
}

When to Use For_each

• Resources require unique configurations.

• You need to manage resources based on dynamic or changing input data.

• Resources need to be uniquely identified by a key.

Example Use Case:

Creating VMs with different configurations for a development and production environment.

resource "azurerm_virtual_machine" "example" {
  for_each = {
    dev  = "Standard_DS1_v2"
    prod = "Standard_DS2_v2"
  }

  name                  = each.key
  location              = azurerm_resource_group.example.location
  resource_group_name   = azurerm_resource_group.example.name
  vm_size               = each.value
}

Conclusion

Both count and for_each are powerful tools in Terraform for managing multiple resources, but they serve different purposes. Use count for simple, uniform resource creation and for_each for more complex, dynamic scenarios. Understanding their differences and use cases will help you write more efficient and maintainable Terraform configurations.

By leveraging the right construct for the right situation, you can optimize your Infrastructure as Code workflows and better manage your cloud resources.

Terraform is a powerful Infrastructure-as-Code (IaC) tool that allows you to manage your cloud infrastructure declaratively. However, scenarios can arise where resources created with Terraform are manually modified in the Azure Portal or via other means. This can lead to a mismatch, or "drift," between Terraforms state and the actual infrastructure.

In this blog, well explore how to handle such situations effectively. Lets consider a scenario where a Virtual Machine (VM) and a Network Security Group (NSG) were created using Terraform but were later manually modified in Azure. For example, additional rules were added to the NSG.

Infrastructure Deployed:

I deployed Linux VM and associated nsg to the VM-Nic.

resource "azurerm_virtual_network" "vnet-clouddevinsights" {
  name                = var.virtual_network_name
  address_space       = var.address_space
  location            = var.resource_group_location
  resource_group_name = var.resource_group_name
}

resource "azurerm_subnet" "vnet-clouddevinsights-subnet" {
  name                 = var.subnet_name
  resource_group_name  = azurerm_resource_group.clouddevinsights.name
  virtual_network_name = azurerm_virtual_network.vnet-clouddevinsights.name
  address_prefixes     = var.subnet_address_prefix
}

resource "azurerm_network_security_group" "nsg-clouddevinsights-nsg" {
  name                = var.network_security_group_name
  location            = var.resource_group_location
  resource_group_name = azurerm_resource_group.clouddevinsights.name

  security_rule {
    name                       = "Allow-SSH"
    priority                   = 1001
    direction                  = "Inbound"
    access                     = "Allow"
    protocol                   = "Tcp"
    source_port_range          = "*"
    destination_port_range     = "22"
    source_address_prefix      = "*"
    destination_address_prefix = "*"
  }

  security_rule {
    name                       = "Allow-HTTP"
    priority                   = 1002
    direction                  = "Inbound"
    access                     = "Allow"
    protocol                   = "Tcp"
    source_port_range          = "*"
    destination_port_range     = "80"
    source_address_prefix      = "*"
    destination_address_prefix = "*"
  }

  security_rule {
    name                       = "Allow-HTTPS"
    priority                   = 1003
    direction                  = "Inbound"
    access                     = "Allow"
    protocol                   = "Tcp"
    source_port_range          = "*"
    destination_port_range     = "443"
    source_address_prefix      = "*"
    destination_address_prefix = "*"
  }
}

resource "azurerm_network_interface" "vm-nic" {
  name                = var.vm-nic-name
  location            = azurerm_resource_group.clouddevinsights.location
  resource_group_name = azurerm_resource_group.clouddevinsights.name

  ip_configuration {
    name                          = "internal"
    subnet_id                     = azurerm_subnet.vnet-clouddevinsights-subnet.id
    private_ip_address_allocation = "Dynamic"

  }

}

resource "azurerm_network_interface_security_group_association" "nsg-association" {
  network_interface_id      = azurerm_network_interface.vm-nic.id
  network_security_group_id = azurerm_network_security_group.nsg-clouddevinsights-nsg.id
}

resource "azurerm_linux_virtual_machine" "linux-vm" {
  name                = var.vm-name
  resource_group_name = azurerm_resource_group.clouddevinsights.name
  location            = var.resource_group_location
  size                = "Standard_B1s"
  admin_username      = "adminuser"
  admin_password = "Password1234!"
  disable_password_authentication = "false"

  network_interface_ids = [
    azurerm_network_interface.vm-nic.id,
  ]

  os_disk {
    caching              = "ReadWrite"
    storage_account_type = "Standard_LRS"
  }

  source_image_reference {
    publisher = "Canonical"
    offer     = "UbuntuServer"
    sku       = "18.04-LTS"
    version   = "latest"
  }
}

Make Changes in Azure:

I will add a NSG rule manually using Azure Portal. I will allow all inbound traffic from internet.

Steps to Sync Terraform with Azure

Step 1: Update Terraforms State File

Terraforms state file does not automatically reflect changes made directly in Azure. To synchronize the state file with the actual infrastructure, use the following command:

terraform refresh

This command fetches the latest state of the resources from Azure and updates the local state file. For example, if you added new rules to the NSG or changed the VM size, these changes will now be reflected in Terraforms state.

Step 2: Detect Drift Using terraform plan

After refreshing the state, run the terraform plan command to identify any differences between the actual resources in Azure and the desired configuration defined in your .tf files:

terraform plan

Terraform will analyze the current state and the configuration files to detect any drift. It will display a plan of actions needed to bring the infrastructure back in line with the desired state. For example, it might show that the VM size or NSG rules differ from the configuration. Please see the image below to see the drift.

Step 3: Apply Changes to Sync Resources

If drift is detected, you can apply the necessary changes to align the resources with your Terraform configuration. Use the following command:

terraform apply

Terraform will prompt you to confirm the changes. Once confirmed, it will update the Azure resources to match the desired configuration. For example, it might:

• Remove any manually added rules in the NSG that are not in the Terraform configuration.

Special Case: Both Terraform and Azure Modify Resources

If both Terraform and Azure have modified the same resources, its crucial to ensure that Terraforms state file is up-to-date before applying changes. Heres why:

1. Terraform Updates the State File: When Terraform applies changes, it updates the state file to reflect the new state of the resources.

2. Manual Changes in Azure: If resources are manually modified after Terraforms state file has been updated, Terraform may overwrite those changes during the next apply operation.

To avoid unintentional overwrites:

• Always run terraform refresh and terraform plan before applying changes.

• Communicate with your team to establish clear guidelines for managing resources.

Key Takeaways

• Avoid Manual Changes: The best practice is to avoid manual changes to resources managed by Terraform. This ensures consistency and reduces the risk of drift.

• Refresh State Regularly: Use terraform refresh to keep the state file in sync with the actual infrastructure.

• Detect and Resolve Drift: Use terraform plan to detect drift and terraform apply to resolve it.

• Establish Governance: Set clear policies to manage resources and avoid conflicts between manual changes and Terraform.

By following these steps, you can ensure that your infrastructure remains consistent, predictable, and aligned with your Terraform configurations. Handling drift effectively is a critical skill for any cloud professional, and it ensures that your IaC workflows remain robust and reliable.

In the ever-evolving world of cloud computing, governance is the cornerstone of a secure, compliant, and efficient Azure environment. As organizations migrate to the cloud, the challenge lies in maintaining control without stifling innovation. Azure Governance provides a framework to address this challenge, ensuring that resources are managed effectively, costs are optimized, and security is upheld.

This blog dives into the essential Azure policies every organization should implement to establish a robust governance model. Whether youre a small business or a global enterprise, these recommendations will help you create a scalable and compliant cloud environment. From cost management to security and resource consistency, well cover practical examples and best practices to guide your governance journey.

Lets simplify Azure Governance and empower your organization to harness the full potential of the cloud responsibly.

Allowed Locations:

This policy restricts the locations your organization can specify when deploying resources. It helps enforce geo-compliance requirements, ensuring resources are deployed only in approved regions.

Exclusions:

• Resource groups.

• Azure Active Directory B2C directories.

• Resources using the global region.

Example Use Case: Restrict deployments to regions that comply with data residency and regulatory requirements, such as GDPR or HIPAA.

Click here to see JSON policy

Allowed Virtual Machine Size SKUs:

This policy enables you to specify a set of virtual machine size SKUs that your organization can deploy. By restricting VM sizes, you can:

• Enforce organizational standards.

• Optimize costs by preventing the use of oversized VMs.

• Simplify management by limiting the variety of deployed VMs.

Example Use Case: Restrict VM SKUs to cost-efficient or approved configurations that meet workload requirements.

Click here to see JSON policy

Allowed Resource Types

This policy specifies which resource types your organization can deploy. It helps reduce complexity and minimizes the attack surface by limiting resource types to those essential for business operations.

Note: Only resource types supporting tags and location are affected. To restrict all resources, duplicate the policy and change its mode to All.

Example Use Case: Allow only approved resource types, such as storage accounts and virtual machines, to simplify governance and improve security.

Click here to see JSON policy

Audit Usage of Custom RBAC Roles

Custom roles can be error-prone and introduce security risks. This policy audits the usage of built-in roles such as Owner, Contributor, and Reader, treating custom roles as exceptions requiring thorough review.

Example Use Case: Ensure adherence to least privilege principles by minimizing the use of custom roles.

Click here to see JSON policy

Custom Subscription Owner Roles Should Not Exist

Prevent the creation of custom subscription owner roles to maintain consistency and security in role assignments.

Example Use Case: Avoid role proliferation and potential misconfigurations.

Click here to see JSON policy

Not Allowed Resource Types

This policy restricts specific resource types from being deployed. It helps reduce the environments complexity and attack surface while managing costs.

Example Use Case: Disallow deployment of expensive or unsupported resource types.

Click here to see JSON policy

A Maximum of 3 Owners for Subscriptions

Limiting the number of subscription owners reduces the potential impact of a compromised owner account.

Example Use Case: Ensure there are no more than three designated subscription owners to enhance security.

Click here to see JSON policy

MFA for Subscription Owners

Multi-Factor Authentication (MFA) should be enabled for all accounts with owner permissions. This prevents unauthorized access to critical resources.

Example Use Case: Enforce MFA for subscription owners to mitigate risks of account breaches.

Click here to see JSON policy

Contact Email Address for Security Issues

Set a security contact email address to ensure the right individuals are notified of potential security breaches.

Example Use Case: Receive alerts from Azure Security Center for prompt action.

Click here to see JSON policy

Require a Tag on Resource Groups

Enforce the existence of a specific tag on resource groups to enhance resource organization and tracking.

Example Use Case: Require a CostCenter tag for resource groups to track expenses.

Click here to see JSON policy

Inherit a Tag from the Resource Group

Automatically add a missing tag to resources from their parent resource group. This ensures consistent tagging across the environment.

Example Use Case: Ensure all resources inherit the Environment tag (e.g., Production, Development).

Click here to see JSON policy

Custom Policy for Resource lock:

Azure doesnt provide built-in policy for resource lock. however, we can create custom policy to deploy a policy to Enforce CanNotDelete Resource Lock using Azure Policy:

{
displayName: "Resource Lock should be enabled",
description: "With this policy: any resource that has the tag key LockLevel with the value CanNotDelete means authorized users can read and modify the resource, but they can t delete it.",
metadata: {
category: "Backup",
version: "1.0.0"
},
mode: "Indexed",
parameters: {
tagName: {
type: "string",
metadata: {
displayName: "Exclusion Tag Name",
description: "Name of the tag to use for excluding resources from this policy. This should be used along with the Exclusion Tag Value parameter."
},
defaultValue: "_MVP_Resource_Lock_should_be_enabled"
},
tagValue: {
type: "string",
metadata: {
displayName: "Exclusion Tag Value",
description: "Value of the tag to use for excluding resources from this policy. This should be used along with the Exclusion Tag Name parameter."
},
defaultValue: "exclude"
},
effect: {
type: "String",
metadata: {
displayName: "Effect",
description: "DeployIfNotExists, AuditIfNotExists or Disabled the execution of the Policy"
},
allowedValues: [
"DeployIfNotExists",
"AuditIfNotExists",
"Disabled"
],
defaultValue: "DeployIfNotExists"
}
},
policyRule: {
if: {
allOf: [
{
field: "tags.LockLevel",
equals: "CanNotDelete"
},
{
value: 🔍"[
    length(
        split(
            field('type'),
           '/'
        )
    )
]",
equals: 2
},
{
not: {
field: 🔍"[
    concat(
       'tags[
           ',
            parameters('tagName'),
           '
        ]'
    )
]",
equals: "[parameters('tagValue')]"
}
}
]
},
then: {
effect: "[parameters('effect')]",
details: {
roleDefinitionIds: [
"/providers/microsoft.authorization/roleDefinitions/8e3af657-a8ff-443c-a75c-2fe8c4bcb635"
],
type: "Microsoft.Authorization/locks",
name: "ResourceLockedByPolicy",
existenceCondition: {
allOf: [
{
field: "Microsoft.Authorization/locks/level",
In: [
"CanNotDelete"
]
},
{
field: "Microsoft.Authorization/locks/notes",
equals: "Locked by Azure Policy"
}
]
},
deployment: {
properties: {
mode: "incremental",
template: {
$schema: "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
contentVersion: "1.0.0.0",
parameters: {
resourceName: {
type: "string"
},
resourceType: {
type: "string"
}
},
variables: {},
resources: [
{
type: "Microsoft.Authorization/locks",
apiVersion: "2016-09-01",
name: "ResourceLockedByPolicy",
scope: 🔍"[
    concat(
        parameters('resourceType'),
       '/',
         parameters('resourceName')
    )
]",
properties: {
level: "CanNotDelete",
notes: "Locked by Azure Policy"
}
}
],
outputs: {}
},
parameters: {
resourceName: {
value: "[field('name')]"
},
resourceType: {
value: "[field('type')]"
}
}
}
}
}
}
}
}

Conclusion:

Azure Governance with Azure Policy empowers organizations to enforce compliance, security, and operational efficiency. By implementing these recommended policies, you can establish a robust governance framework that aligns with your organizations goals and regulatory requirements.

Start defining and enforcing these policies today to ensure your Azure environment remains secure, compliant, and cost-effective.

Data is the lifeblood of modern applications, and Azure provides robust tools to ensure that data is collected, processed, and stored efficiently. One of these tools is Data Collection Rules (DCRs), a feature that enables fine-grained control over data ingestion into Azure Monitor. This blog post will explore the fundamentals of DCRs, their benefits, and how to configure them for effective monitoring.

What Are Data Collection Rules?

Data Collection Rules are configurations that define how telemetry and log data are collected and routed to destinations like Log Analytics workspaces, Azure Storage, or Event Hubs. Introduced to provide greater flexibility and scalability, DCRs are part of Azure Monitors modern data collection architecture.

With DCRs, you can:

• Filter Data: Collect only the data you need, reducing noise and costs.

• Transform Data: Apply transformations to data before ingestion, such as masking sensitive information or enriching fields.

• Route Data: Send data to multiple destinations simultaneously.

Key Benefits of Data Collection Rules

1. Granular Control: Define specific data collection settings for different resource types, such as virtual machines, containers, or PaaS services.

2. Cost Optimization: Reduce ingestion costs by filtering unnecessary data.

3. Flexibility: Route data to multiple destinations without duplicating collection efforts.

4. Compliance: Mask or filter sensitive data to comply with regulatory requirements.

Components of a Data Collection Rule

A DCR consists of the following elements:

1. Data Sources: Specify the resources or telemetry types (e.g., Windows Event Logs, Syslog, or custom logs).

2. Transforms: Apply transformations to modify or filter data before ingestion.

3. Destinations: Define where the collected data will be sent, such as Log Analytics workspaces or Azure Storage.

Configuring a Data Collection Rule

Step 1: Define the Data Sources

Start by identifying the data sources you want to monitor. For example, you might collect Windows Event Logs from virtual machines or Syslog data from Linux servers.

Step 2: Apply Transformations

Use KQL (Kusto Query Language) to define transformations. For instance, you can filter out logs with specific keywords or mask sensitive fields.

Step 3: Set Up Destinations

Choose one or more destinations for the data. Common destinations include:

• Log Analytics Workspace: For querying and analyzing data.

• Azure Storage: For archival purposes.

• Event Hubs: For integration with third-party systems.

Step 4: Create and Deploy the DCR

You can create DCRs using the Azure portal, Azure CLI, or ARM templates. Heres an example using Azure CLI.

az monitor data-collection rule create --location 'eastus' --resource-group 'my-resource-group' --name 'my-dcr' --rule-file 'C:\MyNewDCR.json' --description 'This is my new DCR'

Use local file as source of DCR

DCRs for Syslog events use the syslog data source with the incoming Microsoft-Syslog stream. The schema of this stream is known, so it doesn't need to be defined in the dataSources section. The events to collect are specified in the facilityNames and logLevels properties. See Collect Syslog events with Azure Monitor Agent for further details. To get started, you can use the guidance in that article to create a DCR using the Azure portal and then inspect the JSON using the guidance at DCR definition.

You can add a transformation to the dataFlows property for additional functionality and to further filter data, but you should use facilityNames and logLevels for filtering as much as possible for efficiency at to avoid potential ingestion charges.

The following sample DCR performs the following actions:

• Collects all events from cron facility.

• Collects Warning and higher events from syslog and daemon facilities.

  • Sends data to Syslog table in the workspace.

  • Uses a simple transformation of a source which makes no change to the incoming data.

{
    "location": "eastus",
    "properties": {
      "dataSources": {
        "syslog": [
          {
            "name": "cronSyslog",
            "streams": [
              "Microsoft-Syslog"
            ],
            "facilityNames": [
              "cron"
            ],
            "logLevels": [
              "Debug",
              "Info",
              "Notice",
              "Warning",
              "Error",
              "Critical",
              "Alert",
              "Emergency"
            ]
          },
          {
            "name": "syslogBase",
            "streams": [
              "Microsoft-Syslog"
            ],
            "facilityNames": [
              "daemon",              
              "syslog"
            ],
            "logLevels": [
              "Warning",
              "Error",
              "Critical",
              "Alert",
              "Emergency"           
            ]
          }
        ]
      },
      "destinations": {
        "logAnalytics": [
          {
            "workspaceResourceId": "/subscriptions/aaaa0a0a-bb1b-cc2c-dd3d-eeeeee4e4e4e/resourceGroups/my-resource-group/providers/Microsoft.OperationalInsights/workspaces/my-workspace",
            "name": "centralWorkspace"
          }
        ]
      },
      "dataFlows": [
        {
          "streams": [
            "Microsoft-Syslog"
          ],
          "destinations": [
            "centralWorkspace"
          ],
            "transformKql": "source",
            "outputStream": "Microsoft-Syslog"
        }
      ]
    }
  }

Manage data collection rule associations in Azure Monitor

To view your DCRs in the Azure portal, select Data Collection Rules under Settings on the Monitor menu. Select a DCR to view its details.

Click the Resources tab to view the resources associated with the selected DCR. Click Add to add an association to a new resource. You can view and add resources using this feature whether or not you created the DCR in the Azure portal.

Azure Policy

Using Azure Policy, you can associate a DCR with multiple resources at scale. When you create an assignment between a resource group and a built-in policy or initiative, associations are created between the DCR and each resource of the assigned type in the resource group, including any new resources as they're created. Azure Monitor provides a simplified user experience to create an assignment for a policy or initiative for a particular DCR, which is an alternate method to creating the assignment using Azure Policy directly.

From the DCR in the Azure portal, select Policies (Preview). This will open a page that lists any assignments with the current DCR and the compliance state of included resources. Tiles across the top provide compliance metrics for all resources and assignments.

To create a new assignment, click either Assign Policy or Assign Initiative.

Best Practices for Using Data Collection Rules

1. Start Small: Begin with a limited set of data sources and destinations to understand the impact.

2. Monitor Costs: Use Azure Cost Management to track the costs associated with data ingestion.

3. Test Transformations: Validate KQL queries to ensure they filter or transform data as expected.

4. Use Tags: Apply tags to DCRs for better management and organization.

Common Use Cases

• Application Monitoring: Collect application logs and route them to Log Analytics for troubleshooting.

• Security Auditing: Filter and store security-related logs in Azure Storage for long-term retention.

• Compliance Reporting: Mask sensitive information in logs to meet regulatory requirements.

Conclusion

Data Collection Rules provide a powerful and flexible way to manage data ingestion in Azure Monitor. By leveraging DCRs, you can optimize costs, improve compliance, and ensure that only the most relevant data is collected. Whether youre monitoring applications, auditing security logs, or building compliance workflows, DCRs are a must-have tool in your Azure toolkit.

Start exploring Data Collection Rules today and unlock the full potential of Azure Monitor!

Reference:

Microsoft Article to refer

In this article, I will deploy ARM template using Azure Devops. I will use Continous Integration of ARM template with Azure Pipelines.

source code can be found on Source code and project info

Prepare your project

In This project we have ARM template and Azure DevOps organization ready for creating the pipeline. The following steps show how to make sure you're ready:

You have an Azure DevOps organization. If you don't have one, create one for free. If your team already has an Azure DevOps organization, make sure you're an administrator of the Azure DevOps project that you want to use.

Import GitHub repo to Azure Repos

You have an ARM template that defines the infrastructure for your project. I have ARM tepmate in the repo. the template will create storage account. Virtual Network, Public IP, NSG, subnet, NIC, VM.

Create Pipeline

I will add new pipeline

select Azure repos as repo

select the repo we just imported to Azure Repos Git in this project and then select starter pipeline below.

add two tasks to the pipeline. first one is copying files and second one is publish artifact. please find the code below.

Copy

Copy

# Starter pipeline
# Start with a minimal pipeline that you can customize to build and deploy your code.
# Add steps that build, run tests, deploy, and more:
# https://aka.ms/yaml

trigger:
- main

pool:
  vmImage: ubuntu-latest

steps:
- script: echo Hello, world!
  displayName: 'Run a one-line script'

- script: |
    echo Add other tasks to build, test, and deploy your project.
    echo See https://aka.ms/yaml
  displayName: 'Run a multi-line script'
- task: CopyFiles@2
  inputs:
    SourceFolder: '$(agent.builddirectory)'
    Contents: '**'
    TargetFolder: '$(build.artifactstagingdirectory)'

- task: PublishBuildArtifacts@1
  inputs:
    PathtoPublish: '$(Build.ArtifactStagingDirectory)'
    ArtifactName: 'storagedrop'
    publishLocation: 'Container'

now lets review the pipeline before we run

now lets save and run the pipleline

pipeline has run succesfully.

this pipeline has produced an atrifact. we will use this artifact to create resources mentioned in our template. In our ARM template we are creating a storage account.

let's create a release pipeline and use artifact we got from our build process.

for the release pipeline let's select empty job.

now let's add ARM template to the task

let's fill in the details so that resource get's created.

we need to integrate our build into the release pipeline. the artifact generated will be added.

I added artifact generated into the release pipeline.

now let's add task into the arm template added.

Lets create the release

you can see that resource group is created and storage account is created. please see the logs image below.

I checked in Azure that new resource group is created and storage account is created.

I will add continous deployment and release to the pipeline

now we will add trigger to the release pipeline. this will make sure any changes to the repo will create the new build and will trigger to release pipeline. this will create new resources.

I will enable continoys deployment on the pipleline. when I make change to the repo the pipeline will get executed and resource will be created. Edit the release pipeline

click on the trigger that will let me enable continous deployment

enable create a new release when new build is available and enable pull trigger

Now save the settings. We have enabled continous deployment.

Now we will edit the repo. I will add more resources in the template. I will add storage account, public ip, Virtual network, subnet, NSG, NIC, Disk and Virtual machine.

commit the changes and you will see new bulild will be created and that will trigger release pipeline.

As we have enabled Continous Integration new release will be created

I will check in Azure and all the resources are created.

This is how you can deploy ARM template with Azure Devops. you can enabel contionus deployment to make sure when you add resource to the template they get created.

Create a Project in Azure DevOps First create a project in Azure DevOps.

Push source Code into Azure Repo I pushed .Net Source Code into Azure DevOps Repo. This Repo is part of Microsoft Parts Unlimited Open Source Project.

GitHUb link for Parts Unlimited Project

Build a Pipeline for the application Create a new pipeline in Pipelines

Select Azure Repo as source Repo

select starter pipeline in configure your pipeline

for our pipeline let's add tasks. This tasks will create artifact. task will download required libraries for the application. build the project and create a artifact. first task is Nuget. Nuget is used to download libraries for .net Application.

Second task is we need Visual studio to build the project. I will add Build commands needed for this project below.

/p:DeployOnBuild=true /p:WebPublishMethod=Package /p:PackageAsSingleFile=true /p:SkipInvalidConfigurations=true 
/p:PackageLocation="$(build.stagingDirectory)" /p:IncludeServerNameInBuildInfo=True /p:GenerateBuildInfoConfigFile=true 
/p:BuildSymbolStorePath="$(SymbolPath)" /p:ReferencePath="C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\Extensions\Microsft\Pex"

copy the files to the agent source folder and publish artifact to the staging directory.

Now click on save and run. this will run the build pipeline and create a artifact.

create a release pipeline. click on release and select new release pipeline.

now just like build pipeline we need to add tasks to the release pipeline.

Add ARM Template as task as we need to create the environment for our app.

add varaibles to the pipeline. In the process we are passing some variables to the pipeline.

add App service deployment as task. we will deploy our web app into a app service plan

release pipeline ran sucessfully.

App has been deployed sucessfully.

let's test our pipeline I will change the display name from

<h1>Priya Abilash</h1> <h2>avani subsidiary</h2> ``` to ``` <h1>Sydney Auto Parts</h1> <h2>avani subsidiary</h2>

we have pipeline build so new artifact will be created and we can manually deploy the release pipeline.


you can see build pipeline is running once we save the changes to the layout.cshtml file in our repo

let's create a new release for new build Click on edit pipeline and add the new build artifact as new artifact

The pipeline is running.

Pipeline ran succesfuly.

I will refresh the browser and see the changes made to the repo.