Overview

Apply stack

The SparkleFormation CLI includes functionality to ease the sharing of resources between isolated stacks. This functionality enables groupings of similar resources which can define logical units of a full architecture. The sharing is accomplished using a combination of stack outputs and stack parameters. This feature is commonly referred to as the apply stack functionality.

One drawback of the apply stack functionality is that it introduces dependencies between disparate stacks. These dependencies must be manually resolved when resource modifications occur. When an infrastructure is composed of few stacks, the apply stack approach can be sufficient to manage stack dependencies leaving the user to ensure updates are properly applied. For larger implementations composed of many interdependent stacks, the nested stack functionality is recommended.

Nested stack

Nested stack functionality is a generic feature provided by the SparkleFormation library which the SparkleFormation CLI then specializes based on the target provider. The nested stack functionality utilizes a core feature provided by orchestration APIs. This features allows nesting stack resources within a stack allowing a parent stack to have many child stacks. By nesting stack resources, the provider API will be aware of child stack interdependencies and automatically apply updates when resources have been modified. This removes the requirement of stack updates being tracked and applied manually.

The behavior of the nested stack functionality is based directly on the behavior of the apply stack functionality. This commonality in behavior allows for initial testing development using the apply stack functionality and, once stable, migrating to nested stack functionality without requiring any modifications to existing templates. The commonality also allows for the two functionalities to be mixed in practice.

This guide will first display template implementations using the apply stack functionality. The example templates will then be used to provide a nested stack implementation.

NOTE: This guide targets the AWS provider for simplicity to allow focus on the features discussed. All providers support this behavior.

Apply stack implementation

Lets start by defining a simple infrastructure. Our infrastructure will be composed of a “network” and a collection of “computes” that utilizes the network. This infrastructure can be easily defined within two units:

  1. network
  2. computes

Lets start by creating a simple network template.

Create a new file: ./sparkleformation/network.rb

Template sparkles AWS

SparkleFormation.new(:network) do

  parameters do
    cidr_prefix do
      type 'String'
      default '172.20'
    end
  end

  dynamic!(:ec2_vpc, :network) do
    properties do
      cidr_block join!(ref!(:cidr_prefix), '.0.0/24')
      enable_dns_support true
      enable_dns_hostnames true
    end
  end

  dynamic!(:ec2_dhcp_options, :network) do
    properties do
      domain_name join!(region!, 'compute.internal')
      domain_name_servers ['AmazonProvidedDNS']
    end
  end

  dynamic!(:ec2_vpc_dhcp_options_association, :network) do
    properties do
      dhcp_options_id ref!(:network_ec2_dhcp_options)
      vpc_id ref!(:network_ec2_vpc)
    end
  end

  dynamic!(:ec2_internet_gateway, :network)

  dynamic!(:ec2_vpc_gateway_attachment, :network) do
    properties do
      internet_gateway_id ref!(:network_ec2_internet_gateway)
      vpc_id ref!(:network_ec2_vpc)
    end
  end

  dynamic!(:ec2_route_table, :network) do
    properties.vpc_id ref!(:network_ec2_vpc)
  end

  dynamic!(:ec2_route, :network_public) do
    properties do
      destination_cidr_block '0.0.0.0/0'
      gateway_id ref!(:network_ec2_internet_gateway)
      route_table_id ref!(:network_ec2_route_table)
    end
  end

  dynamic!(:ec2_subnet, :network) do
    properties do
      availability_zone select!(0, azs!)
      cidr_block join!(ref!(:cidr_prefix), '.0.0/24')
      vpc_id ref!(:network_ec2_vpc)
    end
  end

  dynamic!(:ec2_subnet_route_table_association, :network) do
    properties do
      route_table_id ref!(:network_ec2_route_table)
      subnet_id ref!(:network_ec2_subnet)
    end
  end

  outputs do
    network_vpc_id.value ref!(:network_ec2_vpc)
    network_subnet_id.value ref!(:network_ec2_subnet)
    network_route_table.value ref!(:network_ec2_route_table)
    network_cidr.value join!(ref!(:cidr_prefix), '.0.0/24')
  end

end

Here we have an extremely simple VPC defined for our infrastructure. It is important to note the outputs defined within our network template. These outputs are values that will be required for other resources to effectively utilize the VPC. With the network template defined, we can create that unit of our infrastructure:

$ bundle exec sfn create sparkle-guide-network --file network

Having successfully built the network unit of the infrastructure, we can now compose the computes template.

Create a new file: ./sparkleformation/computes.rb

Template sparkles AWS

SparkleFormation.new(:computes) do

  parameters do
    ssh_key_name.type 'String'
    network_vpc_id.type 'String'
    network_subnet_id.type 'String'
    image_id_name do
        type 'String'
        default 'ami-63ac5803'
    end
  end

  dynamic!(:ec2_security_group, :compute) do
    properties do
      group_description 'SSH Access'
      security_group_ingress do
        cidr_ip '0.0.0.0/0'
        from_port 22
        to_port 22
        ip_protocol 'tcp'
      end
      vpc_id ref!(:network_vpc_id)
    end
  end

  dynamic!(:ec2_instance, :micro) do
    properties do
      image_id ref!(:image_id_name)
      instance_type 't2.micro'
      key_name ref!(:ssh_key_name)
      network_interfaces array!(
        ->{
          device_index 0
          associate_public_ip_address 'true'
          subnet_id ref!(:network_subnet_id)
          group_set [ref!(:compute_ec2_security_group)]
        }
      )
    end
  end

  dynamic!(:ec2_instance, :small) do
    properties do
      image_id ref!(:image_id_name)
      instance_type 't2.micro'
      key_name ref!(:ssh_key_name)
      network_interfaces array!(
        ->{
          device_index 0
          associate_public_ip_address 'true'
          subnet_id ref!(:network_subnet_id)
          group_set [ref!(:compute_ec2_security_group)]
        }
      )
    end
  end

  outputs do
    micro_address.value attr!(:micro_ec2_instance, :public_ip)
    small_address.value attr!(:small_ec2_instance, :public_ip)
  end

end

Our computes template will create one micro and one small EC2 instance and output the public IP addresses of each resource. To build the EC2 instances into the VPC created in the sparkle-guide-network stack we need two pieces of information:

  • VPC ID
  • VPC subnet ID

In our computes template we define two parameters for the required VPC information: network_vpc_id and network_subnet_id. When we create a stack using this template we can copy the output values from the sparkle-guide-network stack and paste them into these parameters, but that is extremely cumbersome. The SparkleFormation CLI instead allows “applying” a stack on creation or update.

Notice that the output names in our network template match the parameter names in our computes template.

# From ./sparkleformation/network.rb
  outputs do
    network_vpc_id.value ref!(:network_ec2_vpc)
    network_subnet_id.value ref!(:network_ec2_subnet)
    network_route_table.value ref!(:network_ec2_route_table)
    network_cidr.value join!(ref!(:cidr_prefix, '.0.0/24'))
  end

# From ./sparkleformation/computes.rb
  parameters do
    ssh_key_name.type 'String'
    network_vpc_id.type 'String'
    network_subnet_id.type 'String'
  end

The apply stack functionality will automatically collect outputs from the stack names provided and use them to seed the parameters of the stack being created or updated. This means instead of having to copy and paste the VPC ID and subnet ID values, we can instruct the SparkleFormation CLI to automatically use the outputs of our sparkle-guide-network stack:

$ bundle exec sfn create sparkle-guide-computes --file computes --apply-stack sparkle-guide-network

During the create process, the SparkleFormation CLI will prompt for parameters. The default values for the VPC ID and subnet ID will be automatically inserted, matching the outputs from the sparkle-guide-network.

You can destroy the sparkle-guide-compute and sparkle-guide-network stacks as they will not be used in the next section.

$ sfn destroy sparkle-guide-computes
$ sfn destroy sparkle-guide-network

Nested stack implementation

Now that our infrastructure has been successfully created using disparate stacks, lets combine them to create a single infrastructure unit composed of sub-units. Using nested stacks requires a bucket to store templates. Create a bucket in S3 and then add the following to the .sfn configuration file:

Configuration.new do
  ...
  nesting_bucket 'NAME_OF_BUCKET'
  ...
end

With the required bucket in place and SparkleFormation CLI configured we can now create our infrastructure template.

Create a new file: ./sparkleformation/infrastructure.rb

Template sparkles AWS

SparkleFormation.new(:infrastructure) do
  nest!(:network, :infra)
  nest!(:computes, :infra)
end

This new infrastructure template is using SparkleFormation’s builtin nesting functionality to create stack resources within our infrastructure template composed of our network and computes template. To see the conceptual result of this nesting, we can print the infrastructure template:

$ bundle exec sfn print --file infrastructure

There are a few things of note in this output. First, the Stack property is not a real resource property. It is used by SparkleFormation for template processing and is included in print functions to display the template in its entirety. Next, the generated URLs are not real URLs. This is due to the SparkleFormation CLI not actually storing the templates in the remote bucket. Lastly, and most importantly, the parameters property of the ComputesInfra resource.

"Parameters": {
  "NetworkVpcId": {
    "Fn::GetAtt": [
      "NetworkInfra",
      "Outputs.NetworkVpcId"
    ]
  },
  "NetworkSubnetId": {
    "Fn::GetAtt": [
      "NetworkInfra",
      "Outputs.NetworkSubnetId"
    ]
  }
}

SparkleFormation registers outputs when processing templates and will automatically map outputs to subsequent stack resource parameters if they match. Cross stack resource dependencies are now explicitly defined allowing the orchestration API to automatically determine creation order as well as triggering updates when required.

Now we can create our full infrastructure with a single command:

$ bundle exec sfn create sparkle-guide-infrastructure --file infrastructure

As SparkleFormation processes the nested templates for the create command, the SparkleFormation CLI will extract the nested templates, store them in the configured nesting bucket, and updates the template location URL in the resource.

Using nested templates, update commands follow the same behavior as create commands. All nested templates are extracted and automatically uploaded prior to execution of the update request with the orchestration API. This results in all nested stacks being automatically updated by the API as required based on dependent resource modifications.

Apply nested stack

Nested stacks can be applied to disparate stacks in the same manner described in the apply stack implementation section. When the stack to be applied is a nested stack, SparkleFormation CLI will gather outputs from all the nested stacks, and then apply to the target command. This means using the sparkle-guide-infrastructure stack we previously built can be used for creating new computes stacks without being nested into the infrastructure template.

$ bundle exec sfn create sparkle-guide-computes-infra --file computes --apply-stack sparkle-guide-infrastructure

The ability to apply nested stacks to disparate stacks make it easy to provide resources to new stacks, or to test building new stacks in isolation before being nested into the root stack.

You can destroy the all the infrastructure related stacks with the command:

sfn destroy sparkle-guide-infrastructure