Nested Stacks

Most orchestration API templating systems provide support for a “stack” resource which allows for a stack to define one or more nested stacks within its resources. SparkleFormation expands stack nesting by adding extra functionality when compiling SparkleFormation templates. Currently two styles of expanded functionality are available and are explained in depth below:

The interface for using SparkleFormation’s nested stack functionality is via the nest! helper method. The method accepts a template name and will insert the stack resource into the current template:

SparkleFormation.new(:root_template) do
  nest!(:networking)
  nest!(:applications)
end

Shallow Nesting

Shallow stack nesting is the original style of nesting functionality implemented within SparkleFormation. Key features/restrictions of shallow nesting:

  • Support nesting one level deep
  • Automatic parameter bubbling to root stack
  • Automatic output mapping

Shallow Nesting Depth

Shallow nesting is restricted to single level nesting. This restriction is in place due to the shallow nesting style being the first successfully implemented nesting strategy. The restriction remains due to the unique behavior this style of nesting provides which does not work well past a single level of nesting.

Nested Parameter Bubbling

On compilation SparkleFormation will process nested stacks in a top-down order. It will first extract parameter names from the nested stack. If the root stack has no matching parameter, the parameter will automatically be added to the root stack. For example:

SparkleFormation.new(:template_a) do
  ...
  parameters.fubar do
    type 'String'
    default 'FOOBAR'
  end
end
SparkleFormation.new(:root) do
  nest!(:template_a)
end

when compiled would result in:

...
  "Parameters": {
    "Fubar": {
      "Type": "String",
      "Default": "FOOBAR"
    }
  },
  "Resources": {
    "TemplateA": {
      "Type": "AWS::CloudFormation::Stack",
      "Properties": {
        "Parameters": {
          "Fubar": {
            "Ref": "Fubar"
          }

...

If a second stack is nested and it defines a parameter with the same name as a previously defined parameter, only the original parameter will be used and both stacks will reference it.

This parameter bubbling behavior allows all contained stacks to be controlled from the root stack providing a single point of interaction.

Nested Output Mapping

During compilation and processing of nested stacks, SparkleFormation will also keep list of outputs available from previously processed nested stack resources. If a parameter name on a nested stack matches the name of an output defined in a nested stack, SparkleFormation will automatically update the nested stack resource parameter to use the output value. For example:

SparkleFormation.new(:template_a) do
  ...
  outputs.address do
    description 'Address of thing'
    value ref!(:thing)
  end
  ...
end
SparkleFormation.new(:template_b) do
  ...
  parameters.address do
    type 'String'
  end
  ...
end
SparkleFormation.new(:root_template) do
  nest!(:template_a)
  nest!(:template_b)
end

When the final template file is compiled SparkleFormation will not bubble the Address parameter to the root stack. Because template_b defines an output with a matching name, SparkleFormation automatically uses that output value:

...
    "TemplateB": {
      "Type": "AWS::CloudFormation::Stack",
      "Properties": {
        "Parameters": {
          "Address": {
            "Fn:GetAtt": [
              "TemplateA",
              "Outputs.Address"
            ]
          }
...

Shallow nesting easily exposes the power of nesting stack resources while maintaining a single point of access for managing a stack. This is important to note when looking at the ease of use for updating running stacks. Unless a template change is required, parameter changes can be made via a single update call to the root stack. It also means that parameter based updates can be provided from any acceptable interface, be it a CLI tool, or web based UI.

NOTE: One issue quickly encountered with parameter heavy nested stacks is resource limits on the number of parameters allowed within a single stack. Using deep stack nesting prevents this issue.

Shallow Nesting Usage

Shallow nesting is performed by calling SparkleFormation#apply_nesting. The method expects a block to be provided. This block handles storage of the nested stack template (if required) and any updates to the original stack resource.

sfn = SparkleFormation.compile(template_path, :sparkle)

sfn.apply_nesting(:shallow) do |stack_name, nested_stack_sfn, original_stack_resource|
  template_content = nested_stack_cfn.compile.dump!
  # store the template content as required, set remote location as `template_url`
  original_stack_resource.properites.delete!(:stack)
  original_stack_resource.properties.set!('TemplateURL', template_url)
end

Deep Nesting

Deep stack nesting is an expansion of the shallow stack nesting functionality. It loses some ease of use but gains greater functionality. Key features/ restrictions of deep stack nesting:

  • No parameter bubbling
  • Supports unlimited nesting depths
  • Automatic output mapping
    • Automatic output bubbling

Deep Nested Parameters

Deep stack nesting does not provide parameter bubbling. The biggest issue in providing this type of behavior for deeply nested stacks are the limits applied by the API. It also introduces more complexity to the implementation since parameters would have to be propagated from the root stack to the leaf stacks requiring the parameters.

Instead of bubbling parameters to the root stack, deep nesting behavior does nothing with the parameters defined for nested stacks. It shifts that responsibility to the application which can update resource’s parameters as it decides using its registered callback handler.

Unlimited Nesting Depth

Deep stack nesting does not enforce a limit on the number of levels deep stacks may be nested. This may not be true for the targeted API. Supporting multiple levels of nesting makes it easy to logically compartmentalize related resources into stacks, which can then be collected and compartmentalized into category-style stacks which can be nested into the root stack. This can make it easier to not only develop stacks but easier to reason about as well.

Automatic Output Mapping and Bubbling

Much like the shallow nesting behavior, deep nesting provides automatic output value mapping to parameters of a matching name. This behavior is more challenging when using deep nesting behavior due to the possibility of outputs being defined in a resource tree that is isolated from a nested stack requiring its value. To solve this problem SparkleFormation will automatically add an output entry to the parent stack(s) “bubbling” the value until the output is available at the same level requesting stack. If the requesting stack is nested from the common depth, then parameters are added to the stacks to “push” the value down.

Output Bubbling Behavior

This example will illustrate the behavior seen when outputs are “bubbled”:

SparkleFormation.new(:networking) do
  ...
  outputs.subnet do
    description 'Networking subnet'
    value ref!(:subnet_resource)
  end
  ...
end
SparkleFormation.new(:infrastructure) do
  ...
  nest!(:networking)
  ...
end
SparkleFormation.new(:applications) do
  ...
  nest!(:moneymaker)
  ...
end
SparkleFormation.new(:moneymaker) do
  parameters.subnet do
    type 'String'
  end
  ...
end
SparkleFormation.new(:root) do
  nest!(:infrastructure)
  nest!(:applications)
end

When the root stack is compiled, it will first process the infrastructure nesting, which will in turn process the networking nesting. After processing those stacks, SparkleFormation will know the location of the Subnet output. It will then process the application nesting, which has a Subnet parameter matching a known output. Because the Subnet output from networking stack is not accessible from the root stack to provide to the application stack, SparkleFormation will add an output to the infrastructure stack “bubbling” the output to the root stack. Once it is available at the root stack, it can be passed to the application stack resource:

NOTE: The below example includes the nested stack contents. A real template will simply include a URL endpoint for fetching the document.

{
  "Resources": {
    "Infrastructure": {
      "Type": "AWS::CloudFormation::Stack",
      "Properties": {
        "Stack": {
          "Resources": {
            "Networking": {
              "Type": "AWS::CloudFormation::Stack",
              "Properties": {
                "Stack": {
                  ...
                  "Outputs": {
                    "Subnet": {
                      "Description": "Networking subnet",
                      "Value": {
                        "Ref": "SubnetResource"
                      }
                    }
                  }
                }
              }
            },
            "TemplateURL": "http://example.com/Networking.json"
          },
          "Outputs": {
            "Subnet": {
              "Value": {
                "Fn::Att": [
                  "Networking",
                  "Outputs.Subnet"
                ]
              }
            }
          }
        },
        "TemplateURL": "http://example.com/Infrastructure.json"
      }
    },
    "Applications": {
      "Type": "AWS::CloudFormation::Stack",
      "Properties": {
        "Stack": {
          "Parameters": {
            "Subnet": {
              "Type": "String"
            }
          },
          "Resources": {
            "Moneymaker": {
              "Type": "AWS::CloudFormation::Stack",
              "Properties": {
                "Stack": {
                  "Parameters": {
                    "Subnet": {
                      "Type": "String"
                    }
                  }
                  ...
                },
                "Parameters": {
                  "Subnet": {
                    "Ref": "Subnet"
                  }
                },
                "TemplateURL": "http://example.com/Moneymaker.json"
              }
            }
          }
        },
        "Parameters": {
          "Subnet": {
            "Fn::Att": [
              "Infrastructure",
              "Outputs.Subnet"
            ]
          }
        },
        "TemplateURL": "http://example.com/Applications.json"
      }
    }
  }
}

When the root template is compiled, it nests the infrastructure template, which in turn nests the networking template. The Subnet output is found, registered, and the compilation continues. At this point the networking template is the last of this “branch”, so compilation returns to the root template and starts on the nested applications template. It has moneymaker nested and when the moneymaker template is processed, the parameter Subnet is checked in the registered outputs. Since a match is found, two things happen:

  1. The Subnet output is “bubbled” to the infrastructure template
  2. The Subnet output from the infrastructure template is “dripped” into the applications template and passed to the moneymaker template

When a parameter is encountered and a matching output has been registered, SparkleFormation will add stack outputs to parent templates until a common context can be found between the requesting template (template with the parameter) and the providing template (template with the output). The common context for the two templates may not make it accessible to the requesting template, which is where the “dripping” method is employed.

Since the requesting template may not have access to the common context (as the example above illustrates), SparkleFormation will “drip” the value down to the template. It does this by injecting a Subnet parameter into child templates and passing the value in the stack resource until it reaches a common depth with the requesting template. Once this is completed, the requesting template will have access to the output from the provider template.

Deep Nesting Usage

Deep nesting is performed by calling SparkleFormation#apply_nesting. The method expects a block to be provided. This block handles storage of the nested stack template (if required) and any updates to the original stack resource.

sfn = SparkleFormation.compile(template_path, :sparkle)

sfn.apply_nesting(:deep) do |stack_name, nested_stack_sfn, original_stack_resource|
  template_content = nested_stack_cfn.compile.dump!
  # store the template content as required, set remote location as `template_url`
  original_stack_resource.properites.delete!(:stack)
  original_stack_resource.properties.set!('TemplateURL', template_url)
end