A component module has a focussed job and only interacts with a single cloud provider and only uses the low-level Terraform resources. It might group together an ASG, Launch Configuration and Security Group or a VPC, Subnets, Route Tables and Internet / NAT Gateways.

Component modules are typically the type of module you’ll find on the Terraform Registry. Some good examples are:

So, you’ve started using upstream component modules and you start to realise that you’re doing the same things, adding the same tags, using autoscaling, security-group and alb together to create a service and inputting a lot of default boilerplate values.

This is where Infrastructure modules take over. You can use Terraform modules within other modules. You can create modules that enforce best practice or certain requirements and create things in a consistent way across your infrastructures. They sometimes make use of multiple providers. Some examples:

Modules that do nothing more than give static output for use in other modules. They might be used to enforce available instance types, provide some data like an instance price to use for spot prices or normalise instance sizes across cloud providers. An example of a data module might be something like:

Hopefully some of this makes sense and can help you evolve your Terraform code into something more beautiful than it was previously.

Something I haven’t touched on here is managing Terraform module versions and dependencies. I might braindump a post on this soon. Thanks for reading!

Installation is as simple as: pip install publicholiday

Using publicholiday is easy, the command will exit with a status code of 0 if today is a public holiday and exit with a status code of 1 if today is not a public holiday:

By default, publicholiday uses UK bank holidays. It is possible to pass in a country using either the name or the short code as defined [here](https://pypi.org/project/holidays/). Examples:

Currently, only countries are supported. Province / state level holidays would be reasonably easy to implement and I’d be open to a PR with tests and documentation for that but it’s outside of the scope of my current needs.

Hopefully if you stumble upon this post it helps solve your problem.

I’ve setup automated internal CA’s to handle issuing certificates and re-generating them on expiry. In recent years, I’ve moved to using LetsEncrypt wherever possible.

I’ve found that in cloud environments that often scale up and down, running an agent like certbot on each instance is an anti-pattern.

Each time an instance is provisioned, it requests some certificates, LetsEncrypt will issue a new certificate to each instance. If you’ve ever tried the same thing, you’ll find that you can hit LetsEncrypt limits pretty quickly.

To solve this problem I created a Jenkins job to handle the certificate generation with the Certbot Docker container (lego is a good alternative), build an rpm package containing the certificates and push the rpm up to our private yum repos. From there all instances can have the cp-certs package installed as part of there bootstrapping / initial provisioning.

This job runs on a schedule every 7 days. On the first run of certbot, various configuration files and private keys are generated. Whilst Jenkins itself is relatively secure I don’t like the idea of these files sitting there in the workspace for anyone to get hold of, try to use, accidentally commit, use to decode https traffic…​

Initially I hunted round for a while for some sort of "Secure" or "Hidden" workspace plugin. There’s nothing. I sat and thought: "What is the actual risk I’m trying to reduce?" and came up with:

It dawned on me that the solution might be easier than I thought (damn brain getting in the way).

If I move the relevant sensitive files out of the workspace, there’s no longer a risk of someone borrowing them from the UI and if I encrypt / archive the files with a strong secret then they’ll be reasonably secure on the file system.

Obviously, someone who knows or can access the secret (me) and has access to the file system as the root or Jenkins user (me) could get at them but it’s a reasonable attempt at improving security.

DNS is easy right?! It’s just an IP address and a hostname. Easy…​ We’ve definitely never had an outage or failed to deploy a new application because of a DNS issue have we?

DNS can get a little more interesting when you start chaining CNAMEs, have multiple A records for a hostname and introduce DNSSEC.

PTR records which reverse map an IP to a hostname are often used by various server applications for security purposes (Java + SSL).

If DNS configuration is out of your control and another team forgets to add the records you need correctly you can end up wasting hours troubleshooting why various applications won’t start up, clients fail to connect and you have SSL connection errors.

Testing your DNS with Goss will solve ALL these problems! Okay, that’s a lie. It can however help you identify when DNS records aren’t quite right, have changed, or are missing before deploying a new application.

Goss can validate that any of the following record types are resolveable and can validate the values of the records.

Here are a few examples of DNS record tests:

The above examples will query Google’s public DNS server: 8.8.8.8 for results. You can remove the server parameter which will result in the system DNS resolver being used.

Combining this with the nagios output and creating a monitoring check from it could be helpful in identifying future issues or alerting when a record might have been "cleaned up".

Goss is a tool that let’s you easily and quickly validate infrastructure. Like Serverspec but without all the code. Goss allows you to define what a piece of infrastructure should look like with YAML or JSON. This is made even easier for us with the ability to auto add resources to the Goss configuration on the command line.

Goss allows you to validate many different resource types such as files, users, groups, packages, services and http connectivity. You can read the full Goss documentation here.

Well, a few things make goss an awesome tool for server validation:

All this validation of files, processes, services, ports and connectiovity sounds familiar. It’s something that we quite often try to achieve with our monitoring tools like Nagios, Zabbix or Sensu.

With Goss we can create a single monitoring check that tests many resources at once. Goss has several different outputs. The nagios_verbose output gives you out of the box testing compatible with Nagios or Sensu and gives Nagios long output explaining failures:

Server validation can now become part of your monitoring ecosystem ensuring that any problems are identified quickly with a single monitoring check.

Many applications expose "health" endpoints for applications and services.

Tools like Google’s Borgmon monitoring system and Prometheus use this HTTP(S) scrape or pull model to retrieve monitoring and metrics results from there services.

Goss has a serve command that exposes a http endpoint for scraping. You can then then use something like PHP Server Monitor to show the validation status of each piece of infrastructure.

Hopefully you can see the value of validation testing your infrastructure and building validation into your monitoring systems.

Goss is a young project and currently only supports Linux but it’s very active and open to contributions. You can help out by opening issues, discussing enhancements and submitting pull requests for review.

I’ll cover some advanced Goss usage in future posts. Thanks for reading!

In many beginners tutorials you get taught to put all your code in manifests such as site.pp or nodes.pp. For example:

This is great when you’re just starting out with a few servers to manage. You think you get it. Then you add a few more servers, you start adding more node specific config and before you know it you’ve got 10'000 lines of hand-crafted artisanal Puppet code. This was common in the early days of Puppet use. It was how I started back with Puppet 0.24.

Although, it’s not the best idea to manage your infrastructure in this way it’s actually a reasonably good way to very easily and simply bootstrap cloud instances. with separate manifests based on server type (web.pp, app.pp, lb.pp etc). These can than be applied using cloudinit to create an immutable bootstrapped node.

Quite often I see repos where people have puppet module install’d straight into the `modules directory or they’ve downloaded a module and extracted it there. The whole repo including their own modules mixed in with upstream modules is then committed to source control.

This pattern has a few problems:

Although this way works and you know that your module versions are pinned, tools are out there that make it much easier to manage your Puppet modules such as librarian-puppet and r10k.

When writing Puppet code it’s sometimes tempting to hard-code things like IP addresses or node specific things. For example:

This works, but the code isn’t re-usable. If you deploy to a different network or DC are your DNS servers still the same?

To improve re-usablility, change the variable to be a class parameter with an optional default value:

Then you can specify environment (network, node, DC) specific configuration in Hiera:

dc1.example.com.yaml:

dc2.example.com.yaml:

Now you avoid multiple classes or writing case or if {…​} else {…​} logic in your class file.

In the example above, I’ve mentioned a "profile" class. A common pattern when dealing with Puppet code is the Roles and Profiles Pattern. The idea is that you assign one "Role" per server and the role is made up of individual bite-sized "Profiles". Roles and profiles are just you or your companies custom Puppet modules that make use of upstream modules or Puppet resources to configure systems.

A common and problematic pattern I’ve seen is maintaining an SCM repo for roles, an SCM repo for profiles, an SCM repo for the "control repo" and a separate SCM repo for hieradata. This can then turn into multiple branches of each and before you know it you’re maintaining 12 versions of a diverging code base. It’s common to see this when people have followed some "best practice" blog post without fully thinking things through. You also often see this when developers created the code initially. They love git branches and complexity. Especially if trying to bring Gitflow to Puppet code.

Here’s my top tip when deciding how to organize your repos: KISS - Have no more than one repo for your products Puppet code. Multiple repos and branches leads to managing multiple very different code bases, different module versions in each environment, merge tasks, complex git fixing…​ a nightmare. There is an awesome control-repo by the guys at Example42 here: https://github.com/example42/control-repo

With the r10k tool it’s easy to manage environments dynamically with Git branches. You can then assign subsets of servers to use different environments (branches of Puppet code). This sounds great, we have applications deployed to different environments like test, dev, stage, uat or production. Cool! let’s create all those branches…​ STOP!!

Puppet environments are a powerful thing, but I don’t believe you should confuse Puppet environments with Application environments. You should aim to manage your infrastructure in a single Puppet environment: "production". If you arrange your hieradata hierarchy sensibly you can manage the differences in configuration in a single branch.

Puppet branches should be used when you need to test big changes or new features out in a controlled way (of course you’re developing and testing on Vagrant). Create a branch like "new_feature", develop it locally testing it on Vagrant then test out the changes on a suitable piece of infrastructure by running puppet apply --environment new_feature. Don’t forget your security and perf testing at this point ;) if everything looks good, open a PR, get it reviewed, merged to production and delete the branch.

Most of the opinions I’ve formed are from being forced to work with some painful Puppet code setups and processes. My best advice to anyone developing their infrastructure code: Keep it simple, think about it before you go ahead, keep an open mind and don’t be afraid to change your mind or refactor when necessary.