Scaling terraform environments infracoders sydney 30 nov 2017

33 pages
22 views

Please download to get full document.

View again

of 33
All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.
Share
Description
This presentation looks at scaling patterns for Terraform, an infrastructure provisioning tool/language/framework. I will also demonstrate a code generator that I have written that will ensure that teams can adopt the Terraservices pattern as easily as possible. https://github.com/williamtsoi1/generator-terraform-environments A working example of the terraservices pattern is here: https://github.com/williamtsoi1/terraservices-example
Transcript
  • 1. PATTERNS FOR SCALING TERRAFORMEFFECTIVELY MANAGING YOUR ENVIRONMENTS
  • 2. PATTERNS FOR SCALING TERRAFORM AGENDA ‣ Introduction ‣ Love/Hate relationship with Terraform ‣ Case study – can you relate to the pain? ‣ Something to make your life easier ‣ Things to think about when you get home
  • 3. PATTERNS FOR SCALING TERRAFORM INTRODUCTION ‣ Consultant, Coach and Engineer in Continuous Integration, Infrastructure Automation, Agile and DevOps ‣ Current role: Senior Automation Engineer @ Vibrato ‣ Vibrato is a professional service IT consultancy that specialises in Automation, DevOps, Cloud Migration and Data Engineering
  • 4. PATTERNS FOR SCALING TERRAFORM I LOVE TERRAFORM! ‣ Started in mid-late 2015 (0.6.x) ‣ Lots to love ‣ terraform plan ‣ Dependency management ‣ Domain-specific language ‣ One language, many providers/clouds
  • 5. PATTERNS FOR SCALING TERRAFORM I HATE TERRAFORM! ‣ Live and die by your terraform.tfstate ‣ “Randomly” destroys stuff ‣ Corrupted state files ‣ Sharing state files between your team ‣ No established usage patterns apart from “Buy Terraform Enterprise!”
  • 6. PATTERNS FOR SCALING TERRAFORM LET’S TRY SOME ROLE PLAYING ‣ (scenario largely borrowed from Nicki Watt’s talk from HashiDays London 2017)
  • 7. PATTERNS FOR SCALING TERRAFORM DAY 1 ‣ New green field project ‣ Bastion host ‣ Compute Cluster ‣ Database ‣ Automate all the things! ‣ WIN!
  • 8. PATTERNS FOR SCALING TERRAFORM DAY 1 ‣ terraform.tf ‣ terraform.tfstate
  • 9. PATTERNS FOR SCALING TERRAFORM DAY 2 – PRODUCTION! ‣ Deploy to production… now! ‣ terraform-prod.tf ‣ terraform-test.tf ‣ terraform.tfstate
  • 10. PATTERNS FOR SCALING TERRAFORM DAY 3 – CHANGE TO TEST ‣ terraform-prod.tfbkp ‣ terraform-test.tf ‣ terraform.tfvars ‣ terraform.tfstate
  • 11. PATTERNS FOR SCALING TERRAFORM TERRALITH: CHARACTERISTICS ‣ Single state file ‣ Single definition file ‣ Hard coded config ‣ Local state ‣ Can’t manage environments separately ‣ Config not intuitive ‣ Maintenance nightmare: Duplicate code
  • 12. PATTERNS FOR SCALING TERRAFORM LET’S TRY AGAIN
  • 13. PATTERNS FOR SCALING TERRAFORM MULTI-TERRALITH ‣ Separated state file between staging and production ‣ SLIGHTLY more intuitive (network and VM split into separate files) ‣ Still lots of duplication (networks.tf and vms.tf still duplicated)
  • 14. PATTERNS FOR SCALING TERRAFORM MODULAR! ‣ Database ‣ Amazon RDS ‣ DB Subnet groups ‣ Compute ‣ Instances ‣ Security Groups ‣ Core ‣ VPC ‣ Subnets ‣ Core Routing and Gateways ‣ Bastion Host
  • 15. PATTERNS FOR SCALING TERRAFORM TERRAMOD ‣ Separate out environment management (config) and module definitions (code) ‣ Logical components as reusable modules ‣ No config or hard-coding allowed in modules ‣ Input.tf and output.tf essentially acts as “contracts” of the module
  • 16. PATTERNS FOR SCALING TERRAFORM
  • 17. PATTERNS FOR SCALING TERRAFORM LIFE IS PRETTY GOOD, UNTIL… ‣ You get asked to reduce the size of the bastion box in production ‣ Piece of cake! ‣ Just change the bastion_flav value in the production terraform.tfvars!
  • 18. WHERE’D MY CLUSTER GO? ‣ Someone got lazy and reused the var.bastion_flav variable! ‣ AWS will destroy the instances and reprovision them since the sizes have changed… ‣ Managing environments separately, but not the logical components! PATTERNS FOR SCALING TERRAFORM
  • 19. PATTERNS FOR SCALING TERRAFORM SOLUTION: TERRASERVICES ‣ Expand out environments folder to separate out logical components ‣ Logical components are separated out! Changing config for the bastion will no longer accidentally break things in other modules, even in the same environment
  • 20. PATTERNS FOR SCALING TERRAFORM TERRAMOD RECAP
  • 21. PATTERNS FOR SCALING TERRAFORM HOW TO DO THE SAME THING USING THE TERRASERVICES PATTERN? environments/production/core/compute.tf environments/production/core/output.tf
  • 22. PATTERNS FOR SCALING TERRAFORM HOW TO DO THE SAME THING USING THE TERRASERVICES PATTERN? environments/production/compute/terraform.tf
  • 23. PATTERNS FOR SCALING TERRAFORM TERRASERVICES - IMPLICATIONS ‣ Require additional orchestration effort ‣ Deploy the core (VPC + subnets) before deploying compute (EC2 instance) ‣ tfstate explosion = (number of environments) * (number of logical components) ‣ Need a standard practice on laying out the tfstate files ‣ This setup is for larger teams and enterprises. Smaller teams can just use Terramod ‣ Remote state & distributed locking becomes really important ‣ Requires so much more code to write and maintain just to reference all the remote state files!
  • 24. PATTERNS FOR SCALING TERRAFORM INTRODUCING THE TERRAFORM- ENVIRONMENTS CODE GENERATOR!‣ Yeoman ‣ Supports TerraServices and Terramod patterns ‣ Currently only supports s3 remote state ‣ Future: support anything with distributed state locking (azurerm, gcs, consul)
  • 25. DEMO TIME
  • 26. PATTERNS FOR SCALING TERRAFORM INSTRUCTIONS ‣ Install yeoman ‣ npm install –g yo ‣ Install the generator ‣ npm install –g generator-terraform-environments ‣ Create your project folder, cd to it, then run ‣ yo terraform-environments
  • 27. PATTERNS FOR SCALING TERRAFORM BACKUP FOR IF THE DEMO DOESN’T WORK ‣ https://asciinema.org/a/tUwkFEpuWmR4lJVwYvrKRRkit
  • 28. PATTERNS FOR SCALING TERRAFORM THINGS TO THINK ABOUT WHEN YOU GET HOME ‣ Use either Terramod or TerraServices. Do not Terralith! ‣ How to split modules? Think about: ‣ Team/responsibility structures ‣ Release cadence of various components ‣ Overall architecture of the system ‣ What remote state to use? Distributed locking is important! ‣ Branching model for this repo? Github-flow should be fine ‣ Security for remote state – prevent tampering & accessing secrets
  • 29. PATTERNS FOR SCALING TERRAFORM APPENDIX ‣ Project homepage: https://github.com/williamtsoi1/generator-terraform-environments ‣ Example using TerraServices: https://github.com/williamtsoi1/terraservices-example ‣ Contact details ‣ william.tsoi@vibrato.com.au ‣ @williamtsoi on Twitter ‣ https://about.me/williamtsoi
  • Related Search
    We Need Your Support
    Thank you for visiting our website and your interest in our free products and services. We are nonprofit website to share and download documents. To the running of this website, we need your help to support us.

    Thanks to everyone for your continued support.

    No, Thanks
    SAVE OUR EARTH

    We need your sign to support Project to invent "SMART AND CONTROLLABLE REFLECTIVE BALLOONS" to cover the Sun and Save Our Earth.

    More details...

    Sign Now!

    We are very appreciated for your Prompt Action!

    x