Public cloud, like AWS, which offers a plethora of services brings an extremely dynamic environment to an IT setup. Manually monitoring continuous changes happening beneath several cloud resources is a herculean task for any DevOps and CloudOps engineer. On the other hand, CloudOps, as well as DevOps processes of today, still need to catch up with the time-to-market expectations, with manual processes taking the life out of the engineers.For instance, tasks such as manually creating cloud backups, monitoring resources, managing hundreds of instances of different types with different roles are very challenging. However, with context-aware automation on AWS, it’s just a matter of few steps and one-time scripting.
Here’re few intelligent automation hacks on AWS you as a DevOps or CloudOps can live by:
1. Custom Auto-Scaling
AWS allows for auto-scaling using CloudFormation templates, Amazon Machine Images (AMIs), Puppet scripts, and other configuration management tools such as Chef. It requires a significant investment in time to create an automated, self-healing architecture, which replaces failed instances with new instances that scale out itself with little or no human intervention. Primarily, the effectiveness of any environment’s auto-scaling process will purely depend on the skills of cloud automation engineers.
And to reach the point of true auto-scaling, a cloud engineer needs to work with JSON in CloudFormation, install Puppet master, attach to those cloud resources the node requires to operate them (like Elastic IPs, network interfaces, or additional block storage), and then integrate deploy process to auto-scaling. For instance, updating new EC2 instances added to auto-scaling groups automatically. Post deployment, AMIs can be used to spin up exact replicas of instances, create a virtual machine within EC2, and then serve as the basic unit of deployment.
Even though this needs some effort, it is one of the few intelligent automation hacks on AWS every DevOps or CloudOps can live by.
2. Automating CRON and Other Jobs/Tasks
Every AWS resource costs money. Hence, optimal use of resources is pivotal in keeping a check on TCO. For instance, your company might have paid to run several RDS instances round the clock. There’s a higher chance that few RDS instances might go unused by developers in a non-production environment for a long period of time. Yet, your company is billed for the entire duration. Even though you terminate them, you will still be on the hook for provisioned storage, manual snapshots, and automated backup storage. And when you stop an instance, it will be paused only for seven days. So, after seven days, you will once again begin to accrue fees.
This is just the scenario for RDS. How about other resources running on your AWS inventory?
All resources residing in your AWS inventory cost money.
So, we’ve jotted down a quick cheat sheet of several jobs you can create, which help a DevOps and CloudOps to be on the bleeding edge of cloud automation on AWS:
For EC2 AMIs:
- De-register Old EC2 AMIs
- Create EC2 AMIs based on instance tags
- Copy EC2 AMIs based on instance tags across regions
- Copy EC2 AMIs based on EC2 Instance tags across accounts
- Copy EC2 AMIs based on EC2 AMIs tags across regions
For EBS Snapshots:
- Delete Old EBS Snapshots jobs
- Copy EBS snapshot based on EBS volume tags across regions
- Create EBS snapshot based on EC2 instance tags & IDs
- Copy EBS snapshot based on EC2 instance tags across regions & accounts
For RDS Snapshots:
- Copy RDS snapshot based on DB Instance tags across regions
- Create Redshift snapshots based on Redshift cluster tags
Route 53 hosted zone:
- Create backups for hosted zones
3. Customizing and Automating Monitoring of Alerts
Creating alarms, such as CloudWatch Alarms, plays a vital role in AWS cloud optimization. For example, AWS EC2 instances that use less than 10% daily average CPU on at least five of the previous 14 days are marked as low CPU utilization EC2 instances. Even the network IOs that are less than 5MB are marked as low utilization. These instances do not require higher configurations. In such scenarios, automate the task of changing the type of such instances to have a lesser configuration. By doing so, you will aid in drastically reducing the overall cost of AWS cloud infrastructure.
And with a plethora of such alarms being created, day in, day out, for ITOps, monitoring them leads to alerts fatigue. Of course there are innumerable monitoring tools available in the market that help prioritize alerts, however it’s taxing as the business scales on the cloud. The best way forward is to automate these IT alerts and intelligently monitor them.
The Wrap: Context-Aware Automation on AWS is the Way Forward
All your AWS cloud management and optimization efforts, either made in silos or with an integrated approach, such as AWS cost management, AWS governance, AWS cloud security and compliance, AWS cloud operations automation, and DevOps practices, need to be regularly monitored, nurtured, and evolved from time to time. More so, the DevOps practices have to adapt to the current cloud-driven world. Context-aware AWS automation is one such weapon that can help you win the battle.
Of course, there are several AWS services at your disposal to engineer a fully automated, continuous integration and delivery system, and help you be at the bleeding edge of DevOps practices and context-aware automation. It is, however, easier said than done.