When The Buggles released their debut single “Video Killed the Radio Star” in 1979, their prescient video would ironically become the first music video played on MTV. A very real example of Digital Transformation causing a massive shift in culture and business. But digital transformation never ends, and the advent of high-speed internet would continue to radically change the music industry; where bands once toured to sell records, they now give away music to make money touring.
With the relentless pace of innovation using Lean software development, we’ve seen an explosion in the use of serverless technologies with a mantra of “NoOps.” Will this mean the death of the DevOps star? Unlikely. Research has shown teams implementing modern DevOps practices are significantly higher performing than those who don’t.1 And the gap is significant:
- 46 times more frequent code deployments
- 440 times faster lead time from commit to deploy
- 170 times faster mean time to recover from downtime
- 5 times lower change failure rate
Developers have plenty of choices for their serverless solutions today: AWS Lambda, Google Cloud Functions, Microsoft Azure Functions and Cloudflare Workers. All good choices each with their own strengths and weaknesses. For those concerned with vendor-lock, Serverless Framework may be a good choice. Honestly, however, vendor-lock is far less a problem than the cost of not pursuing a cloud-based agile digital transformation strategy. As long as you have vibrant innovation and competition between industry heavyweights like Microsoft, Google and Amazon, you’re unlikely to be taken advantage of.
The Maturation of Serverless
Serverless has matured a lot over the last 4 years. One big development are new powerful tools to aid building and deploying cloud-based serverless solutions, namely the Serverless Framework and the AWS Serverless Application Model. Both are open source. The Serverless Framework is a cloud-provider agnostic system that supports AWS, Azure, Tencent Cloud, Google Cloud, Cloudflare, Alibaba Cloud and twillo. Obviously, AWS SAM supports AWS products.
As an AWS Select Partner, we at Modus are big fans of AWS Lambda and their many serverless managed solutions. For this post, we’ll mainly focus on AWS Lambda.
The promise of “NoOps” with serverless is terribly misleading; it’s not that DevOps goes away, it’s more of a democratization by pushing it down to developers. The tools and practices may change, but the need does not. But this new model is fraught with danger; while serverless may ‘hide’ DevOps it opens the door to dangerous practices. Serverless or not, the best teams tightly integrate their DevOps practice with development early and continuously. As we’ll show, today’s DevOps star has an opportunity to be a massive force multiplier in Agile delivery for teams using serverless. One thing is for sure, as The Buggles chorus goes, “we can’t rewind we’ve gone too far…”
At Modus, we love serverless. Often cited benefits of serverless:
- No servers!
- No OS patches
- No containers
- No container orchestration
- Scales with usage
- No over or under provisioning
- Never pay for unused provisioning
- Highly available
- Lower latency when deployed to the edge
- Loose coupling for faster and easier development and deployment
- Easier and less expensive A/B testing and Blue/Green deployments
- Fine security granularity with Identity Access Management
- Serverless is Agile by design – perfect for Lean product development
Honing DevOps for Serverless
Obviously, as with any new technology, everything isn’t always sunshine, lollipops and rainbows. While not inherent flaws with serverless, the ease of serverless deployments can mask critical DevOps roles, responsibilities, and pitfalls that don’t go away with serverless. For example:
- Dealing with IAM users and roles in a secure fashion
- Dealing with the stateless nature of serverless
- Cold start latency and correctly provisioning concurrency
- Carefully using reserved concurrency (or not)
- Safely managing keys and certificates
- Dealing with challenging debugging
- Standardization of naming for management, logging and finance
- Standardized CI/CD pipelines
- Ensuring teams leverage extant performance visualization tools (e.g. X-Ray)
- Standardized Canary, Blue/Green and A/B testing approaches
- Account management and limits (Don’t DDOS yourself with tests)
- Micro-account strategies for better reliability and security
- Best use-case patterns (when to use step functions and SQS over Lambda)
- Solving networking issues – like firewalls – that can cause problems for serverless
- Templates for dealing with database connection limits
- Standardized caching patterns to deal with stateless serverless
- Standardized division of work between API Gateways and Lambdas
Obviously, DevOps isn’t going away. But how we ensure our development and operational quality may change. There are a couple approaches to achieving the goal: use processes – like DevOps code reviews and Agile tickets – or try to ‘bake-in’ DevOps best practices through templatizing repo creation and the deployment process. Or use a combination of both.
Consider how easy the AWS SAM CLI makes it to spin up an AWS Lambda service with a couple commands:
sam init cd sam-app sam build sam deploy --guided
That’s a pretty powerful tool to put in the hands of your developers; also, kind of scary. Let your development teams adopt this pattern and you can easily lose manageability, security and accountability (both personal and financial). One of Modus’ prospective clients got hit with a $320k unexpected bill from AWS and MSFT because of a lack of financial accountability in using serverless.
But the SAM CLI serves as a model of how to solve our many DevOps needs by templatizing all best practices.
Just as SAM builds on CloudFormation, we can use the same approach to build tools to automate the project creation. An excellent example is Expedia’s early work (before SAM) in templatizing creation of Lambda-based microservices. In 2016, Expedia was running 600 Lambda- based microservices with 2.3 billion computations per month for $550 per month. In order to codify their best practices and standards, they built a dashboard called Primer that allows developers to pick and choose predefined project templates that address specific needs. The benefits aside from cost: Creating governance standards so who owns what can easily be determined. Enforcing unit and e2e testing standards. Standardizing your integration tools, e.g. CodeBuild, CodePipeline, CodeDeploy and CodeStar. Standardizing your deployment pipeline with development, QA, staging and production instances. Creating patterns for blue/green deployments and monitoring. Creating Patterns for A/B testing.
Challenges to Serverless Solutions
Some have suggested Serverless (like Lambda) doesn’t fulfill the promise of no over-provisioning or under-provisioning because there can be hidden costs or problems. For example, the problem of running Lambda functions in VPC could consume all of your elastic IP addresses in the VPN. That’s not an argument against serverless in general – it’s just a challenge. Which, by the way, has now been mitigated by changes to ENI creation (or, rather, ‘somewhat’ mitigated – 14.8 seconds down to 933ms which still needs improvement). Or, we may have cold start latency issues that drive us to use provisioned concurrency, which breaks the promise of only paying for what you use. Arguable. But it certainly points to the need for DevOps stars helping our teams with the most extant and cost-efficient technology solutions.
While you automatically get CloudWatch logs for your services, the reality is that these are just not going to be an effective way to get debugging and performance visibility into multiple levels of stateless chains of microservices. Again, by creating templatized project solutions we can provide baked-in AWS X-Ray for both development and production environments. Development teams could certainly do this on their own, but it is a DevOps role to make this an organizational standard (and save developers a bunch of work).
Well-meaning people have suggested that cloud providers are selling a bill of goods that you can outsource your DevOps or SRE to the cloud provider. Are they? Cloud providers are actually selling the fact that you can create a tremendous level of homogeneity among your microservices. Cloud is a shared responsibility model; while they’ve made dealing with availability, scalability, and security easier and more uniform, we still have the responsibility of ensuring our CI/CD and site reliability are robust. By building on templatized SAM-like repository creation, we take tested and proven patterns and make them easily implemented so developers can spend their time delivering business value without having to reinvent the wheel for every project. It turns our DevOps and SRE teams into force multipliers.
And, if you’re thinking, ‘gosh, guess I don’t need Docker anymore…’ rest assured it’s more important than ever. Creating Lean rapid development pipelines for serverless cloud projects is inherently fraught with real challenges – especially when having to debug our serverless functions. AWS SAM used Docker, for example, to provide a local testing environment for Lambda. By building on Serverless Framework or SAM-like patterns, our DevOps teams can be a huge asset for our boots-on-the-ground developers. We’re going to see more and more tools and open source projects that should move this to an even higher-level of state-of-art.
Conclusion
So, no, serverless won’t kill the DevOps star. But they’re certainly going to be learning new approaches. It will be exciting to see what gets built on SAM to provide better DevOps capabilities in the the serverless world. Be sure to keep your eye on AWS Serverless Repository for new innovative solutions. While video certainly wasn’t going to help the career of Shane MacGowan of The Pogues – with a face made for radio – serverless can make our DevOps people become real stars.
Modus is an Amazon Web Services Select Consulting Partner, with special experience planning for, migrating to, and optimizing for the AWS cloud. To read more about our partnership and the benefits of working with Modus + AWS, go to our partner page.
1 Nicole Forsgren, et al., Accelerate (Portland: IT Revolution: 2018), 9-10
