Read on Twitter
In my previous company some teams celebrated every production deployment because a lot of long-awaited changes were shipped each time.
If you worked with me you know how excited and proud I felt about our team's efforts and release milestones, *however* https://twitter.com/lizthegrey/status/1340091870584573953
If you worked with me you know how excited and proud I felt about our team's efforts and release milestones, *however* https://twitter.com/lizthegrey/status/1340091870584573953
In my opinion this is a sign that you might need to update your feature release strategy and CI/CD pipelines.
You'd want to get to a point when a production deployment is not a celebration, but a very boring, frequent occurrence.
You'd want to get to a point when a production deployment is not a celebration, but a very boring, frequent occurrence.
Assuming you are building a new microservice, this is what helped the teams I was part of:
(When it makes sense) your MVP should include, in each env:- monitoring (dashboards, alerts, metrics) & logging
- automated tests as part of your CI/CD (unit, integration, ...)
- resilience, automated error handling
*Create tickets* for this stuff & estimate them as team.
Plan your release strategy to make it incremental & with fast iterations. To achieve this, the tickets you create need to take that into account.- 1st release: basic infra with an hello world, no logic, one region
- 2nd: logic, tests for just 1 API endpoint (and repeat)
- ...
Implement (automated) error handling: - Retry strategies when calling dependencies
- Dead Letter Queues / error destinations
- Log errors and capture them in metrics/dashboards
- Autoscaling
- Failover (in a different available zone or AWS region)
Use dashboards in pre-prod envs to find anomalies:- Check for memory leaks or bottlenecks
- Once the business logic is finished and before getting real traffic, run load and stress tests to see how the system behaves with high traffic at spikes
Perform "dark deployments" to prod as much as possible. This means you are not getting real traffic yet, but the service is there in standby.
Make sure everyone in the team gets used to deploy to prod without fear and learns what to do before, during and after a prod deployment. Folks less used to deploy should be the ones that need to deploy the most. Every person in the team should feel equally empowered.
There is absolutely a lot more that I am forgetting, but this is what comes on top of my mind. Not all of what I listed makes sense for each case. You should apply only what makes sense for your requirements and scope.
[end]
[end]
