Challenges
- Handle and consolidate data coming from a variety of sources
- Implement Microservices for various business functions
- Automate the entire deployment process to reduce the time to market. Multiple services caused high time to market due to testing effort involved in multiple environments
- Optimize operational cost of the setup through better utilization of resources on AWS
Solutions
- Implemented Lambda functions to transform data coming from multiple sources into the standard format
- Implemented Microservices and deployed to ECS (Docker Containers)
- Modified the design to use AWS Spot instances instead of the on-demand EC2 instances for running one of their daily jobs
- Improved the build and deployment pipeline through Jenkins
Tools & Technologies
AWS, Jenkins, Github, Python, React, NodeJS, Docker
Key benefits
- Streamlined the data transformation process using serverless architecture
- Since the build and deployment process was automated, we were able to roll out 2-3 builds per week with no deployment errors
- Operational costs came down by ~30% through Spot instances
- Using AWS ECS (Container Service) also ensured that resource utilization was better
