Challenges
- Handle and consolidate data coming from a variety of sources
- Implement microservices for various business functions.
- Automate the entire deployment process thereby reducing the time to market.
- Given that there were multiple services involved (microservices architecture), time to market was usually high due to the testing effort involved in multiple environments.
- Optimize the operational cost of the setup through better utilization of resources on AWS.
Solutions
- Implemented Lambda functions to handle data coming from multiple sources.
- Implemented microservices and deployed to ECS (Docker Containers)
- Modified the design to use Spot Instances instead of the on-demand EC2 instances for running one of their daily jobs.
- Improved the build and deployment pipeline through the use of Jenkins.
- Automated verification using Jasmine, Mocha, Karma.
Tools & Technologies
AWS, Jenkins, Github, Python, React, Node JS, Docker
Key benefits
- Streamlined the data transformation process using a serverless architecture.
- Since the build and deployment process was automated, changes could be rolled out faster and without errors.
- Test automation ensured better coverage of test cases.
- Operational costs came down significantly through the use of Spot Instances.
- ECS also ensured that resource utilization was better.
