Challenges
- The client wanted to scale up the legacy data warehouse system built on RDBMS based MPP system and perform real-time analytics on the marketing data in order to generate useful business insights
- The client was receiving data from various data sources like SAP ERP, JDA, IRI, Acousta and many others, and was facing challenges in integrating the data from these data sources & performing real time analytics on the given data
- The client wanted to switch to a scalable EDM system which :
- Is easy to integrate with the other enterprise applications
- Minimizes the effort spend on data migration and processing
Solutions
- Analyzed the various data sources and targets and designed and implemented a suitable Hadoop architecture for the same
- Designed near real-time ingestion system using Attunity and Kafka to perform real-time batch processing
- Designed a data warehouse on Hive and HBase for all transactional data processing
- Used Talend ETL tool to design and develop Spark Streaming and batch jobs
- Designed OLAP cubes for reporting
Tools & Technologies
Hadoop, Talend, SAP Hana, Kafta, Hortonworks, Hive
Key benefits
- Production rate can now be matched with order rate, thereby enabling faster and more informed decisions for sales, pricing, product, and manufacturing teams
- Provides real-time operational visibility
- Able to take quicker and informed decisions
- Lower data integration cost
- Easily scalable architecture
- Ability to drill across cubes/data subject in pre-defined and ad-hoc ways
- Attunity replicates the integrated data into Kafta streams thereby providing a comprehensive view of it to the managers
