Analytical Compute Grid for a Managed Cloud Services Provider

By partnering with AccionLabs, a leading technology company in the US built and deployed a powerful data warehousing solution. The solution integrated data from disparate sources and enhanced the analytical capabilities for its customers.

This company served more than 200,000 business customers across four continents. The leading technology company provided managed cloud services and promoted a popular open source cloud platform. As a result, the client was drowning in data. Due to the large volume, velocity and variety of billing data, the company was unable to drill down to individual service levels with actionable insights. The lack of analytical tools to make business sense of these data made it difficult to take customer support to the next level. Billing analytics became a critical business requirement for this client.

The client’s legacy cloud based storage platform enabled enterprises to focus on generating revenue by tapping the power of the cloud, without the pain of hiring experts in dozens of complex technologies. The data warehousing solution used a set of 3 clusters made up of Relational (PostgreSQL), Columnar (Cassandra) and HDFS (Haddop+Hive) data stores. These clusters, in turn, had ‘n’ number of nodes.

The operational parameters and attributes of each cluster and its corresponding nodes were exposed through a variety of metrics: free memory, CPU utilization, cache memory, etc. Although these metrics were captured in the backend, there was no simplified UI to display this to the end user. As a result, the client required a web application that interfaced with these metrics and users seamlessly. This would help the client monitor these data and take suitable action. Additionally, the client required a solution in a compressed time frame. As a result, the client decided to use the services of Accion as an end-to-end product development partner.

DIGITAL TRANSFORMATION TO A DATA-DRIVEN ORGANIZATION

The existing system handled billing of independent services handled by different divisions, with consolidated billing completed through the centralized data warehouse. The objective of the project was to integrate billing data for all services across the company, and gain the ability to drill down from unified billing into individual service details using big data analytics.

AccionLabs CTO Ashutosh Bijoor started with a deep dive session into the legacy architecture of the client. The session detailed the technology challenges of the architecture, such as how to integrate the three data stores, how to view node status and metrics on each cluster type, and how to explore the best ways to display individual metrics for each node.

This initial consultation process highlighted how enhanced data representation and insights cloud drive improved decision making for the billing department. The resulting report outlined action points for how to provide analytics for each metric, configure metric preferences, provide a single ODBC/JDBC driver to access all data stores and support ad-hoc reporting using a standard SQL based reporting tool. Bijoor started scoping for a right-fit solution that solved their business problem by integrating multiple services and related data into a unified repository.

Accion proposed an EDC model to the client, which provided complete transparency in their work-order execution. The engagement set up two teams: one with expertise in backend technology such as big data and web services, and the other with frontend expertise such as HTML5, CSS, JS and AngularJS.

The Accion team incorporated a crucial feature in the web application, which allowed users to easily set sub-values or thresholds for different metrics through the interface. This was designed to help users decipher between normal and abnormal readings through a color coding scheme for every cluster/node object. The feature also allows users to either enable or disable a particular metric during front-end UI display.

The project completed data analysis of all existing reporting requirements for each service provider. Following this, the team built a Data Lake architecture with Polyglot Persistence to store raw data of each service data model. This was followed by Data Ingestion using batch and online operations, in-stream analysis to extract pre-calculated aggregates, alerts and notifications of outliers and abnormalities based on predefined rules. The team designed visualizations through continuous graphs for stronger analysis and decision making. The solution also delivered custom unified ODBC/JDBC driver development and RESTful API based integration for all individual service provider reporting requirements.

The project created a Cloud Hosted Data Warehouse Platform that combined three different storage models of HDFS, Columnar and Relational. The solution included an Indexing Engine that maps various external data sources to data storage engines, which allowed the user to seamlessly map external data sources to any internal data store. This included an integrated ODBC/JDBC access layer plus a REST API to access any data store for storing, retrieving or querying data.

This framework allowed all conventional Data Analytics tools and applications to access the data warehouse with minimal changes. The solution provided an implemented replication in PostgreSQL to allow horizontally distributed relational data. A management Console UI allowed for monitoring all the data store nodes.

The application also provided rich visualization in the form of continuous configuration of mappings of data sources to data stores, as well as usage statistics and graphs. This presented a platform for stronger decision making and in-depth analysis.

EXTENDED DELIVERY MODEL - THE ENGINE OF INNOVATION

At Accion, we understand the challenges that organizations face when it comes to digital transformation and technology upgrades. We are focused on ensuring our client goals are met, while also providing flexible models to plug and play based on the project requirements.

Our Extended Development Center (EDC) is one module that fits a company looking to add a new technology capability without investing manpower and operational bandwidth. The Center is built upon the philosophy of co-development and virtual teams, standing opposed to the traditional ‘black box’ model of outsourcing.

Under this model we provide our clients a range of services, such as new development, sustenance, QA, support, etc. These services are built on a strong operational framework and a mature governance model. This kind of partnership of trust and commitment through joint ownership creates equal investment and ensures success. The EDC model is built on the principles of talent creation and retention, brand development, 100% transparency and control, complete integration of teams, a customized delivery and execution methodology, and a deep focus on IP protection.

The technology solution for this client was built using Polyglot Persistence Architecture, combining a backend team with big data and web services and a frontend team with Rich Internet Application (RIA). The teams leveraged lightweight and an open source HTML5 based chart .js library for visualization. The solution used Java-based RESTful web services using JSON for interfacing with the backend platform.

BUSINESS VALUE

Accion successfully eliminated complex ETL processes and replaced them with a Big Data Lake, in the form of a horizontally scalable Analytical Compute Grid. The project implemented Deep Business Analytics with fine grain service detail; legacy reporting capabilities were migrated with no loss of data granularity. Seamless integration of external reporting applications and consolidated billing generated were at the core of the project, with no delay in billing cycles.

Unlock Business Value in your enterprise by using power of emerging technologies. Contact us to learn more.