Architecture and Implementation of a serverless platform for Data Science
Our client, the product branch of a top-tier management consultancy, required a platform for calculating short- and long-term calculations. niologic planned and implemented a calculation platform based on the design principle of a serverless architecture.
Challenge
The product development team had successfully launched a prototype of their analysis software a year earlier. In the meantime, further parameters and analysis steps have been added, so that despite further optimization the calculations were no longer possible immediately.
The previously synchronous calculation resulted in waiting times and failures. After investigating them, it became clear, that they were caused by the underlying network infrastructure (timeouts), among other things. At the same time, the range of distinct and differently complex simulations and calculations had increased significantly. A way to trigger calculation jobs and query progress at regular intervals was needed. At the same time, the platform had to be horizontally scalable so that bottlenecks could quickly be eliminated by scaling as the overall product continued to grow.
Procedure
niologic moderated the requirements for a calculation platform and evaluated several asynchronous worker engines / calculation platforms. One part of these platforms was based on classical service-oriented architectures (SOA), another part on the modern serverless architecture.
Based on the requirements catalog, three serverless platforms were shortlisted for a feasibility analysis (Proof-of-Concept / PoC). In the PoC it quickly became clear that only one candidate could meet the requirements of a stable and scalable calculations platform. In the end, a project was chosen, which could be integrated into the existing Kubernetes infrastructure at the same time.
niologic developed an API for the administration of calculation jobs, so that calculations received a unique key and could be referenced in the later progress display. Furthermore several basic images for Kubernetes were created, so that R, Python and Java-Worker could be executed on the platform.
Result and customer value
Within 4 weeks, the customer’s development team received a feasibility assessment, as well as planning and implementation of a serverless architecture, that was integrated with the existing systems. Through design principles such as asynchrony, the software could be extended by a status communication to the user. Other principles such as serverless and containerization encapsulated the algorithms and calculations, so that data scientists no longer had to worry about the execution of their algorithms. Rather, the execution of their program code is now encapsulated and can be loaded and modified as a package into the serverless platform.
The customer was able to save development time due to the short evaluation phase and the fast implementation. Furthermore, they could concentrate on the elements usability / communication of progress and calculation. Thus, the limitations of the original architecture were immediately eliminated by a redesign.
