Within just a few days, the customer’s employees were able to run Big Data analyses that were as easy to use as a spreadsheet. At the same time, employees received further support through the adapted logging and alerting of the multipliers. This showed the strength of custom logging based on fluentd and Google Stackdriver.
By further conversion of data formats to column-based and compressing formats, the performance of Spark InMemory Processing could be increased by a factor of 100. Among other things, the multi-client capability of the system was achieved through horizontal scaling and an optimization of the Spark configuration. At the same time Dataiku could be offered important ideas for a further optimization of the software, so that for example today Kubernetes is optimally supported by Dataiku.
The customer thus found both a technically stable platform and an agile solution adapted to the consulting processes. The consultants are able to process customer data within a short period of time and to add further data through their own uploads (Big Data Enablement).
The close cooperation with the customer’s IT security numerous security measures could to be taken to regulate data security and the accessibility of the systems.