Identifying social media trends with deep learning
The interest in news services is being essentially influenced by social media posts. In order to monitor these new media channels, trends and viral topics should be identifiable in real-time. Editorial departments linked to our customer’s recommendations model were now able to stronger focus on editorial work, because of algorithmic monitoring.
Challenge
With the increasing popularity of social media, the medium “news” has changed fundamentally. The interest in news services is being essentially influenced by social media posts. Our customer, who works as a consulting firm for media companies was facing the task of analyzing trends and viral topics in real-time with the help of social media challenges. Consequently, their customers should be provided with future and currently relevant news via a new platform.
Procedure
An algorithm based on deep learning methods was implemented, due to the complexity of the problem.
Consequently, a correlation analysis was conducted first, to exclude irrelevant factors and therefore reduce training effort. We developed a multistate neuronal network, based on multi dimensional influencing factors. The content and information of social media posts, for example, tweets, served as input parameters for said network.
Based on this, niologic created a scoring model, which could calculate the range and relevance of a particular post for the next 24 hours.
The training of the scoring model was implemented with Google TensorFlow and Kubernetes. By using containers for training and future scoring, the algorithm could be integrated into a CI/CD pipeline. Therefore daily updates for the algorithm were made possible through continuous learning or continuous deployment.
Results and customer value
Finally, we successfully developed a software as a service solution, which supports our customer’s media enterprises in the extraction of potentially relevant news content from social media.
The deep learning model, which has been implemented with Google TensorFlow and hosted in the Google Cloud forms the core of the project.
To evaluate the enormous amounts of data from social media almost in real-time using the deep learning algorithm, dynamic scalability has been used within the cloud and combined with Kubernetes.
Thanks to using Kubernetes and container technologies the intense training of the deep learning algorithm was completed in no time, paving the way for the introduction of continuous integration and a deployment pipeline. This way, we allowed our customer to train new deep learning models and to easily integrate them into the current system.
The solution developed by us, enables our customers to identify news topics, which can quickly gain relevance in the future, providing a competitive advantage in the creation of news services.
By using cloud technologies, initial capital costs were avoided. Costs merely incurred for the usage of cloud infrastructures (pay as you go). Therefore, our customers reached a return on investment within three months.
