Identifying social media trends with deep learning

The interest in news services is being essentially influenced by social media posts. In order to monitor these new media channels trends and viral topics should be identifiable in real time. Editorial departments linked to our customers recommendations model were now able to stronger focus on editorial work, because of algorithmic monitoring.

Challenge

With the increasing popularity of social media the medium “news” has undergone a fundamental process of change. The interest in news services is being essentially influenced by social media posts. Our customer, who works as a consulting firm for media companies was facing the task of analyzing trends and viral topics in real time with the help of social media challenges. Thus, their customers should be provided with future and currently relevant news via a new platform.

Procedure

An algorithm based on deep learning methods was implemented, because of the problem’s complex nature.

Consequently, a correlation analysis was conducted first, in order to exclude irrelevant factors and therefore reduce training effort. We developed a multistate neuronal network, based on multi dimensional influencing factors. The content and information of social media posts, for example tweets, served as input parameters for said network.

Out of this, a scoring model was created, which could calculate a particular post’s range and relevance for the next 24 hours.

The training of the scoring model was implemented with Google Tensorflow and Kubernetes. By using containers for training and future scoring, the algorithm could be integrated into a CI/CD pipeline. Therefore daily updates for the algorithm were made possible through continuous learning or continuous deployment.

Results and customer value

Thus, we successfully developed a software as a service solution, which supports our customer’s media in the extraction of potentially relevant news content from social media.

The deep learning model, which has been implemented with Google Tensorflow and hosted in the Google Cloud forms the core of the project.

To evaluate the enormous amounts of data from social media almost in real time using the deep learning algorithm, dynamic scalability has been used within the cloud and combined with Kubernetes.

Thanks to using Kubernets and container technologies the intense training of the deep learning algorithm was completed in no time, paving the way for the introduction of continuous integration and a deployment pipeline. This way, we allowed our customer to train new deep learning models and to easily integrate them into the current system.

The solution developed by us, enables our customers to identify news topics, which can quickly gain relevance in the future, providing a competitive advantage in the creation of news services.

By using cloud technologies, initial capital costs were avoided. Costs merely incurred for the usage of cloud infrastructures (pay as you go). Therefore, our customers reached return on investment within three months.