Twitter Ranking Tweets With Machine Learning
Machine learning is entering production at Twitter as a way of ranking tweets and boosting engagement.
Twitter engineers this week unveiled the social media platform’s ranking algorithm driven by deep neural networks. In a blog post, company engineers said their approach leverages an in-house artificial intelligence platform that includes new modeling capabilities.
Among the results, wrote Nicolas Koumchatzky, a software engineer with Twitter’s AI team called Cortex, are “more relevant timelines now, and in the future, as this opens the door for us to use more of the many novelties that the deep learning community has to offer, especially in the areas of [natural language processing], conversation understanding and media domains.”
Currently, Twitter (NYSE: TWTR) timelines are arranged chronologically based on a user’s last visit. That alone is a daunting infrastructure task, the company notes. The new ranking algorithm gathers all tweets from accounts being followed by an individual user. It then scores those tweets using a relevance model, with the goal of predicting the most relevant comments. The highest-ranked tweets are then display at the top of a user’s timeline, Koumchatzky explained.
Twitter’s new ranking model takes into account factors such as the number of re-tweets, likes, the inclusion of images and video and other responses to a post. It also attempts to gauge a user’s past interactions with authors along with the “strength” and origin of the follower’s relationship to an author.
“This scoring step imposes an even greater computational demand on timelines serving infrastructure, as we are now scoring thousands of tweets per second to satisfy all of the timeline requests,” the Twitter engineer noted. “The unique challenge is to perform scoring quickly enough to instantly serve tweets back to the people viewing their timelines, yet have powerful enough models to allow for the best possible quality and future improvements.”
Along with prediction models, Twitter said similar requirements are applied to its machine learning frameworks. A set of tools is used to train and launch a prediction model, with particular attention given to training speed and scaling when dealing with very large datasets, much of it unstructured.
Based on the breadth of research into new AI algorithms and model architectures, Koumchatzky argued, “betting on a platform that natively supports deep learning and complex graphs is key to leveraging the promises of that work.”
The Cortex team of data scientists and machine-learning researchers works on Twitter’s deep learning platform. New members were added last year when Twitter acquired London-based AI startup Magic Pony Technology. These and other deals are part of a push by social media and other hyper-scalers to train rather than just program algorithms. It also underscores how AI technology is finding new automation applications required to organize huge datasets.
Having worked out most of the kinks in its deep learning models and platform, Twitter’s Koumchatzky said “online experiments have also shown significant increases in metrics such as tweet engagement and time spent on the platform.”
August 22, 2019
- VMware Signs Definitive Agreement to Acquire Pivotal Software
- VMware Enters Definitive Agreement to Acquire Carbon Black
- AWS Announces General Availability of Amazon Forecast
- Splunk to Acquire SignalFx
- SAP Positioned as a Leader in 2019 Gartner “Magic Quadrant for Data Integration Tools”
- Spark + AI Summit Returns to Amsterdam as the Largest Data and Machine Learning Conference in Europe
- Deep Learning Enables Scientists to Identify Cancer Cells in Blood in Milliseconds
August 21, 2019
- Paxata Recognized in the Gartner Peer Insights ‘Voice of the Customer’: Data Preparation Tools
- Syncsort Acquires SQData
- Okera Adds Visual, Natural Language Policy Creation Workflow to Data Lake Security and Governance Platform
- Databricks Offers Automation Throughout the End-to-End Data and Machine Learning Lifecycle
- Capacity Unveils AI Knowledge Sharing Platform to Boost Workplace Productivity
- ArangoDB Boosts Multi-Model Database Scalability Across Distributed Environments with Release of ArangoDB 3.5
- Siren Extends Scope of Platform with AI Functionality
August 20, 2019
- H2O.ai Raises $72.5M Led by Goldman Sachs and Ping An to Democratize AI
- Tuxera and Microsoft Strengthen Collaboration Through New exFAT Licensing
- Archive Document Data Storage Acquired by OASIS Group
- H2O.ai Releases its Automatic Machine Learning Platform
- Resolve Acquires FixStream for $24M
- NetApp Releases Storage Array Solution for Analytics and HPC Applications
Most Read Features
- Is Python Strangling R to Death?
- Big Data File Formats Demystified
- Re-Imagining Big Data in a Post-Hadoop World
- What HPE Sees in MapR Technologies
- Is Hadoop Officially Dead?
- 10 Big Data Trends to Watch in 2019
- Why Knowledge Graphs Are Foundational to Artificial Intelligence
- How to Build a Better Machine Learning Pipeline
- Big Data Is Still Hard. Here’s Why
- Data Catalogs Seen as Difference Makers in Big Data
- More Features…
Most Read News In Brief
- HPE Acquires MapR
- LinkedIn Data Helps to Create the First-Ever Global Map of Labor Flow
- MapR Says It’s Close to Deal to Sell Company
- Corporate Culture Continues to Stymie Data Insights
- War Unfolding for Control of Elasticsearch
- AWS Debuts PartiQL for Query Agnosticism
- California’s New Data Privacy Law Takes Effect in 2020
- Gartner Sees AI Democratized in Latest ‘Hype Cycle’
- LinkedIn Unleashes ‘Nearline’ Data Streaming
- Domo’s Latest ‘Data Never Sleeps’ Infographic – Just How Much Data Are We Generating Now?
- More News In Brief…
Most Read This Just In
- SAS Fulfills Pledge to Support HBCUs with Software and Partnerships
- New Graph Database Performance Benchmark Confirms Graph Databases are Ready for Solving Real-World Business Intelligence, Data Challenges
- Cray ARM-based System ‘Ookami’ to Serve as Testbed for Computational Studies at Stony Brook
- Illumina to Share their Data Virtualization Journey at Gartner Catalyst Conference
- Report: SAS Sees 105% Growth in AI Revenue
- Accenture to Acquire Analytics8, Australian Analytics and Data Specialists
- MLOps NYC19 Conference to Promote the Standardization of ML Operations
- SnapLogic Delivers AI-powered Pipeline Recommendations and Azure Databricks Support with Latest Platform Release
- USC Marshall Convenes Workshop on Fintech and Big Data
- DataRobot Launches its First AI Investment Workflow with FactSet
- More This Just In…
September 11 - September 12New York NY United States
September 23 - September 26New York United States
October 20 - October 22Charlotte NC United States
October 23 - October 24Berlin Germany