Twitter Ranking Tweets With Machine Learning
Machine learning is entering production at Twitter as a way of ranking tweets and boosting engagement.
Twitter engineers this week unveiled the social media platform’s ranking algorithm driven by deep neural networks. In a blog post, company engineers said their approach leverages an in-house artificial intelligence platform that includes new modeling capabilities.
Among the results, wrote Nicolas Koumchatzky, a software engineer with Twitter’s AI team called Cortex, are “more relevant timelines now, and in the future, as this opens the door for us to use more of the many novelties that the deep learning community has to offer, especially in the areas of [natural language processing], conversation understanding and media domains.”
Currently, Twitter (NYSE: TWTR) timelines are arranged chronologically based on a user’s last visit. That alone is a daunting infrastructure task, the company notes. The new ranking algorithm gathers all tweets from accounts being followed by an individual user. It then scores those tweets using a relevance model, with the goal of predicting the most relevant comments. The highest-ranked tweets are then display at the top of a user’s timeline, Koumchatzky explained.
Twitter’s new ranking model takes into account factors such as the number of re-tweets, likes, the inclusion of images and video and other responses to a post. It also attempts to gauge a user’s past interactions with authors along with the “strength” and origin of the follower’s relationship to an author.
“This scoring step imposes an even greater computational demand on timelines serving infrastructure, as we are now scoring thousands of tweets per second to satisfy all of the timeline requests,” the Twitter engineer noted. “The unique challenge is to perform scoring quickly enough to instantly serve tweets back to the people viewing their timelines, yet have powerful enough models to allow for the best possible quality and future improvements.”
Along with prediction models, Twitter said similar requirements are applied to its machine learning frameworks. A set of tools is used to train and launch a prediction model, with particular attention given to training speed and scaling when dealing with very large datasets, much of it unstructured.
Based on the breadth of research into new AI algorithms and model architectures, Koumchatzky argued, “betting on a platform that natively supports deep learning and complex graphs is key to leveraging the promises of that work.”
The Cortex team of data scientists and machine-learning researchers works on Twitter’s deep learning platform. New members were added last year when Twitter acquired London-based AI startup Magic Pony Technology. These and other deals are part of a push by social media and other hyper-scalers to train rather than just program algorithms. It also underscores how AI technology is finding new automation applications required to organize huge datasets.
Having worked out most of the kinks in its deep learning models and platform, Twitter’s Koumchatzky said “online experiments have also shown significant increases in metrics such as tweet engagement and time spent on the platform.”
September 13, 2019
- Duality Technologies Named a “Cool Vendor” by Gartner for Privacy Preservation in Analytics
- Valen Analytics Announces InsureRight Manage 2.0
- HPCwire and EnterpriseAI to Cover Silicon & Systems for Deep Learning at AI Hardware Summit as Headline Media Partners
September 12, 2019
- Odaseva Announces Growth with One Trillion Documents Supported and Over 10 Million Users
- IBM and ŠKODA AUTO University Collaborate on new Digital Skills for Students
- Trifacta Raises $100M to Support Explosive Growth of Data Wrangling for AI and the Cloud
- Snowflake and Fedresults Bring Cloud Smart Technology to Federal Government
- IBM Unveils z15 With Industry-First Data Privacy Capabilities
- StorageCraft Research Reveals Rampant Data Growth, and Inadequate IT Infrastructures are a Source of Global Concern and Risk
- Sumo Logic Accelerates Continuous Intelligence for Modern Enterprises with New Product Innovations
September 11, 2019
- StackRox Launches New Sumo Logic App for Kubernetes Security
- Sumo Logic Showcases the Intelligence Economy at Illuminate 2019
- Multi-Cloud on the Rise and Open Source Tech Like Kubernetes is Disrupting the Modern Application Stack, According to Sumo Logic Research
- Accenture Acquires Pragsis Bidoop
- Lucidworks Fusion 5.0 Features Data Science Toolkit Integration & Microservices Architecture Orchestrated by Kubernetes
- TIBCO and Asia Pacific University of Technology and Innovation Announce Enriched Collaboration
- Looker Brings the Data Community Together at JOIN 2019
- InfluxData Launches InfluxDB Cloud 2.0
- Nationwide Drives Data-enabled Culture with ‘Fit to Fly’ Analytics Strategy
- Hazelcast Enhances Real-Time Capabilities for Financial Services Industry
Most Read Features
- Is Python Strangling R to Death?
- Can We Stop Doing ETL Yet?
- Big Data File Formats Demystified
- Seeing the Big Picture on Big Data Market Shift
- Re-Imagining Big Data in a Post-Hadoop World
- How to Build a Better Machine Learning Pipeline
- Is Hadoop Officially Dead?
- 10 Big Data Trends to Watch in 2019
- Why Knowledge Graphs Are Foundational to Artificial Intelligence
- AutoML Tools Emerge as Data Science Difference Makers
- More Features…
Most Read News In Brief
- HPE Acquires MapR
- R Backers Tout Funding Milestone, Seek Comeback
- H2O.ai Tops Off Funding to Accelerate AI Adoption
- Startup Rockset Adds SQL to DynamoDB
- AI, Analytics Help to Propel Wind Power
- MapR Says It’s Close to Deal to Sell Company
- War Unfolding for Control of Elasticsearch
- StreamSets Eases Spark-ETL Pipeline Development
- California’s New Data Privacy Law Takes Effect in 2020
- How IBM Is Turning Db2 into an ‘AI Database’
- More News In Brief…
Most Read This Just In
- Cray ARM-based System ‘Ookami’ to Serve as Testbed for Computational Studies at Stony Brook
- Illumina to Share their Data Virtualization Journey at Gartner Catalyst Conference
- Cloudera Agrees to Acquire Arcadia Data
- Report: SAS Sees 105% Growth in AI Revenue
- Ascend Introduces Queryable Dataflows for Faster Pipeline Development and Overall Time to Big Data Success
- SnapLogic Delivers AI-powered Pipeline Recommendations and Azure Databricks Support with Latest Platform Release
- Accenture to Acquire Analytics8, Australian Analytics and Data Specialists
- New Graph Database Performance Benchmark Confirms Graph Databases are Ready for Solving Real-World Business Intelligence, Data Challenges
- SAS Fulfills Pledge to Support HBCUs with Software and Partnerships
- NVIDIA vComputeServer with NGC Containers Brings GPU Virtualization to AI, Deep Learning and Data Science
- More This Just In…
September 23 - September 26New York United States
October 20 - October 22Charlotte NC United States
October 23 - October 24Berlin Germany