What Can Enterprises Learn From Genome Sequencing?
The data handing requirements for genetic discovery are increasing as the tools become more sophisticated, causing data sets to become increasingly large. However, dealing with these large data sets is nothing new for practitioners in traditional science, and enterprises can learn from the strategies and processes that these disciplines have put into play.
Mario Caccamo, head of Bioinformatics at The Genome Analysis Center (TGAC) based in the UK, recently outlined the work and challenges that researchers face, explaining the explosive nature of the data and the processes that that genome scientists use to wrestle with it.
“We have super exponential growth in data generation,” explained Caccamo. “To give you a concrete example, with two sequencing instruments in 2010, we generated 1.2 terabases of data. We can now generate half of that (600 gigabases) in two weeks in only one sequencing round with our current instruments.”
This, of course, is a familiar refrain that enterprise executives are hearing as they consider their own data plans and consider what their own data requirements will be down the line. But having an infrastructure to handle and store this data isn’t the same thing as having a strategy to turn it into predictive intelligence.
“This is very much hitting the point where you need a strategy to cope with more data today than you have ever generated before,” says Caccamo, explaining the growing challenges TGAC faces as their data volumes increase in size and complexity. The organization looks at these challenges holistically, explained Caccamo in describing how they have built a cultural infrastructure to support their data-driven science goals.
Caccamo explains that a technology focus drives their cultural underpinnings. “In one end, developing the state of the art intensive platform, both sequencing and computational – this is what I call the ‘hard infrastructure.’ The other focus of our activities is on what we can develop between intensive algorithms and databases – this is what we call the ‘soft infrastructure,’”
Caccamo explains that TGAC’s challenge is on developing these components together through what he calls a systems approach in order to produce “predictive assumptions” that help them towards discovery and ultimately understanding biology.
Included in their base equation is the development of skills, says Caccamo, being very sensitive to the talent acquisition challenges that organizations face in the exploding data science fields. “We really take training into this,” explained Caccamo. “Technologies and training toward developing new strategies and tools is a very important part of what we do.”
Putting this organizational foundation into place, explained Caccamo, they are then able to give focus to what their true aim is: data driven science. Their process is something that other organizations can learn from, suggests Caccamo, explaining TGAC’s view of the pipeline that starts with data on the input end, and biology science as the output.
The process starts with huge volumes of sequencing data and using efficient algorithms to tackle the data, endeavor to distill it into what can then be classified as “information.” In the case of genomic research, the information are the ~150 letter base strings that the researchers use to assemble genome maps with. The research head says that once they have achieved the information step in the process, the focus, they shift priorities from efficiency to quality. At this step, says Caccamo, they’re focused on turning “information” into “knowledge” which can then be transformed into biology.
In the case of genome sequencing, the research head used the discovery process of the wheat genome as an example. The first step is in extracting and sequencing the DNA into strands of information called “bases,” which are ready for assembling into something more actionable.
These bases are then passed through sophisticated hardware (TGAC uses an array of SGI UV supercomputers for much of their assembly work) and assembled into enormous graphs. When mapped out in a graph, wheat has 10 billion nodes of sequencing information, says Caccamo. These graphs are now considered knowledge that can be used by biologists use for the purposes of obtaining better crops.
Caccamo explains that having the right strategy in place is essential for the long term success of TGAC because their data growth is exponential. He notes that as the data becomes cheaper and cheaper to process, the scope of the research tends to expand. In the example of wheat genetics, he explains that the research has expanded to examine environmental factors.
“A recent addition of the toolkit of the bioinformatician is that we can look now into what is present as well in the soil,” says Caccamo. “In this case, it’s not going to be about one species, but instead about a community of species – what we call a microbial community. That’s what we call metagenomics.”
While the concept of metagenomics is contained within the esoteric domain of the genomic community, the concept of the run-away project is not. Virtually every enterprise has experienced the resource drain that happens when a project expands beyond its original scope. In the case of big data, these run-away project can be very costly if there aren’t strategies in place to govern the direction they take.
As enterprises ramp up their big data initiatives, TGAC’s example suggests that organizations would be wise to consider the culture and processes already in place where traditional sciences have already blazed a trail in managing and processing enormous amounts of data.
October 21, 2021
- Dremio Announces New Dart Initiative Release
- Hex Technologies Raises $16 Million Series A to Help Data Teams Do More
- 2021 GigaOm Radar Report for Data Warehouses Names Yellowbrick Data an Outperformer
- DataRobot Research Finds 86% of Organizations Prioritize AI and ML
- Terrafuse AI Launches New Platform to Visualize California Wildfire Risk
- New Relic Launches In-IDE Observability and Code Collaboration Experience
- KX Announces Launch of KX Academy On-Demand Training Portal
- KDD 2021 Celebrates Winning Teams of 25th Annual KDD Cup
- Global Survey Reveals 8 in 10 Companies Struggle to Unify Data Assets
October 20, 2021
- OctoML Announces Collaboration with Arm for ML Models
- VAST Data Introduces VASTOS Version 4
- DAS42 and AtScale Partner to Deliver Advanced Data Technology Solutions
- Iguazio MLOps Platform Now Supports Amazon FSx for NetApp ONTAP
- Credo AI Emerges from Stealth to Help Organizations Build Ethical AI
- Exxact Partners with SoftIron to Provide Ceph-Based Software Defined Storage Solutions
- TigerGraph, HPE, and Xilinx Announce Graph Analytics Solution for the Enterprise
October 19, 2021
- Quantum Announces Partnership with IBM for Next Generation of LTO Technology
- Scality Delivers Comprehensive Portfolio for Splunk SmartStore Deployments
- Splunk Announces Enhancements to its Enterprise Observability Portfolio
- Datatron Introduces New Features to MLOps and AI Governance Solution
Most Read Features
- Google Cloud Gives Spanner a PostgreSQL Interface
- What Is Data Science? A Turing Award Winner Shares His View
- Big Data File Formats Demystified
- One on One with Google Cloud Product Director Irina Farooq
- We’re In the Moneyball 3.0 Era. Here’s What It Means for Live Sports
- What’s the Difference Between AI, ML, Deep Learning, and Active Learning?
- Who’s Winning In the $17B AIOps and Observability Market
- SambaNova Brings Custom Silicon To Bear on High-End AI Workloads
- Composite AI: What Is It, and Why You Need It
- OpenTelemetry Gains Momentum as Observability Standard
- More Features…
Most Read News In Brief
- Data and AI Salaries Continue Upward March, O’Reilly Says
- Bigeye Observes $45 Million in Funding
- Data Prep Still Dominates Data Scientists’ Time, Survey Finds
- Why Is SAS Going Public?
- Gartner Shuffles the Technology Deck with Latest ‘Hype Cycle’ Report
- LinkedIn Open Sources Tech Behind 10,000-Node Hadoop Cluster
- Feature Stores Emerging as Must-Have Tech for Machine Learning
- The Next Breakthrough in Long-Term Data Storage is….Gold?
- Sisu Nabs $62M to Grow Data Analytics Biz
- Logistics Operators Look to Data, Technology for Advantage
- More News In Brief…
Most Read This Just In
- TIBCO NOW 2021 Showcases Limitless Power of Data
- Esri Releases ArcGIS GeoBIM, Bringing Spatial Context to AEC Operations
- Databricks Acquires Low-code/No-code Company to Expand its Lakehouse Platform
- PrivaceraCloud 4.0 Enables Governed Data Sharing Across the Open Cloud
- MicroAI to Bring AI Training to Renesas MCUs
- TIBCO Delivers a Comprehensive, Connected Platform for the Adaptable Digital Business
- Scality Awarded New U.S. Patent for Breakthrough Technology in Hyper-Scale Data Protection
- BriefCam Introduces Video Analytics Enabled on Deep Learning Cameras from Axis Communications
- NetApp to Acquire CloudCheckr and Expand its Spot by NetApp CloudOps Platform
- OneTrust Enhances First-Party Data Solution to Strengthen Holistic Consent and Preference Management Platform
- More This Just In…
Sponsored Partner Content
October 27 - October 28
November 29 - December 3
December 6 - December 10San Diego CA United States
February 7, 2022 - February 9, 2022Houston TX United States
June 26, 2022 - June 30, 2022Hollywood FL United States