Analyze This: How to Prepare Your Unstructured Data in Six Steps
We live in a data-rich world where information is ours for the taking. But throwing just any data at your algorithm is a bad idea. With AI, small inconsistencies quickly become big ones. And those mistakes affect your decision-making, reputation, and bottom line. That’s why you need to prepare your data before you hand it over to your algorithms.
Here’s how to put quality data in—so you get quality data out.
Step: 1 Clean Your Data
Junk data is part of life, especially with qualitative (text-based) data. Before you hit “upload” to your analytics platform, find and strip out low- or no-value data. You’ll improve your quality—and avoid wasting valuable processing credits.
Remove fields containing nothing, n/a, and gibberish. You can generally also remove very short text responses. Exceptions are when a question specifically asks for a very short response, or when users write something in a “further comments” box.
Sometimes you might want to add data, such as the term “N/A” to an empty cell. This will let the system process those data points instead of just skipping over blank text fields. Check how your system handles special characters—some will skip over text fields consisting of a certain percentage of these. Adjust these cells if you need to.
Step 2: Combine Like Data
Scraped or exported data often arrives in multiple files. You’ll get better results by combining your data into fewer files – or even just one. When deciding how to combine your data, know:
- What you’re looking for
- If you’re comparing and contrasting, and if so what your main comparison point is
- If you’ll aggregate, then sort data
- If your data sources need to be separated, and if so whether you’ll build separate dashboards for each source
- How big the files are (NB, combining sources creates very large files that take longer to process)
For example, say you’re comparing reviews from “App A,” “App B,” and “App C” from both the Apple App Store and the Google Play Store. The review data will be 6 files: one for each app from each source.
You can combine this data in a few different ways. You could collate the data from the Apple App store in one file and the data from the Google Play store data in another. Or save the reviews from each app across both stores into three separate files. Or you could combine all of the data into one large file.
Why one over the other? It depends on your goals. If you want to contrast the Apple App reviews with the Google App reviews, then two files makes sense. If you’re comparing the Apps themselves, then three files might make more sense. If you only need to process the data once, a single huge file is fine.
Step 3: Add Metadata
Metadata provides information about your data. It helps you find, use, filter, sort and preserve your data. The more metadata, the better—just so long as it’s good quality.
Always add the essentials:
- Document source
- Date(s) created
- Date(s) scraped/pulled
- Author
You can also add:
- URLs
- Groupings
- Notes
- Names
- Locations
- Tags
- Other relevant info
You can upload as much or as little metadata as you want. But more metadata makes it easier to sort and filter your data.
Step 4: Sort Your Metadata
Consistency matters. Properly format your metadata so that you can find and filter it in your system. Unformatted metadata just makes life harder for you. To get started:
- Check date formats
- Standardize formatting
- Fix misspellings or variations (Apple vs apple vs Apple Inc)
Uploading multiple documents to analyze, filter and graph? The formatting must be consistent across all the files so you can sort and compare them. In the above App store example, you’d want the source field to always be “Apple App Store” or “Google Play Store” across every document.
Step 5: Save Your File
If you’re using Excel or Google Docs to prepare your data, save two copies of your cleaned and prepared data: one in a native file type and one in a CSV. CSVs tend to upload faster. They’re also easy to open and edit on the fly, no matter what OS or software you use.
Step 6: Tune Your Sample Sets
Planning to tune and reprocess your data multiple times? Create special tuning sets. These are small selections of your data used to “tune” your configurations. Because they’re smaller, Kyour system can quickly process them. You’ll get timely feedback without burning through processing credits. Once you’ve tuned with your smaller sets, move on to a larger set to confirm that your results align with your expectations. Or repeat the process with a new tuning set.
Getting your data in great shape before you feed it to an algorithm will net you better results and let you get more from your tech. Use the data prep best practices above, and you’ll spend less time fixing and more time analyzing.
About the author: Paul Barba is the chief scientist at Lexalytics, an InMoment company and a provider of analytic solutions for structured and unstructured data. Paul has 10 years of experience developing, architecting, researching and generally thinking about machine learning, text analytics and NLP software. He earned a degree in Computer Science and Mathematics from UMass Amherst.
Related Items:
Stemming vs Lemmatization in NLP
Five Ways Big Data Projects Can Go Wrong (And What You Can Do About Them)
April 25, 2024
- Cleanlab Launches New Solution to Detect AI Hallucinations in Language Models
- University of Maryland’s Smith School Launches New Center for AI in Business
- SAS Advances Public Health Research with New Analytics Tools on NIH Researcher Workbench
- NVIDIA to Acquire GPU Orchestration Software Provider Run:ai
April 24, 2024
- AtScale Introduces Developer Community Edition for Semantic Modeling
- Domopalooza 2024 Sets a High Bar for AI in Business Intelligence and Analytics
- BigID Highlights Crucial Security Measures for Generative AI in Latest Industry Report
- Moveworks Showcases the Power of Its Next-Gen Copilot at Moveworks.global 2024
- AtScale Announces Next-Gen Product Innovations to Foster Data-Driven Industry-Wide Collaboration
- New Snorkel Flow Release Empowers Enterprises to Harness Their Data for Custom AI Solutions
- Snowflake Launches Arctic: The Most Open, Enterprise-Grade Large Language Model
- Lenovo Advances Hybrid AI Innovation to Meet the Demands of the Most Compute Intensive Workloads
- NEC Expands AI Offerings with Advanced LLMs for Faster Response Times
- Cribl Wins Fair Use Case in Splunk Lawsuit, Ensuring Continued Interoperability
- Rambus Advances AI 2.0 with GDDR7 Memory Controller IP
April 23, 2024
- G42 Selects Qualcomm to Boost AI Inference Performance
- Veritas Strengthens Cyber Resilience with New AI-Powered Solutions
- CERN’s Edge AI Data Analysis Techniques Used to Detect Marine Plastic Pollution
- Alteryx and DataCamp Partner to Bring Analytics Upskilling to All
- SymphonyAI Announces IRIS Foundry, an AI-powered Industrial Data Ops Platform
Most Read Features
Sorry. No data so far.
Most Read News In Brief
Sorry. No data so far.
Most Read This Just In
Sorry. No data so far.
Sponsored Partner Content
-
Get your Data AI Ready – Celebrate One Year of Deep Dish Data Virtual Series!
-
Supercharge Your Data Lake with Spark 3.3
-
Learn How to Build a Custom Chatbot Using a RAG Workflow in Minutes [Hands-on Demo]
-
Overcome ETL Bottlenecks with Metadata-driven Integration for the AI Era [Free Guide]
-
Gartner® Hype Cycle™ for Analytics and Business Intelligence 2023
-
The Art of Mastering Data Quality for AI and Analytics
Sponsored Whitepapers
Contributors
Featured Events
-
AI & Big Data Expo North America 2024
June 5 - June 6Santa Clara CA United States -
AI Hardware & Edge AI Summit Europe
June 18 - June 19London United Kingdom -
AI Hardware & Edge AI Summit 2024
September 10 - September 12San Jose CA United States -
CDAO Government 2024
September 18 - September 19Washington DC United States