AutoTrader Zooms Ahead with Automated ETL

If you’re in the market for a car, you’re probably using the Internet to check inventory and compare prices. From the consumer point of view, it all appears to work seamlessly, but keeping all that automotive data accurate and up-to-date is no easy task. For the folks at AutoTrader and its data integration subsidiary, VinSolutions, the move to an automated ETL process proved beneficial.

Founded in 1996, AutoTrader.com today is one of the most popular sources for market data on new and used automobiles for sale. But it’s not just a single website, and today the AutoTrader Group of companies is a $1-billion operation that spans AutoTrader.com, Kelley Blue Book, and various automotive data management subsidiaries, including VinSolutions. (A majority of AutoTrader Group itself is owned by Cox Enterprises.)

VinSolutions bears the brunt of the data integration challenges for AutoTrader and other websites that dealers use to advertise their vehicles. It’s essentially a data-focused middleman that sits between the car dealers on the one hand, and outfits like AutoTrader.com and Cars.com that market those cars to a broad Internet audience on the other.

For each car, there may be more than 100 individual pieces of data that need to be updated every day. This can include obviously critical piece of data, such as the list price, along with various other pieces of metadata that determine how the vehicle should be listed. With listings on more than a million cars requiring updates every day, the scale of the problem for VinSolutions approaches big-data territory.

VinSolutions has grown quickly over the past few years, as new and use car dealers and dealership groups look to get a leg up in this extremely competitive field. In addition to being the data middleman, VinSolutions offers a range of value-added solutions aimed at bolstering a dealer’s positioning across marketing, finance, search optimization, mobile, and social media.

At any given time, scores of consultants and trainers are fanning out from the company’s headquarters near Kansas City, Missouri, to get dealerships up and running with the software. Besides training salespeople how to use the new tools, much of the work involves hooking VinSolutions software the into the dealership management system (DMS), the core software applications that automate car dealerships.

This is where the difficulties started for VinSolutions. Many of the biggest dealers in the country have standardized on a handful of DMS applications, such as those from ADP, DealerTrack, and Reynolds. That’s the good news for VinSolutions. The bad news is that just about every ADP, DealerTrack, and Reynolds customer uses their DMS in different ways. And then there are another 60 to 80 smaller DMSs that VinSolutions works with, not to mention the custom-built DMSs that it must interface with.

About two years ago, the company sought a way to automate the homegrown ETL process and get out of the custom development business. It settled on the Centerprise offering from data integration software vendor Astera. According to VinSolution’s director of integration Doug Maskill, the software has dramatically reduced the need to custom code the integration into each dealer’s DMS.

“One of the big advantages in using Astera is it allowed us in development to build the core data flows, which then allows non-developers in our implementation department to use Astera to build sub-flows to connect into those core data flows,” Maskill tells Datanami. “That allowed us to create an environment where we don’t have developers doing the actual configuration for every single independent dealer. It allows the variances from dealer to dealer to be configured in Astera by our implementation team building data sub-flows. Trying to write code to do all that–that’s a lot of heavy lifting.”

According to VinSolutions senior programmer Mike Ethetton, it could take up to two months for a developer to do the custom-coding to ensure the accurate capture of data from a new client’s DMS into the VinSolutions network. With Astera, that has been cut to about an hour’s worth of work. With 70 to 100 car dealers going online with VinSolutions every month, the company has eliminated a lot of custom data mapping work on the part of its development team.

The new system doesn’t necessarily make the data more accurate. A consumer shopping for a car won’t notice any differences. But for VinSolutions, maintaining the high level of accuracy it historically delivered is easier now.

“It’s not that the data went to production bad,” Ethetton says. “It’s just that every time we bring in a new dealer, there was a lot of development effort that had to be taken into account, whereas with Astera, it’s a lot faster process, and it doesn’t require a developer to be involved every step of the way.”

The company is currently looking to take its ETL process to the next level with real-time data integration via REST Web services. This form of integration is highly dependent on whether a given DMS supports the specific APIs.

The hope is that, eventually, any change made to car data at the DMS level will automatically trigger an update into the VinSolutions systems. The company is starting to roll it out with the dealerships that support them, such as the corporate websites for Nissan, Kia, Toyota, and Lexus.

Related Items:

Auto Data Being Culled to Understand Driver Preferences

Report: Big Data Will Represent Billions in Automotive

Ford Looks to Hadoop, Innovative Analytics