Ohio Council of Retail Merchants

A single-purpose web scraper that reformats the expected data into an ideal format

About the project:

While data integration is something we're very familiar with at Software Verde, this project was unique because the target integration point was not formatted for easy consumption and was potentially subject to unexpected changes. After analyzing the data and developing an understanding of how it would be used, we put together a set of guidelines for how each piece of data should be extracted, largely using regular expressions. From there, we were able to build a tool around that data.

By separating data capture from data usage, we were able to better prepare for future changes to the data source and build the application in a way that could be more easily migrated if a new source became available. We always strive to build our applications to be as "future-proof" as possible. Unexpected changes are always possible, and not all scenarios can be anticipated, but the more flexibly we build systems, the more likely they are to hold up over time.

Services Provided

  • Data Integration
  • Unstructured Data Extraction
  • Regular Expression Processing
  • Data Parsing & Transformation
  • System Design
  • Data Pipeline Architecture
Looking for a similar solution?