External Service Parsing (In Development)

This page is in development. Although the below accurately describes this project, I hope to have a working demo published soon.

The Scenario

A company requires large amounts of data from an external industry site, available in .csv files. This data is required to update the company’s live database on demand with current data periodically – once every few months or once a year. The required data from these files may exist in only some of the files or all of them. Normally this is a manual process, the requirement is to automate this process. Users should be able to initiate a download/parse of the external data, be able to compare values of the new data with current data, choose to update all or only some of the values on demand, and commit the new data to the live database.

The Logic

To implement this system requires:

  • On selection of a "Refresh Data" users initiate the retrieval of the external data in which it is downloaded to a temporary directory. The data is a .zip archive and unzips to multiple .csv files.
  • Using a configuration file which maps the .csv file names and their columns to the temporary database columns, the .csv files are searched for the relevant data and stored in an array.
  • On completion of the insertion of the refreshed data into the temporary database, the GUI reloads to show the current live values next to the new refreshed values, for example 
        Live Company Name | Current Company Name | Live Company CEO | Current Company CEO (etc.)
  • Users can choose to import all the new data, or select one or many rows to update.
  • The list may be very long, so the GUI must support pagination.
  • There must be some rollback capability.
  • The file names downloaded and columns in which the required data exist, and in which files, may change at any time. The system should have a configurable portion of the GUI where users can view the downloaded files and set the mapping for the required data columns to the columns of the database.

The Code

The demo for this is in development. It will use mock data and no database but mostly demonstrate the GUI. The server side of the app is driven by a number of PHP class objects which I will demonstrate when the demo is operational. Stay tuned!