The Big Data Transition

Thinking Highways
By Tip Franklin September 1, 2014 15:33

The Big Data Transition

Until now the development and evolvement of Intelligent Transportation Systems has been thwarted by the limitations of the data currently available. But, says Tip Franklin, this is no longer the case… 

All too often ITS has been more about simply collecting all available data without really understanding how best to process and use the data that is available, and therefore how to transition raw data to information and then onto actionable intelligence. Instead, we are thwarted by our own inability to understand the data that is available and simply do not understand systems and processes well enough to be able to process the raw data and the quantity of raw data that is actually available.  As a result our focus has been on the mantra of “We just don’t have all of the data required to effectively operate and manage the surface transportation networks”. But in the world of “Big Data” this is no longer true.


As an underpinning to the following I would postulate that there is a continuum which establishes the utilization of data in any decision process. Data, once collected, becomes Information; Information, once processed, becomes Intelligence; Intelligence, once distributed, becomes Actionable.  This is the Data Processing Stream (DPS). NOTE: for clarity throughout the rest of this paper the term “data” may be used regardless of the stage along the continuum under discussion

Each of these general descriptors – data, information, intelligence, actionable – has a set of characteristics, attributes and parameters that must be applied. For example, the parameters for data require that it be accurate, timely and above all, complete. If a bit of data does not meet these criteria singularly and in composite, the entire ensuing flow along the continuum becomes suspect.

As can be imagined the attribute of “accurate” is “not” constant throughout the process, but is an attribute of “timely”, or of “how” the data is to be used, and as we transition into this big data world, this data may acutely alter.

It is the steps at each of these transition points that we have to solidify. At each point along this DPS process we have to identify potential sources, the appropriate filters, the necessary processing (which could include transmission to another DPS) and storage requirements. Plus we are going to have to establish a set of priorities for collection, transmission, processing, analysis, distribution and storage.

To date there has been a little, but slowly increasing, flow of data between the major players in this space – the traffic managers, the transit properties and the tolling authorities. That is now changing with the advent of integrated corridor management deployments that have brought freeway and arterial traffic managers together and joined them to the transit authorities within the corridor. Similarly, managed lanes have caused the tolling authorities and the traffic management community to work together.

The data we have had in the past could best be categorized as historic or real-time (which for a long time was considered to be the Holy Grail), yet the emergence of new data sources must force us to reassess how best to use the data that is becoming increasing available in ever-increasing amounts.  The emergence of the Connected Vehicle as an example, will revolutionize the amount of raw data that is available, the data, as with all data sources will need careful analysis, need to be run through algorithms tuned to a specific location, and ultimately could provide a level of accuracy until now only dreamt of.  The latest estimate is that the full deployment of the Connected Vehicle will generate 11.1 Petabytes (equivalent to 1m GB) of data points by 2020.

Given the current state of the industry in which all of the data we currently have is neither fully understood nor particularly well utilized, how will DOTs respond to data sources which will ultimately provide 100s of times more raw data than ever previously received?

In order to understand and address this increase in data availability and successfully make transition from data to information we must:

  1. Better understand the data sources.
  2. Understand data limitations.
  3. Understand data gaps.
  4. Understand how data gaps can be filled to provide smoothed data.
  5. Deliver automated pre-processing systems that convert in real time.

In order to use information as actionable intelligence we must:

  1. Better integrate data from multiple data sources.
  2. Have the ability to (both automatically and manually) remove or add additional data sets to complete an otherwise incomplete picture.
  3. Better visualize data, including data from disparate data sources.
  4. Provide automated updates to TMC Operators and Managers.

We are going to be very data rich – but we are simply not ready! We have no filters; we have no integration schema; we have no method of prioritizing this onrush of data to make it usable in a timely and efficient manner.

Some of the questions that must be addressed:

  • How will you use data?
  • How will you process, store and distribute it?
  • What will be your litmus test as to viability, usability and credibility?
  • How do you identify and control your sources?
  • How will you process visual and aural data (ie CCTV, machine vision and telephonic reports?
  • How will you prioritize establishment, restoration and maintenance of data sources?
  • How will you prioritize the flow of data through the various communication media?
  • What is the evaluation process for inclusion/exclusion of sources?
  • Who else can use your data; how do you get it to them and in what form (raw or processed); and what is the timeline to provide it?
  • What are the external sources of data that could be useful to you?
  • What is the filter process for identifying the correct data to store for planning processes?
  • What changes, if any, need to be made within your personnel organization and individual attributes (skills, knowledge and abilities –SKA) to be able to handle this?

These are just some of the litany of questions and considerations we are going to face as we move into this “Big Data” realm.

Pages: 1 2

Thinking Highways
By Tip Franklin September 1, 2014 15:33