Viajeo Validation Plan D5 1 Final Version · E-mail: [email protected] Abstract The deliverable...
Transcript of Viajeo Validation Plan D5 1 Final Version · E-mail: [email protected] Abstract The deliverable...
D5.1 Validation plan
28/02/2011 Page 1 of 63 Version 2.0
VVIIAAJJEEOO
D5.1 Validation plan
Author(s) Philipp Böhnke (DLR), Dr. Matthias Hetscher (DLR), Annie Kortsari
(CERTH), Mai Sloth (KeyResearch)
Project VIAJEO – International Demonstrations of Platform for Transport
Planning and Travel Information
Date Contractual: 30 Aug 2010 Actual: 01 Nov 2010
Project Coordinator
Yanying Li ERTICO – ITS Europe Tel: +32 2 400 07 37
E-mail: [email protected]
Abstract The deliverable D5.1 defines indicators and procedures for the validation
and the impact assessment of the Viajeo project.
Keyword list Validation, Impact assessment, Indicators, Demo sites
Nature of deliverable
Report
Dissemination Public1
Project financially supported by
European Commission DG Research
Project number 233745 FP7- SST.2008.3.1.6
1 This is either: Public, restricted to other programme participants, restricted to a group specified by the
consortium, confidential
D5.1 Validation plan
28/02/2011 Page 2 of 63 Version 2.0
Control sheet
Version history
Version number Date Main author Summary of changes
1.0 14.06.2010 Philipp Böhnke (DLR) Draft Version
2.0 25.08.2010 Philipp Böhnke (DLR) Final draft Version
3.0 15.10.2010 Philipp Böhnke (DLR) Final Version
Approval
Name Date
Prepared Philipp Böhnke (DLR) 15 Oct 2010
Reviewed All demo site leaders 01 Nov 2010
Authorized Yanying Li (ERTICO) 01 Nov 2010
Circulation
Recipient Date of submission
European Commission 05 Nov 2010
D5.1 Validation plan
28/02/2011 Page 3 of 63 Version 2.0
Table of content
1. Introduction ......................................................................... 5
1.1. About the Viajeo project ............................................................ 5
1.2. Target audience ....................................................................... 5
1.3. Role of this report in Viajeo ........................................................ 6
1.4. Structure of the document .......................................................... 6
2. Basis and definitions of the validation plan ................................... 7
2.1. Used documents ....................................................................... 7
2.2. MAESTRO Evaluation Guidelines ................................................... 7
2.3. Open platform requirements ....................................................... 7
2.4. Definitions for WP5 ................................................................... 8
2.5. Demo sites and planned extensions ............................................... 8
3. Approach of the overall validation plan ...................................... 11
3.1. Validation (technical) .............................................................. 11
3.2. Impact assessment .................................................................. 11
3.3. Perspectives for European ITS technologies in China and Brazil ......... 11
3.4. Relation between components and testing phases........................... 12
4. Validation ........................................................................... 13
4.1. Objectives – (technical) validation .............................................. 13
4.2. Approach and methodology – (technical) validation ......................... 13
4.3. Validation - Testing phase I (Laboratory test) ............................... 14
4.4. Validation – Testing Phase II (Field test) ...................................... 17
4.4.1. Open platform - requirement ......................................................... 17
4.4.2. Validation of the new developed/expanded and implemented modules ... 19
4.4.3. Validation of selected and implemented European standards (adaptability and practicability) .................................................................................... 44
5. Impact assessment ................................................................ 45
5.1. Objectives ............................................................................ 45
5.2. Target groups ........................................................................ 45
5.3. Approach and methodology – impact assessment ............................ 45
5.3.1. Impact assessment indicators – Athens ............................................. 47
5.3.2. Impact assessment indicators - Sao Paulo ......................................... 53
5.3.3. Impact assessment indicators – Beijing ............................................. 55
5.3.4. Impact assessment indicators – Shanghai .......................................... 59
6. Perspectives for European ITS technologies in China and Brazil ......... 61
6.1. Objectives ............................................................................ 61
D5.1 Validation plan
28/02/2011 Page 4 of 63 Version 2.0
6.2. Approach and Methodology ....................................................... 61
7. Data (collection, analysis, validation, tables and reports) ................ 62
7.1. Data - collection .................................................................... 62
7.2. Data - analysis ....................................................................... 62
7.3. Data - validation .................................................................... 62
7.4. Data – charts of responsibility ................................................... 62
7.5. Reporting system for WP5 ........................................................ 63
D5.1 Validation plan
28/02/2011 Page 5 of 63 Version 2.0
1. Introduction
1.1. About the Viajeo project
The Viajeo project designs, demonstrates and validates an open platform which facilitates data sharing and exchange from different sources and also provides data processing and management to support a variety of services. The project integrates the open platform with local components and demonstrates its applications in four cities: Athens, Beijing, Shanghai and São Paulo. Viajeo (September 2009 – August 2012) is co- funded by the EC DG Research for Specific International Cooperation Actions (SIGA). Cities face ever increasing demands on their transportation systems, especially in developing regions with growing car ownership and rapid urban migration. Even more than heavy infrastructure investment, strategic mobility management is becoming the most important tool for meeting this demand. Mobility management is the domain of activities that combine long term planning and implementation activities with real time and historical information. To be able to deliver effective transport planning and implement the plans, huge amounts of information about usage of the transportation network are needed. Currently, traditional methods (such as roadside units to collect traffic data) and new methods (probe vehicles) are being used to provide traffic data. However, integration of such data is essential in order to support innovative long-term planning and short-term proactive and reactive management of the transport network. The open platform will benefit travelers, transport planners and transport system operators and managers. The open platform will be able to support cross-modal journey planning and seamless traveler information service. Data exchange between operators will lead to a more optimized operation of all systems and individual operation strategies can be harmonized by the data exchanges and information sharing. The platform will enable all data and information used for real-time operation to be used for transport planning. Transport planners will have access to traffic data and information on traffic management strategies. Such information will lead more detailed and comprehensive historical database for transport modeling development and long term policy evaluation The objectives of the Viajeo project are to:
• Design of an open platform with interfaces to a wide range of mobility services.
• Implement of the open platform in Europe, and in the emerging economies, i.e. China and Brazil.
• Validation of the open platform.
• Assess of social and transport impacts of the implementation and demonstration of the open platform.
1.2. Target audience
Target audiences of the deliverable D5.1 “Validation plan” are mainly all partners of the Viajeo project as well as all involved participants/working parties in the demo sites in Athens, Beijing, Shanghai and Sao Paulo. Beside this the deliverable D5.1 is also part of the official documentation and reporting of the Viajeo project to the European Commission. Deliverable D5.1 “Validation plan” is primarily part of the working level. Final results will be given in the deliverables D5.2 “Assessment and validation results” and D5.3 “Best practice guidelines for business models” which will be based on D5.1 “Validation plan” during the further progress of the Viajeo project.
D5.1 Validation plan
28/02/2011 Page 6 of 63 Version 2.0
1.3. Role of this report in Viajeo
The report D5.1 provides the validation and impact assessment plan of the Viajeo project. The focus of the report D5.1 is the Viajeo objectives “Validation of the open platform” and “Assessment of social and transport impacts of the implementation and demonstration of the open platform”. The structure of the overall validation plan, especially e.g. chosen indicators, questions and requirements will help all project partners to focus their research activities within the project. This is of particular importance because of the geographical positions of the demo sites within Europe, China and South America. In each demo site the Viajeo project faces new boundary conditions (e.g. social, cultural and technical). It is important to show that the developed open platform system is highly adaptable but it is also necessary to set up a universally valid validation plan which is strict enough to make the results of the different demo sites comparable but which also allows to go into the specifics. The validation plan is cross-linked with the work packages 2, 3, 4, 6 and 7 of the Viajeo project to ensure that the overall project objectives given in the description of work (DoW) will be reached at the end of the project. At the end the validation plan is an instrument which will help to evaluate / validate the project results of the Viajeo project.
1.4. Structure of the document
Following a brief overview about the structure of the document is given. The report starts with a short summary of the used documents, guidelines, definitions and assumptions in chapter 2. Chapter 3 explains the approach and methodology of the overall validation plan. Chapters 4, 5 and 6 deal each with one of the main parts of the overall validation parts (Validation, Impact assessment, Perspectives for European ITS technologies in China and Brazil) whereas chapter 7 gives short information about data collection, data validation and data analysis.
D5.1 Validation plan
28/02/2011 Page 7 of 63 Version 2.0
2. Basis and definitions of the validation plan
To guarantee a good quality of results in the further progress of the Viajeo project and to avoid misunderstandings based on different wordings the present chapter gives a brief overview of used documents, the MAESTRO Evaluation Guidelines and definitions as well as a short description of the planned services within the demo sites (inside the Viajeo project).
2.1. Used documents
Beside the information (which was) generated specially for deliverable D5.1” by the WP5 partners five important documents form mainly the basis for the validation plan:
• Description of Work (DoW)
• D2.1 Investigation report
• D6.1.1 Investigation final (Athens)
• D6.2.1 Investigation draft (Sao Paulo)
• D6.3.1 Investigation final (Beijing)
• D6.4.1 Investigation final (Shanghai)
2.2. MAESTRO Evaluation Guidelines
MAESTRO (Monitoring Assessment and Evaluation of Transport Policy Options in Europe) was a project funded by DG TREN, the Transport and Energy Directorate of the European Commission. The MAESTRO project brought together a team of leading evaluation experts from 11 EU member states. The purpose of the project was to develop a common framework and methodology for the selection, design and evaluation of pilot and demonstration projects within the specific research program on transport. Although these Guidelines focus on EU-level research projects, they may be applied to P/D projects at all levels, even down to the most localized level. The MAESTRO Guidelines follow a comprehensive approach which means that they cover all specific stages and processes within a P/D project. In the field of evaluation they provide – beside guidance about how to define new / specific impacts and indicators, including necessary specifications / details of an indicator – a list of potential project impacts and associated indicators.
All indicators of the following Viajeo validation plan are defined according to the MAESTRO Evaluation Guidelines. This ensures a comprehensive definition of each indicator, a broad scale of important impacts as well as a general compatibility with the requirements of P/D projects funded by the EU. All aspects are very important to archive a found basis for the following phases of the validation process and also to archive best results for the whole Viajeo project.
2.3. Open platform requirements
Open platform” is neither an official standard nor does a universal valid definition exist. Therefore the partners of the Viajeo project worked out an own definition. This definition forms the basis of the overall validation plan as well as of the following validation and impact assessment:
• A well defined set of core processes accessible via published open external interfaces which are accessible by external parties.
• Standardized, openly described interfaces for data exchange between processes in the realm of the open platform. This set of interfaces is to be used by any new functionality implemented within the open platform concept.
• An “Open Platform” does not mean it is “Open Source”.
D5.1 Validation plan
28/02/2011 Page 8 of 63 Version 2.0
• Related to Open Standards, an open platform means the platform is publicly available and has various rights to use associated with it.
2.4. Definitions for WP5
Overall validation plan: In the following document “D5.1 Validation plan” is also called “overall validation plan”. It consists of different components. One component is called “Validation” and is just a part of the overall validation plan. The other two components are called “Impact assessment” and “Perspectives for European ITS technologies in China and Brazil”.
Validation: Validation is defined as the technical evaluation of the system which will be developed within the Viajeo project. Validation deals with e.g. the system architecture, specifications, open platform requirements, interfaces and all kind of technical parts.
Impact assessment: Impact assessment is defined as the evaluation of impacts to users (end users, transport planners and operators), stakeholder and decision makers. For the impact assessment an already implemented systems is obligatory.
2.5. Demo sites and planned extensions
Athens
Athens (Municipality of Athens), which is the Greek capital, has a population of 745,514 inhabitants (according to the census of 2001) within its administrative limits and a land area of 39 km2. The urban area of Athens, however, extends beyond the administrative city limits with a population of 3,130,841 inhabitants (2001 census) and a land area of 412 km2. According to Eurostat, the Athens Larger Urban Zone (LUZ) is the 7th most populous LUZ in the European Union (the 5th most populous capital city of the EU) with a population of 4,013,368 (in 2004). Athens is the central to economic, financial, industrial, political and cultural life in Greece and is rapidly becoming a leading business centre in the European Union. Planned extensions within the Viajeo project:
The three services that are planned to be implemented for Athens test site are the following:
• Taxi fleet management and traffic information
• End user multi-modal trip planning and traffic information
• Observatory for Public Authorities and Traffic Planners
All three services comprise of both existing and new modules and interfaces. Most of the existing elements are to be developed further in the project.
Beijing
Beijing, as the capital city and a municipality of the People's Republic of China (PRC), is a very large city, quickly becoming one of the main business centers in the world. Its statistics are quite impressive too: a 735 km2 urban area, more than 16 million permanent resident people, 3.9 million vehicles, more than 20.000 public transport buses, and more than 200 kilometers of underground line. Traffic in the city centre is often gridlocked, with rush hour lasting 11 hours a day as of recent years, and smooth traffic only available at night. The target area of the demonstration project covers the northwest corner of the area between the 3rd and 4th ring roads, thus comprising both bus and metro lines. The approximate size of the area is 140 square kilometers. Planned extensions within the Viajeo project:
The Beijing test site is going to include services in the following fields:
D5.1 Validation plan
28/02/2011 Page 9 of 63 Version 2.0
• Within the Viajeo project a new source of floating car data will be introduced (by T-systems) and thus improving the available information for traffic management and planning. Besides the basic GPS information (position, speed, course), data on fuel consumption will also be registered.
• The existing public transport websites are to be extended with a co-modal journey planner. Improved bus timetables and information at bus stops are also part of the Viajeo project.
Shanghai
Shanghai is the largest city in China, and one of the largest metropolitan areas in the world. Located on China's central eastern coast just at the mouth of the Yangtze River, the city has a population of 19,213,200 inhabitants within its administrative limits and a land area of 6,340 km2. However, the urban area of Shanghai extends to 5,299 km2. After 1990, the economic reforms introduced by Deng Xiaoping resulted in intense re-development and financing in Shanghai, and in 2005 Shanghai became the world's largest cargo port. Shanghai will hold the World Expo 2010, the largest event in China since the 2008 Olympics. Planned extension within the Viajeo project:
The Shanghai test site is going to include services in the following fields:
• Within the Viajeo project, a new source of environmental data will be introduced and thus to give a real time vision on air quality over the test site. These data with traffic data will be contributed to calibrate and validate a traffic related environmental model which can simulate environmental impacts of road traffic. The traffic environmental model will be used for the appraisal of transport policy and planning. During the demonstration, a number of environmental sensors with GPS and GPRS units will be installed on buses and other cars depending on the actual situation to measure air pollution along the bus route.
• Besides this, the Shanghai demonstration will improve the bus information at bus stops such as forecasting bus arrival times. Existing interfaces will be also improved in accordance with open standards. The bus arrival times are also going to be used to form dynamic timetables which will improve bus operation.
São Paulo
São Paulo is the largest city in Brazil and is the world’s 7th largest metropolitan area. With a population of 11,105,249 residents within an area of 1,523 km2, the city is the largest city in the Southern hemisphere in terms of population. The Greater São Paulo, on the other hand, consists of 39 municipalities, including the capital, São Paulo, with around 27,640,577 population spread over 23,062 km2.
Greater São Paulo has over 6.2 million cars, 4 million of them in São Paulo city. São Paulo is a heavily congested city with a record of 270 km traffic queue in 2010. Demand management has been applied, i.e. 20% of cars are not allowed to drive in the extended centre area between 7am and 10am and between 5pm and 8pm on weekdays. São Paulo has a comprehensive public transport service consisting of metro, train and buses. However, 45% of daily journeys use private cars. Traffic Engineering Company (CET) manages and maintains the traffic system in São Paulo city. The network consists of 5500 traffic lights of which 1000 are connected through ATCs in 8 zones. As expected, CET has been having difficulties controlling the network due to the equipment’s age (around 20 years old) and lack of a more thorough maintenance.
The Sao Paulo test site is going to include services in the following fields:
D5.1 Validation plan
28/02/2011 Page 10 of 63 Version 2.0
• Proof-of-Concept of integration of probe vehicles and urban traffic control centre;
• Integrating urban traffic control centre data with motorway traffic data; The open platform will connect data from urban traffic control centre and data from motorway tolling system in order to estimate real-time traffic condition in urban and motorway networks.
• Use of urban traffic control centre for real-time traffic information. The real-time data will be stored into a historical database. The real-time information and historical database will be used to support a web-based journey planner.
• Real-time traffic information service delivered to on-board units.
D5.1 Validation plan
28/02/2011 Page 11 of 63 Version 2.0
3. Approach of the overall validation plan
To fulfill the multilayer objectives of WP5 given in the description of work the overall validation plan is subdivided into three different components:
1. Validation
2. Impact assessment
3. Perspectives for European ITS technologies in China and Brazil
Each component can be assessed more or less individually but all components are needed to evaluate the complex structure of the whole Viajeo project. Below a short overview of each of the different components is given. A detailed description of the proceeding is given in the corresponding chapters.
3.1. Validation (technical)
Following the definition within the Viajeo project, “Validation” is defined as the technical evaluation of the system which will be developed within the Viajeo project. Validation deals with, among other things, the system architecture, specifications, open platform requirements, interfaces and other technical parts.
Technical systems, especially complex software systems, need to be tested during their development to guarantee that the whole system is running. Normally, like in Viajeo, they consist of various single modules and interfaces. Each module and interface as well as the interaction between them has to be tested. Within the Viajeo there are two defined testing phases:
1. Testing phase I (laboratory test) In the laboratory tests all new designed parts of the system will be tested as well as the whole core system. All laboratory tests are related to the non implemented system.
2. Testing phase II (field test) In the field tests the planned and already implemented system will be tested at each demo site. The field tests are integration tests which will help to confirm whether data sources and services are integrated into the developed open platform.
3.2. Impact assessment
Following the definition within the Viajeo project, “Impact assessment” is defined as the evaluation of impacts to users (professionals as transport and traffic planners as well as transport operators, non-professionals, stakeholders and decision makers). In order to perform the impact assessment, an already implemented systems is obligatory. This leads to the fact that the impact assessment can also be assigned to the testing phase II (field test).
3.3. Perspectives for European ITS technologies in China and Brazil
This component is little detached from the other components. It assesses the potential for further expansion of the demonstrated technologies to other parts of China and Brazil and looks at the potential in general for the use of European ITS technologies in the two countries.
An existing open platform system is not at all mandatory although the experiences and input of the Viajeo project are valuable and necessary for the preparation of the best practice guidelines.
This component cannot be assigned to one of the testing phases.
D5.1 Validation plan
28/02/2011 Page 12 of 63 Version 2.0
3.4. Relation between components and testing phases
Figure 1 illustrates the relation between components and testing phases. Each component of the overall validation plan is clearly represented. The three separate columns are partly overlapped by two rows which represent the different testing phases. The overlapping area of the columns and rows show that there is a relation between the components and the testing phases. As testing phase I deals only with the non-implemented systems (laboratory test), while impact assessment needs an already implemented system, this overlapping area is crossed out. The three overlapping areas, which build the main parts of the overall validation plan, as well as the last column “Perspectives for the European ITS technologies in China and Brazil” will be described in detail in the following chapters.
Perspectives forEuropean ITS
technologies in China and Brazil
Impact assessment
Labora
tory
te
st
Testi
ng
ph
ase
I
Fie
ldte
st
Tes
tin
gp
has
eII
Validation(technical)
xModule test
Integration test
System test
Demonstrator tests
• Open platform
• Processing chain
• Multi-/Intermodal
routing
Impact assessmentplan
• Athens
• Beijing
• Shanghai
• Sao Paulo
Perspectives forEuropean ITS
technologies in China and Brazil
Impact assessment
Labora
tory
te
st
Testi
ng
ph
ase
I
Fie
ldte
st
Tes
tin
gp
has
eII
Validation(technical)
xModule test
Integration test
System test
Demonstrator tests
• Open platform
• Processing chain
• Multi-/Intermodal
routing
Impact assessmentplan
• Athens
• Beijing
• Shanghai
• Sao Paulo
Perspectives forEuropean ITS
technologies in China and Brazil
Impact assessment
Labora
tory
te
st
Testi
ng
ph
ase
I
Fie
ldte
st
Tes
tin
gp
has
eII
Validation(technical)
xModule test
Integration test
System test
Demonstrator tests
• Open platform
• Processing chain
• Multi-/Intermodal
routing
Impact assessmentplan
• Athens
• Beijing
• Shanghai
• Sao Paulo
Figure 1: Relation between components and testing phases
D5.1 Validation plan
28/02/2011 Page 13 of 63 Version 2.0
4. Validation
Following the technical validation, which comprise of a laboratory and a field test phases, is described. Each phase can be assessed more or less individually but both are needed for the validation of the whole project.
4.1. Objectives – (technical) validation
The goal of the validation is to evaluate if the developed open platform works correctly and if it conforms to the technical requirements of the description of work. The technical validation of the whole system will be done through the validation of the modules and interfaces integrated into the open platform system combined with a validation of the implemented service within each demo site.
4.2. Approach and methodology – (technical) validation
A brief overview of the chosen tests and testing proceedings for the open platform system is given below. The chosen tests consist of four stages:
I. Module test
II. Integration test
III. System test
IV. Test during the final demonstration in each demo site (Acceptance test)
I. Module test Goal of this test is to test the software functionality at the level of the single module itself. It provides the developer with information about the validity of algorithms, functions and routines. This test is planned and done by the module developer himself. To specify the test cases the module interface specifications are used. II. Integration test Goal of this test is to test the interaction between several single modules. These modules are put together/composed to create subsystems to test a bigger part of the whole software system. It ensures the validity of the I/O routines, interfaces and the consistency of the software body. This test is also planned and carried out by the software developer in all versions. III. System test Goal of this test is to compare the whole developed software system with the technical specification (DoW). This test is planned between the different parties providing system modules. It is focused on the implementation of the proposed functionalities and basic features. This test pays attention on:
• Performance
• Reliability
• Handling
• Functionality
• Documentation
and provide both parties with measurable indicators. IV. Test during the final demonstration (acceptance tests) Goal of this final test is to demonstrate that the delivered system fulfils the originally and currently defined requirements. It is performed at the demo sites after the integration of the developed open platform system into the existing infrastructure. This test is planned by
D5.1 Validation plan
28/02/2011 Page 14 of 63 Version 2.0
WP5 and the developer and will be executed and evaluated by the software developer and the demo site operators. Within the Viajeo project the module, integration and system tests are dedicated to the “Validation – Testing Phase I (Laboratory test)” while the tests during the final demonstrations (acceptance tests) are dedicated to the “Validation – Testing Phase II (Field test)”. The module and integration tests will be done only by the software developer which means that they can chose a testing procedure. Test results and reports will be handed over to WP5. The system and field tests (acceptance test) are developed according to the Viajeo requirements.
4.3. Validation - Testing phase I (Laboratory test)
The module and integration tests – in contrast to the system test – will be developed and done only by the software developers themselves. As already mention there will be no specifications for these tests defined by WP5. This approach makes sense because number, complexity and indicators of those tests depend highly on the realization of the final software platform as well as on the newly developed modules. The necessary degree of knowledge and information about the modules and the essential tests will (only) be combined by the developers at the end of the software work. Results and reports of these tests will be transferred by the developers to the WP5 leader who will take them into the WP5 reports.
However, this approach does not mean that there will be no independent validation of the modules and/or interaction at all (because they are developed and tested only by the developers). Within the system test the functionality of the whole software platform will be tested which means that therefore also the single modules as well as the interactions between them are indirectly tested.
Goal of the system test is to compare the developed software modules with the technical requirements and specifications defined in Viajeo. These tests will be done in the laboratories of the developing companies (Mizar/Ptv) corresponding to the specifications defined in the following. All results and reports will be transferred to the WP5 leader who will take them into the WP5 report. Main focuses of these tests are:
• Performance
• Reliability
• Handling
• Functionality
• Documentation quality as acceptance criterion
The whole system including the peripheral components will be tested in the final test for the Viajeo project which is the acceptance test during the demonstrations within the demo sites (field test). Following evaluation methods and indicators for the system test (laboratory test) of the core system are defined, related to the MAESTRO evaluation guidelines of the EU:
Test Indicator Description
Data attributes
Sources of data Unit
Type of data
acquisition
Method of data
acquisition
Performance test
Time response
Mean time between raw data input and data output
Seconds
Quantitative Measurement Special reference data set / test data
D5.1 Validation plan
28/02/2011 Page 15 of 63 Version 2.0
set
Actuality Mean actuality of output data – time difference between oldest used data set and output data
Seconds Quantitative Measurement Time stamp of used data sets and output data
Correctness Percentage of accuracy of data output (data output structure)
% Quantitative Measurement / Survey
Special reference data set / test data set
Reliability Percentage of error-free run-time
% Quantitative Measurement /Survey
Special reference data set / test data set
Goal: Validation of the software platform performance. Does the measured performance fit the requirements (dynamic online services possible, real-time capabilities)
Test Indicator Description
Data attributes Sources of
data
Unit Type of data acquisition
Method of data
acquisition
Reliability test
Validation time (Maturity)
Mean time of data validation (Maturity of results)
Seconds
Quantitative Measurement Time stamp of used data sets and output data
Fault tolerance
Percentage of temporal data gaps (no input at all) with which the system still is running
% Quantitative Survey / Measurement
Special reference data set / test data set with temporal data gaps
robustness Percentage of incomplete input data (incomplete data set / e.g. data of one sensor is missing) with which the system still is running
% Quantitative Measurement / Survey
Special reference data set / incomplete test data set
Recoverability
Mean time for a system recovery after a system crash
Seconds Quantitative Measurement
Conformity Percentage of erroneous data (obvious wrong/impossible data) with
% Quantitative Measurement / Survey
Special reference data set / erroneous test data
D5.1 Validation plan
28/02/2011 Page 16 of 63 Version 2.0
which the system still is running
set
Goal: Verification of the enhancement of run time stability
Test Indicator Description
Data attributes
Sources of data Unit
Type of data
acquisition
Method of data
acquisition
Handling test
Comprehensibility Percentage of people who understand the system
%
Qualitative Survey System itself
Operability Percentage of people who are able to operate the system
%
Qualitative Survey System itself
Conformity Conformity of the two different systems (Ptv/Mizar)
% Qualitative Survey Systems itself
Goal: Demonstration of an easy handling
Test Indicator Description
Data attributes
Sources of data
Unit
Type of data
acquisition
Method of data
acquisition
Functionality test
Suitability / correctness
Percentage of suitable and correct output data
%
Quantitative
Measurement
Special reference data set / test data set
Security Rating given to security of system (How easy is a manipulation of the system)
% Qualitative Survey System itself
Interoperability
Maximum/minimum no. of different input data sources and services with which the system (traffic engine) is able to run
No. Quantitative
Measurement /Survey
System itself / input data sources / services
Goal: Demonstration of output correctness and operation security of the software
Documentation Indicator Description
Data attributes
Sources of data Unit
Type of data
acquisition
Method of data
acquisition
Documentation quality as acceptance criterion
Documentation Percentage to which all important methods and
%
Qualitative Survey Documentation
D5.1 Validation plan
28/02/2011 Page 17 of 63 Version 2.0
software components of the system (following the before defined open platform standard) are sufficient described
Goal: To guarantee that the defined open platform standard is kept
4.4. Validation – Testing Phase II (Field test)
The tests during the final demonstration (acceptance tests) belong to the “Validation – Testing Phase II (Field test)”. The goal of these tests is to evaluate / validate if the implemented demonstrator systems in the test sites are running and if they fit the technical requirements of the Viajeo project. Therefore a (tripartite) validation and testing procedure has been developed by WP5 (based on the description of work and the defined services respectively the use cases defined for the demo sites) which focus on the following requirements:
I. Validation of the “open platform” requirement
II. Validation of the new developed/expanded and implemented modules
III. Validation of selected and implemented European standards (adaptability and practicability)
The first requirement “open platform” verifies that it is (easily – only with the published information) possible to integrate new data sources or services into the platform which means that the platform is indeed an open platform. The second requirement is to validate the new developed or expanded and implemented modules in the demo sites. Modules which already exist and modules which will be developed outside the Viajeo project will not be validated within the WP5. Finally the third requirement focuses on the adaptability and practicability of the selected and implemented European standards to existing local components and services within the demo sites in Europe, China and Brazil.
4.4.1. Open platform - requirement
One of the main requirements for the platform used in the Viajeo project is that it should be an open platform. To evaluate if the requirement is fulfilled, an additional fictitious data source (raw data set) will be integrated into the already implemented and running platforms within the sites. This will be done according to the official documentation and information. The generated information of the corresponding service (e.g. Traffic information) will be checked and evaluated.
To keep the additional amount of work within an economically justifiable effort, the additional fictitious data source (dummy data) should be defined according to the interface specifications like an already implemented data source (in favor analog FCD). Therefore no (possibly) complex preprocessing or adjustments of the additional data set will be necessary. The data/content within the additional data set will be created / manipulated in a certain way so that the results - after processing and integrating it into the platform – change in an expected manner. As a result of the manipulation it will be possible to verify if the additional data source is fully integrated into the platform and if the functionalities operate according to the specification. The kind of additional data set as well as the testing area (even if our partners will try hard to realize large-scale data coverage within the whole demo site it is sufficient to check the open platform requirement within a small
D5.1 Validation plan
28/02/2011 Page 18 of 63 Version 2.0
area) will be chosen by the demo sites in cooperation with the platform developer (MIZAR and PTV). Intended goals are:
• Evaluation of the documentation
• Evaluation of the open platform accessibility via interfaces
• Evaluation of the core process functionalities (e.g. data fusion) of the open platform as specified by WP3
Open issues to be defined by the demo sites and the platform developer during the Viajeo project:
• Kind of additional data set
• Testing area
Example sheet:
The example sheet has to be filled in by the demo site leaders (in cooperation with the platform developer) during the further progress of the Viajeo project.
Validation “Open platform”
(demo site - name)
Selected data source (additional):
Filled in at a later date by the demo sites in cooperation with the platform developer!
Testing area: Filled in at a later date by the demo sites in cooperation with the platform developer!
Data set: • Reference data set
• Additional and manipulated data set
Evaluation method / indicator:
• Comparison of results of the original versus the enhanced implemented system (added by selected data source).
• Simulation of e.g. congestion or free traffic flow by manipulation of the raw data set � Verification: Traffic information services should display the expected traffic situation
Goal: • Evaluation of the documentation
• Evaluation of the open platform accessibility – core system adjustment via interfaces
• Evaluation of the integration possibilities like e.g. data fusion of additional and already existing data
D5.1 Validation plan
28/02/2011 Page 19 of 63 Version 2.0
4.4.2. Validation of the new developed/expanded and implemented modules
All modules which will be developed or expanded within the Viajeo project are already identified by WP2 “Investigation”. Based on the investigation reports prepared by the demo sites themselves (D6.1.1, D6.1.3 and D6.1.4) the overall investigation report of WP2 (Version: v26) contains service diagrams of all planned services. The service diagram consists of modules ordered into five layers (data collection, processing, applications, services and devices) with data transfer interfaces between the layers. Within these service diagrams all modules are marked as already existing modules, as modules which have to be new developed or as modules which have to be expanded within the Viajeo project. All new or expanded modules which are likely (depending on data availability) to be implemented in the sites and for this reason need to be validated are marked in the following with a pink
star ( ). Modules which are currently not likely to be implemented are not listed below.
Figure 2: The five layers of the service diagram. Squares indicate modules and circles indicate interfaces
Due to the fact that the data collection itself is not part of the Viajeo project the validation focuses only on the layers processing, application, services and devices. Each layer has been related to several of the following focuses which are also used in the testing phase I (laboratory test) added by the focus of economy:
• Performance - Does the measured performance fit the requirements (dynamic online services possible, real-time capabilities)
• Reliability – validate stability and robustness of the system
• Functionality / user perception – safe operation and output correctness
• Handling – easy to use
• Economy – cost benefit ratio
Table 3 gives an overview of the relation between layers and dedicated validation focuses:
Layer Focus Processing • Performance & Reliability
• Functionality
Application • Performance & Reliability
• Functionality
Service • User perception
• Handling
• Economics
Device • User perception
• Handling
• Economics Table 3: Relation between layers and focuses
D5.1 Validation plan
28/02/2011 Page 20 of 63 Version 2.0
According to the MAESTRO Evaluation Guidelines indicators for the different validation focuses are defined, including the name of the indicator, description, data attributes and source of data.
Following service diagrams and indicator lists for each demo site related to corresponding service and module are listed.
4.4.2.1. Athens – Indicator list (field test)
Athens - Taxi fleet management and traffic information (Service A):
1.2
FCD collected by
GPS (current
position)
Stakeholder: Taxi
company
2.1
Taxi data
processing
Stakeholder:
ANCO
3.1
Calculate
Travel times
Stakeholder:
Infotrip
4.1
Professional
service
gateway
(Communicatio
n gateway)
Stakeholder:
ANCO
5.1
Taxi navigation
unit
Stakeholder:
Taxi driver
1.1
Proprietary
XML
SERVICE A:
Taxi Fleet Management and Traffic Information
1.4
Real time traffic
data collected by
road side
equipment
Stakeholder:
Ministry of
Infrastructures
1.1
Taxi assignment
and destination
Stakeholder: Taxi
company
(dispatch center)
1.2
Proprietary
XML
1.4
Proprietary
XML
2.2
Real time
traffic data
processing and
model
Stakeholder:
Infotrip
2.1.2
Proprietary
XML
2.2.1
Proprietary
XML
3.2
Taxi route
engine & Route
monitoring
Stakeholder:
Infotrip
3.2
Proprietary
XML
4.1
Proprietary
XML
1.3
Static data (e.g.
road network)
Stakeholder:
Infotrip
1.3
Proprietary
XML
2.1.1
Proprietary
XML
3.3
Traffic engine
(traffic alerts)
Stakeholder:
Infotrip
2.2.2
Proprietary
XML
3.1
Proprietary
XML
3.3
Proprietary
XML
Figure 4: Diagram of the service chains in Service A (Athens)
Module 2.1: Taxi data processing
Indicator Description
Data attributes
Source of data Comments / Remarks Unit
Type of data
acquisition
Method of data
acquisition
Performance and Reliability test
Status Mean no. of measuring devices working well related to number of active devices per mode (taxi assignment / FCD)
% Quantitative Survey / Measurement
Measured data set
Actuality Mean time between data updates (taxis assignment / FCD system)
Seconds
Quantitative Survey / Measurement
Measured data set
Reliability Percentage of error-free run-time related to total runtime (whole system, single devices/equipment may not work)
% Quantitative Measurement /Survey
Measured data set
User Perception
Correctness / quality
Percentage of accuracy of data output (taxi processing)
% Qualitative Measurement / Survey
Measured data set as well as data of existing system
D5.1 Validation plan
28/02/2011 Page 21 of 63 Version 2.0
Module 2.2: Real time traffic data processing and model
Indicator Description
Data attributes Source of data
Comments / Remarks Unit
Type of data
acquisition
Method of data
acquisition
Performance and Reliability test
Status Mean no. of measuring devices working well related to number of active devices per mode (statistic data / real time data)
% Quantitative Survey / Measurement
Measured data set
Actuality Mean time between data updates (taxis assignment / FCD system)
Seconds
Quantitative Survey / Measurement
Measured data set
Reliability Percentage of error-free run-time related to total runtime (whole system, single devices/equipment may not work)
% Quantitative Measurement /Survey
Measured data set
Functionality test
Correctness / quality
Percentage of accuracy of data output (real time traffic data)
% Qualitative Measurement / Survey
Measured data set as well as data of a reference measuring system
Module 3.1 Calculate travel time
Indicator Description
Data attributes Source of data
Comments / Remarks Unit
Type of data
acquisition
Method of data
acquisition
Performance and Reliability test
Status Percentage of availability of input data (all possible input data are available)
% Quantitative Survey / Measurement
Measured data set
Actuality Mean time between data updates (travel time)
Seconds
Quantitative Survey / Measurement
Measured data set
Reliability Percentage of error-free run-time related to total runtime (whole system, single devices/equipment may not work)
% Quantitative Measurement /Survey
Measured data set
Functionality test
Correctness / quality
Percentage of accuracy of data output (travel time)
% Qualitative Measurement / Survey
Measured data set as well as data of a reference measuring system
Module 3.2 Taxi route engine & route monitoring
Indicator Description
Data attributes Source of data
Comments / Remarks Unit
Type of data
acquisition
Method of data
acquisition
Performance and Reliability test
Status Percentage of availability of input data (all possible input data are available)
% Quantitative Survey / Measurement
Measured data set
D5.1 Validation plan
28/02/2011 Page 22 of 63 Version 2.0
Actuality Mean time between data updates (taxi route / route monitoring)
Seconds
Quantitative Survey / Measurement
Measured data set
Reliability Percentage of error-free run-time related to total runtime (whole system, single devices/equipment may not work)
% Quantitative Measurement /Survey
Measured data set
Functionality test
Correctness / quality
Percentage of accuracy of data output (taxi route / route engine)
% Qualitative Measurement / Survey
Measured data set as well as data of a reference measuring system
Module 3.3 Traffic engine (traffic alerts)
Indicator Description
Data attributes Source of data
Comments / Remarks Unit
Type of data
acquisition
Method of data
acquisition
Performance and Reliability test
Status Percentage of availability of input data (all possible input data are available)
% Quantitative Survey / Measurement
Measured data set
Actuality Mean time between data updates (traffic alerts)
Seconds
Quantitative Survey / Measurement
Measured data set
Reliability Percentage of error-free run-time related to total runtime (whole system, single devices/equipment may not work)
% Quantitative Measurement /Survey
Measured data set
Functionality test
Correctness / quality
Percentage of accuracy of data output (no. total alerts related to no. of correct alerts)
% Qualitative Measurement / Survey
Measured data set as well as data of a reference measuring system
Module 4.1 Professional service gateway (Communication gateway)
Indicator Description
Data attributes Source of data
Comments / Remarks Unit
Type of data
acquisition
Method of data
acquisition
Functionality test
Utilizability Mean percentage of error free data communication time between system and gateway
% Quantitative Measurement / Survey
Data stream
Conformity Mean percentage of information available after conversion of transmission protocols
% Quantitative Measurement / Survey
Data stream
Handling test
Operability Percentage of people who are able to operate the gateway (settings)
%
Qualitative Survey System itself
D5.1 Validation plan
28/02/2011 Page 23 of 63 Version 2.0
Module 5.1 Taxi navigation unit
Indicator Description
Data attributes Source of data
Comments / Remarks Unit
Type of data
acquisition
Method of data
acquisition
User perception
Actuality Mean time between data updates (traffic alerts)
Seconds
Quantitative /Qualitative
Survey / Measurement
Measured data set
Reliability Percentage of error-free run-time related to total runtime (whole system, single devices/equipment may not work)
% Quantitative /Qualitative
Measurement /Survey
Measured data set
Utilizability Mean percentage of error free data communication time between gateway and taxi navigation units
% / unit Quantitative /Qualitative
Measurement / Survey
Measured data set
Status Percentage of availability of input data (all possible input data are available)
% Quantitative /Qualitative
Survey / Measurement
Measured data set
Handling test
Operability Percentage of people who are able to operate the system (settings)
%
Qualitative Survey System itself
Comprehensibility Percentage of people who understand the given output data of the system
%
Qualitative Survey System itself
Satisfaction Percentage of user who are satisfied with the system
%
Qualitative Survey System itself
D5.1 Validation plan
28/02/2011 Page 24 of 63 Version 2.0
Athens – End-user Multi-modal Trip Planning and Trafic Information (Service B)
SERVICE B:
End-user Multi-modal Trip Planning and Traffic Information
3.1
Traffic engine
(traffic
information)
Stakeholder:
Infotrip
4.1
Consumer
service
gateway
Stakeholder:
Infotrip
5.1
Mobile, LBS
app
Stakeholder:
End-user
1.5
Static data
(public transport)
Stakeholder:
Public transport
company,
CERTH/HIT,
Infotrip
1.5
Proprietary
XML
2.2
Real time
traffic data
processing and
model
Stakeholder:
Infotrip
2.2.1
Proprietary
XML
3.1
Proprietary
XML
4.1.1
Proprietary
XML
1.3
Static data (e.g.
road network)
Stakeholder:
Infotrip
1.3
Proprietary
XML
1.2
Real time traffic
data collected by
road side
equipment
Stakeholder:
Ministry of
Infrastructures
1.2
Proprietary
XML
5.2
Web
Stakeholder:
End-user3.3
Multi-modal
route planner
(multi-modal
routing)
Stakeholder:
Infotrip
2.2.2
Proprietary
XML
3.3
Proprietary
XML
4.1.2
Proprietary
XML
1.1
FCD collected by
GPS (current
position)
Stakeholder: Taxi
company
1.4
Proprietary
XML
1.4
Historical data
(Portal)
Stakeholder:
CERTH/HIT
1.1
Proprietary
XML
3.2
Calculate Travel
times
Stakeholder:
Infotrip
3.2
Proprietary
XML
Figure 5: Diagram of the service chains in Service B (Athens)
Module 2.1 Real time traffic data processing and model
Indicator Description
Data attributes Source of data
Comments / Remarks Unit
Type of data
acquisition
Method of data
acquisition
Performance and Reliability test
Status Mean no. of measuring devices working well related to number of active devices per mode (statistic data / real time data)
% Quantitative Survey / Measurement
Measured data set
Actuality Mean time between data updates (taxis assignment / FCD system)
Seconds
Quantitative Survey / Measurement
Measured data set
Reliability Percentage of error-free run-time related to total runtime (whole system, single devices/equipment may not work)
% Quantitative Measurement /Survey
Measured data set
Functionality test
Correctness / quality
Percentage of accuracy of data output (real time traffic data)
% Qualitative Measurement / Survey
Measured data set as well as data of a reference measuring system
D5.1 Validation plan
28/02/2011 Page 25 of 63 Version 2.0
Module 3.1 Traffic engine (traffic information)
Indicator Description
Data attributes Source of data
Comments / Remarks Unit
Type of data
acquisition
Method of data
acquisition
Performance and Reliability test
Status Percentage of availability of input data (all possible input data are available)
% Quantitative Survey / Measurement
Measured data set
Actuality Mean time between data updates (traffic information)
Seconds
Quantitative Survey / Measurement
Measured data set
Reliability Percentage of error-free run-time related to total runtime (whole system, single devices/equipment may not work)
% Quantitative Measurement /Survey
Measured data set
Functionality test
Correctness / quality
Percentage of accuracy of data output (quality of traffic information)
% Qualitative Measurement / Survey
Measured data set as well as data of a reference measuring system
Module 3.2 Calculate travel times
Indicator Description
Data attributes Source of data
Comments / Remarks Unit
Type of data
acquisition
Method of data
acquisition
Performance and Reliability test
Status Percentage of availability of input data (all possible input data are available)
% Quantitative Survey / Measurement
Measured data set
Actuality Mean time between data updates (travel time)
Seconds
Quantitative Survey / Measurement
Measured data set
Reliability Percentage of error-free run-time related to total runtime (whole system, single devices/equipment may not work)
% Quantitative Measurement /Survey
Measured data set
Functionality test
Correctness / quality
Percentage of accuracy of data output (travel time)
% Qualitative Measurement / Survey
Measured data set as well as data of a reference measuring system
Module 3.3 Multi-modal route planner (multi-modal routing)
Indicator Description
Data attributes Source of data
Comments / Remarks Unit
Type of data
acquisition
Method of data
acquisition
Performance and Reliability test
Status Percentage of availability of input data (all possible input data are available)
% Quantitative Survey / Measurement
Measured data set
D5.1 Validation plan
28/02/2011 Page 26 of 63 Version 2.0
Actuality Mean time between data updates (multi-modal routing)
Seconds
Quantitative Survey / Measurement
Measured data set
Reliability Percentage of error-free run-time related to total runtime (whole system, single devices/equipment may not work)
% Quantitative Measurement /Survey
Measured data set
Functionality test
Correctness / quality
Percentage of accuracy of data output (multi-modal routing)
% Qualitative Measurement / Survey
Measured data set as well as data of a reference measuring system
Module 4.1 Consumer service gateway
Indicator Description
Data attributes Source of data
Comments / Remarks Unit
Type of data
acquisition
Method of data
acquisition
User perception
Utilizability Mean percentage of error free data communication time between system and gateway
% Quantitative /Qualitative
Measurement / Survey
Data stream
Conformity Mean percentage of information available after conversion of transmission protocols
% Quantitative /Qualitative
Measurement / Survey
Data stream
Handling test
Operability Percentage of people who are able to operate the gateway (settings)
%
Qualitative Survey System itself
Module 5.1 Mobile, LBS, app
Indicator Description
Data attributes Source of data
Comments / Remarks Unit
Type of data
acquisition
Method of data
acquisition
User perception
Utilizability Mean percentage of error free data communication time between gateway and mobile, LBS, app
% / unit Quantitative /Qualitative
Measurement / Survey
Measured data set
Handling test
Operability Percentage of people who are able to operate the system (settings)
%
Qualitative Survey System itself
Comprehensibility Percentage of people who understand the given output data of the system
%
Qualitative Survey System itself
Satisfaction Percentage of user who are satisfied with the system
%
Qualitative Survey System itself
D5.1 Validation plan
28/02/2011 Page 27 of 63 Version 2.0
Module 5.2 Web
Indicator Description
Data attributes Source of data
Comments / Remarks Unit
Type of data
acquisition
Method of data
acquisition
User perception
Utilizability Mean percentage of error free data communication time between gateway and web
% Quantitative /Qualitative
Measurement / Survey
Measured data set
Handling test
Operability Percentage of people who are able to operate the system (settings)
%
Qualitative Survey System itself
Comprehensibility Percentage of people who understand the given output data of the system
%
Qualitative Survey System itself
Satisfaction Percentage of user who are satisfied with the system
%
Qualitative Survey System itself
D5.1 Validation plan
28/02/2011 Page 28 of 63 Version 2.0
Athens – Observatory for Public Authorities and Traffic Planners (Service C)
SERVICE C:
Observatory for Public Authorities and Traffic Planners
3.1
Observatory
(Portal)
Stakeholder:
CERTH/HIT
4.1
Public
authorities
gateway
(Portal)
Stakeholder:
CERTH/HIT
5.1
Public
authority
1.3
Transport and
traffic indicators
(Portal)
Stakeholder:
CERTH/HIT
2.1
Data
aggregation
and processing
Stakeholder:
CERTH/HIT4.1.2
Proprietary
XML
1.2
Historical data
(Portal)
Stakeholder:
CERTH/HIT
1.1
Real time traffic
data
Stakeholder:
Infotrip
1.1
Proprietary
XML
5.2
Traffic planner
4.1.1
Proprietary
XML
Figure 6: Diagram of the service chains in Service C (Athens)
Module 2.1 Data aggregation and processing
Indicator Description
Data attributes Source of data
Comments / Remarks Unit
Type of data acquisition
Method of data
acquisition
Performance and Reliability test
Actuality Mean time between data updates (real time / traffic indicators)
Seconds
Quantitative Survey / Measurement
Measured data set
No of accurate DB updates
Ratio of total devices vs correct devices (statistical analysis)
% Quantitative Survey / Measurement
No of inconsistent DB matchings
No Quantitative Survey / Measurement
Functionality test
Correctness / quality
Percentage of accuracy of data output (data aggregation / data fusion)
% Qualitative Measurement / Survey
Measured data set as well as data of a reference measuring system
D5.1 Validation plan
28/02/2011 Page 29 of 63 Version 2.0
Module 3.1 Observatory (Portal)
Indicator Description
Data attributes Source of data
Comments / Remarks Unit
Type of data
acquisition
Method of data
acquisition
Performance and Reliability test
No of inaccurate calculationa
No of inadequate calculation vs total no.
% Quantitative Measurement /Survey
Measured data set
More to be defined
Functionality test
Correctness / quality
Percentage of accuracy of data output (traffic indicators)
% Qualitative Measurement / Survey
Measured data set as well as data of a reference measuring system
Module 4.1 Public authorities gateway (Portal)
Indicator Description
Data attributes Source of data
Comments / Remarks Unit
Type of data
acquisition
Method of data
acquisition
User perception
Utilizability Mean percentage of error free data communication time between system and gateway
% Quantitative /Qualitative
Measurement / Survey
Data stream
Conformity Mean percentage of information available after conversion of transmission protocols
% Quantitative /Qualitative
Measurement / Survey
Data stream
Handling test
Operability Percentage of people who are able to operate the gateway (settings)
%
Qualitative Survey System itself
Module 5.1 Public authority
Indicator Description
Data attributes Source of data
Comments / Remarks Unit
Type of data
acquisition
Method of data
acquisition
User perception
No of false downloads of data
Mean percentage of error free data communication
% Quantitative /Qualitative
Measurement / Survey
Measured data set
Utilizability Mean percentage of error free data communication time between gateway and web
% Quantitative /Qualitative
Measurement / Survey
Measured data set
Handling test
Operability Percentage of people who are able to operate the system (settings)
%
Qualitative Survey System itself
Comprehensibility Percentage of people who understand the given output data of the system
%
Qualitative Survey System itself
D5.1 Validation plan
28/02/2011 Page 30 of 63 Version 2.0
Module 5.2 Traffic planner
Indicator Description
Data attributes Source of data
Comments / Remarks Unit
Type of data
acquisition
Method of data
acquisition
User perception
No of false downloads of data
Mean percentage of error free data communication
% Quantitative /Qualitative
Measurement / Survey
Measured data set
Utilizability Mean percentage of error free data communication time between gateway and web
% Quantitative /Qualitative
Measurement / Survey
Measured data set
Handling test
Operability Percentage of people who are able to operate the system (settings)
%
Qualitative Survey System itself
Comprehensibility Percentage of people who understand the given output data of the system
%
Qualitative Survey System itself
D5.1 Validation plan
28/02/2011 Page 31 of 63 Version 2.0
Data Type
4.4.2.2. Sao Paulo – Indicator list (field test)
Viajeo São Paulo site
Service 1: Real time traffic data processing with floating vehicle data
Module 2.1 Real time traffic data processing and model
Indicator Description
Data attributes Source of data
Comments / Remarks Unit
Type of data
acquisition
Method of data
acquisition
Performance and Reliability test
Status Mean no. of measuring devices working well related to number of active devices per mode (statistic data / real time data)
% Quantitative Survey / Measurement
Measured data set
Actuality Mean time between data updates ( FCD)
Seconds
Quantitative Survey / Measurement
Measured data set
Reliability Percentage of error-free run-time related to total runtime (whole system, single devices/equipment may not work)
% Quantitative Measurement /Survey
Measured data set
Data Source Users
Services Data processing
Viajeo Platform
Historical traffic
database
Floating Car Data
(FCD
Fleet Management
Operators
Real time traffic
data
GPS equipped cars
Real time
information
Web based
journey planner
Dynamic traffic
information
systems
End-user on-board
unit & website
Mizar, IPT
Magnetti Marelli Magnetti Marelli
Traffic Engine
(traffic info &
alerts)
Traffic data
processing & model
Mizar
Mizar
Magnetti Marelli, Mizar
Magnetti Marelli
D5.1 Validation plan
28/02/2011 Page 32 of 63 Version 2.0
Functionality test
Correctness / quality
Percentage of accuracy of data output (real time traffic data)
% Qualitative Measurement / Survey
Measured data set as well as data of a reference measuring system
Service 2: Traffic Engine (traffic information) Module 3.1 Traffic engine (traffic information)
Indicator Description
Data attributes Source of data
Comments / Remarks Unit
Type of data
acquisition
Method of data
acquisition
Performance and Reliability test
Status Percentage of availability of input data (all possible input data are available)
% Quantitative Survey / Measurement
Measured data set
Actuality Mean time between data updates (traffic information)
Seconds
Quantitative Survey / Measurement
Measured data set
Reliability Percentage of error-free run-time related to total runtime (whole system, single devices/equipment may not work)
% Quantitative Measurement /Survey
Measured data set
Functionality test
Correctness / quality
Percentage of accuracy of data output (quality of traffic information)
% Qualitative Measurement / Survey
Measured data set as well as data of a reference measuring system
Service 3: Traffic information for end users (web-based and mobile applications)
Module 5.1 Mobile application
Indicator Description
Data attributes Source of data
Comments / Remarks Unit
Type of data
acquisition
Method of data
acquisition
User perception
Utilizability Mean percentage of error free data communication time between gateway and mobile applications
% / unit Quantitative /Qualitative
Measurement / Survey
Measured data set
Handling test
Operability Percentage of people who are able to operate the system (settings)
%
Qualitative Survey System itself
Comprehensibility Percentage of people who understand the given output data of the system
%
Qualitative Survey System itself
Satisfaction Percentage of user who are satisfied with the system
%
Qualitative Survey System itself
D5.1 Validation plan
28/02/2011 Page 33 of 63 Version 2.0
Module 5.2 Web
Indicator Description
Data attributes Source of data
Comments / Remarks Unit
Type of data
acquisition
Method of data
acquisition
User perception
Utilizability Mean percentage of error free data communication time between gateway and web
% Quantitative /Qualitative
Measurement / Survey
Measured data set
Handling test
Operability Percentage of people who are able to operate the system (settings)
%
Qualitative Survey System itself
Comprehensibility Percentage of people who understand the given output data of the system
%
Qualitative Survey System itself
Satisfaction Percentage of user who are satisfied with the system
%
Qualitative Survey System itself
D5.1 Validation plan
28/02/2011 Page 34 of 63 Version 2.0
4.4.2.3. Beijing – Indicator list (field test)
Beijing – Floating car data (T-System+BTRC)
1.2 FCD T-
systems
1.2 Proprie-
tary
kml/xml
4.2 Consumer
Service Gateway
T-systems
3.1 Traffic
Engine and model
T-systems
2.2.1 Mobile Real-
Time Data
Processing
2.2.2 DB processing
T-systems
3.1 Proprie-
tary
kml/xml
2.2 Proprie-
tary
kml/xml
4.1 Public
Authority
Gateway
T-systems
4.2
Proprie-
tarykml/xml
4.1
Proprie-
taryKml/xml
5.3 Information
in the Internet
T-systems
5.2 Information
in the Internet
T-systems
Figure 7: Service diagram - floating car data from private vehicles (Beijing)
Module 2.2.1 Mobile real time processing
Indicator Description
Data attributes Source of data
Comments / Remarks Unit
Type of data
acquisition
Method of data
acquisition
Performance and Reliability test
Setup No. of measuring devices / equipment (FCD)
No. Quantitative Survey / Measurement
Measured data set
Status Mean no. of measuring devices working well related to number of active FCD devices
% Quantitative Survey / Measurement
Measured data set
Actuality Mean time between data updates (Mobile real-time data processing)
Seconds
Quantitative Survey / Measurement
Measured data set
Reliability Percentage of error-free run-time related to total runtime (whole system, single devices/FCD may not work)
% Quantitative Measurement /Survey
Measured data set
Functionality test
Correctness / quality
Percentage of accuracy of data output (Mobile real-time data processing)
% Qualitative Measurement / Survey
Measured data set as well as data of a reference measuring system
Module 2.2.2 DB processing
Indicator Description
Data attributes Source of data
Comments / Remarks Unit
Type of data
acquisition
Method of data
acquisition
Performance and Reliability test
Setup No. of measuring devices / equipment (FCD)
No. Quantitative Survey / Measurement
Measured data set
Status Mean no. of measuring devices working well related to number of active FCD devices
% Quantitative Survey / Measurement
Measured data set
Actuality Mean time between data updates (DB processing)
Seconds
Quantitative Survey / Measurement
Measured data set
D5.1 Validation plan
28/02/2011 Page 35 of 63 Version 2.0
Reliability Percentage of error-free run-time related to total runtime (whole system, single devices/FCD may not work)
% Quantitative Measurement /Survey
Measured data set
Functionality test
Correctness / quality
Percentage of accuracy of data output (DB processing)
% Qualitative Measurement / Survey
Measured data set as well as data of a reference measuring system
Module 3.1 Traffic Engine and model
Indicator Description
Data attributes Source of data
Comments / Remarks Unit
Type of data
acquisition
Method of data
acquisition
Performance and Reliability test
Setup No. of input data/sources No. Quantitative Survey / Measurement
Measured data set
Status Percentage of availability of input data (all possible input data are available)
% Quantitative Survey / Measurement
Measured data set
Actuality Mean time between data updates (Traffic data and model)
Seconds
Quantitative Survey / Measurement
Measured data set
Reliability Percentage of error-free run-time related to total runtime (whole system, single devices/equipment may not work)
% Quantitative Measurement /Survey
Measured data set
Functionality test
Correctness / quality
Percentage of accuracy of data output (Traffic data and model)
% Qualitative Measurement / Survey
Measured data set as well as data of a reference measuring system
Module 4.1 Public Authority Gateway
Indicator Description
Data attributes Source of data
Comments / Remarks Unit
Type of data
acquisition
Method of data
acquisition
User perception
Utilizability Mean percentage of error free data communication time between system and gateway
% Quantitative /Qualitative
Measurement / Survey
Data stream
Conformity Mean percentage of information available after conversion of transmission protocols
% Quantitative /Qualitative
Measurement / Survey
Data stream
Handling test
Operability Percentage of people who are able to operate the gateway (settings)
%
Qualitative Survey System itself
Economic test
Cost of operation Mean costs of operation and maintenance costs
€/month Quantitative Survey
D5.1 Validation plan
28/02/2011 Page 36 of 63 Version 2.0
Module 4.2 Consumer Service Gateway
Indicator Description
Data attributes Source of data
Comments / Remarks Unit
Type of data
acquisition
Method of data
acquisition
User perception
Utilizability Mean percentage of error free data communication time between system and gateway
% Quantitative /Qualitative
Measurement / Survey
Data stream
Conformity Mean percentage of information available after conversion of transmission protocols
% Quantitative /Qualitative
Measurement / Survey
Data stream
Handling test
Operability Percentage of people who are able to operate the gateway (settings)
%
Qualitative Survey System itself
Economic test
Cost of operation Mean costs of operation and maintenance costs
€/month Quantitative Survey
Module 5.2 Information in the internet (public authorities & Operators)
Indicator Description
Data attributes Source of data
Comments / Remarks Unit
Type of data
acquisition
Method of data
acquisition
User perception
Utilizability Mean percentage of error free data communication time between gateway and internet
% Quantitative /Qualitative
Measurement / Survey
Measured data set
Handling test
Operability Percentage of people who are able to operate the system (settings)
%
Qualitative Survey System itself
Comprehensibility Percentage of people who understand the given output data of the system
%
Qualitative Survey System itself
Satisfaction Percentage of user who are satisfied with the system
%
Qualitative Survey System itself
Economic test
Cost of operation Mean costs of operation and maintenance costs
€/month Quantitative Survey
Module 5.3 Information in the internet (Public)
Indicator Description
Data attributes Source of data
Comments / Remarks Unit
Type of data
acquisition
Method of data
acquisition
User perception
Utilizability Mean percentage of error free data communication time between gateway and web
% Quantitative /Qualitative
Measurement / Survey
Measured data set
Handling test
Operability Percentage of people who are able to operate the system (settings)
%
Qualitative Survey System itself
D5.1 Validation plan
28/02/2011 Page 37 of 63 Version 2.0
Comprehensibility Percentage of people who understand the given output data of the system
%
Qualitative Survey System itself
Satisfaction Percentage of user who are satisfied with the system
%
Qualitative Survey System itself
Economic test
Cost of operation Mean costs of operation and maintenance costs
€/month Quantitative Survey
D5.1 Validation plan
28/02/2011 Page 38 of 63 Version 2.0
Beijing – Data from buses equipped with Automatic Vehicle Monitoring system (AVM)
Figure 8: Service diagram, data from buses equipped with AVM (Beijing)
Module 2.5 Realtime Bus Data processing BPT
Indicator Description
Data attributes
Source of data Comments / Remarks Unit
Type of data
acquisition
Method of data
acquisition
Performance and Reliability test
Setup No. of measuring devices / equipment (AVM)
No. Quantitative Survey / Measurement
Measured data set
Status Mean no. of AVM devices working well related to number of active AVM
% Quantitative Survey / Measurement
Measured data set
Actuality Mean time between data updates (Real time bus data processing)
Seconds
Quantitative Survey / Measurement
Measured data set
Reliability Percentage of error-free run-time related to total runtime (whole system, single devices / bus data may not work)
% Quantitative Measurement /Survey
Measured data set
Functionality test
Correctness / quality
Percentage of accuracy of data output (Real time bus data processing)
% Qualitative Measurement / Survey
Measured data set as well as data of a reference measuring system
Module 5.1 Bus stop
Indicator Description
Data attributes Source of data
Comments / Remarks Unit
Type of data
acquisition
Method of data
acquisition
User perception
Utilizability Mean percentage of error free data communication time between gateway and bus stop information
% Quantitative Measurement / Survey Measured data set
Handling test
Operability Percentage of people who are able to operate the system (settings)
%
Qualitative Survey System itself
D5.1 Validation plan
28/02/2011 Page 39 of 63 Version 2.0
Comprehensibility Percentage of people who understand the given output data of the system
%
Qualitative Survey System itself
Satisfaction Percentage of user who are satisfied with the system
%
Qualitative Survey System itself
Economic test
Cost of operation Mean costs of operation and maintenance costs
€/month Quantitative Survey
Module 5.2 Onboard
Indicator Description
Data attributes Source of data
Comments / Remarks Unit
Type of data
acquisition
Method of data
acquisition
User perception
Utilizability Mean percentage of error free data communication time between gateway and onboard information
% Quantitative /Qualitative
Measurement / Survey
Measured data set
Handling test
Operability Percentage of people who are able to operate the system (settings)
%
Qualitative Survey System itself
Comprehensibility Percentage of people who understand the given output data of the system
%
Qualitative Survey System itself
Satisfaction Percentage of user who are satisfied with the system
%
Qualitative Survey System itself
Economic test
Cost of operation Mean costs of operation and maintenance costs
€/month Quantitative Survey
D5.1 Validation plan
28/02/2011 Page 40 of 63 Version 2.0
4.4.2.4. Shanghai – Indicator list (field test)
Shanghai – data from bus equipped with GPS
Figure 9: Service diagram, data from buses equipped with GPS
Module 2.1 Bus data processing
Indicator Description
Data attributes Source of data
Comments / Remarks Unit
Type of data
acquisition
Method of data
acquisition
Performance and Reliability test
Setup No. of measuring devices / equipment (Bus data)
No.
Quantitative
Survey / Measurement
Measured data set
Status Mean no. of measuring devices working well related to number of active Bus data devices
% Quantitative Survey / Measurement
Measured data set
Actuality Mean time between data updates (Real time data processing)
Seconds
Quantitative Survey / Measurement
Measured data set
Reliability Percentage of error-free run-time related to total runtime (whole system, single devices / bus data may not work)
% Quantitative Measurement /Survey
Measured data set
Functionality test
Correctness / quality
Percentage of accuracy of data output (Bus data processing)
% Qualitative Measurement / Survey
Measured data set as well as data of a reference measuring system
Module 3.1 Bus arrival time
Indicator Description
Data attributes Source of data
Comments / Remarks Unit
Type of data
acquisition
Method of data
acquisition
Performance and Reliability test
Setup No. of input data/sources No. Quantitative Survey / Measurement
Measured data set
Status Percentage of availability of input data (all possible input data are available)
% Quantitative Survey / Measurement
Measured data set
Reliability Percentage of error-free run-time related to total runtime (whole system, single devices/equipment may not work)
% Quantitative Measurement /Survey
Measured data set
Functionality test
Correctness / quality
Percentage of accuracy of data output (public transport journey planner)
% Qualitative Measurement / Survey
Measured data set as well as data of a reference measuring system
D5.1 Validation plan
28/02/2011 Page 41 of 63 Version 2.0
Module 5.1 Display systems at bus stops
Indicator Description
Data attributes Source of data
Comments / Remarks Unit
Type of data
acquisition
Method of data
acquisition
User perception
Utilizability Mean percentage of error free data communication time between gateway and onboard information
% Quantitative /Qualitative
Measurement / Survey
Measured data set
Handling test
Operability Percentage of people who are able to operate the system (settings)
%
Qualitative Survey System itself
Comprehensibility Percentage of people who understand the given output data of the system
%
Qualitative Survey System itself
Satisfaction Percentage of user who are satisfied with the system
%
Qualitative Survey System itself
D5.1 Validation plan
28/02/2011 Page 42 of 63 Version 2.0
Shanghai - Environmental data from mobile sensors
Figure 10: Service diagram, environmental data from mobile sensors
Module 2.2 Road Environment data processing
Indicator Description
Data attributes Source of data
Comments / Remarks Unit
Type of data
acquisition
Method of data
acquisition
Performance and Reliability test
Setup No. of measuring devices / equipment (road environment detector)
No. Quantitative Survey / Measurement
Measured data set
Status Mean no. of measuring devices working well related to number of active road environment detectors
% Quantitative Survey / Measurement
Measured data set
Actuality Mean time between data updates
Seconds
Quantitative /Qualitative
Survey / Measurement
Measured data set
Reliability Percentage of error-free run-time related to total runtime (whole system, single devices / road environmental equipment may not work)
% Quantitative /Qualitative
Measurement /Survey
Measured data set
Module 3.2 Realtime monitoring
Indicator Description
Data attributes Source of data
Comments / Remarks Unit
Type of data
acquisition
Method of data
acquisition
Functionality test
Correctness / quality
Percentage of accuracy of data output (environmental model)
% Qualitative Measurement / Survey
Measured data set as well as data of a reference measuring system
Module 4.2 Web based information for user
Indicator Description
Data attributes Source of data
Comments / Remarks Unit
Type of data
acquisition
Method of data
acquisition
User perception
Actuality Mean time between data updates
Days
Quantitative /Qualitative
Survey / Measurement
Measured data set
D5.1 Validation plan
28/02/2011 Page 43 of 63 Version 2.0
Reliability Percentage of error-free run-time related to total runtime (whole system, single devices/equipment may not work)
% Quantitative /Qualitative
Measurement /Survey
Measured data set
Status Percentage of availability of input data (all possible input data are available)
% Quantitative /Qualitative
Survey / Measurement
Measured data set
Module 4.3 Environment model
Indicator Description
Data attributes Source of data
Comments / Remarks Unit
Type of data
acquisition
Method of data
acquisition
User perception
Actuality Mean time between data updates
Minutes
Quantitative /Qualitative
Survey / Measurement
Measured data set
Reliability Percentage of error-free run-time related to total runtime (whole system, single devices/equipment may not work)
% Quantitative /Qualitative
Measurement /Survey
Measured data set
Status Percentage of availability of input data (all possible input data are available)
% Quantitative /Qualitative
Survey / Measurement
Measured data set
D5.1 Validation plan
28/02/2011 Page 44 of 63 Version 2.0
4.4.3. Validation of selected and implemented European standards (adaptability and practicability)
The open platform itself as well as the integrated local components and services consist of modules and their dedicated interfaces. One of the Viajeo requirements is that – as far as possible – all interfaces correspond to European standards. Due to the fact that Viajeo is a R&D project with demo sites in Europe, China and Brazil it is likely that their might be difficulties regarding the implementation of European standards. Difficulties can be caused e.g. by a lack of suitable European standards (because of new technologies which are still in the research phase) or by incompatibility of the European standards and the local components and services which can not be solved within the Viajeo project. However the table below gives an overview about the selected standards which are likely to be integrated into the implemented systems in the demo sites. Beside columns for the interface name, designation, typically transmitted data and standard to be used there are also columns for remarks, demo sites and comments. The remarks are related mainly to general difficulties (non implemented system) and are made by WP3 while the comments are related to specific difficulties during the implementation of the systems in the sites. Comments will be given during the implementation (at later stage of the project) by those partners who are responsible for the implementation.
Interface Name Desig-nation
Data Typically Transmitted Standard to be used
Remark Demo sites Comments
FCD I1 Vehicle ID Time Stamp Position Speed
SIMONE SIMONE does not count as an European Standard (see chapter 4.3 for explanation, D3.1) but is used in the market
Athens, Brazil, Beijing, Shanghai
Roadside Traffic Count Data
I2 Detector ID Lane N° Direction Traffic Count (passenger cars) Traffic Count (lorries) Time Gap (between vehicles)
DATEX 2 Road side units (especially in case of older systems) may feature low computing power and low bandwidth connections. DATEX 2 is designed for centre to centre communication and rather heavy, hence a level 3 compliant interface is used.
Athens, Brazil
Roadside Environmental Data
I3 Detector ID Direction Measurement CO2 Measurement NOx
DATEX 2 Shanghai
Centre Road Traffic Data Interface
I4 Traffic Events Traffic Load per Segment
DATEX 2 Classic DATEX 2 application Athens, Brazil
Static Public Transport Information
I5 Timetables Stop Positions
VDV 452 SIRI does not cover static data, the German VDV 452 standard is based on the same principles and widely used in Europe
Beijing, Shanghai
Dynamic Public Transport Information
I6 Vehicle ID Time Stamp Delays (line,vehicle)
SIRI Beijing, Shanghai
Traffic Information over TPEG
I7 Traffic Messages Traffic Load
TPEG RTM/TEC
TPEG RTM defined as widely used standard, TPEG TEC is widely used in the European automotive sector.
Athens, Brazil, Beijing
Comodal Routes I8 Waypoints Travel Time on Links Traffic Mode
IN TIME / eMotion
No public standard available, the European Project IN TIME defined a well received approach (see chapter 4.7 for explanation).
Athens, Brazil, Beijing
Table 1: Interface – standard – demo site relation (Source: PTV)
D5.1 Validation plan
28/02/2011 Page 45 of 63 Version 2.0
5. Impact assessment
The present section deals with the “Impact assessment – Testing Phase II (Field test)” that refers to the evaluation of expected impacts of the Viajeo project. For the impact assessment an already implemented systems is obligatory.
5.1. Objectives
Goal of the impact assessment is to evaluate the expected impacts of the Viajeo project to users (professionals, non-professionals), stakeholder and decision makers in the demo sites. A broad overview about the impacts in the different sites is important to evaluate the whole Viajeo project and to validate if the project requirements are met.
5.2. Target groups
The following three target groups have been identified to participate in the impact assessment of the Viajeo Platform:
• End Users: These are the daily users that are expected to use most of the services of the Viajeo platform once they are finalized. Members of this group have already been involved in the project during the identification of user needs in the framework of WP2 (residents, visitors, taxi drivers)
• Transport and traffic planners (incl. stakeholder and decision maker): These are planners or decision makers responsible for the transport and traffic planning not only of the involved cities but of other cities as well. They can either be high level civil servants working for relevant ministries, municipalities and authorities and responsible for the decision making, but also experts as well as researchers.
• Transport operators: This group involves people from transport companies providing transport services to citizens.
5.3. Approach and methodology – impact assessment
A brief overview of the chosen approach and methodology is given in the following. Basis for the impact assessment are the finally implemented use cases in the demo sites. These use cases are developed in WP 2 based on the user needs identified during the investigation in the demo sites. Related to the use cases services/extensions are developed and planned by the demo sites themselves.
Not all use cases defined in WP 2 are going to be finally implemented in the demo sites. The following table gives an overview about the use cases which will be implemented in the Viajeo project related to the corresponding demo sites.
For each implemented use case a set of indicators has been developed according to the MAESTRO Evaluation Guideline which means that each indicator is related to an evaluation category, a sub category and an impact. Furthermore there is a short description of the indicator as well as information about (expected) data attributes and the intended target groups.
D5.1 Validation plan
28/02/2011 Page 46 of 63 Version 2.0
Implemented use cases Demo site for the use cases
Number Description Athens Sao Paulo
Beijing Shanghai
1 Real time bus arrival information on displays at bus stops
X X
2 Web-based journey planner
Multi-model journey planer (pre-trip) X X
Car only journey planer (pre-trip) X
Multi-modal journey planer on buses (on-trip) X
3 Dynamic assignment of taxies X
4 Traffic alerts for taxi drivers (realtime) X
5 Traffic information to end users (realtime)
Mobile application / traffic information / route planer
X
Mobile application X
Traffic information X
6 Processed traffic information and indicators to transport and traffic planners
With specific users (transport and traffic planner) X
Interface without specific users (Sao Paulo) X
7 Real time bus operation (to operators) X
8 Environmental data monitoring
Real time monitoring X
Historic data base and modelling X
5 3 5 3
Table 2: Relation between implemented use cases and demo sites
D5.1 Validation plan
28/02/2011 Page 47 of 63 Version 2.0
5.3.1. Impact assessment indicators – Athens
Use case 2: Web-based multi modal journey planner
Number Evaluation
Sub-Category Impact Indicator Description
Data attributes
Target group Unit
Type of data acquisition
Method of data
acquisition
Acceptance
2.1.1 General attitude (operator)
Operator attitude
Operator acceptance
Increase of frequency of use of advanced technologies in urban mobility
Index Qualitative Survey Transport and traffic planer
2.1.2 Degree to which the transport operators can adapt to the changes caused by the new services
Index Qualitative Survey Transport and traffic planer
2.1.3 General attitude (user)
New technology acceptance
User Acceptance
Increase of satisfaction of the users when travelling in the urban network due to offered services
Index
Qualitative Survey End user
Accessibility
2.2.1 Accessibility Service integration
Interchange points
Possibility of change of route of users due to the provided information
Index Qualitative Measurement End user
2.2.2 Service integration
Achievement of better coordination between different transport modes
Index Qualitative Survey End user
Transport patterns
2.3.1 Route change Route change Average route change trips
Increase of frequency of use of advanced technologies in urban mobility
Index Qualitative Survey End user
2.3.2 Route guidance
Improved journey time
Reduction of travel times Index Quantitative Qualitative
Measurement or modelled
End user
D5.1 Validation plan
28/02/2011 Page 48 of 63 Version 2.0
Quality of service
2.4.1 Information Availability of information
Information accessibility
Percentage of people who know to access information
% Index
Quantitative Qualitative
Survey End user
2.4.2 Ease of use Ease of use Clarity of information
Increase of overall satisfaction / understanding of information of special categories of users (elderly people, mobility impaired, etc) by the provided services of public transport modes
Index Qualitative Survey End user
Use case 3: Dynamic assignment of taxies
Number Evaluation
Sub-Category Impact Indicator Description
Data attributes
Target group Unit
Type of data acquisition
Method of data
acquisition
Acceptance
3.1.1 General attitude (operator)
Operator attitude
Operator acceptance
Increase of the use of new technologies by the taxi companies
Index Qualitative Survey Transport operators
3.1.2 Satisfaction of taxi companies by the available information for the route planning
Index Qualitative Survey Transport operators
3.1.3 General attitude (user)
User acceptance of demonstrated service
Acceptance rating
Degree to which the use of more up to date devices by taxi drivers will increase as a result of the services offered through them.
Index
Qualitative Survey End user
3.1.4 Satisfaction of taxi drivers due to the more efficient distribution of their routes
Index
Qualitative Survey End user
D5.1 Validation plan
28/02/2011 Page 49 of 63 Version 2.0
Accessibility
3.2.1 Accessibility Accessibility System access distance
Increase in quality of provided services in terms of quick response to a call for a taxi due to the decrease of traffic congestion
Minutes Index
Quantitative Qualitative
Measurement Survey
End user / Transport operators
Capacity
3.3.1 Network efficiency
Network efficiency
Passenger load factor
No. occupied taxis in relation to available taxis
% / hr Index
Quantitative Qualitative
Collected / Measurement Survey
End user / Transport operators
3.3.2 More optimized transport network due to the more efficient planning of taxi routes
Index Qualitative Collected / Survey
End user / Transport operators
Quality of service
3.4.1 Information Availability of information
Information accessibility
Percentage of taxi driver who can access information
Index Qualitative Survey End user
3.4.2 Ease of use Ease of use Clarity of information
Rating given to ease of understanding information
Index Qualitative Survey End user
Use case 4: Traffic alerts for taxi drivers
Number Evaluation
Sub-Category Impact Indicator Description
Data attributes
Target group Unit
Type of data acquisition
Method of data
acquisition
Acceptance
4.1.1 General attitude (user)
User acceptance of demonstrated service
Acceptance rating
Increase of frequency of use of advanced technologies in urban mobility
Index
Qualitative Survey End user/ Taxi Driver
D5.1 Validation plan
28/02/2011 Page 50 of 63 Version 2.0
4.1.2 Satisfaction Increase of satisfaction of the users when travelling in the urban network due to offered services
Index
Qualitative Survey End user/ Taxi Driver
Transport patterns
4.2.1 Route change Route change Average route change trips
Increase of taxi drivers with changed route
% Index
Quantitative Qualitative
Measurement or Survey
Taxi driver
4.2.2 Route guidance
Improved journey time
Journey time saved due to revised route choice
Index Quantitative Qualitative
modelled / Survey
Taxi driver
Quality of service
4.3.1 Ease of use Ease of use Clarity of information
Rating given to ease of understanding information
Index Qualitative Survey Taxi driver
Use case 5: Traffic information to end users
Number Evaluation
Sub-Category Impact Indicator Description
Data attributes
Target group Unit
Type of data acquisition
Method of data
acquisition
Acceptance
5.1.1 General attitude (user)
User acceptance of demonstrated service
Acceptance Increase of frequency of use of advanced technologies in urban mobility
Index
Qualitative Survey End user/ Taxi Driver
5.1.2 Satisfaction Increase of satisfaction of the users when travelling in the urban network due to offered services
Index
Qualitative Survey End user/ Taxi Driver
Transport patterns
5.2.1 Reduction of traffic
Reduction of traffic
Route changes
Possibility of change of route of users due to the provided information
Index Qualitative Survey End user/ Taxi Driver
D5.1 Validation plan
28/02/2011 Page 51 of 63 Version 2.0
Quality of service
5.3.1 Information Availability of information
Information accessibility
people who know to access information
Index Qualitative Survey End user/ Taxi Driver
5.3.2 Success of advertising
people who know about service
Index Qualitative Survey End user/ Taxi Driver
Use case 6: Processes traffic information and indicators to transport and traffic planners
Number Evaluation
Sub-Category Impact Indicator Description
Data attributes
Target group Unit
Type of data acquisition
Method of data
acquisition
Acceptance
6.1.1 General attitude (operator)
Operator attitude
Operator Acceptance
Increase of satis-faction of planners from kind of available information, traffic models and data accuracy
Index Qualitative Survey Transport and traffic planer
6.1.2 General attitude (user)
New technology acceptance
User Acceptance
Increase in the number of users
Index Qualitative Survey Transport and traffic planer
Accessibility
6.2.1 Accessibility Better transport systems accessible for all
Increase of special user groups i.e. seniors
Increase of number of users and satisfaction by transport and planning
Index Qualitative Survey Transport and traffic planer
Transport Pattern
6.3.1 Optimisation More Optimised transport systems
Reduction of travel times
Reduction of travel time due to the better coordination of transport modes
Index Qualitative Survey Transport and traffic planer
D5.1 Validation plan
28/02/2011 Page 52 of 63 Version 2.0
6.3.2 Congestion More Optimised transport systems
Decrease of Congestions
Decrease in the traffic congestion due to the better coordination of transport modes
Index Qualitative Survey Transport and traffic planer
D5.1 Validation plan
28/02/2011 Page 53 of 63 Version 2.0
5.3.2. Impact assessment indicators - Sao Paulo
Use case 2: Web-based journey planner
Number Evaluation
Sub-Category Impact Indicator Description
Data attributes
Target group Unit
Type of data acquisition
Method of data
acquisition
Acceptance
2.1.1 General attitude (user)
User acceptance of demonstrated data
Acceptance rating
Attitude survey of current and potential utility
Index
Qualitative Survey End user
Transport patterns
2.3.2 Route guidance Improved journey time
Reduction of travel times Index Qualitative Measurement or modelled
End user
Capacity
2.2.1 Information Quality index
Composition rating of pre defined criteria / set of information
Index Qualitative Collected / Measurement or Survey
End user
Use case 5: Traffic information to end users
Number Evaluation
Sub-Category Impact Indicator Description
Data attributes
Target group Unit
Type of data acquisition
Method of data
acquisition
Acceptance
5.1.1 General attitude (user)
User acceptance of demonstrated service
Acceptance Increase of frequency of use of advanced technologies in urban mobility
Index
Qualitative Survey End user/ Taxi Driver
D5.1 Validation plan
28/02/2011 Page 54 of 63 Version 2.0
5.1.2 Satisfaction Increase of satisfaction of the users when travelling in the urban network due to offered services
Index
Qualitative Survey End user/ Taxi Driver
Quality of service
5.3.1 Information Availability of information
Information accessibility
people who know to access information
Index/% Qualitative/ Quantitative
Survey End user/ Taxi Driver
5.3.2 Success of advertising
people who know about service
Index/% Qualitative Quantitative
Survey End user/ Taxi Driver
Use case 6: Processes traffic information and indicators to transport and traffic planners
Number Evaluation
Sub-Category Impact Indicator Description
Data attributes
Target group Unit
Type of data acquisition
Method of data
acquisition
Acceptance
6.1.1 General attitude (operator)
Operator attitude
Operator Acceptance
Increase of satis-faction of planners from kind of available information, traffic models and data accuracy
Index Qualitative Survey Transport and traffic planer
6.1.2 General attitude (user)
New technology acceptance
User Acceptance
Increase in the number of users
Index Qualitative Survey Transport and traffic planer
Accessibility
6.2.1 Accessibility Better transport systems accessible for all
Increase of special user groups i.e. seniors
Increase of number of users and satisfaction by transport and planning
Index Qualitative Survey Transport and traffic planer
D5.1 Validation plan
28/02/2011 Page 55 of 63 Version 2.0
5.3.3. Impact assessment indicators – Beijing
Use case 1: Real time bus arrival information on displays at bus stops
Number Evaluation
Sub-Category Impact Indicator Description
Data attributes
Target group Unit
Type of data acquisition
Method of data
acquisition
Acceptance
1.1.1 General attitude (user)
User acceptance of demonstrated Data
Acceptance rating
Attitude survey of current and potential utility
Index
Qualitative Survey End user
1.1.2 Accessibility Access Service access Information readability and information content
Index Qualitative Survey End user
Quality of service
1.2.1 Information Availability of information
Information accessibility
Percentage of people who know how to access information
Index Qualitative Survey End user
1.2.2 Information of advertising
Percentage of people who know or have recognized the new service
Index Qualitative Survey End user
Use case 2: Web-based journey planner (pre trip)
Number Evaluation
Sub-Category Impact Indicator Description
Data attributes
Target group Unit
Type of data acquisition
Method of data
acquisition
Acceptance
2.1.1 General attitude (operator)
Operator attitude
Operator acceptance
Acceptance confidence rating given by operator to demonstration parameters
Index Qualitative Survey Transport and traffic planer
D5.1 Validation plan
28/02/2011 Page 56 of 63 Version 2.0
2.1.2 General attitude (user)
User acceptance of demonstrated data
Acceptance rating
Attitude survey of current and potential utility
Index
Qualitative Survey End user
2.1.4 Ease of modal transfer
Level of conflict between modes
Conflict rating
Public survey to measure parameters of conflict between modes
Index Qualitative Survey End user
Capacity
2.2.1 Information Quality index
Composition rating of pre defined criteria / set of information
Index Qualitative Collected / Measurement or Survey
End user
Quality of service
2.4.1 Information Availability of information
Information accessibility
Percentage of people who know to access information
Index Qualitative Survey End user
2.4.2 Success of advertising
Percentage of people who know about service
Index Qualitative Survey End user
Use case 2: Web-based journey planner (on trip)
Number Evaluation
Sub-Category Impact Indicator Description
Data attributes Target group Unit
Type of data acquisition
Method of data acquisition
Acceptance
2.1.1 General attitude (user)
User acceptance of demonstrated data
Acceptance rating
Attitude survey of current and potential utility
Index
Qualitative Survey End user
2.1.4 Access for mobility-impaired PAX
System access
Survey about the possibility of use by mobility-impaired PAX (e.g. location inside/ free space in front / height of installation)
Index Qualitative Survey End user
D5.1 Validation plan
28/02/2011 Page 57 of 63 Version 2.0
Quality of service
2.2.1 Information Availability of information
Information accessibility
Percentage of people who know to access information
Index Qualitative Survey End user
2.2.2 Success of advertising
Percentage of people who know about service
Index Qualitative Survey End user
Use case 5: Traffic information to end users
Number Evaluation
Sub-Category Impact Indicator Description
Data attributes
Target group Unit
Type of data acquisition
Method of data
acquisition
Acceptance
5.1.1 General attitude (user)
User acceptance of demonstrated service
Acceptance Increase of frequency of use of advanced technologies in urban mobility
Index
Qualitative Survey End user
5.1.2 Satisfaction Increase of satisfaction of the users when travelling in the urban network due to offered services
Index
Qualitative Survey End user
Quality of service
5.3.1 Information Availability of information
Information accessibility
people who know to access information
Index Qualitative Survey End user
5.3.2 Success of advertising
people who know about service
Index Qualitative Survey End user
Use case 7: Real time bus operation
Number Evaluation
Sub-Category Impact Indicator Description
Data attributes Target group Unit
Type of data acquisition
Method of data
acquisition
Acceptance
7.1.1 General Operator Operator Acceptance confidence Index Qualitative Survey Transport
D5.1 Validation plan
28/02/2011 Page 58 of 63 Version 2.0
attitude (operator)
attitude acceptance rating given by operator to demonstration parameters
operator
Quality of service
7.3.9 Ease of use Ease of use Clarity of information
Rating given to ease of understanding information
Index Qualitative Survey End user
D5.1 Validation plan
28/02/2011 Page 59 of 63 Version 2.0
5.3.4. Impact assessment indicators – Shanghai
Use case 1: Real time bus arrival information on displays at bus stops
Number Evaluation
Sub-Category Impact Indicator Description
Data attributes Target group Unit
Type of data acquisition
Method of data acquisition
Acceptance
1.1.1 General attitude (user)
User acceptance of demonstrated Data
Acceptance rating
Attitude survey of current and potential utility
Index
Qualitative Survey End user
1.1.2 Accessibility Access Service access Information readability and information content
Index Qualitative Survey End user
Quality of service
1.2.1 Information Availability of information
Information accessibility
Percentage of people who know how to access information
Index Qualitative Survey End user
1.2.2 Information of advertising
Percentage of people who know or have recognized the new service
Index Qualitative Survey End user
1.2.3 Service frequency
Service frequency
Average update time
Average time between information updates
Min. Quantitative Measurement End user
1.2.4 Reliability reliability Accuracy of timekeeping
Percentage of services arriving within given arrival time
Index Qualitative Survey End user
1.2.5 Ease of use Ease of use Clarity of information
Rating given to ease of understanding information
Index Qualitative Survey End user
D5.1 Validation plan
28/02/2011 Page 60 of 63 Version 2.0
Use case 8.1: real time environmental data monitoring
Number Evaluation
Sub-Category Impact Indicator Description
Data attributes
Target group Unit
Type of data acquisition
Method of data
acquisition
Acceptance
8.1.1 General attitude
General attitude
Acceptance Acceptance confidence rating to demonstration parameters
Index Qualitative Survey End user
Quality of service
8.1.2 Accuracy Accuracy of information
Information accuracy
Comparison mobile to fixed stations
Index Qualitative Survey End user and Operator
Use case 8.2: environmental data and modelling
Number Evaluation
Sub-Category Impact Indicator Description
Data attributes
Target group Unit
Type of data acquisition
Method of data
acquisition
Acceptance
8.2.1 General attitude
Planer attitude
Acceptance Acceptance confidence rating given by operator to demonstration parameters
Index Qualitative Survey Transport and traffic planer
Quality of service
8.2.2 Accuracy Accuracy of information
Information accuracy
Comparison measured and modelled dataset
Index Qualitative Survey Transport and traffic planer
D5.1 Validation plan
28/02/2011 Page 61 of 63 Version 2.0
6. Perspectives for European ITS technologies in China and Brazil
Following a brief overview of the proceeding used for the assessment of the perspectives for European ITS technologies in China and Brazil are given.
6.1. Objectives The partners in the Viajeo project aim to ensure that the project will have long lasting effects on the cooperation between European companies dealing in ITS technologies and users at the test sites and other markets. To help achieve this goal, task 5.5 will assess the future potential for export of European technology to the Chinese and Brazilian markets and analyse how entrance into these markets will be able to have a successful outcome.
6.2. Approach and Methodology
The analysis will be conducted on two parallel tracks.
• Firstly, existing models normally used by European firms when considering entering the two markets in question will be discussed. Focus will be placed on both the strategic reflections a company must go through before expansion and practical behaviour necessary to obtain success when first established on the market. Even though China and Brazil have historically been popular for outsourcing, this analysis will only examine the countries in their capacity of markets and ability to purchase European ITS technology. Business models for exporting to China and Brazil will not necessarily be alike for different product groups because buyer segments vary and competition may be fierce in some industries and lax in others. Thus the analysis will as far as possible be centered around export of technologies and ITS technology more precisely.
• Secondly, concrete business models considered by the partners at the test sites will be surveyed. These surveys will be conducted through interviewing the partners who are already on the Chinese and Brazilian markets. The interviews will seek to establish how the partners practically handle the culturally and legislatively different markets and how they plan to expand their operations, if indeed they have such plans. Naturally the interviews will focus on export rather than outsourcing and on the issues specifically faced when offering ITS technology.
The two analyses will result in a guideline for the successful commercial utilization of European ITS technologies in Sao Paulo, Shanghai and Beijing. The aim is that the guideline will be used for opening other Chinese and Brazilian cities for European companies, so European technology will gain a more solid market share in these countries and boost European export.
D5.1 Validation plan
28/02/2011 Page 62 of 63 Version 2.0
7. Data (collection, analysis, validation, tables and reports)
Below a short description about the collection, validation and analysis of the data used in WP5 is given.
7.1. Data - collection Goal of the Viajeo project is to implement an open platform system in different demo sites. This will be done close to the end of the project duration during the demonstration of the demo site. It’s important to note that the data collection will take place during a limited period. Comprehensive measurements like long term measurements are not part of WP5.
The data collection will be carried out at the different test sites, for each of the use cases considered in the validation and impact assessment. The collection itself will be done in the demonstration WP.
The finally choice of the method of data collection is within the responsibility of the corresponding partner in the demo site. This makes sense because these partners know the locally situation. However for each indicator a method of data collection is proposed by WP5 which should (if possible) be used.
7.2. Data - analysis A data analysis can only be as good as the used data basement. Since the data basement within the Viajeo project is not that comprehensive (compare: data-collection) the generated results need to be interpreted carefully. All partners in the demo sites are requested to collect the data as carefully as possible. WP5 will put the results into the context which allows a more general interpretation.
7.3. Data - validation Due to the fact that Viajeo is a research project all collected validation and impact assessment data have to be used carefully related to the circumstances of their collection. A universal validity of the collected data and/or the generated results is therefore not possible. Based on the independent results of the four different demo sites it is very well possible to get a competent knowledge of difficulties and possibilities.
7.4. Data – charts of responsibility The charts of responsibility give information (description, related to, format, responsibility, delivery date) about needed / required data sets which are used in the validation and impact assessment process of the Viajeo project (WP5). There will be one chart for each demo site. The charts will be filled during the process of implementation of functionality and services and will be updated on a regular basis, corresponding to available and/or required data. First contacts within the sites are the site representatives who have two possibilities: (1) to deliver the data or (2) to appoint a finally responsible contact person who deliver the data.
All data sets will be given to the WP5 leader (DLR) who collects and (if necessary) provides the data. This guarantees that all necessary data sets (including possible updates) are available in one place.
The structure of the charts of responsibility is exemplary shown in Table 2.
Athens Data set Related to Data
format Responsibility Delivery date
Description (Name, short description)
Module (Number)
Indicator (Name)
Format (e.g. Excel,
Word, data base)
Organisation (Name)
Person (Name, email, phone-nr.)
Intended (date)
Real (date)
FCD single car Service A all Data base Infotrip Mr. / Ms.
FCD accumulated Service B all Data base Infotrip Mr. / Ms.
… … … … … … Table 3: Chart of responsibility - Example
D5.1 Validation plan
28/02/2011 Page 63 of 63 Version 2.0
7.5. Reporting system for WP5 Within the Viajeo project the open platform system will be implemented in four cities which belong to three different continents. Data collection and at least a first data analysis and validation will be organized by the demo sites. The results and reports will be part of the “Assessment and validation results (D5.2)”. To harmonize/standardize the results and reports of the four demo sites a kind of guideline (e.g. table of content) will be prepared by WP5. All WP5 partners are required to follow the guidelines. This guarantees a constant quality of the reports and makes it easier to integrate the results into the final deliverable of WP5 (D5.2).