Cross-border Interoperability TESTFEST · KPI IVS & Cellular IVS & Cellular Use-case count Dynamic...
Transcript of Cross-border Interoperability TESTFEST · KPI IVS & Cellular IVS & Cellular Use-case count Dynamic...
Cross-border Interoperability TESTFEST
Initial evaluation resultsMarch 28th, 2019
Content
• Logistics
• Technical evaluation• Introduction• TESTFEST UK• TESTFEST FR• TESTFEST BE/NL:
• Results from the app logging• Results from OBU logging• Results from the teams
• Final remarks and closing
Results from the UK TESTFESTMarch 18-19, 2019
Rochester/London, United Kingdom
Overview
Sessions 18th March 19th March
Country In UK with EU partners
(FR, BE, NL)
In UK with EU partners (BE
and NL)
Location Urban (London) Motorway (A2M2) and Rural
(Kent)
Participants 8 vehicles (BeMobile x 1,
BE x 1, RWS x 2, V-
TRON x 1, HE x 1, FR x
1, TfL x1)
6 vehicles (BeMobile x 1, BE
x 1, RWS x 2, V-TRON x 1,
HE x 1)
Providers TfL HE
Protocol and
Services
Live Cellular IVS and Live
Cellular GLOSA
Hybrid IVS, Hybrid RWW,
ITS-G5 GLOSA
Architecture
18/03 London (Urban)19/03 A2M2 (Motorway) + Kent (Rural)
Architecture
Services and Uses cases
Services ProtocolUse cases in London
(Urban)
Use cases in
A2M2 (Motorway) and Kent
(Rural)
Security ITS-G5
IVS CellularDynamic Speed Limit Information
and Dynamic Lane Management
IVS HybridDynamic Speed Limit Information and
Dynamic Lane Management
RWW Hybrid Lane Closure
GLOSA CellularTime-to-green (speed advice)
Time-to-red (speed advice)
GLOSA ITS-G5Time-to-green (speed advice)
Time-to-red (speed advice)
Test statistics
Security tests: “certified/uncertified RSU sent message”
KPI A2M2 (Motorway)
Use-case count AA certificates
Participation 6
Success use-cases 4
Unsuccessful use-cases 2*
*Note: AA certificates were loaded manually by 4 participants. 2 Were unable to load
AA certificates
Test statistics
In-Vehicle Signage – London (Urban)KPI IVS & Cellular IVS & Cellular
Use-case
count
Dynamic Speed Limit Information
Southbound (Showed 50 Speed limit
and Speed Limit End)
Dynamic Lane Management Northbound
(VMS 1 & 2 showed Red “X” lane message
and VMS3 showed “Sign under Test” free
text)
Participation 8 8
Success
use-cases4* 2*
Unsuccesful
use-cases
• RWS x 2 – Unable to receive IVI
messages via cellular.
• FR x 1 – Received IVS, but not
displayed. HMI issue.
• V-TRON ?
TOTAL = 3 +?
• RWS x 2 – Unable to receive IVI
messages via cellular.
• BE x 2 (incl. BeMobile)– No messages
VMS. HMI is not configured to see text.
• FR x 1 – Received IVS, but not
displayed. HMI issue.
• V-TRON ?
TOTAL = 5 +?
*Note: 1 Southbound gantry (G2) was faulty, so HMI replicated only 2 lanes, not 3.
Test statistics
In-Vehicle Signage – A2M2 (Motorway)
KPI IVS & Hybrid IVS & Hybrid
Use-case count
Dynamic Speed Limit Information
Northbound (Display 60 speed
limit until National Speed Limit
just before exit).
Dynamic Lane Management Southbound
(Display 50 speed limit southbound to
gantry 8487A until RWW. Display of
national speed limit for final gantry section).
Participation 6 6
Success
use-cases6 6
Unsuccesful
use-casesTOTAL = 0 TOTAL = 0
Test statistics
Road Works Warnings (RWW)
KPIRWW & Hybrid –
A2M2 (Motorway)
Use-case count1 (Lane closure or other restrictions - Displayed with
50 speed limit prior to national speed limit)
Participation 6
Success use-cases 5
Unsuccesful use-cases
• BE x 1 – Lanes not inverted. OBU/HMI issue.
TOTAL = 1
*Note: BeMobile suggested RWW trace was short.
Test statistics
GLOSA – London (Urban)
KPI GLOSA & Cellular
Use-case countTime-to-green
Time to red
Participation 7 (note: FR since doesn’t handle GLOSA Service)
Success
use-cases1
Unsuccesful use-cases
• RWS x 2 – Unable to receive cellular service.
• 4 (see notes*)
TOTAL = 6
*Notes:
- GLOSA Service was inaccurate.
- Speed advice on HMI display may not have reflected the actual units km/h.
Test statistics
GLOSA – Kent (Rural)
KPI GLOSA & G5 (petrol station an main carriageway)
Use-case countTime-to-green
Time-to-red
Participation 6
Success
use-cases
3 (displayed accurate time)
V-TRON ?
Unsuccesful use-cases
• RWS x 2 – Some issues for receiving GLOSA on
main carriageway.
TOTAL = 2
*Notes:
- Speed advice on HMI display may not have reflected the actual units km/h.
- Time to green occurred only sometimes. This is likely due to petrol station access road only
triggering a phase change occasionally.
Results from FrenchTestfest
March 21-22, 2019
Reims, France
Participation
• 4 countries: FR, NL, UK, BE
• 9 institutions (UK, BE, NL, V-Tron, Renault, PSA, Université Valenciennes, UniversitéReims, SANEF, DIT)
• 14OBUs, 4 RUSs
• PKIs: Common CTL, eCTL (including AA of the four countries)
• 38 registered participants
Scenarios
• 11 events sent for each scenario on a trajectory of 23 kms
• I2V, V-road operator-2-V, V-user-2-V events
• DENM on RWW and OHLN and IVI messages
• PVD has not been run
• Communications on G5 and/or cellular networks
• 4 tours for each OBU
Coverage
Events
Senders and receivers
• For I2V communications:• For V2V communications: SANEF
• From road operators: SANEF, NeoGLS
• From user vehicles: PSA, Renault, NéoGLS
• All OBUS have displayed events at least once (some after correcting configurations on their software)
• 10 vehicles receiving events
• The first day we had a platoon of 10 cars
Some results
•TOUR 1 (10 OBUs)
•Event Result
•1 90%
•2 90%
•3 10% (IVI over G5)
•4 70%
•5 80%
•6 50%
•7 60%
•8 60%
•9 20% (IVI over cellular)
•10 NA
•11 60%
•TOUR 2 (13 OBUs)
•Event Result
•1 85%
•2 77%
•3 23% (IVI over G5)
•4 54%
•5 54%
•6 18%
•7 17%
•8 25%
•9 8% (IVI over cellular)
•10 55%
•11 33%
•TOUR 3 (10 OBUs)
•Event Result
•1 70%
•2 80%
•3 30% (IVI over G5)
•4 60%
•5 60%
•6 63%
•7 30%
•8 70%
•9 0% (IVI over cellular)
•10 30% (G5 coverage )
•11 70%
•TOUR 4 (8 OBUs)
•Event Result
•1 63%
•2 75%
•3 25% (IVI over G5)
•4 38%
•5 75%
•6 43%
•7 38%
•8 63%
•9 13% (IVI over cellular)
•10 NA
•11 25% (G5 coverage)
Lessons learned
• Still some troubles to handle security• ITS-AID on used certificates
• SSP with wrong values (rights)
• Certificate request on the air not implemented by some partners
• Time window to consider any received message (too strict for some partners messages are dropped)
Lessons learned
• Message structure• IVI message not well implemented by somepartners
• Some use cases on IVI not implemented as multi-lane case, …
• IF2 for cellular communications• Communications between National Nodes seemto work well
Next steps
• Analysis of log files
• Analysis of pcaps
• Target Measure IF2 performances
Results from the BE/NL TESTFEST
Overview
Overview
Countries: 6 (FR, UK, BE, NL, DE, CZ)
Locations: Highways + GLOSA site
Participants/providers: 15 teams (4G/G5/IF2)
Test site (BE)
Test site (NL)
RW trailers
RW trailers
Tunnel closure
Lane closure
GLOSASecurity, RWW-DENM, IVS
HLN
vehicle
Use cases
Example of GLOSA
Example of speed advice
Overview of the participating teams
• Alps• Agentschap Wegen
en Verkeer • Rijkswaterstaat 1&2• Be-Mobile• Cohda Wireless• Dynniq NL• Dynniq BE• TNO• Transport for London
& HE• Université Reims
• UniversitéValenciennes UPHF
• V-Tron• Viveris• YoGoKo
Data analysis setup
• Analyses based on 3 main sources:
• Logging data from the app
• Logging data from the NL OBU
• Service-specific information from the teams• RWW• IVI• GLOSA• Other
• Geographical analysis, app usage, messages transmissions/receptions, …
Results from the BE/NL TESTFEST
Analysis of app logging
Logging app
App log data(time of day)
• Number of registered records (every 2 seconds):• Monday 25 / Tuesday 26 / Wednesday 27 March• 309 740 in total• 219 959 between 9h-20h (71%)
-
5 000
10 000
15 000
20 000
25 000
30 000
35 000
40 000
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
Nu
mb
er
of
regi
ste
red
re
cord
s
App log data (time between message start stop)
0:00:35
App log data(day of week)
939
485
764
-
100
200
300
400
500
600
700
800
900
1 000
Monday Tuesday Wednesday
App log data(per team)
- 50 100 150 200 250 300 350 400
Be-Mobile
V-Tron
Flanders G5
Alps
Rijkswaterstaat 2
Dynniq NL
TfL and HE
TNO
Dynniq IF2 (by RWS)
YoGoKo
Evaluation team
Rijkswaterstaat 1
UnivValenciennes UPHF
Flanders (Dynniq BE)
Université Reims
App logs (per message type)
0
200
400
600
800
1000
1200
1400
1600
IVI DENM THUMB SPAT
App log data(per message detail)
0 50 100 150 200 250 300 350
RWW
lane closed
lane free
speedLimitMax 70
clear lane to right
speedLimitMax 100
speedLimitMax 90
speedLimitMax 50
thumbs-up
accident
speedLimitEnd
clear lane to left
SPAT straight
any other warning
thumbs-down
SPAT general
speedLimitMax 110
SPAT left
Free Text
speedLimitMax 30
SPAT right
speedLimitMax 60
speedLimitMax 80
App log data(spatial scope, all teams)
App log data (spatial scope,(de)activation of messages)
• Example: RWW messages:
• Example: Lane closed/free:
App log data (spatial scope,(de)activation of messages)
• Example: GLOSA
App log data(thumbs up and down)
App log data QGis Cases
• RWW (case with dynamic maintenance vehicles)
• RWW (Classic info)
25/03 26/03 27/03
#Passes 6 5 3
Detected 6 3* 2*
Succesrate 100% 60% 66%
25/03 26/03 27/03
#Passes 6 5 3
Detected 6 3* 2*
Succesrate 100% 60% 66%
App log data QGis Cases
• IVS (lane closed)
• Warnings (accident)
25/03 26/03 27/03 Total
#Passes 7 8* 9 24
Detected 4 4 6 14
Succesrate 57% 50% 67% 58%
25/03 26/03 27/03 Total
#Passes 7 7 7 21
Detected 6 5 4 15
Succesrate 85% 71% 57% 71%
Results from the BE/NL TESTFEST
Analysis of NL OBU logging
Logging from OBU (& RSU)
Interoperability in communication, application and driver support
Intermediate results – still collecting and analysing data!
GLOSA event
Test statistics
GLOSA
Observations:
• Absolute waypoints in MAP (compliant but not used before)
• SPAT not sent (temporary technical issue)
• Static / adaptive TLC (realistic, - affects advices)
KPI G5 Hybrid
Use-case count 2 intersections N12
Participation (#OBU) 2 2
Success use-cases 2 2
# OBU events 103 133
Test statistics
In-Vehicle Signage
Observations:
• > 1700 events broadcasted (including real matrix sign changes)
• Same IVI sent by multiple stations (not uncompliant)
• Non-standard traffic codes used (configuration issue solved)
KPI G5 Hybrid
Use-case count E34 – E313 - A16
Participation (#OBU) 3 3
Success use-cases 3 3
# OBU events 256 284
Test statistics
Road Works Warnings - DENM
KPI G5 Hybrid
Use-case count E34 – E313 - A16
Participation (#OBU) 3 3
Success use-cases 3 3
# OBU events 289 383
Outlook
• Calculate communication delays • Alternative (member state) data provisioning services (IF2)
• ITS-G5 versus 4G
• Security• Certificate validation (authorities)
• Driver advice quality• Timing and location accuracy of warnings
• Updating of matrix signs
Team-specificresults and feedback
Team
Team
• RWW• Received warnings from moving ‘botsers’
• IVI• Signs on the highway received via ITS-G5 and
cellular• GLOSA
• Fixes made on OBU to suppress bicycle paths• Worked both via ITS-G5 and cellular
• Other• Cross border testing in Helmond (NL)• Cellular feed via Talking Traffic• ITS-G5 in Helmond with complex intersections
Team V-Tron & Rijkswaterstaat
ZOOF CITS app
•Right hybrid
•Left Wifi-P only
• RWW • IVI• Other
Team V-Tron & Rijkswaterstaat
ZOOF GLOSA app
•Hybrid
Team V-Tron & Rijkswaterstaat
• RWWWorked well on both channels (Cellular & WIFI-P)
• IVIWorked well on both channels (Cellular & WIFI-P)
• GLOSAWe made changes during the test days to fix things that doesn’t worked at the start of TestFest BE. We left the TestFest with a matching ZOOF GLOSA app!
• OtherNo further notes
Team TNO/NL PKI
• BE (25/03-26/03) – Working properly• Received unknown AA certificates from RSU’s via
CAM Certificate Request procedures
• NL (27/03) – Working mostly properly• Appeared to be an issue with the RSU’s, where
we did not receive a reply in all cases, but once the AA was cached, the requests where no longer necessary
Team TNO/NL GLOSA
• BE • Only one prediction -> end time of current
phase, no future phases are present in the SPAT
• Before 16:30• Perfectly stable countdown on time to phase
change• Perfectly stable speed advices can be calculated
• After 16:30• Unknown timing value from VRi was 36000, should
have been 36001 (was later fixed)• Countdown timers are unpredictable causing
calculation of speed advices to also be unpredictable
• NL• Very unpredictable phase timing, much
more than BE after 16:30• VRi is sending more predictions, although
because of the unpredictability of the phase timing this cannot be benefitted to it’s full potential
Team TNO/NL DENM&IVI
• BE• Latency for real-time IVI update ~5
seconds (rough guess)• Second day had some overlapping
traces(?) for IVI, causing our HMI to display ~8 lanes when there were 4
• 2x DENM for same location/event was sent out (was later fixed)
• NL• Messages for lanes from Rotterdam ->
Dordrecht were only sent out over G5 (was later fixed)
Team Alps
Team Alps
• RWW• No particular problems detected on the Antwerp/E313 site, but our UI is very simple.• Tested both ITS-G5 and Hybrid.
• IVI• Initially some problems with timing on Hybrid, but it was caused by our (test-only) IF2
client. No problems on ITS-G5.
• GLOSA• Our UI only displays TTG. In general we had very good match with real signals. • We had problems with MAPEMs over ITS-G5 on Helmond 502, 505 intersections,
reasons are still to be analyzed.• On system level, we had problems with ITS-G5 and cellular messaging overlap.
• Other:• Very good test event for us, we’ve found many problems with our implementation
(mostly GUI side)• The Helmond site with high traffic of complex MAPEMs was causing system overload in one of our low-power embedded targets, This problem would probably go unnoticed for some time if it wasn’t for this event.
Team France (Viveris)Pilot vehicle / French Ministry delegation
• Vehicle : Renault Megane• Validation of the PKI ; 2 runs on 27th March :
E313 + E34
• Interface overview: On-board HMI
Team France (Viveris)Pilot vehicle / French Ministry delegation
• Interface overview: DENM for RWW, received on E313, East direction
Team France (Viveris)Pilot vehicle / French Ministry delegation
• RWW• DENM received and displayed
• IVI• Events received but not displayed. Multilane not
supported yet• GLOSA
• Out of scope on this vehicle, for Renault current development
• Other: PKI• Interoperability works well with either DENM and
IVI messages• Use of FL RCA for signature verification• Use of FR AA1 for message signature
Team France (Viveris)Pilot vehicle / French Ministry delegation
Team Be-Mobile
Team YoGoKo
• Objective for this TESTFEST: Interoperability of Hybrid architecture (C-ITS-S + OBU)
IF2 IF2
Quad-tree tiling over MQTT
Direct reception
over ITS-G5
YoGoKo Central ITS-Station
Built for scalability and interoperability
Team YoGoKo
• Messages tested• Spat/Map (GLOSA)• DENM (RWW, RHS)
• Modes tested• Hybrid mode: both ITS-G5 and cellular connected• Cellular only mode: ITS-G5 antenna disconnected
• Results• Good behavior experienced• Supports a high load of messages• Further analysis to be done
Architecture already deployed in Norway(E8Borealis project) with Aventi
Team UK
Team UK
• IVS• All IVS events displayed correctly on the HMI on
the E313 & E34• All IVS events displayed correctly on HMI in
Dordrecht
• RWW• RWW events received and displayed on the HMI
on test runs through Dordrecht• Received DENM messages whilst on the E313 &
E34. Further investigation regarding their display
Team UK
• GLOSA• All GLOSA events displayed correctly and
accurately on the HMI in Schilde• Received GLOSA events in Helmond were
correct and accurately displayed on the HMI• GLOSA in Helmond was intermittent. Possible
OBU cellular issue
• Other• IF2 provided services in all member states• HMI unable to display ITS-G5 messages in
Flanders. Possible issue during certificate preloading
Conclusions
TESTFEST goals
• TESTFEST 1 (NL): G5
• TESTFEST 2 (FR): G5 + security
• TESTFEST 3 (UK): G4/4G hybrid
• TESTFEST 4 (all): cross-border interoperability
Conclusions
• Overall• The events that were transmitted by the Road Side Units were received:
speed limits, road works, lane closures/opening, …
• GLOSA worked• Participants received the status of ahead traffic lights (red/green)
including speed advice and timing
• ITS-G5 and cellular data are consistent• The information received trough both channels is consistent and
generally received synchronously
• PKI• Generally worked well
Conclusions
Cross-border interoperability achieved!
• Problems that were registrated were mainly programming issues and not directly related to interoperability
• More detailed results on interoperability will be processed
• Thanks to all the teams for this successful TESTFEST!
Thank you!