Research in Soft Real-Time and Virtualized Applications on Linux

Real-Time Systems LaboratoryReal-Time Systems Laboratory

RETIS LabRETIS Lab

Research inResearch in

Soft Real-Time and Virtualized ApplicationsSoft Real-Time and Virtualized Applicationson Linuxon Linux

Tommaso CucinottaTommaso Cucinotta

Real-Time Systems Lab (RETIS)Real-Time Systems Lab (RETIS)

Center for Excellence in Information, Communication and Perception Engineering Center for Excellence in Information, Communication and Perception Engineering (CEIICP)(CEIICP)

Scuola Superiore Sant'Anna, Pisa (Italy)Scuola Superiore Sant'Anna, Pisa (Italy)

© 2007 Scuola Superiore Sant’Anna Tommaso Cucinotta – Real Time Systems Laboratory (ReTiS) – Scuola Superiore Sant'AnnaTommaso Cucinotta – Real Time Systems Laboratory (ReTiS) – Scuola Superiore Sant'Anna

Introduction and motivationsIntroduction and motivations


MotivationsMotivations

General-Purpose Operating SystemsGeneral-Purpose Operating Systems

Very effective for storing & managing multimedia contents

Designed for

• average-case performance

• serving applications on a best-effort basis

They are not the best candidate for serving real-time applications with tight timing constraints

• nor for real-time multimedia


MotivationsMotivations

Overcoming limitations of a GPOS for multimediaOvercoming limitations of a GPOS for multimedia

Large buffers used to compensate unpredictability

• ==> poor real-time interactivity and no low-latency multimedia

One-application one-system paradigm

• For example, for low-latency real-time audio processing (jack),

gaming, CD/DVD burning, plant control, etc...

POSIX real-time extensions

• Priority-based, no temporal isolation

• Not appropriate for deploying the multitude of (soft) real-time

applications populating the systems of tomorrow


Real-time theoryReal-time theory

Reservation-based scheduling: (QReservation-based scheduling: (Qii, P, P

ii))

“Qi time units guaranteed on a CPU every P

i time units”

Independently of how others behave(temporal isolation)

t

(5, 9)

(2, 6)

~88.9%OverallLoad

t


Temporal IsolationTemporal Isolation

Is needed despite blocks/unblocksIs needed despite blocks/unblocksNot only EDF scheduling

t

(5, 9)

(2, 6)

block unblock

deadline-miss

t



Is needed despite blocks/unblocksIs needed despite blocks/unblocksNot only EDF scheduling

See CBS “unblock rule”See CBS “unblock rule”

t

(5, 9)

(2, 6)

block unblock

t



Enforcement of temporal isolationEnforcement of temporal isolationNot only EDF scheduling

t

(5, 9)

(2, 6)

t

deadline missed by good job

bad job



Enforcement of temporal isolationEnforcement of temporal isolationOnce budget exhausted, delay to next activation period

t

(5, 9)

(2, 6)

t

Deadline missed by bad jobbad job


Linux and T.I.Linux and T.I.

Need for T.I. evident in mainline LinuxNeed for T.I. evident in mainline LinuxA (buggy) real-time task can starve the entire OS

Real-time throttling prevents this to happen

Real-Time ThrottlingReal-Time ThrottlingDifferent design principles

• Priority-based scheduling

• Constraints to “no more than Qi every system-wide P”

• Behaves like “Deferrable Servers”


Making Linux a better placeMaking Linux a better place

(for multimedia and soft real-time applications)


Real-Time Scheduling in Real-Time Scheduling in LinuxLinux

LinuxLinuxNot a Hard Real-Time OS

Monolithic structure• Device drivers may adversely affect responsiveness

Advances in Linux temporal behavior (responsiv.)Advances in Linux temporal behavior (responsiv.)Preemptability of kernel-space code

High-resolution timers

Increasing use of RCU primitives

Nearly complete support for POSIX real-time extensions• Sporadic Server is missing (!)

IRQ handlers as kernel threads (preempt-rt branch)


Real-Time Scheduling in Real-Time Scheduling in LinuxLinux

Research on Linux for Real-Time SystemsResearch on Linux for Real-Time Systemsat RETISat RETISApplying techniques from Real-Time Systems to a GPOS Improving responsiveness and stability of performance

for time-sensitive applications Investigating effectiveness of Adaptive ReservationsSynchronization protocols for real-time multimediaStabilising performance of virtualized applications Isolating computing and I/O traffic for Virtual MachinesEffective exploitation of multi-cores in real-time systemsProgrammability of multi-threaded, distributed, real-time

applications (API)


Real-Time SchedulersReal-Time Schedulersfor Linuxfor Linux

Resource reservation scheduling in LinuxResource reservation scheduling in LinuxAdaptive Quality of Service Architecture

• Single-processor embedded systems

• Multi-threaded applications

Hybrid Deadline/Priority Scheduler• Multi-processor systems

• Multi-threaded virtualizedapplications

Partitioned/Global EDF SchedulerSCHED_DEADLINE

• Multi-processor systems

• Single-threaded controlapplications


Adaptive Quality of Service Adaptive Quality of Service Architecture for LinuxArchitecture for Linux


Framework for Real-time Embedded Framework for Real-time Embedded Systems based on COntRactsSystems based on COntRacts

Soft and hard real-timeSoft and hard real-time

Support for CPU, disk, networkSupport for CPU, disk, network

Distributed real-time systemsDistributed real-time systems

Portability across RTOSsPortability across RTOSs

Adaptive real-time systemsAdaptive real-time systems

Application-level QoS controlApplication-level QoS control

QoS power-aware optimizationQoS power-aware optimization

Atomic negotiationsAtomic negotiations

http://www.frescor.orghttp://www.frescor.org

http://www.frescor.org/


Experimental resultsExperimental results(Pentium 4 @ 2GHz, Linux 2.6.29.1 + AQuoSA)(Pentium 4 @ 2GHz, Linux 2.6.29.1 + AQuoSA)

Response-time CDF for the Response-time CDF for the real-time task with the real-time task with the shortest period under light shortest period under light load (48%)load (48%)

Response-time CDF for the Response-time CDF for the real-time task with the real-time task with the shortest period under heavy shortest period under heavy load (84%)load (84%)

dea

dli

ne

dead

lin

e


Experimental resultsExperimental results(Pentium 4 @ 2GHz, Linux 2.6.29.1 + AQuoSA)(Pentium 4 @ 2GHz, Linux 2.6.29.1 + AQuoSA)

Average normalized tardiness Average normalized tardiness for the real-time task with the for the real-time task with the shortest period, varying the shortest period, varying the number of real-time tasks in number of real-time tasks in the system.the system.

Maximum and 95th percentile Maximum and 95th percentile of the normalized tardiness of the normalized tardiness for the real-time task with the for the real-time task with the shortest period, varying the shortest period, varying the number of real-time tasks in number of real-time tasks in the system.the system.

deadline

deadline


SynchronizationSynchronization

Bandwidth Inheritance (BWI / M-BWI)Bandwidth Inheritance (BWI / M-BWI)Deals with Priority Inversion in EDF scheduling

Deadline inherited from lock owner


Adaptive ReservationsAdaptive Reservations


Need for adaptivityNeed for adaptivity

Computation times for decoding MPEG Computation times for decoding MPEG framesframes

Scheduling error with an over-Scheduling error with an over-allocation of 30% over the averageallocation of 30% over the average

Scheduling error with allocation tuned Scheduling error with allocation tuned on maximumon maximum

deadline

deadline



Feedback-based schedulingFeedback-based schedulingSense: tracking of workload fluctuations

Compute:• Prediction/estimation of workload for next period(s)

• Compensation for possible current delays

Actuate: adapt the scheduling parameters (Qi, P

i)



Workload predictionWorkload predictionMoving average

Percentile estimation over moving window

FIR decorrelation + error estimation

Budget controllerBudget controllerTarget region around

deadline

Stochastic approaches• Probability of deadline-miss

• Optimum error/bandwidth trade-offs


Experimental resultsExperimental results

PMF of scheduling error with various budget PMF of scheduling error with various budget control strategiescontrol strategies


Automatic identification of scheduling parametersAutomatic identification of scheduling parameters


Automatic identificationAutomatic identificationof scheduling parametersof scheduling parameters

Recent developments in GPOS CPU schedulingRecent developments in GPOS CPU schedulingVarious APIs for accessing the enhanced functionality

• For example, the FRSH API

• For example, the AquoSA API

They require modifications of the applications

• at the source-code level

Can we provide real-time guarantees to unmodified applications ?

• For example, for legacy multimedia applications ?

• Or, simply because of no time to modify the applications


ObjectivesObjectives

We actually canWe actually can

allow (legacy) real-time periodic applications

to benefit of real-time scheduling facilities increasingly

available on a GPOS

without any change in the application source-code


LFS++LFS++

MonitorMonitor

ApplicationApplication

SchedulingParametersSchedulingParameters

Real-TimeSchedulerReal-TimeScheduler

Kernel


Proposed approachProposed approach

Legacy Feedback Scheduling (LFS++)Legacy Feedback Scheduling (LFS++)

An appropriate tracing mechanism observes the

application, inferring main parameters affecting a (periodic)

multimedia application temporal behaviour:

• job execution time

• period

Scheduling guarantees are automatically provisioned by

the OS, according to proper scheduling parameters• Based on sound arguments from real-time theory



Comprehensive viewComprehensive viewApplication tracing

Period estimation (events analysis)

On-line WCET estimation

Automagic provisioning of scheduling guarantees



Reservation-based scheduling: (QReservation-based scheduling: (Qii, P, P

ii))

“Qi time units guaranteed on CPU every P

i time units”

Real-time throttling on Linux is differentReal-time throttling on Linux is differentOnly constraint to “no more than Q

i every system-wide P”

t

(5, 9)

(2, 6)



What are the reservation What are the reservation parameters needed for parameters needed for correctly scheduling a real-correctly scheduling a real-time periodic task with time periodic task with assigned WCET and period ?assigned WCET and period ?

The The minimumminimum reservation reservation utilization is achievedutilization is achieved

when the reservation period equals the task period

• or any integer sub-multiple


Core problemCore problem

How to detect the application period ?How to detect the application period ?

Of a legacy real-time application (no source-code availability, no modifications)


Tracing the application behaviour at run-time

For the purpose of identifying “periodicity patterns”


Period detectionPeriod detection

The tracer produces a sequence of The tracer produces a sequence of time-stampstime-stamps

Time-stamps used to compute a Time-stamps used to compute a Fourier-transformFourier-transform

A A heuristicheuristic catches the catches the first harmonicfirst harmonic


Budget identificationBudget identification

““Feedback-based scheduling” budget control loopFeedback-based scheduling” budget control loop



Set-upSet-upLinux 2.6.29, with an implementation of the CBS schedulerFeedback-scheduling by means of AQuoSA

mplayer• modified to monitor the Inter-Frame Time (IFT) and

Application tracing by using qtrace (kernel-level)

Validation metricsValidation metrics Inter-Frame Time (for mplayer)

A/V desynchronisation (for mplayer)

Response-time (for synthetic application)

Allocated bandwidth on the Real-Time scheduler



Using the correct reservation period is betterUsing the correct reservation period is better


Benefits for the applicationBenefits for the application(LFS++ improves over Linux)(LFS++ improves over Linux)

A/V desynchronisation in A/V desynchronisation in mplayermplayer while starting while starting the Eclipse IDEthe Eclipse IDE90% reduction of the peak A/V desynchronisation


Benefits for the applicationBenefits for the application(LFS++ improves over Linux)(LFS++ improves over Linux)

CDF of response-times with a 80% loaded systemCDF of response-times with a 80% loaded systemUsing a synthetic real-time periodic application load


Multiple LFS++ instancesMultiple LFS++ instances

Frequency misdetection degreeFrequency misdetection degree with other LFS++ with other LFS++ controllers already activecontrollers already active

New application runs without guarantees while traced


Precision of frequency Precision of frequency detectordetector

PMF of detectedPMF of detectedfrequency whenfrequency whenobservation-timeobservation-timegoesgoesFrom 0.2 ms

To 2 s


Cost of tracingCost of tracing

Tracing overheadTracing overhead

When using strace: +5,51%

When using qostrace: +2,69% (previous paper)

When usinig qtrace: +0.63% (kernel-level tracer)

88.57% overhead reduction with the custom tracer


Cost of Fourier TransformCost of Fourier Transform

Fourier Transform OverheadFourier Transform Overhead 1ms to 25ms, depending on parameters (and accuracy)

Linear in the observation-time and f parameter (as expected)

Tommaso Cucinotta – ReTiS Lab – Scuola Superiore Sant'Anna – Pisa – Italy 45

Research Activities at ReTiSResearch Activities at ReTiS

Tommaso Cucinotta – Tommaso Cucinotta – ReTiS – CEIICP – Scuola Superiore Sant'AnnaReTiS – CEIICP – Scuola Superiore Sant'Anna

Real-Time Service-Oriented ArchitecturesReal-Time Service-Oriented Architectures

Virtualized Real-Time MultimediaVirtualized Real-Time Multimedia


IntroductionIntroduction

Virtualization & real-time increasingly interestingVirtualization & real-time increasingly interesting

Wide availability of broadband connections ==>shift in computing paradigms towards distributed computing

Not only remote storage and batch processing

But also remote processing for interactive applications

Examples

Virtual Reality with heavyweight physics simulations

Distributed editing of HD video (film post-production)


IntroductionIntroduction

Service-Oriented ArchitecturesService-Oriented Architectures

Promising approach to distributed computing

Taking advantage from virtualization techniques:

Location independence

Security

Fault-Tolerance

Distributed Interactive/Real-Time Applications may benefit from SOA design


ProblemsProblems

Issues in deploying RT SW Components in VMsIssues in deploying RT SW Components in VMs

Execution time

Virtualisation overhead: intensive I/O on virtualised peripherals

Benchmarking

A VM may be deployed on different HW (SOA scenario)

– How to achieve predictable performance ?

VMs may be deployed on General-Purpose HW (with cache)

– How to account for HW-level interferences ?


ProblemsProblems

Issues in deploying RT SW Components in VMsIssues in deploying RT SW Components in VMs Scheduling and timing

Host scheduling algorithm and VM layer impacts on the vision of time by guest OSes– Time granularity (for measuring time and setting timers)– Scheduling capabilities– Non-uniform progress-rate of applications

SMP-enabled guests– Spin-lock primitives assume release of locks within very short time-frames

» What happens if the lock-owner VM is descheduled ? Proper management of shared resources

– What MP resource-sharing protocol is appropriate ?» Proper management of priority inversion» Reduced overheads (limited number of preemptions)» Run-time schedulability analysis and admission control


ApproachApproach

Use of RR techniques forUse of RR techniques forscheduling multiple VMsscheduling multiple VMs Temporal isolation among

VMs Deadline guarantees to tasks

within VMs hierarchical scheduling

Experimental set-upExperimental set-up VMs scheduled by a

modified Linux kernel implementing resource reservations

Hard Reservation


ApproachApproach

Validation on modified Linux kernelValidation on modified Linux kernelwith SMP Resource Reservation Schedulerwith SMP Resource Reservation Scheduler


Experimental ResultsExperimental Results

SOA ScenarioSOA Scenario Response-times of Apache2 Web Server

From unpredictable and highly variable To predictable and very stable (low fluctuations)

– At the cost of an increased minimum response-time


Security for EmbeddedSecurity for Embedded(and Real-Time) Systems(and Real-Time) Systems

Real-Time and SecurityReal-Time and Security Temporal encapsulation as a security feature Secure access to Real-Time scheduling services by

unprivileged applications Optimum trade-offs between:

Security, Quality of Service and Power Consumption


Relevant PublicationsRelevant Publications(from year 2009)(from year 2009)

JournalsJournals A robust mechanism for adaptive scheduling of multimedia applications, Tommaso Cucinotta, Luca

Abeni, Luigi Palopoli, Giuseppe Lipari, accepted on ACM Transactions on Embedded Computing Systems

QoS Control for Pipelines of Tasks using Multiple Resources, Tommaso Cucinotta, Luigi Palopoli, IEEE Transactions on Computers, Vol. 53, No. 3, pp. 416--430, March 2010, IEEE Computer Society Digital Library

A Real-time Service-Oriented Architecture for Industrial Automation, Tommaso Cucinotta, Antonio Mancina, Gaetano F. Anastasi, Giuseppe Lipari, Leonardo Mangeruca, Roberto Checcozzo, Fulvio Rusinà, to appear on IEEE Transactions on Industrial Informatics, Aug 2009

AQuoSA - Adaptive Quality of Service Architecture, Luigi Palopoli, Tommaso Cucinotta, Luca Marzario, Giuseppe Lipari, Software: Practice and Experience, Vol. 39, No. 1, January 2009


Relevant PublicationsRelevant Publications(from year 2009)(from year 2009)

Conferences/WorkshopsConferences/Workshops The Multiprocessor BandWidth Inheritance Protocol, Dario Faggioli, Giuseppe Lipari, Tommaso

Cucinotta, to appear at the 22nd Euromicro Conference on Real-Time Systems (ECRTS 2010), Bruxelles, July 2010

Self-tuning Schedulers for Legacy Real-Time Applications, Tommaso Cucinotta, Fabio Checconi, Luca Abeni, Luigi Palopoli, European Conference on Computer Systems (EuroSys 2010), Paris, April 2010

The Wizard of OS: a Heartbeat for Legacy Multimedia Applications, Tommaso Cucinotta, Luca Abeni, Luigi Palopoli, Fabio Checconi, 7th IEEE Workshop on Embedded Systems for Real-Time Multimedia, Grenoble (ESTIMedia 2009), Grenoble, October 2009

Multi-level feedback control for Quality of Service Management, Tommaso Cucinotta, Giuseppe Lipari, Luigi Palopoli, Luca Abeni, Rodrigo Santos, 14th IEEE International Conference on Emerging Technologies and Factory Automation (ETFA 2009), Mallorca, Spain, September 2009

Exception-Based Management of Timing Constraints Violations for Soft Real-Time Applications, Tommaso Cucinotta, Dario Faggioli, Alessandro Evangelista, 5th International Workshop on Operating Systems Platforms for Embedded Real-Time Applications (OSPERT 2009), Dublin, Ireland, June 2009

Hierarchical Multiprocessor CPU Reservations for the Linux Kernel, Fabio Checconi, Tommaso Cucinotta, Dario Faggioli, Giuseppe Lipari, 5th International Workshop on Operating Systems Platforms for Embedded Real-Time Applications (OSPERT 2009), Dublin, Ireland, June 2009

Real-time guarantees in flexible advance reservations, Kleopatra Kostanteli, Dimosthenis Kyriazis, Theodora Varvarigou, Tommaso Cucinotta, Gaetano Anastasi, 2nd IEEE International Workshop on Real-Time Service-Oriented Architecture and Applications (RTSOAA 2009), Seattle, Washington, July 2009

Respecting temporal constraints in virtualised services, Tommaso Cucinotta, Gaetano Anastasi, Luca Abeni, in Proceedings of the 2nd IEEE International Workshop on Real-Time Service-Oriented Architecture and Applications (RTSOAA 2009), Seattle, Washington, July 2009


S(o)OSS(o)OS

Service-Oriented Operating Systems – S(o)OSService-Oriented Operating Systems – S(o)OS Focus on large-scale multi-cores of tomorrow Heterogeneous multi-core systems What OS model is appropriate ? What kind of applications are expected ? How to deploy heterogeneous workloads ? How to program in such systems ? How to mix H.P. with G.P. Computing ? etc.


Thanks for your attentionThanks for your attention

Questions ?Questions ?

http://retis.sssup.it/people/tommasohttp://retis.sssup.it/people/tommaso

http://retis.sssup.it/people/tommaso

Research in Soft Real-Time and Virtualized Applications on Linux

Technology

Transcript of Research in Soft Real-Time and Virtualized Applications on Linux