Ols2008 Luis Claudio Goncalves Application Testing
-
Upload
denisjolson -
Category
Documents
-
view
216 -
download
0
Transcript of Ols2008 Luis Claudio Goncalves Application Testing
-
8/10/2019 Ols2008 Luis Claudio Goncalves Application Testing
1/37
Application Testing under
Realtime Linux
Luis Claudio R. GonalvesRed Hat Realtime Team
Software Engineer
-
8/10/2019 Ols2008 Luis Claudio Goncalves Application Testing
2/37
Agenda
* Realtime Basics
* Linu and t!e "REE#"T$RT %atc!
* &'out t!e Tests
* Loo(ing for 'ad %rogramming %ractices
* Bad "riorit) &ssignment
* Resource &llocation
* C!anging &%%lication Be!avior
* Com%aring &%%les to &%%les
Conclusion
-
8/10/2019 Ols2008 Luis Claudio Goncalves Application Testing
3/37
Realtime Basics
-
8/10/2019 Ols2008 Luis Claudio Goncalves Application Testing
4/37
Realtime Basics
* eterminism a'ilit) to sc!edule t!e !ig!est%riorit) tas(s in a consistent and %redicta'le wa).
* Latenc) time 'etween a given event and t!edesired effect.
-
8/10/2019 Ols2008 Luis Claudio Goncalves Application Testing
5/37
'Almost' is not enough...
* +!en determinism matters...
* ,av)- inancial Services- ederal sector-Telco/,etwor(
* inancial Services01 first to answer !as 'etter c!ances on t!e 'ids1 legal re2uirements for consistenc) in o%erations
* Telco / ,etwor(1 networ( %ac(et management1 3oS
-
8/10/2019 Ols2008 Luis Claudio Goncalves Application Testing
6/37
Latency...
-
8/10/2019 Ols2008 Luis Claudio Goncalves Application Testing
7/37
Latency in detail...
-
8/10/2019 Ols2008 Luis Claudio Goncalves Application Testing
8/37
Sources of latency
* &%%lication "riorit)1 one a%%lication 'loc(s 4or %reem%ts5 anot!er1 !olds a contended resource 4loc(5
* T!e 6ernel1 w!en t!e (ernel runs a%%s sto%1 t!e longer t!e (ernel runs t!e longer a%%lications wait1 determinism gets limited ') t!e longest code%at!
Hig! "riorit)&%%lication
7nterru%t
&%%lication continues
6ernel0 interru%t !andler 8
sc!eduler
-
8/10/2019 Ols2008 Luis Claudio Goncalves Application Testing
9/37
Linux + PREEMPT_RT
-
8/10/2019 Ols2008 Luis Claudio Goncalves Application Testing
10/37
PREEMPT_RT patch
* #et!odical im%lementation- focusing u%stream ado%tion
* Started ') t!e 'uilding 'loc(s t!at would ena'le t!eim%lementation
* Com%lete RT- not 9ust a :#/sc!eduler !ac(
* #ature features- t!at 'enefits 'ot! RT and u%stream-!ave 'een merged
* Communit) 8 sand'o 4linu1rt1users5
-
8/10/2019 Ols2008 Luis Claudio Goncalves Application Testing
11/37
Building Blocks
* T!readed 7R3s
* T!readed Softir2s
* Re%laced sema%!ores and s%inloc(s ') 4rt5mutees
* Slee%ing s%inloc(s
* "riorit) 7n!eritance
-
8/10/2019 Ols2008 Luis Claudio Goncalves Application Testing
12/37
Strategies
* #easure and 7dentif) t!e longest code %at! in t!e (ernel* En!ance algorit!ms to ena'le more concurrenc)* S!orten t!e time t!e (ernel runs as non1%reem%ti'le* efer %rocessing to lower %riorit) (ernel t!reads
Hig!"riorit)&%%lication
7nterru%t
&%%licationcontinuesand finis!es
7nterru%t Handlersc!edules lower%rio t!read to
!andle t!einterru%t
7nterru%t Handler0lower %rio t!readcom%letes data
transfer
,et !ig!%riorit)&%%lication
-
8/10/2019 Ols2008 Luis Claudio Goncalves Application Testing
13/37
About the Tests
-
8/10/2019 Ols2008 Luis Claudio Goncalves Application Testing
14/37
About the tests
* :alidation of t!e 6ernel and Customer &%%s
* ;7t runs slower on RTt s!ow )ou or c!ange t!e code...=
* +!ere t!e %ro'lem lies? 4(ernel- a%%- com%arison5
-
8/10/2019 Ols2008 Luis Claudio Goncalves Application Testing
15/37
Bad Practices
-
8/10/2019 Ols2008 Luis Claudio Goncalves Application Testing
16/37
Bad Coding Practices
* Bad coding %ractices 4focus on Realtime51 some %ractices are 'ad onl) in t!e contet of "REE#"T$RT1 a%%s %orted from ot!er/older @S1 tunings t!at no longer ma(e sense
* How to s%ot 'ad %rogramming %ractices?1 s)stem calls t!at s!ould 'e avoided1 'ad %arameter assignment
* Reading t!e source code
* S)stemta% scri%ts
* Strace
* Eam%le0 avoid sc!ed$)ield45
-
8/10/2019 Ols2008 Luis Claudio Goncalves Application Testing
17/37
Example
* How to s%ot a given s)scall usage
A cat test$)ield.st%
%ro'e s)scall.sc!ed$)ield
%rintf4 Ds4Dd0Dd5 Called sc!ed$)ield45n- eecname45- %id45-tid455F
A sta% 1v test$)ield.st%rs)slogd4IJ0IK5 Called sc!ed$)ield45
rs)slogd4IJ0JMIN5 Called sc!ed$)ield45rs)slogd4IJ0JMIN5 Called sc!ed$)ield45
-
8/10/2019 Ols2008 Luis Claudio Goncalves Application Testing
18/37
Priority Assignment
-
8/10/2019 Ols2008 Luis Claudio Goncalves Application Testing
19/37
The Short Version...
* "rioritiOe w!at is im%ortant1 c!rt is )our friend1 sc!ed$setsc!eduler451 alwa)s c!ec( >man M% PfunctionQ>1 avoid SCHE$7@ %riorit)
-
8/10/2019 Ols2008 Luis Claudio Goncalves Application Testing
20/37
Priorities
* 7s t!e &%% suffering non1voluntar) %reem%tion?1 Tuna1 c!ec( /%roc/P%idQ/status 1 4nonvoluntar)$ctt$switc!es51 use t!e function getrusage451 create a s)stemta% scri%t to verif) t!is 4and ot!er5 information1 c!ec( t!e %riorit) of t!e a%%lication and its t!reads
rootvoid UVA %s 1emo %id-tid-%olic)-%ri-rt%rio-cmd
...MWJ 1 1 1 1 /usr/li'N/.../firefo1'in 1 MWJ TS 1 1
1 MWNW TS 1 1 1 MWN TS 1 1 1 MWKK TS 1 1 1 MWK TS 1 1 1 MWK TS 1 1 1 MW TS 1 1 1 WWKN TS 1 1
...
-
8/10/2019 Ols2008 Luis Claudio Goncalves Application Testing
21/37
Priorities
*Eam%le s)stemta% scri%t
A if )ou !ave %ro'lems !oo(ing on eit$mm-A use %rofile$tas($eit instead.
%ro'e (ernel.function4eit$mm5 %rintf4Ds4Dd0Dd5 stats0n- eecname45- %id45- tid455
%rintf4t"ea(RSS0 Dd6B t"ea(:#0 Dd6Bn- Xtas(1Qmm1Q!iwater$rss*N- Xtas(1Qmm1Q!iwater$vm*N5
%rintf4tS)stem Time0 Ddms tYser time0 Ddmsn- Xtas(1Qstime- Xtas(1Qutime5
%rintf4t:oluntar) Contet Switc!es0 Dd t7nvoluntar)0 Ddn- Xtas(1Qnvcsw- Xtas(1Qnivcsw5
%rintf4t#inor "age aults0 Dd t #a9or "age aults0 Ddn- Xtas(1Qmin$flt- Xtas(1Qma9$flt5
F
-
8/10/2019 Ols2008 Luis Claudio Goncalves Application Testing
22/37
Priorities
* Results0
ss!d4MIW0MIW5 stats0 "ea(RSS0 KWN6B "ea(:#0 NN6B S)stem Time0 ms Yser time0 Mms
:oluntar) Contet Switc!es0 MJ 7nvoluntar)0 I #inor "age aults0 MNI #a9or "age aults0 I
id4MIJ0MIJ5 stats0 "ea(RSS0 K6B "ea(:#0 N6B S)stem Time0 Jms Yser time0 Ims :oluntar) Contet Switc!es0 M 7nvoluntar)0 I #inor "age aults0 JJ #a9or "age aults0 I
-
8/10/2019 Ols2008 Luis Claudio Goncalves Application Testing
23/37
Priorities
* ,otes a'out t!e s)stemta% scri%t
1 gat!ers information from tas($struct
1 eit45 1Q do$eit45
1 eit$mm45
* #ore o%tions are coming
-
8/10/2019 Ols2008 Luis Claudio Goncalves Application Testing
24/37
Priorities
* "rioriOe (ernel t!reads 47R3s- Softir2s- ...57.e.0 &%%lications de%ending on networ( and not using dis( 7/@
N %osi$c%u$timerV
K softir21!ig!/IV softir21timer/IV softir21net1t/V W softir21net1r/V softir21'loc(/IV I softir21tas(letV softir21sc!ed/IV J softir21!rtimerV
WW (ac%idV I 7R31V W (sus%end$us'dV JN aio/IV MKI 7R31WV MWN 7R31NV NI 7R31V
NI %ccarddV
-
8/10/2019 Ols2008 Luis Claudio Goncalves Application Testing
25/37
Resource Allocation
-
8/10/2019 Ols2008 Luis Claudio Goncalves Application Testing
26/37
Good Habits
* &llocate Resources in advance1 minor / ma9or %age faults1 memor) 4mloc(51 create t!read %ool in advance
* Bind %rocesses to C"Ys and 7solate C"Ys1 tas(set1 isolc%us in t!e (ernel command line1 Tuna
-
8/10/2019 Ols2008 Luis Claudio Goncalves Application Testing
27/37
Resource Allocation
* #emor) &llocation1 minor %age faults1 ma9or %age faults1 mloc(45 and friends1 don>t forget touc!ing memor) 'efore mloc(>ing1 can 'e c!ec(ed via /%roc- getrusage45 or s)stemta% scri%ts
* T!read creation1 can ta(e more t!an IIus1 can suffer %age faults
* T!reads1 some s)stem calls can generate %age faults 4i.e. fo%en51 select45 !as a ms resolution
-
8/10/2019 Ols2008 Luis Claudio Goncalves Application Testing
28/37
CPU Affinity
* &llocate %rocesses to C"Y1 i.e. avoid running sender and receiver on t!e same C"Y
1 sometimes- tr) running sender and receiver on t!e same C"Y1 tas(set- eidus1 some (ernel t!reads can>t 'e moved1 tr) Tuna
* C"Y 7solation1 'oot %arameter- isolc%us1 removes all 4%ossi'le5 %rocesses from t!e C"Y
-
8/10/2019 Ols2008 Luis Claudio Goncalves Application Testing
29/37
Application Behavior
-
8/10/2019 Ols2008 Luis Claudio Goncalves Application Testing
30/37
Changing App Behavior
* ,o access to source code1 +!en s)stem tuning is not enoug!...1 +!at if t!is or t!at configuration was in use?1 !ac(ing t!e (ernel
1 Ysing s)temta%1 Li'rar) %reload tric(s1 6ernel !ac(s
-
8/10/2019 Ols2008 Luis Claudio Goncalves Application Testing
31/37
Changing App Behavior
* Li'autocor(1 "erformance is (ing1 ,agle &lgorit!m1 TC"$,@EL&Z1 TC"$C@R6
A L$"REL@&[li'autocor(.so ./customer$a%%
* Li'localiOe
-
8/10/2019 Ols2008 Luis Claudio Goncalves Application Testing
32/37
Comparing Results
-
8/10/2019 Ols2008 Luis Claudio Goncalves Application Testing
33/37
Apples to Apples
* &7T1 Have )ou attended &rnaldo>s %resentation?1 +!ic! (no's were set?
1 id t!e tests run on t!e same environment?1 +!at was different?1 How to re%roduce a given test on a different s)stem?1 How to com%are all t!e tests t!at !ave a given set of features?
-
8/10/2019 Ols2008 Luis Claudio Goncalves Application Testing
34/37
Conclusion
-
8/10/2019 Ols2008 Luis Claudio Goncalves Application Testing
35/37
Conclusion
* Some features of "REE#"T$RT ma(es it uni2ue wit! regardsto s)stem tuning
* Several test cases solved wit! minimal s)stem and a%%licationtuning
* T!ere are cases w!ere a%% 'e!avior (ills t!e 'enefits oftunings
* Clues for finger%ointing
-
8/10/2019 Ols2008 Luis Claudio Goncalves Application Testing
36/37
Questions...?
* T!e answer is NJ
-
8/10/2019 Ols2008 Luis Claudio Goncalves Application Testing
37/37
References
L7B&YT@C@R6!tt%0//git.(ernel.org/%[linu/(ernel/git/acme/li'autocor(.git\a['lo'$%lain\f[tc%$nodela).tt
Real1Time +i(i "age!tt%0//rt.wi(i.(ernel.org
Tec!ni2ues t!at can !ave its 'e!avior c!anged w!en t!e (ernel is re%laced!tt%0//oo%s.g!ost%rotocols.net0W/acme/un'e!aved.tt
,etta%s!tt%0//oo%s.g!ost%rotocols.net0W/acme/netta%s.tar.'OJ
http://rt.wiki.kernel.org/http://rt.wiki.kernel.org/