SMP, 64bit Unix and Kernel Compilation

SMP, 64bit Unix and Kernel Compilation

Guntis BarzdinsGirts Folkmanis

K palaist MPI[guntisb@zars mpi]$ ls -ltotal 392-rw-rw-r-- 1 guntisb guntisb 122 Apr 28 07:08 Makefile-rw-rw-r-- 1 guntisb guntisb 13 May 17 14:33 mfile-rw-rw-r-- 1 guntisb guntisb 344 May 12 09:28 mpi.jdl-rw-rw-r-- 1 guntisb guntisb 2508 Apr 28 07:08 mpi.sh-rwxrwxr-x 1 guntisb guntisb 331899 May 17 14:48 passtonext-rw-rw-r-- 1 guntisb guntisb 3408 Apr 28 07:08 passtonext.c-rw-rw-r-- 1 guntisb guntisb 2132 May 17 14:48 passtonext.o[guntisb@zars mpi]$ more mfilelocalhost:4

[guntisb@zars mpi]$[guntisb@zars mpi]$ makempicc passtonext.c -o passtonext -lmpich -lm[guntisb@zars mpi]$ mpirun -np 2 -machinefile mfile passtonextguntisb@localhost's password:Nodename=zars.latnet.lv Rank=0 Size=2INFO: zars.latnet.lv (0 of 2) sent 73 value to 1 of 2INFO: zars.latnet.lv (0 of 2) received 74 value from 1 of 2Nodename=zars.latnet.lv Rank=1 Size=2INFO: zars.latnet.lv (1 of 2) received 73 value from 0 of 2INFO: zars.latnet.lv (1 of 2) sent 73+1=74 value to 0 of 2[guntisb@zars mpi]$ Paldies Jnim Tragheimam!!

3b mjas darba variantsJa ir izpildits 3a mjas darbs, tad 3b da var uzrakstt un notestt pats savu MPI programmu, kas rins ar vismaz 4 procesiem. MPI programmu drkst nedarbint Grid!

Message-passing performance comparisonSMP MPI performanceGigabitEthernet MPI performance

Chart1

0.2814757.3713573.0639870.176333

0.6139914.5477865.9856520.392785

0.92692821.1595078.9933650.499961

1.21414428.79959411.8925040.786296

1.83384643.72531817.8275751.274626

2.46317158.0383723.7161581.606568

3.6870285.2211534.2666492.016553

3.96072594.55610236.6271222.438502

4.885092113.81749546.3351842.71544

5.790828137.76145653.0607584.026536

6.413572152.97400959.1526473.350554

7.368076170.28560270.079453.831907

8.203419197.61352375.3156854.55265

8.79215206.89368681.7001455.655495

9.646528228.85791292.3903995.426617

10.644633250.41229498.9083398.508124

13.79629324.127558120.6036099.147833

14.647996350.064451133.3405619.74361

15.435718364.706047122.82228610.552526

18.507878437.192419167.62115713.154808

19.545046462.939828181.35529713.079803

20.402282369.348908171.11352211.692634

27.974254497.147959242.17765119.103481

29.138987572.44007262.04488619.511098

29.968021543.207771257.42705220.192376

36.983623707.531119308.7725729.997626

38.64132760.725658323.31975125.90242

38.328893612.109993319.56274929.035245

54.872792870.859807446.5656941.735663

56.489209961.219338458.96678537.578825

57.595077758.687038435.19020541.791783

75.9039381000.434619534.24871658.094607

73.9393711083.917552541.85102359.023802

74.141053913.813495542.00898462.554444

107.2251671227.421415764.05805967.970935

109.4224611355.946837780.96204787.74085

109.1032071178.534485775.98643673.465665

138.19411466.745324953.502881119.542283

142.6563121404.286583966.562794104.978561

142.341211389.989651941.8648289.494004

202.3576311724.1559481255.968218158.880963

209.525711711.4644591267.68997156.118055

198.0298071624.1463221252.544659134.762982

255.8777441852.7836731492.127754177.606592

267.9571191915.5399471503.721693223.523028

254.3999591764.7446591482.758837228.529378

349.5140142036.4580141770.020829338.73584

371.7808262159.7597951783.600697287.490738

354.7484071975.1681731743.762752327.066776

432.8338192128.3082541968.873759433.921543

474.4960482311.1110551958.28841435.18533

435.208942079.100731945.381387336.127091

554.8894472213.626022207.641599619.964505

618.2340462507.7581132241.552252552.195503

554.2370712191.7474152244.964955582.984516

650.4876092181.7784452377.219095638.214068

729.588772592.9058752389.618441610.301585

650.566222167.0314052358.087989705.164857

765.9324252237.1045892551.238118937.898778

943.00732514.9710382729.443958905.954846

786.4208942246.075452552.757392966.494976

863.5860212263.3207512715.5842041078.395605

1079.3777912498.8959292705.4491911140.150726

859.0415782377.8022662691.7776891125.115603

957.9282142383.4496542835.1465531584.461976

1234.1848442528.6624522809.3028341479.862384

960.1059712375.5531182828.596321484.113874

924.5415982382.3680952884.6070581666.423811

1228.9351812505.5730372926.8699251654.527238

1209.3194432222.7868753007.519671668.604029

1383.5845982203.3100033068.9415112060.832715

1467.8976292908.2160863090.16132060.31493

1426.425272315.6772643062.6003622075.648832

1411.6546092224.351943105.0237012414.289072

1618.0847883086.5931663017.3169612375.29744

1396.1111552374.6092842993.1870862377.499718

1559.086882261.1548843010.5708052418.599859

1629.0289223269.3216933140.9253072513.455461

1504.5789052426.237463137.0899022396.62052

1579.8937422273.3931113138.3684532104.189811

1722.5120243424.914482999.2903892217.107943

1437.5914282467.9654873036.4436532252.584054

1458.6236572315.8918793141.1677972084.208735

1553.5302363512.4810613063.1059972160.070803

1488.3655052490.6695163012.3403092178.419039

1136.6307692337.3150423112.7162261420.054821

1401.5058513541.2801773091.6515941428.255358

1373.5910522536.0024243037.9628351468.932822

1298.3921072395.6589253162.9171571483.917741

1405.7753363459.8289723033.4987171445.590028

1353.6340382593.4647493111.864971478.731663

1234.0712182405.7469213210.4584971469.677393

1386.0168963452.0462533084.962141485.791533

1218.9794852629.5764433155.1917761487.170593

1394.1168032450.4991373251.3063071482.944557

1433.8852383361.2400483170.4554921480.853378

1400.613612670.2955073145.1721751453.466099

1265.734952457.3278873214.8091651494.811133

1439.7992843300.7374813168.2213281496.274828

1296.2451442679.7980013166.7579851496.034623

1451.7242482487.3027553236.435031515.591772

1479.1986763211.7686163176.7876871493.953952

1438.5847892721.8745063135.4162181501.742444

1488.5055382494.7687333252.837781525.985278

1568.8330063213.4976083121.858291525.859675

1306.9560542743.5634093159.1721251569.776899

1379.9223092931.1768333247.5097781539.631626

1404.7885063176.6771083163.1444721550.712877

1381.6956042904.2720843191.868481533.622164

1369.7835662839.5826223184.66811632.316899

1404.002223157.6462493180.108151557.434898

1385.2076312840.9146013156.4815841571.268858

1388.0243192956.5964363221.7604121523.095538

1411.7928213151.5442383205.9580471501.430489

1386.5368832930.1163523224.534791513.396937

1380.8771952977.7796943206.6093681555.372109

1390.1360423169.5327493171.8352771596.992117

1385.5022882971.4928883221.706451556.496431

1375.8831243026.3377413200.6374611490.720961

1405.8311083159.558423219.4764381514.251193

1377.964953044.1174053225.61681511.613287

1380.2121033061.1509753184.3158711496.591076

1412.1232133147.7486063204.7017611497.268366

1348.2443283061.4463433159.4273731508.278525

MPICH25 us

MP_Lite3 us

PVM33 us

LAM/MPI 1 us

MPICH

LAM

MP_Lite

PVM

Message size in Bytes

Throughput in Mbps

np.mini.fe

MPICHLAMMP_LitePVM

10.2814757.3713573.0639870.176333

20.6139914.5477865.9856520.392785

30.92692821.1595078.9933650.499961

41.21414428.79959411.8925040.786296

61.83384643.72531817.8275751.274626

82.46317158.0383723.7161581.606568

123.6870285.2211534.2666492.016553

133.96072594.55610236.6271222.438502

164.885092113.81749546.3351842.71544

195.790828137.76145653.0607584.026536

216.413572152.97400959.1526473.350554

247.368076170.28560270.079453.831907

278.203419197.61352375.3156854.55265

298.79215206.89368681.7001455.655495

329.646528228.85791292.3903995.426617

3510.644633250.41229498.9083398.508124

4513.79629324.127558120.6036099.147833

4814.647996350.064451133.3405619.74361

5115.435718364.706047122.82228610.552526

6118.507878437.192419167.62115713.154808

6419.545046462.939828181.35529713.079803

6720.402282369.348908171.11352211.692634

9327.974254497.147959242.17765119.103481

9629.138987572.44007262.04488619.511098

9929.968021543.207771257.42705220.192376

12536.983623707.531119308.7725729.997626

12838.64132760.725658323.31975125.90242

13138.328893612.109993319.56274929.035245

18954.872792870.859807446.5656941.735663

19256.489209961.219338458.96678537.578825

19557.595077758.687038435.19020541.791783

25375.9039381000.434619534.24871658.094607

25673.9393711083.917552541.85102359.023802

25974.141053913.813495542.00898462.554444

381107.2251671227.421415764.05805967.970935

384109.4224611355.946837780.96204787.74085

387109.1032071178.534485775.98643673.465665

509138.19411466.745324953.502881119.542283

512142.6563121404.286583966.562794104.978561

515142.341211389.989651941.8648289.494004

765202.3576311724.1559481255.968218158.880963

768209.525711711.4644591267.68997156.118055

771198.0298071624.1463221252.544659134.762982

1021255.8777441852.7836731492.127754177.606592

1024267.9571191915.5399471503.721693223.523028

1027254.3999591764.7446591482.758837228.529378

1533349.5140142036.4580141770.020829338.73584

1536371.7808262159.7597951783.600697287.490738

1539354.7484071975.1681731743.762752327.066776

2045432.8338192128.3082541968.873759433.921543

2048474.4960482311.1110551958.28841435.18533

2051435.208942079.100731945.381387336.127091

3069554.8894472213.626022207.641599619.964505

3072618.2340462507.7581132241.552252552.195503

3075554.2370712191.7474152244.964955582.984516

4093650.4876092181.7784452377.219095638.214068

4096729.588772592.9058752389.618441610.301585

4099650.566222167.0314052358.087989705.164857

6141765.9324252237.1045892551.238118937.898778

6144943.00732514.9710382729.443958905.954846

6147786.4208942246.075452552.757392966.494976

8189863.5860212263.3207512715.5842041078.395605

81921079.3777912498.8959292705.4491911140.150726

8195859.0415782377.8022662691.7776891125.115603

12285957.9282142383.4496542835.1465531584.461976

122881234.1848442528.6624522809.3028341479.862384

12291960.1059712375.5531182828.596321484.113874

16381924.5415982382.3680952884.6070581666.423811

163841228.9351812505.5730372926.8699251654.527238

163871209.3194432222.7868753007.519671668.604029

245731383.5845982203.3100033068.9415112060.832715

245761467.8976292908.2160863090.16132060.31493

245791426.425272315.6772643062.6003622075.648832

327651411.6546092224.351943105.0237012414.289072

327681618.0847883086.5931663017.3169612375.29744

327711396.1111552374.6092842993.1870862377.499718

491491559.086882261.1548843010.5708052418.599859

491521629.0289223269.3216933140.9253072513.455461

491551504.5789052426.237463137.0899022396.62052

655331579.8937422273.3931113138.3684532104.189811

655361722.5120243424.914482999.2903892217.107943

655391437.5914282467.9654873036.4436532252.584054

983011458.6236572315.8918793141.1677972084.208735

983041553.5302363512.4810613063.1059972160.070803

983071488.3655052490.6695163012.3403092178.419039

1310691136.6307692337.3150423112.7162261420.054821

1310721401.5058513541.2801773091.6515941428.255358

1310751373.5910522536.0024243037.9628351468.932822

1966051298.3921072395.6589253162.9171571483.917741

1966081405.7753363459.8289723033.4987171445.590028

1966111353.6340382593.4647493111.864971478.731663

2621411234.0712182405.7469213210.4584971469.677393

2621441386.0168963452.0462533084.962141485.791533

2621471218.9794852629.5764433155.1917761487.170593

3932131394.1168032450.4991373251.3063071482.944557

3932161433.8852383361.2400483170.4554921480.853378

3932191400.613612670.2955073145.1721751453.466099

5242851265.734952457.3278873214.8091651494.811133

5242881439.7992843300.7374813168.2213281496.274828

5242911296.2451442679.7980013166.7579851496.034623

7864291451.7242482487.3027553236.435031515.591772

7864321479.1986763211.7686163176.7876871493.953952

7864351438.5847892721.8745063135.4162181501.742444

10485731488.5055382494.7687333252.837781525.985278

10485761568.8330063213.4976083121.858291525.859675

10485791306.9560542743.5634093159.1721251569.776899

15728611379.9223092931.1768333247.5097781539.631626

15728641404.7885063176.6771083163.1444721550.712877

15728671381.6956042904.2720843191.868481533.622164

20971491369.7835662839.5826223184.66811632.316899

20971521404.002223157.6462493180.108151557.434898

20971551385.2076312840.9146013156.4815841571.268858

31457251388.0243192956.5964363221.7604121523.095538

31457281411.7928213151.5442383205.9580471501.430489

31457311386.5368832930.1163523224.534791513.396937

41943011380.8771952977.7796943206.6093681555.372109

41943041390.1360423169.5327493171.8352771596.992117

41943071385.5022882971.4928883221.706451556.496431

62914531375.8831243026.3377413200.6374611490.720961

62914561405.8311083159.558423219.4764381514.251193

62914591377.964953044.1174053225.61681511.613287

83886051380.2121033061.1509753184.3158711496.591076

83886081412.1232133147.7486063204.7017611497.268366

83886111348.2443283061.4463433159.4273731508.278525

np.mini.fe

MPICH25 us

MP_Lite3 us

PVM33 us

LAM/MPI 1 us

MPICH

LAM

MP_Lite

PVM


Throughput in Mbps

Chart1

0.0381610.0381450.0381180.0381320.0381260.038127

0.0763020.0762610.0762670.0762680.0762830.076301

0.1144530.1144140.1144150.1143730.1144250.114405

0.1525440.1527130.152450.1526710.1525650.152617

0.2289820.2287530.2286350.2287520.2289690.228918

0.3053960.305290.3054070.3051840.3052490.305144

0.4581630.4574640.4580360.4577430.4579850.457569

0.4958890.4956890.4961750.495670.4958270.495861

0.6116420.6103650.6106370.6103760.6105160.61029

0.7249960.7250920.7245420.724540.7246820.724713

0.8012740.8010740.8010890.8013380.8013070.801036

0.9154480.9155140.9155440.9154860.915530.915308

1.029591.0299481.0299541.0290441.0298171.029561

1.1080891.1063371.1067961.1064071.1060731.10712

1.2216741.2207491.2201671.2208651.2205061.220066

1.3355461.3345961.3361011.3356121.3358591.334988

1.7178291.7159821.7171491.7167461.7169511.716428

1.8309071.8310431.8315881.8315111.830781.830857

1.9459031.9454451.9449291.9465781.9457791.943578

2.3280232.3270962.3270842.3269772.3265852.330301

2.4413662.4415552.4424812.4413572.4410192.441029

2.5575852.5558972.5568792.5541022.555442.554578

3.5487383.5477343.5465583.5480113.5471013.547311

3.661793.6620323.6588813.659893.6615713.661703

3.7779913.7765063.7765393.776363.7748653.775011

4.7681634.7710534.7686354.7687394.7676354.765248

4.8864644.8831864.8837984.8828524.8800534.885023

4.9968794.9979834.9993034.9971944.9964454.99865

7.2092517.2128127.2151137.1985047.2077757.209027

7.3236257.3214957.3263967.3094587.3230977.321207

7.4424747.4253857.440837.4346687.4420077.428943

9.6522379.6130089.6570019.653359.6512389.622536

9.7648029.7657829.7668819.7611929.7639399.764603

9.8794369.8764959.880429.8759839.8784159.879043

14.53867414.53416814.53998814.337214.53245314.530454

14.65129714.64826214.64839214.64251714.51142312.948124

14.76172914.76257214.76274714.76221614.76055912.799333

19.41578719.35014719.42957817.76957416.62942918.938847

19.53775319.53164519.5237414.19078319.52795614.573324

19.64435519.64647319.64608415.98338819.64248319.633313

29.1912329.04698729.1945821.75095829.14253829.179417

29.28500729.234929.30567216.38225827.52674321.304491

29.41757629.31410229.45549723.78372924.50057329.396421

38.99019838.95085338.97323727.9363938.93946433.92745

39.06016239.06327839.08003835.43551839.05562234.029107

39.17397139.1935439.13162233.33621337.39399928.967598

40.42404541.29397553.02596952.99856542.64145955.766694

58.59628258.59549841.68271739.30369746.2875246.176994

42.55584744.88391334.823830.90072558.60732241.359307

60.33208262.4131156.40987340.11293456.39788378.005301

77.43879860.79363558.09532670.24980665.05398165.253829

70.88820772.12204458.96681264.54488458.53885559.551467

105.31351978.95525277.433515106.92361589.23702875.658416

61.55672980.9819394.68205780.902905110.51769174.053444

100.618714100.843534108.88967172.252514117.27657281.175949

150.52657190.919694116.653497100.196237103.1209881.038003

127.827082104.68865129.619563112.914765142.38791480.156104

145.13887398.204626116.486837120.49974397.48576989.868656

127.383841138.892796128.619426117.590748117.973491121.26169

181.174485127.791055131.301311138.860339176.990375135.233866

129.835391118.53153152.884766150.314208149.432742118.779276

180.634156154.084244203.064639153.702312160.12104154.436174

174.68441153.942764204.330629153.703529161.0131153.736802

227.98184156.797503157.511262154.007173155.925881154.496155

225.636208233.632903226.578122230.726719227.896725231.194076

226.724455232.644153232.228915231.852167230.743621228.647683

226.728076228.594239227.780522229.009944228.378106228.977752

274.027817269.996498278.828066179.231819284.26482151.900991

277.982945276.351733280.036482179.809828298.113839151.71794

283.153327290.893004293.326027167.589783295.438224149.266285

359.192948282.308403217.982385150.260275301.604907138.479871

340.159738281.911633212.810675185.411671302.483101146.782264

353.768143285.559468218.5164166.170952300.548004149.177851

351.665585298.956127218.861579167.213347324.164554185.567085

352.58368298.694194192.81117166.452061331.36703171.02341

352.558225298.940859209.847921166.116185328.275981173.483823

380.88954337.75593227.562565175.649182382.758364158.354173

393.521036339.017825234.7294175.347077383.206287175.782227

399.47588340.433648228.860379205.706406385.353738181.838338

431.790947354.198557258.153917203.928035404.706746172.823034

435.09208354.918884251.96351191.906937404.334847168.731342

431.57609355.824888217.435538191.047828404.319383160.425975

468.765745366.325772242.374732202.360242440.321872141.959444

465.535377366.756995232.581276206.707359439.660667156.283084

466.830917367.14946230.972724210.697712439.836357178.403186

489.291443345.743653245.804565208.269152452.386669187.23302

489.482042347.012124244.922947207.366292454.11793187.256014

489.573917346.881379238.217864204.455973450.701848159.658811

508.635057371.103069244.949404210.622987465.000979158.873301

509.192669369.845791256.20503206.692665467.058584152.366628

510.653796370.128594261.274386218.483467465.63005171.929814

519.250495374.481988250.074219227.347614480.507328193.309226

518.899329378.16968250.487368227.165802479.198842193.938595

519.862082376.559703254.481736222.787036480.306937189.632506

528.946367376.746706269.960148229.009271491.955472172.189377

528.862536377.886078260.652606228.316978492.637716152.419228

528.850547376.280992260.235466231.023518491.916126175.261832

536.336321381.906719264.72252240.330672499.874496193.202924

535.795764383.392169265.781876237.849558500.014481185.231732

535.325936381.979839263.210258241.231965500.467805174.561879

539.890965396.546122260.627791241.693723479.873969164.869483

539.000543394.814811271.640694243.593383480.917512186.811975

539.783632393.624866276.810349242.460141480.066764190.284062

543.273793400.516715267.08372242.294513490.194555167.17891

542.786753400.374608277.078206243.552917491.21092185.975476

543.346043398.107209269.754803237.199879491.630663191.492171

545.049287402.142993270.437248236.900217498.11037174.897829

545.238477402.332566270.114694243.542479498.10248188.466939

545.113937401.885761269.094139247.621159497.817105176.994108

546.336638404.973404274.842188249.018302503.096422187.048445

546.628151404.84732269.798585244.891435503.526966188.930202

546.346304404.582477267.452855242.786406503.356768183.256671

547.66063407.674074268.646931251.142982507.58838190.032713

547.249052408.122113279.05632244.323469507.446331193.622106

547.299578407.603619281.26667242.413983507.485904179.115823

548.556058409.868781276.949382247.657302510.078797191.512844

548.308041409.063322273.889194245.47475509.655568192.755163

548.371515408.070974277.321616248.626628509.817165189.158894

549.383346411.255734277.619735250.95497512.06178188.311532

549.132201411.299219274.050815248.575501512.168575185.084858

549.071005409.721312281.301741248.521657511.807629181.936362

549.587164411.011102276.236099246.914032513.202478186.431618

549.75903411.306707276.982361249.116277512.940591186.61904

549.516123409.421407277.877982247.577508512.938407183.386672

MPICH

MP_Lite

raw TCP

PVM

LAM/MPI

MPI/Pro

raw TCP

MPICH

LAM-MPI

MPI/Pro

MP_Lite

PVM


Throughput in Mbps

np.mini.fe

raw TCPMPICHLAM-MPIMPI/ProMP_LitePVMTCGMSG

10.0381610.0381450.0381180.0381320.0381260.0381270.038136

20.0763020.0762610.0762670.0762680.0762830.0763010.076288

30.1144530.1144140.1144150.1143730.1144250.1144050.114432

40.1525440.1527130.152450.1526710.1525650.1526170.152642

60.2289820.2287530.2286350.2287520.2289690.2289180.228721

80.3053960.305290.3054070.3051840.3052490.3051440.305232

120.4581630.4574640.4580360.4577430.4579850.4575690.457736

130.4958890.4956890.4961750.495670.4958270.4958610.496135

160.6116420.6103650.6106370.6103760.6105160.610290.610336

190.7249960.7250920.7245420.724540.7246820.7247130.724769

210.8012740.8010740.8010890.8013380.8013070.8010360.801041

240.9154480.9155140.9155440.9154860.915530.9153080.91528

271.029591.0299481.0299541.0290441.0298171.0295611.029977

291.1080891.1063371.1067961.1064071.1060731.107121.106242

321.2216741.2207491.2201671.2208651.2205061.2200661.221164

351.3355461.3345961.3361011.3356121.3358591.3349881.335104

451.7178291.7159821.7171491.7167461.7169511.7164281.717095

481.8309071.8310431.8315881.8315111.830781.8308571.832572

511.9459031.9454451.9449291.9465781.9457791.9435781.947142

612.3280232.3270962.3270842.3269772.3265852.3303012.326965

642.4413662.4415552.4424812.4413572.4410192.4410292.442341

672.5575852.5558972.5568792.5541022.555442.5545782.555708

933.5487383.5477343.5465583.5480113.5471013.5473113.54747

963.661793.6620323.6588813.659893.6615713.6617033.661912

993.7779913.7765063.7765393.776363.7748653.7750113.777633

1254.7681634.7710534.7686354.7687394.7676354.7652484.773444

1284.8864644.8831864.8837984.8828524.8800534.8850234.884467

1314.9968794.9979834.9993034.9971944.9964454.998654.997509

1897.2092517.2128127.2151137.1985047.2077757.2090277.224662

1927.3236257.3214957.3263967.3094587.3230977.3212077.325907

1957.4424747.4253857.440837.4346687.4420077.4289437.438439

2539.6522379.6130089.6570019.653359.6512389.6225369.651957

2569.7648029.7657829.7668819.7611929.7639399.7646039.76535

2599.8794369.8764959.880429.8759839.8784159.8790439.886943

38114.53867414.53416814.53998814.337214.53245314.53045414.550331

38414.65129714.64826214.64839214.64251714.51142312.94812414.643259

38714.76172914.76257214.76274714.76221614.76055912.79933314.761997

50919.41578719.35014719.42957817.76957416.62942918.93884719.439401

51219.53775319.53164519.5237414.19078319.52795614.57332419.530778

51519.64435519.64647319.64608415.98338819.64248319.63331319.645301

76529.1912329.04698729.1945821.75095829.14253829.17941729.181284

76829.28500729.234929.30567216.38225827.52674321.30449129.319403

77129.41757629.31410229.45549723.78372924.50057329.39642129.409113

102138.99019838.95085338.97323727.9363938.93946433.9274538.855765

102439.06016239.06327839.08003835.43551839.05562234.02910739.072867

102739.17397139.1935439.13162233.33621337.39399928.96759839.174591

153340.42404541.29397553.02596952.99856542.64145955.76669444.715573

153658.59628258.59549841.68271739.30369746.2875246.17699443.108397

153942.55584744.88391334.823830.90072558.60732241.35930746.706283

204560.33208262.4131156.40987340.11293456.39788378.00530163.633343

204877.43879860.79363558.09532670.24980665.05398165.25382978.124246

205170.88820772.12204458.96681264.54488458.53885559.55146760.235903

3069105.31351978.95525277.433515106.92361589.23702875.658416108.19779

307261.55672980.9819394.68205780.902905110.51769174.05344483.082819

3075100.618714100.843534108.88967172.252514117.27657281.175949109.38065

4093150.52657190.919694116.653497100.196237103.1209881.03800394.941585

4096127.827082104.68865129.619563112.914765142.38791480.156104156.220141

4099145.13887398.204626116.486837120.49974397.48576989.868656138.246904

6141127.383841138.892796128.619426117.590748117.973491121.26169145.545558

6144181.174485127.791055131.301311138.860339176.990375135.233866185.499308

6147129.835391118.53153152.884766150.314208149.432742118.779276230.62151

8189180.634156154.084244203.064639153.702312160.12104154.436174199.179803

8192174.68441153.942764204.330629153.703529161.0131153.736802157.535419

8195227.98184156.797503157.511262154.007173155.925881154.496155159.716391

12285225.636208233.632903226.578122230.726719227.896725231.194076228.916008

12288226.724455232.644153232.228915231.852167230.743621228.647683228.720924

12291226.728076228.594239227.780522229.009944228.378106228.977752226.109776

16381274.027817269.996498278.828066179.231819284.26482151.900991204.22593

16384277.982945276.351733280.036482179.809828298.113839151.71794165.187003

16387283.153327290.893004293.326027167.589783295.438224149.266285194.67264

24573359.192948282.308403217.982385150.260275301.604907138.479871193.443439

24576340.159738281.911633212.810675185.411671302.483101146.782264198.480244

24579353.768143285.559468218.5164166.170952300.548004149.177851198.810479

32765351.665585298.956127218.861579167.213347324.164554185.567085207.638848

32768352.58368298.694194192.81117166.452061331.36703171.02341197.471557

32771352.558225298.940859209.847921166.116185328.275981173.483823223.991818

49149380.88954337.75593227.562565175.649182382.758364158.354173212.481713

49152393.521036339.017825234.7294175.347077383.206287175.782227213.439091

49155399.47588340.433648228.860379205.706406385.353738181.838338215.478994

65533431.790947354.198557258.153917203.928035404.706746172.823034211.890004

65536435.09208354.918884251.96351191.906937404.334847168.731342210.615019

65539431.57609355.824888217.435538191.047828404.319383160.425975205.46014

98301468.765745366.325772242.374732202.360242440.321872141.959444216.044434

98304465.535377366.756995232.581276206.707359439.660667156.283084217.783247

98307466.830917367.14946230.972724210.697712439.836357178.403186221.781306

131069489.291443345.743653245.804565208.269152452.386669187.23302225.34062

131072489.482042347.012124244.922947207.366292454.11793187.256014218.400171

131075489.573917346.881379238.217864204.455973450.701848159.658811213.730804

196605508.635057371.103069244.949404210.622987465.000979158.873301215.751186

196608509.192669369.845791256.20503206.692665467.058584152.366628219.242509

196611510.653796370.128594261.274386218.483467465.63005171.929814219.242179

262141519.250495374.481988250.074219227.347614480.507328193.309226216.127069

262144518.899329378.16968250.487368227.165802479.198842193.938595223.008491

262147519.862082376.559703254.481736222.787036480.306937189.632506216.962009

393213528.946367376.746706269.960148229.009271491.955472172.189377229.949015

393216528.862536377.886078260.652606228.316978492.637716152.419228229.654106

393219528.850547376.280992260.235466231.023518491.916126175.261832224.686418

524285536.336321381.906719264.72252240.330672499.874496193.202924221.069087

524288535.795764383.392169265.781876237.849558500.014481185.231732227.453615

524291535.325936381.979839263.210258241.231965500.467805174.561879236.505843

786429539.890965396.546122260.627791241.693723479.873969164.869483239.104815

786432539.000543394.814811271.640694243.593383480.917512186.811975228.067488

786435539.783632393.624866276.810349242.460141480.066764190.284062222.706801

1048573543.273793400.516715267.08372242.294513490.194555167.17891231.057719

1048576542.786753400.374608277.078206243.552917491.21092185.975476234.834827

1048579543.346043398.107209269.754803237.199879491.630663191.492171226.092428

1572861545.049287402.142993270.437248236.900217498.11037174.897829237.208959

1572864545.238477402.332566270.114694243.542479498.10248188.466939239.285902

1572867545.113937401.885761269.094139247.621159497.817105176.994108233.896438

2097149546.336638404.973404274.842188249.018302503.096422187.048445230.26663

2097152546.628151404.84732269.798585244.891435503.526966188.930202226.257586

2097155546.346304404.582477267.452855242.786406503.356768183.256671230.949342

3145725547.66063407.674074268.646931251.142982507.58838190.032713243.539449

3145728547.249052408.122113279.05632244.323469507.446331193.622106241.340654

3145731547.299578407.603619281.26667242.413983507.485904179.115823244.265444

4194301548.556058409.868781276.949382247.657302510.078797191.512844242.37043

4194304548.308041409.063322273.889194245.47475509.655568192.755163243.887175

4194307548.371515408.070974277.321616248.626628509.817165189.158894241.627226

6291453549.383346411.255734277.619735250.95497512.06178188.311532246.143897

6291456549.132201411.299219274.050815248.575501512.168575185.084858250.358308

6291459549.071005409.721312281.301741248.521657511.807629181.936362250.065428

8388605549.587164411.011102276.236099246.914032513.202478186.431618247.296077

8388608549.75903411.306707276.982361249.116277512.940591186.61904250.734921

8388611549.516123409.421407277.877982247.577508512.938407183.386672241.921636

np.mini.fe

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

000000

MPICH

MP_Lite

raw TCP

PVM

LAM/MPI

MPI/Pro

raw TCP

MPICH

LAM-MPI

MPI/Pro

MP_Lite

PVM


Throughput in Mbps

AMD Opteron 800 HPCProcessing NodeHPC StrengthsFlat SMP like Memory Model:All four reside with the same 248 memory mapExpandable to 8P NUMA

Glue-less Coherent multi-processing:low Latency and high Bandwidth ~1600M T/sec (6.4 GB/s)

32GB of High B/W external memory bus (>5.3GB/sec.)

Native high B/W memory map I/O (>25Gbits/sec.)

Sufficiently Uniform Memory Organization (SUMO)Disadvantages3P and 4P nodes work better if the OS is aware of the memory map>4P may require a NUMA (Non-uniform memory architecture) aware OS if the CACHE hit rate is low2.6.9 kernel needed to take full advantage of NUMA architecture of OpteronsAdvantages Software view of memory is SMPLatency difference between local & remote memory is a function of the number of processors in the node1P and 2P look like a SMP machine3P and 4P are NUMA like but can still be viewed as a ccUMA or asymmetric SMP node>4P can be viewed as ccUMA and depending on CACHE hit rate, may or may not required NUMA aware OSPhysical address space is flat and can be viewed as fully coherent or not (MOEIS state)DRAM can be contiguous or interleavedAdditional processor nodes bring true increased memory bandwidthDesigned for lower overall system chip count (glue-less interface)

Future NUMA Systems Scaling beyond 8 ProcessorScaling beyond 8P is enabledExternal Coherent HyperTransport switch Coherent Interconnect Snoop filter Data cachingUp to 16 processors within the same 240 SPM memory spaceInterconnect Fabric

High Density HPC ClusterSprayCool Technology from ISR 16 cards

16G-flops/card

256G-flops peak throughput

64GB of memory per card

1TerraByte of sys. Memory

240 cubic inches114M-flops/cubic inch4.27GB of memory storage cubic inch

~6K watts ~3 watts/cubic inch141016

AMD OpteronBeowulf 4P SMP Processing NodeAMD Opteron200-333MHz9 byte Reg. DDR8GB DRAMAMD Opteron200-333MHz9 byte Reg. DDRFLASHLPCLegacy PCIUSB1.0AC97UDMA133MII10/100 Phy100 BaseTManagement LANManagementSIOPCIGraphicsVGAAMD-8111TMI/O Hub16x16 HyperTransport @ 1600MT/sPCI-XPCI-XAMD-8131TMPCI-X TunnelTo AMD 8131 TunnelTo AMD 8131 TunnelOne 4P SMP node16G-flops32GB DRAM10GB/sec. Memory BW

Single Instruction, Multiple Data (SIMD)

Extending 32-bit instruction setIntel and AMD scheme very similar48-bit virtual address space64-bit General Purpose RegistersSupport 64-bit addressing and integer mathEight extra GPR addedEight extra XMM addedDifferenceEM64T supports SSE3 instructions, Opteron has 3DNow!

Status of 64bit Linux64-bit Linux OS is a stable operating platformOpteron CPU and associated platforms have sufficient reliabilityOpteron CPU gives slightly better performance for significantly less power draw as Xeon.Using 64-bit compilation and optimization can lead to significant performance gains on AMD and Intel.

Increased Memory for 32-bit Applications32-bit server, 4 GB DRAM64-bit server, 12 GB DRAMShared256 TB32-bit App0 GB4 GB0 GB4 GB32-bit App256 TBNotsharedNotsharedNotshared OS & App share small 32-bit VM space 32-bit OS & applications all share 4GB DRAM Leads to small dataset sizes & lots of paging App has exclusive use of 32-bit VM space 64-bit OS can allocate each application large dedicated portions of 12GB DRAM OS uses VM space way above 32-bits Leads to larger dataset sizes & reduced paging

CompilersOpteron 250Legacy executable, i386: 1290 VUPSGcc 3.4.2 optimized: 2440 VUPSPathscale compiler: 2677 VUPSXeon 3.6Legacy executable, i386: 1386 VUPSGcc 3.4.2 optimized: 2309 VUPSIntel 8.1 compiler:2910 VUPSIntel 8.1 compiler with profile feedback: 4332 VUPSIntel Fortran and C 8.1 uses SSE3 instructions to optimize, makes it incompatible with Opterons.For comparison PentiumIII 1.0 GHz=568 VUPS.

64-bit Compilers

Computing Strategy: x86-64Legacy: 32-bit OSBoth AMD Athlon 64 and AMD Opteron processors run any 32-bit legacy O/SCompatible all legacy Drivers, OS & BIOSNo application recompile required, no emulation layer64-bit OSDesired applications can be written/ported to leverage the full 64-bit capabilities of x86-64Migrate only where warranted, and at the users pace32-bit applications run under 64-bit OSBIOS is standard x86 32-bit code. Transfer to 64-bit operation occurs under OS load/startup control64-bit mode does not use segmentation - Flat addressing

Compatibility Thunking LayerAMD64 Operating SystemAMD64 Device DriversAMD64 Application64-bit ProcessThunking LayerIA32 Application64-bit Process

64-bit OS & Application Interaction32-bit Compatibility Mode64-bit OS runs existing 32-bit APPs with leading edge performanceNo recompile required, 32-bit code directly executed by CPU64-bit OS provides 32-bit libraries and thunking translation layer for 32-bit system calls. 64-bit ModeMigrate only where warranted, and at the users pace to fully exploit AMD6464-bit OS requires all kernel-level programs & drivers to be ported.Any program that is linked or plugged in to a 64-bit program (ABI-level) must be ported to 64-bits.

Problems 64/32In 64bit Linux you can't run binary only programs which are compiled for IA32 or applications which haven't been ported to AMD64 yet (e.g. OpenOffice.org). This is because you can't mix 32bit applications and 64bit libraries. You would also need the 32bit versions of the libraries to run a 32bit application.Multiarcharchitectures like sparc64 or powerpc64, which provide lib for default 32bit libraries and lib64 for extra 64bit libraries, default to executing 32bit applicationsamd64 defaults to 64bit binaries because of the performance benefits it offers in 64bit mode. Thus, not wanting to rewrite virtually every binary-arch package's creation rules to install libs in lib64 not lib, and wanting to find a solution for all multiarch capable platforms, various people are working on so-called multiarch support.

Sun Solaris 64bitThe Solaris Operating System supports both 32-bit and 64-bit hardware. Customers with 32-bit hardware can run the Solaris Operating System and take advantage of the many features in the Solaris Operating System that are not explicitly related to 64-bits (e.g., dynamic reconfiguration, scalability enhancements, performance improvements). Customers can run a 32-bit application on 64- or 32-bit hardware with the Solaris Operating System without any change to the application.

Note that Solaris for x86, prior to Solaris 10, supports only a 32 bit kernel.

MacOS X xcodeAt the heart of Xcode 2.0 is Apples version of gcc 3.5, the next generation of the industry-standard gcc compiler. The new compiler helps you get more performance from your existing code by using a number of advanced optimization techniques. Auto-vectorization, a technique borrowed from the world of supercomputing, helps you to unlock the power of the Velocity Engine in every PowerPC G4 and G5 system without writing vectorized code.

With the new 64-bit support in Mac OS X Tiger, Xcode gives you the ability to create applications such as computation and rendering engines that use 64-bit memory addressing. This is ideal for data-intensive applications, which can run faster by accessing data in memory, rather than via disk access. Xcode gives you the tools for building and debugging 64-bit applications for PowerPC G5 and Mac OS X Tiger, as well as letting you create Fat Binaries that contain both 32-bit and 64-bit executables.

Kernel Overview

Kernel sourceDownload source from www.kernel.org or Mirrors.Unpack:cd /usr/srctar xzvf linux-.tar.gzln s linux-.tar.gz linuxSource-Root:/usr/src/linux

Linux Source Tree Layout

/usr/src/linuxDocumentationarchfsinitkernelincludeipcdriversnetmmlibscriptsalphaarmi386ia64m68kmipsmips64ppcs390shsparcsparc64acornatmblockcdromchardiofc4i2ci2oideieee1394isdnmacintoshmiscnetadfsaffsautofsautofs4bfscodecramfsdevfsdevptsefsext2fathfshpfsasm-alphaasm-armasm-genericasm-i386asm-ia64asm-m68kasm-mipsasm-mips64linuxmath-emunetpcmciascsivideo adfsaffsautofsautofs4bfscodecramfsdevfsdevptsefsext2fathfshpfs 802appletalkatmax25bridgecoredecneteconetethernetipv4ipv6ipxirdakhttpdlapb

Sizes (linux-2.4.0-test2)sizedirectoryentriesfiles loc90M/usr/src/linux/1976452.6M4.5MDocumentation97380 na16.5March121685466K54Mdrivers3122561.5M5.6Mfs70489150K14.2Minclude192262285K28Kinit221K120Kipc664.5K332Kkernel252512K80Klib882K356Kmm191912K5.8Mnet33453162K400Kscripts264212K

linux/archSubdirectories for each current port.Each contains kernel, lib, mm, boot and other directories whose contents override code stubs in architecture independent code.lib contains highly-optimized common utility routines such as memcpy, checksums, etc.arch as of 2.4:alpha, arm, i386, ia64, m68k, mips, mips64.ppc, s390, sh, sparc, sparc64.

linux/driversLargest amount of code in the kernel tree (~1.5M).device, bus, platform and general directories.drivers/char n_tty.c is the default line discipline.drivers/block elevator.c, genhd.c, linear.c, ll_rw_blk.c, raidN.c.drivers/net specific drivers and general routines Space.c and net_init.c.drivers/scsi scsi_*.c files are generic; sd.c (disk), sr.c (CD-ROM), st.c (tape), sg.c (generic).General: cdrom, ide, isdn, parport, pcmcia, pnp, sound, telephony, video.Buses fc4, i2c, nubus, pci, sbus, tc, usb.Platforms acorn, macintosh, s390, sgi.

linux/fsContains:virtual filesystem (VFS) framework.subdirectories for actual filesystems.vfs-related files:exec.c, binfmt_*.c - files for mapping new process images.devices.c, blk_dev.c device registration, block device support.super.c, filesystems.c.inode.c, dcache.c, namei.c, buffer.c, file_table.c.open.c, read_write.c, select.c, pipe.c, fifo.c.fcntl.c, ioctl.c, locks.c, dquot.c, stat.c.

linux/includeinclude/asm-*:Architecture-dependent include subdirectories.include/linux:Header info needed both by the kernel and user apps.Usually linked to /usr/include/linux.Kernel-only portions guarded by #ifdefs#ifdef __KERNEL__ /* kernel stuff */#endifOther directories:math-emu, net, pcmcia, scsi, video.

linux/initJust two files: version.c, main.c.version.c contains the version banner that prints at boot.main.c architecture-independent boot code.start_kernel is the primary entry point.

linux/ipcSystem V IPC facilities.If disabled at compile-time, util.c exports stubs that simply return ENOSYS.One file for each facility:sem.c semaphores.shm.c shared memory.msg.c message queues.

linux/kernelThe core kernel code.sched.c the main kernel file:scheduler, wait queues, timers, alarms, task queues.Process control:fork.c, exec.c, signal.c, exit.c etcKernel module support:kmod.c, ksyms.c, module.c.Other operations:time.c, resource.c, dma.c, softirq.c, itimer.c.printk.c, info.c, panic.c, sysctl.c, sys.c.

linux/libkernel code cannot call standard C library routines.Files:brlock.c Big Reader spinlocks.cmdline.c kernel command line parsing routines.errno.c global definition of errno.inflate.c gunzip part of gzip.c used during boot.string.c portable string code.Usually replaced by optimized, architecture-dependent routines.vsprintf.c libc replacement.

linux/mmPaging and swapping:swap.c, swapfile.c (paging devices), swap_state.c (cache).vmscan.c paging policies, kswapd.page_io.c low-level page transfer.Allocation and deallocation:slab.c slab allocator.page_alloc.c page-based allocator.vmalloc.c kernel virtual-memory allocator.Memory mapping:memory.c paging, fault-handling, page table code.filemap.c file mapping.mmap.c, mremap.c, mlock.c, mprotect.c.

linux/scripts Scripts for:Menu-based kernel configuration.Kernel patching.Generating kernel documentation.

Linux Kernel ConfigurationDownload the source code and extract it under /usr/srcCustomize kernel configurationmake configmake menuconfigmake xconfigTwo menu items: Networking options and Network device supportDevice supportThree options: y, m, n

Kernel ConfigGet sources from kernel.orgUnpack in your home directorygzip -cd linux-2.4.XX.tar.gz | tar xvf -make menuconfigmake depmake bzimage

Building and Installing KernelCompiling the kernel.make dep [ make clean ]make bzImagemake modulesInstalling the kernel.make modules_install[ make install ]Kernel stored in /usr/src/linux-2.4/arch/i386/boot/bzImage and copied to /bootcp arch/i386/boot/bzImage /boot/Edit your lilo.conf and run /sbin/liloReboot.

LILO v.s. GRUBLILORun LILO to modify mini-bootloader in the MBRCannot read file system itselfGRUBMultistage loaderCan read file-system itself

Parameter passing (runlevel, init) to kernelActually hacking modifies address and name inside kernel for the process to start

Linux Loader (LILO)# sample /etc/lilo.confboot = /dev/hda delay = 40 password=SOME_PASSWORD_HERE default=vmlinuz-stable vga = normal root = /dev/hda1 image = vmlinuz-2.5.99 label = net test kernel restrictedimage = vmlinuz-stable label = stable kernel restricted other = /dev/hda3 label = Windows 2000 Professional restricted table = /dev/hda

GRUB # /etc/grub.conf generated by anacondatimeout=10splashimage=(hd0,1)/grub/splash.xpm.gzpassword --md5 $1$pd$J08sMAcfyWW.C3soZpHkh.title Red Hat Linux (2.4.18-3custom) root (hd0,1) kernel /vmlinuz-2.4.18-3custom ro root=/dev/hda5 initrd /initrd-2.4.18-3.imgtitle Red Hat Linux (2.4.18-3) Emergency kernel (no afs) root (hd0,1) kernel /vmlinuz-2.4.18-3 ro root=/dev/hda5 initrd /initrd-2.4.18-3.imgtitle Windows 2000 Professional rootnoverify (hd0,0) chainloader +1

/bootunix grub # more grub.confdefault 0# How many seconds to wait before the default listing is booted.timeout 5

title=gentoo root (hd0,0) kernel /kernel-2.4.26-gentoo-r9 root=/dev/ram0 init=/linuxrc ramdisk=8192 real_root=/dev/hda2 initrd /initrd-2.4.26-gentoo-r9

title GNU/Linux (2.4.25) root (hd0,4) kernel (hd0,4)/boot/kernel-2.4.25_pre6-gss root=/dev/ram0 init=/linuxrc real_root=/dev/hda5 vga=0x317 splash=verbose initrd (hd0,4)/boot/initrd-2.4.25_pre6-gss

unix grub # unix boot # pwd; ls -lRp/boot.:total 1582lrwxrwxrwx 1 root root 1 Sep 23 14:11 boot -> ./drwxr-xr-x 2 root root 1024 Sep 23 15:34 grub/-rw-r--r-- 1 root root 458622 Sep 23 14:58 initrd-2.4.26-gentoo-r9-rw-r--r-- 1 root root 1137878 Sep 23 14:50 kernel-2.4.26-gentoo-r9drwx------ 2 root root 12288 Sep 23 13:49 lost+found/

./grub:total 846-rw-r--r-- 1 root root 30 Sep 23 15:34 device.map-rw-r--r-- 1 root root 11264 Sep 23 15:34 e2fs_stage1_5-rw-r--r-- 1 root root 10256 Sep 23 15:34 fat_stage1_5-rw-r--r-- 1 root root 9216 Sep 23 15:34 ffs_stage1_5-rw-r--r-- 1 root root 245 Sep 23 15:34 grub.conf-rw-r--r-- 1 root root 1495 Sep 23 15:32 grub.conf.sample-rw-r--r-- 1 root root 11456 Sep 23 15:34 jfs_stage1_5lrwxrwxrwx 1 root root 9 Sep 23 15:32 menu.lst -> grub.conf-rw-r--r-- 1 root root 9600 Sep 23 15:34 minix_stage1_5-rwxr-xr-x 1 root root 196836 Sep 23 15:32 nbgrub-rwxr-xr-x 1 root root 197860 Sep 23 15:32 pxegrub-rw-r--r-- 1 root root 12864 Sep 23 15:34 reiserfs_stage1_5-rw-r--r-- 1 root root 33856 Sep 23 15:32 splash.xpm.gz-rw-r--r-- 1 root root 512 Sep 23 15:34 stage1-rw-r--r-- 1 root root 135148 Sep 23 15:34 stage2-rwxr-xr-x 1 root root 196900 Sep 23 15:32 stage2.netboot-rw-r--r-- 1 root root 8896 Sep 23 15:34 vstafs_stage1_5-rw-r--r-- 1 root root 12840 Sep 23 15:34 xfs_stage1_5

./lost+found:total 0unix boot #

Build kernel for other platformunix linux # make menuconfig ARCH=x86_64/usr/src/linux/archalpha cris ia64 mips parisc ppc64 s390x sh64 sparc64 arm i386 m68k mips64 ppc s390 sh sparc x86_64Cross-compile make HOSTCC="gcc -m32" ARCH="x86_64" bzImageget a gcc wrapper in order to crosscompile on a i386 hosthttp://www.jukie.net/~bart/debian/amd64/scripts/gcc.bart

Linux modulesdriver modules in /lib/module[root@dafinn net]# pwd; ls/lib/modules/2.4.22-1.2166.nptlsmp/kernel/drivers/net3c509.o b44.o eepro100.o netconsole.o pppox.o tg3.o3c59x.o bonding epic100.o ns83820.o ppp_synctty.o tlan.o8139cp.o de4x5.o ethertap.o pcmcia r8169.o tulip8139too.o dl2k.o fealnx.o pcnet32.o sis900.o tun.o82596.o dmfe.o irda ppp_async.o sk98lin typhoon.o8390.o dummy.o mii.o ppp_deflate.o slhc.o via-rhine.oacenic.o e100 natsemi.o ppp_generic.o smc9194.o wirelessamd8111e.o e1000 ne2k-pci.o pppoe.o starfire.oRecompiling the kernelMake the kernel smallerAdd a new deviceModify a system parameter

Hello World !#define MODULE#include int init_module() { printk("Hello, world\n"); return 0; }void cleanup_module() { printk("Goodbye !\n"); }

Loaded and Unloadedroot# gcc -c hello.croot# insmod ./hello.oHello, worldroot# rmmod helloGoodbye !root#MUST be root to load/unload moduleCheck /var/log/messages if nothing shownCan use module_init(my_init) & module_exit(my_cleanup) in Linux 2.2 and later

Source Code ControlThe entire source code for FreeBSD is stored in a CVS repositoryThe logs, and individual changes for each file can be traced back to 1994.The source tree can be checked out at any state, or corresponding to any releaseCDs are available taking the history back a further 20 years

Building WorldThe entire operating system, including all libraries and utilities can be built with a single command : make worldThe source code for the system is placed in /usr/src during installation.Much easier to secure a system if a bug is found in a key library like OpenSSL.More information in build(7) and Handbook.

Building ReleasesYou can even build a complete release of FreeBSD, including FTP install directories, floppy images, and ISO images for CDROMs with one command.make release is used by many large companies to produce special versions of FreeBSD with special patches or additional software installed by default.It is also the well documented way in which the release engineering team makes all official releases of FreeBSD.

Release Engineeringmake release makes it much easier to deploy thousands of systems pre-configured for a specific environment.The release engineering team for FreeBSD publishes schedules, identifies QA issues that must be resolved before release, and publishes documents to help other people build FreeBSD based products.See release(7) and www.freebsd.org/releng

Running a FreeBSD binaryCode likefd = open(/etc/passwd, O_RDONLY);Becomessyscall(5, ...)Kernel knows its a FreeBSD binary, uses freebsd_syscalls[ ] arrayfreebsd_syscalls[5] = freebsd_open();File is opened

Running a Linux binaryCode likefd = open(/etc/passwd, O_RDONLY);Becomessyscall(5, ...)Kernel knows its a Linux binary, uses linux_syscalls[ ] arraylinux_syscalls[5] = linux_open();File is openedAll Linux file operations redirected to /compat/linux first

Linux modulesList installed moduleslsmodModule dependenciesThe meaning of (autoclean)Load the module mannually: insmod [k] 3c509modprobe smc-ultraGenerate the dependencydepmod aRemove module: rmmodDevice driver can be dynamically loaded to compiled into the kernel.

Function pointers

LKM Utilities cmdinsmod Insert an LKM into the kernel.rmmod Remove an LKM from the kernel.depmod Determine interdependencies between LKMs.kerneld Kerneld daemon programksyms Display symbols that are exported by the kernel for use by new LKMs.lsmod List currently loaded LKMs.modinfo Display contents of .modinfo section in an LKM object file.modprobe Insert or remove an LKM or set of LKMs intelligently. For example, if you must load A before loading B, Modprobe will automatically load A when you tell it to load B.

Two ways for loading a modulemanual way : using insmod command.

automatic way : the kernel discovers the need for loading a module (for example, user mounts a file system) => requests kerneld daemon to load the needed module. => kerneld loads module using insmod.

The insmod mechanism for loading a module

reads the module into its virtual memory fixes up unresolved references to kernel routines and resources using the exported symbols from the kernel.requests the kernel for enough space to hold the new kernelthe kernel allocates a new module data structure and enough kernel memory to hold the new module and puts it at the end of the kernel modules list

The insmod mechanism for loading a module

insmod copies the module into the allocated space and relocates it so that it will run from the kernel address that it has been allocatedthe new module exports sysmbols to the kernel and insmod builds a table of these exported symbolsif the new module depends on another module, that module has the reference of the new module.the kernel calls the modules initialization routine and carries on installing the module

Unloadig a moduleTwo ways for unloading a moduleManual way : uses rmmod commandautomatic way : when idle timer expires, the kerneld calls the service routines for all unused loaded modulesthe mechanism of unloadingif the module can be unloaded, its cleanup routine is called to free up the kernel resources that it has allocatedthe module data structure is unlinked from the list of kernel modulesall of the kernel memory that the module needed is deallocated.

User Space DriverAdvantages Full C library can be linked inRun in conventional debuggerProblem in driver unlikely to hang entire systemUser memory is swapping (more to use)Still allow concurrent access to a device for well-designed driverDisadvantages Interrupts are not available in user spaceDirect access to memory by mmapping /dev/mem (only privileged user)Access to I/O ports after calling ioperm or iopl (not all platform support), and access to /dev/port can be too slow (only privileged user)Response time is slower (context switch, driver swapped to disk)Most important devices cant be handled in user space (block, network)

The Pentium III 1.0 GHz came out 4 years ago (June 2001).Thunking is a term used to describe calling cross platform code and can mean calling 32 or 16bit code from a 64bit platform or calling 64bit code from a 32bit platform. In essence, a thunk is a translation layer that sits between the 32bit application and the 64bit process code.There are two basic methods used for thunking, which is a function of the calling platform. In the case of 32bit application calling 64bit application, the generic thunking functions in the 64bit kernel can be used. For the opposite scenario, the 64bit process must use a flat thunk to get the job done. It is also worth noting that flat thunks can be used to make 32bit to 64bit calls but because flat thunks are far more difficult, generic thunks are preferred for this situation.Process management creating and destroying processes and handling heir connection to the outside world. Process communication and scheduler.

Memory management builds up a virtual addressing space. Different parts of kernel interact with malloc/free

Filesystems almost everything can be treated as a file. Build a structured filesystem on top of unstructured hardware. Support multiple filesystems such as ext2, FAT.

Device Control kernel must have embedded in its a device driver for every peripheral present on a system.

Networking most network operations are not specific to a process, incoming packets are asynchronous events.

MODULE is needed except for drivers that are linked directly to the kernel.

Printk() is defined in Linux kernel and behaves similarly to printf. The kernel runs by itself without the help of C library. No floating point support before Linux 2.2The string is the priority of the message so it can show on console. (depending on configuration of klogd daemon)

static int __init my_init(void)static void __exit my_cleanup(void)Advantages: (i)Each init and cleanup have unique name to helps debugging.(ii) Make life easier for those writing drivers that work either as a module or built directly into the kernel (no need to change function name)

User program that RW directly to device portsBlock device, network interfacesExample is X server, gpm mouse driver

SMP, 64bit Unix and Kernel Compilation

Documents

Transcript of SMP, 64bit Unix and Kernel Compilation