New 20160926 - Introducing pScheduler - Feit · 2016. 10. 24. · Introducing pScheduler...
Transcript of New 20160926 - Introducing pScheduler - Feit · 2016. 10. 24. · Introducing pScheduler...
![Page 1: New 20160926 - Introducing pScheduler - Feit · 2016. 10. 24. · Introducing pScheduler perfSONAR’sNew Scheduler Presented by the perfSONAR Project ... • Round-Trip Ping ...](https://reader036.fdocuments.in/reader036/viewer/2022081617/604a71f004b410263575e2e1/html5/thumbnails/1.jpg)
IntroducingpSchedulerperfSONAR’s NewScheduler
PresentedbytheperfSONAR Projecthttp://www.perfsonar.net
Internet2TechnologyExchange2016September,2016ThisdocumentisaresultofworkbytheperfSONAR Project(http://www.perfsonar.net)andislicensedunderCCBY-SA4.0
(https://creativecommons.org/licenses/by-sa/4.0/).
![Page 2: New 20160926 - Introducing pScheduler - Feit · 2016. 10. 24. · Introducing pScheduler perfSONAR’sNew Scheduler Presented by the perfSONAR Project ... • Round-Trip Ping ...](https://reader036.fdocuments.in/reader036/viewer/2022081617/604a71f004b410263575e2e1/html5/thumbnails/2.jpg)
pScheduler• pScheduler isnewsoftwareforschedulingandsupervisingmeasurements.
• SuccessortotheBandwidthTestController(BWCTL)
October24,2016©2016,http://www.perfsonar.net 2
![Page 3: New 20160926 - Introducing pScheduler - Feit · 2016. 10. 24. · Introducing pScheduler perfSONAR’sNew Scheduler Presented by the perfSONAR Project ... • Round-Trip Ping ...](https://reader036.fdocuments.in/reader036/viewer/2022081617/604a71f004b410263575e2e1/html5/thumbnails/3.jpg)
WhyreplaceBWCTL?• BWCTLhasservedperfSONARwellbutisshowingitsage.– Maintenanceandexpansionhavebecomedifficult.
• Demandfornewfeaturesandadditionalsophisticationcalledforaclean-slateimplementationbuiltwithaneyetowardthefuture.
October24,2016©2016http://www.perfsonar.net 3
![Page 4: New 20160926 - Introducing pScheduler - Feit · 2016. 10. 24. · Introducing pScheduler perfSONAR’sNew Scheduler Presented by the perfSONAR Project ... • Round-Trip Ping ...](https://reader036.fdocuments.in/reader036/viewer/2022081617/604a71f004b410263575e2e1/html5/thumbnails/4.jpg)
WeGetLetters• TestsandTools.CanperfSONARmeasure ____?• Schedule.Whatran?What’srunning?What’splannedtorun?
• StatusandDiagnostics.Whathappened?WhydoIhavetologinandpickthroughthelogs?
October24,2016©2016,http://www.perfsonar.net 4
![Page 5: New 20160926 - Introducing pScheduler - Feit · 2016. 10. 24. · Introducing pScheduler perfSONAR’sNew Scheduler Presented by the perfSONAR Project ... • Round-Trip Ping ...](https://reader036.fdocuments.in/reader036/viewer/2022081617/604a71f004b410263575e2e1/html5/thumbnails/5.jpg)
WeGetLetters• RegularTesting. Canitworkwith____?Canitbebetter?Lesskludgy?
• Archiving. Canresultsbesentto____?Reliably?• Persistence.Whydoestheschedulenotsurviveareboot?
• OpenIssuesinGitHub.Lotsofthem.
October24,2016©2016,http://www.perfsonar.net 5
![Page 6: New 20160926 - Introducing pScheduler - Feit · 2016. 10. 24. · Introducing pScheduler perfSONAR’sNew Scheduler Presented by the perfSONAR Project ... • Round-Trip Ping ...](https://reader036.fdocuments.in/reader036/viewer/2022081617/604a71f004b410263575e2e1/html5/thumbnails/6.jpg)
OurGoals1. Lookateveryaspectofmeasurement
throughasmanylensesaspossible.
2. Designsomethingadaptabletonewneeds.
3. Buildittolastalongtime.
October24,2016©2016,http://www.perfsonar.net 6
![Page 7: New 20160926 - Introducing pScheduler - Feit · 2016. 10. 24. · Introducing pScheduler perfSONAR’sNew Scheduler Presented by the perfSONAR Project ... • Round-Trip Ping ...](https://reader036.fdocuments.in/reader036/viewer/2022081617/604a71f004b410263575e2e1/html5/thumbnails/7.jpg)
NOMENCLATUREANDCONCEPTS
October24,2016©2016,http://www.perfsonar.net 7
![Page 8: New 20160926 - Introducing pScheduler - Feit · 2016. 10. 24. · Introducing pScheduler perfSONAR’sNew Scheduler Presented by the perfSONAR Project ... • Round-Trip Ping ...](https://reader036.fdocuments.in/reader036/viewer/2022081617/604a71f004b410263575e2e1/html5/thumbnails/8.jpg)
pScheduler Terminology• Test Definitionofameasurement• Participant Pointinvolvedinthetest• Tool Programthatdoesthemeasuring• Run Themakingofameasurement• Result Theendproductofarun• Archiver Awaytosendresulttostorage• Task AjobforpScheduler todo
October24,2016©2016,http://www.perfsonar.net 8
![Page 9: New 20160926 - Introducing pScheduler - Feit · 2016. 10. 24. · Introducing pScheduler perfSONAR’sNew Scheduler Presented by the perfSONAR Project ... • Round-Trip Ping ...](https://reader036.fdocuments.in/reader036/viewer/2022081617/604a71f004b410263575e2e1/html5/thumbnails/9.jpg)
TestsintheAbstract• Measurementhasbeenabstracted.– Testparameters Standardformat– Result foreachtest.
• Testsdescribetheparameters,toolsdothemeasuring.
• Tool-specificfeaturesareaccommodated.October24,2016©2016,http://www.perfsonar.net 9
![Page 10: New 20160926 - Introducing pScheduler - Feit · 2016. 10. 24. · Introducing pScheduler perfSONAR’sNew Scheduler Presented by the perfSONAR Project ... • Round-Trip Ping ...](https://reader036.fdocuments.in/reader036/viewer/2022081617/604a71f004b410263575e2e1/html5/thumbnails/10.jpg)
WHAT’SNEW
October24,2016©2016,http://www.perfsonar.net 10
![Page 11: New 20160926 - Introducing pScheduler - Feit · 2016. 10. 24. · Introducing pScheduler perfSONAR’sNew Scheduler Presented by the perfSONAR Project ... • Round-Trip Ping ...](https://reader036.fdocuments.in/reader036/viewer/2022081617/604a71f004b410263575e2e1/html5/thumbnails/11.jpg)
Extensibility• ManyofthethingsBWCTLandotherpartsofperfSONARusedtodointernallyhavebeendelegatedtoplugins:– Tests Measurenewthings.– Tools Usenewthingstodomeasurements.– Archivers Sendresultswhereyouwant.– Otherthingstocomeinthefuture
October24,2016©2016,http://www.perfsonar.net 11
![Page 12: New 20160926 - Introducing pScheduler - Feit · 2016. 10. 24. · Introducing pScheduler perfSONAR’sNew Scheduler Presented by the perfSONAR Project ... • Round-Trip Ping ...](https://reader036.fdocuments.in/reader036/viewer/2022081617/604a71f004b410263575e2e1/html5/thumbnails/12.jpg)
Extensibility• DocumentedAPI– Therewillbetraining.• Minimalinvolvementfromcoredevelopmentteamrequired– Weaskearlyadopterstoworkwithus
• Supportfornicheapplications– DomorewithyourperfSONARdeployments
• Shorterdevelopmentcycle
October24,2016©2016,http://www.perfsonar.net 12
![Page 13: New 20160926 - Introducing pScheduler - Feit · 2016. 10. 24. · Introducing pScheduler perfSONAR’sNew Scheduler Presented by the perfSONAR Project ... • Round-Trip Ping ...](https://reader036.fdocuments.in/reader036/viewer/2022081617/604a71f004b410263575e2e1/html5/thumbnails/13.jpg)
TestsandTools• Throughput Iperf2,Iperf3,Nuttcp• Latency OWAMP,Powstream• Round-Trip Ping• Trace Traceroute,tracepath,ParisTraceroute• DevelopmentandDemo:– Simplestream Simplestreamer– Idle Sleep,Snooze
October24,2016©2016,http://www.perfsonar.net 13
![Page 14: New 20160926 - Introducing pScheduler - Feit · 2016. 10. 24. · Introducing pScheduler perfSONAR’sNew Scheduler Presented by the perfSONAR Project ... • Round-Trip Ping ...](https://reader036.fdocuments.in/reader036/viewer/2022081617/604a71f004b410263575e2e1/html5/thumbnails/14.jpg)
ToolSelection• JustOne owamp• PreferredOrder iperf3,iperf2• Automatic– Toolavailabilityandhard-wiredpreferences(e.g.,traceroute overtracepath).
– Allowssmoothdeprecationofoldertoolsandeasyintroductionofnewones.
October24,2016©2016,http://www.perfsonar.net 14
![Page 15: New 20160926 - Introducing pScheduler - Feit · 2016. 10. 24. · Introducing pScheduler perfSONAR’sNew Scheduler Presented by the perfSONAR Project ... • Round-Trip Ping ...](https://reader036.fdocuments.in/reader036/viewer/2022081617/604a71f004b410263575e2e1/html5/thumbnails/15.jpg)
Archivers• Shipping Esmond,Syslog• DevandDemo BitBucket,Failer• NearFuture HTTP,RabbitMQ,Others
• Archiverscanbeselectedpertaskorappliedgloballytoeveryresult.
October24,2016©2016,http://www.perfsonar.net 15
![Page 16: New 20160926 - Introducing pScheduler - Feit · 2016. 10. 24. · Introducing pScheduler perfSONAR’sNew Scheduler Presented by the perfSONAR Project ... • Round-Trip Ping ...](https://reader036.fdocuments.in/reader036/viewer/2022081617/604a71f004b410263575e2e1/html5/thumbnails/16.jpg)
RepetitiveTesting• Bakedintothecoreofthesystem.• Nowpartofschedulingrunsforevery task.• Alltasksworkthesameway:– Somerepeat,somedon’t.– Alltestfeaturesareavailable,repeatingornot.– Nospecialdevelopmentrequiredfornewtestsortools.
October24,2016©2016,http://www.perfsonar.net 16
![Page 17: New 20160926 - Introducing pScheduler - Feit · 2016. 10. 24. · Introducing pScheduler perfSONAR’sNew Scheduler Presented by the perfSONAR Project ... • Round-Trip Ping ...](https://reader036.fdocuments.in/reader036/viewer/2022081617/604a71f004b410263575e2e1/html5/thumbnails/17.jpg)
LimitSystem• Allnew– Therewillbetraining• FourPhases– Identification Who’sasking?– Classification Whatgroup(s)?– Limits Whataretherestrictions?– Application Forcetesttoconformto…
October24,2016©2016,http://www.perfsonar.net 17
![Page 18: New 20160926 - Introducing pScheduler - Feit · 2016. 10. 24. · Introducing pScheduler perfSONAR’sNew Scheduler Presented by the perfSONAR Project ... • Round-Trip Ping ...](https://reader036.fdocuments.in/reader036/viewer/2022081617/604a71f004b410263575e2e1/html5/thumbnails/18.jpg)
LimitSystem• Identification Who’sasking?– CIDRs(explicitlist,download/update),ReverseDNS,Bogons/Martians,Hints(IP,protocol),Forced
• Classification Whatgroups?– E.g.,bogons,selectedIPsandalistdownloadedfromIDSfallintoahostile classification.
– IPsinadownloadedlistofR&ECIDRsfallintoaresearch classification.
October24,2016©2016,http://www.perfsonar.net 18
![Page 19: New 20160926 - Introducing pScheduler - Feit · 2016. 10. 24. · Introducing pScheduler perfSONAR’sNew Scheduler Presented by the perfSONAR Project ... • Round-Trip Ping ...](https://reader036.fdocuments.in/reader036/viewer/2022081617/604a71f004b410263575e2e1/html5/thumbnails/19.jpg)
LimitSystem• Limits Whataretherestrictions?– Testtypeandparameters,scheduleattributes
• Applications Howarethelimitsapplied?– Denyrequestsinthehostile classification– Allowrequestsintheresearch classificationthatpasslimitsresearch-throughput orinert-tests
October24,2016©2016,http://www.perfsonar.net 19
![Page 20: New 20160926 - Introducing pScheduler - Feit · 2016. 10. 24. · Introducing pScheduler perfSONAR’sNew Scheduler Presented by the perfSONAR Project ... • Round-Trip Ping ...](https://reader036.fdocuments.in/reader036/viewer/2022081617/604a71f004b410263575e2e1/html5/thumbnails/20.jpg)
ComputerandHumanInterfaces• REST– Allotherinterfacesoperatethroughthisone.– 100%JavaScriptObjectNotation(JSON)
• Command-Line– Notidenticaltobw* familyofcommandsbutnotadifficultadjustment
• GUI(Future)October24,2016©2016,http://www.perfsonar.net 20
![Page 21: New 20160926 - Introducing pScheduler - Feit · 2016. 10. 24. · Introducing pScheduler perfSONAR’sNew Scheduler Presented by the perfSONAR Project ... • Round-Trip Ping ...](https://reader036.fdocuments.in/reader036/viewer/2022081617/604a71f004b410263575e2e1/html5/thumbnails/21.jpg)
CLIFeatures• Exportandimporttasksto/fromJSON– Editparametersonthefly
• Watchrepeatingtasksastheyrun• Retrievepreviously-storedresults• Querytheschedule
October24,2016©2016,http://www.perfsonar.net 21
![Page 22: New 20160926 - Introducing pScheduler - Feit · 2016. 10. 24. · Introducing pScheduler perfSONAR’sNew Scheduler Presented by the perfSONAR Project ... • Round-Trip Ping ...](https://reader036.fdocuments.in/reader036/viewer/2022081617/604a71f004b410263575e2e1/html5/thumbnails/22.jpg)
OtherFeatures• Per-taskreferencedata– AddarbitraryJSONtoanytask.– QueryagainstthisandothertaskparametersviatheRESTAPI
• Extensivediagnosticsanddebug– Reducedemailvolleysrequiredtosolveproblems
October24,2016©2016,http://www.perfsonar.net 22
![Page 23: New 20160926 - Introducing pScheduler - Feit · 2016. 10. 24. · Introducing pScheduler perfSONAR’sNew Scheduler Presented by the perfSONAR Project ... • Round-Trip Ping ...](https://reader036.fdocuments.in/reader036/viewer/2022081617/604a71f004b410263575e2e1/html5/thumbnails/23.jpg)
BackwardCompatibility• pScheduler isincompatiblewithBWCTL• BWCTLwillbekeptatleastthroughfall,2017• Duringthetransition:– ToolpluginsthatcantalktoBWCTL-onlysystemsforthroughput,latency,round-triptimeandtraceroute
– perfSONAR3.xRegularTestingandMeshConfig havebeenadaptedtoworkwithpScheduler.
October24,2016©2016,http://www.perfsonar.net 23
![Page 24: New 20160926 - Introducing pScheduler - Feit · 2016. 10. 24. · Introducing pScheduler perfSONAR’sNew Scheduler Presented by the perfSONAR Project ... • Round-Trip Ping ...](https://reader036.fdocuments.in/reader036/viewer/2022081617/604a71f004b410263575e2e1/html5/thumbnails/24.jpg)
TECHNICALFEATURES
October24,2016©2016,http://www.perfsonar.net 24
![Page 25: New 20160926 - Introducing pScheduler - Feit · 2016. 10. 24. · Introducing pScheduler perfSONAR’sNew Scheduler Presented by the perfSONAR Project ... • Round-Trip Ping ...](https://reader036.fdocuments.in/reader036/viewer/2022081617/604a71f004b410263575e2e1/html5/thumbnails/25.jpg)
RelationalDatabaseBackend• PostgreSQL9.5– Selectedforscaleup/down,maturityandfullfeatureset
• Heavyliftingthroughqueriesinsteadofhand-rolledcode
• Integrityfeaturescutdownonbugsandconsistencyproblemsinducedbysoftwarebugsandsystemcrashes.
• Upgradeprocessis100%transactional.
October24,2016©2016,http://www.perfsonar.net 25
![Page 26: New 20160926 - Introducing pScheduler - Feit · 2016. 10. 24. · Introducing pScheduler perfSONAR’sNew Scheduler Presented by the perfSONAR Project ... • Round-Trip Ping ...](https://reader036.fdocuments.in/reader036/viewer/2022081617/604a71f004b410263575e2e1/html5/thumbnails/26.jpg)
Plug-inArchitecture• APIforextendingpartsofthesystem• NomodificationstopScheduler needed• Processboundary– Developinanylanguage– AllcommunicationisJSONviastandardI/O– pScheduler isprotectedfromplug-in-relatedcrashes
October24,2016©2016,http://www.perfsonar.net 26
![Page 27: New 20160926 - Introducing pScheduler - Feit · 2016. 10. 24. · Introducing pScheduler perfSONAR’sNew Scheduler Presented by the perfSONAR Project ... • Round-Trip Ping ...](https://reader036.fdocuments.in/reader036/viewer/2022081617/604a71f004b410263575e2e1/html5/thumbnails/27.jpg)
SupportingCode• WritteninPython2.x– Well-usedandwell-understoodbythecommunity
• Recommendedlanguageforplug-ins– Notrequired– pscheduler Pythonmodulewillmakecommonutilitiesavailabletospeeddevelopment(e.g.,inhalingJSON,standardexitmethods,saferunningofexternalprocesses).
October24,2016©2016,http://www.perfsonar.net 27
![Page 28: New 20160926 - Introducing pScheduler - Feit · 2016. 10. 24. · Introducing pScheduler perfSONAR’sNew Scheduler Presented by the perfSONAR Project ... • Round-Trip Ping ...](https://reader036.fdocuments.in/reader036/viewer/2022081617/604a71f004b410263575e2e1/html5/thumbnails/28.jpg)
AQUICKLOOKATTHECLI
October24,2016©2016,http://www.perfsonar.net 28
![Page 29: New 20160926 - Introducing pScheduler - Feit · 2016. 10. 24. · Introducing pScheduler perfSONAR’sNew Scheduler Presented by the perfSONAR Project ... • Round-Trip Ping ...](https://reader036.fdocuments.in/reader036/viewer/2022081617/604a71f004b410263575e2e1/html5/thumbnails/29.jpg)
% pscheduler task throughput \--dest ps.bar.net --duration PT15S \--interval PT3S
Submitting task...Task URL:
https://ps.foo.net/pscheduler/tasks/9d30dbf7-746f-4fef-9640-814896b2b8a5Fetching first run...Next run:https://ps.foo.net/pscheduler/tasks/9d30dbf7-746f-4fef-9640-814896b2b8a5/runs/a785d67e-08af-4614-ab91-593425ee04d5
October24,2016©2016,http://www.perfsonar.net 29
![Page 30: New 20160926 - Introducing pScheduler - Feit · 2016. 10. 24. · Introducing pScheduler perfSONAR’sNew Scheduler Presented by the perfSONAR Project ... • Round-Trip Ping ...](https://reader036.fdocuments.in/reader036/viewer/2022081617/604a71f004b410263575e2e1/html5/thumbnails/30.jpg)
Starts 2016-09-12T01:29:06Z (~3 seconds)
Ends 2016-09-12T01:29:26Z (~19 seconds)Waiting for result...
* Stream ID 4Interval Throughput Retransmits Current Window 0.0 - 3.0 854.60 Mbps 630 1.40 MBytes 3.0 - 6.0 599.98 Mbps 12 849.98 KBytes
6.0 - 9.0 551.65 Mbps 0 923.82 KBytes9.0 - 12.0 575.41 Mbps 0 931.06 KBytes12.0 - 15.0 464.74 Mbps 3 750.06 KBytes
SummaryInterval Throughput Retransmits
0.0 - 15.0 609.29 Mbps 645
October24,2016©2016,http://www.perfsonar.net 30
![Page 31: New 20160926 - Introducing pScheduler - Feit · 2016. 10. 24. · Introducing pScheduler perfSONAR’sNew Scheduler Presented by the perfSONAR Project ... • Round-Trip Ping ...](https://reader036.fdocuments.in/reader036/viewer/2022081617/604a71f004b410263575e2e1/html5/thumbnails/31.jpg)
FINALLY…
October24,2016©2016,http://www.perfsonar.net 31
![Page 32: New 20160926 - Introducing pScheduler - Feit · 2016. 10. 24. · Introducing pScheduler perfSONAR’sNew Scheduler Presented by the perfSONAR Project ... • Round-Trip Ping ...](https://reader036.fdocuments.in/reader036/viewer/2022081617/604a71f004b410263575e2e1/html5/thumbnails/32.jpg)
BearWithUs• pScheduler representsaverysignificantchange.
• Weexpecttherewillbesometeething.
October24,2016©2016,http://www.perfsonar.net 32
![Page 33: New 20160926 - Introducing pScheduler - Feit · 2016. 10. 24. · Introducing pScheduler perfSONAR’sNew Scheduler Presented by the perfSONAR Project ... • Round-Trip Ping ...](https://reader036.fdocuments.in/reader036/viewer/2022081617/604a71f004b410263575e2e1/html5/thumbnails/33.jpg)
Training,SeminarsandShortSubjects• pScheduler Overview(LongVersion)• Command-LineInterface• PluginDevelopment• Archiving Moretocome.
• Announcementsonperfsonar-users
October24,2016©2016,http://www.perfsonar.net 33
![Page 34: New 20160926 - Introducing pScheduler - Feit · 2016. 10. 24. · Introducing pScheduler perfSONAR’sNew Scheduler Presented by the perfSONAR Project ... • Round-Trip Ping ...](https://reader036.fdocuments.in/reader036/viewer/2022081617/604a71f004b410263575e2e1/html5/thumbnails/34.jpg)
Feedback• Yourcomments,questionsandideasarethesourceofmanypScheduler features.– Pleasekeepthemcoming.
• perfSONARDeveloperMailingList:https://lists.internet2.edu/sympa/info/perfsonar-developer
• Emailmedirectly:[email protected],2016©2016,http://www.perfsonar.net 34
![Page 35: New 20160926 - Introducing pScheduler - Feit · 2016. 10. 24. · Introducing pScheduler perfSONAR’sNew Scheduler Presented by the perfSONAR Project ... • Round-Trip Ping ...](https://reader036.fdocuments.in/reader036/viewer/2022081617/604a71f004b410263575e2e1/html5/thumbnails/35.jpg)
TheEnd
Questions?Comments?
October24,2016©2016,http://www.perfsonar.net 35