Stackless Python in EVE Kristján Valur Jónsson [email protected] CCP Games inc

Click here to load reader

  • date post

  • Category


  • view

  • download


Embed Size (px)

Transcript of Stackless Python in EVE Kristján Valur Jónsson [email protected] CCP Games inc

  • Stackless Python in EVEKristjn Valur [email protected] Games inc.

  • EVEMMORPG Space gameClient / serverSingle shard massive server120.000 active players, >24.000 concurrent usersWorld concurrency record on a shardRelies on Stackless Python

  • The Tranquility cluster400 GHz CPU / 200 Gb RAM2 Routers (CISCO Alteon)14 Proxy servers (IBM Blade) 55 Sol servers (IBM x335)2 DB servers (clustered, IBM Brick x445)FastT600 Fiber, 56 x FC 15k disks, DS4300 + 3*EXP700Windows 2000, MS SQL ServerCurrently being upgradedAMD x64

  • EVE ArchitectureCOM-like basic architecturePython tighly integrated at an early stageHome-grown wrapping of BLUE objects






    blue modules



    python modules

  • Stackless PythonTaskletsThreads of execution. Not OS threadsLightweightNo pre-emptionChannelsTasklet rendezvous pointData passingSchedulingSynchronization

  • Stackless?No C stackPython stack in linked frame objectsTasklet switching by swapping frame chainCompromisestackless where possible.C stack whisked away if necessary



    def MyMain(self): cmodule.Frobnicate(self.Callback)

    PyFrobnicate() { PyObject_CallObject(cb, ); }

    def Callback(self):

    Stack switch

  • Channels




    Tasklet 1

    Tasklet 2

  • Channel semanticsSend on a channel with no receiver blocks tasklet.Send on a channel with a (blocked) receiver, suspends tasklet and runs receiver immediately. Sender runs again in due course.Symmetric wrt. Send and Receive.balance, can have a queue of readers or writers.Conceptually similar to Unix pipes

  • Channel semantics, cont.Scheduling semantics are precise:A blocked tasklet is run immediatelyUsable as a building block:semaphoresmutexcritical sectioncondition variables

  • Stackless in EVEBLUE foundation: robust, but cumbersomeRADStackless Python: Python and so much moreEVE is inconceivable without StacklessEveryone is a programmer

  • The main loopEstablish stackless contextint WinMain(...) {PyObject *myApp = new EveApp();PyObject *r = PyStackless_CallMethod_Main(MyApp, WinMain, 0);return PyInt_AsLong( r );

  • The main loop cont.Regular Windows message loopRuns in Stackless contextThe Main Tasklet

    PyObject* EveApp::WinMain(PyObject *self, PyObject *args) { PyOS->ExecFile("script:/sys/"); MSG msg; while(PeekMessage(&msg, 0, 0, 0, PM_REMOVE)){TranslateMessage(&msg);DispatchMessage(&msg); } for (TickIt i = mTickers.begin(; i != mTickers.end(); i++)i->mCb->OnTick(mTime, (void*)taskname);


  • import bluedef Startup(): import service srvMng = service.ServiceManager() run = ["dataconfig", "godma", ui", ] srvMng.Run(run)

    #Start up the client in a tasklet!if CheckDXVersion(): import blue blue.pyos.CreateTasklet(Startup, (), {})

  • TickersTickers are BLUE modules:Trinity (the renderer)NetclientDB (on the server)AudioPyOS (special python services)

  • The PyOS tick:Runs fresh tasklets(sleepers awoken elsewhere)

    Tick() { mSynchro->Tick() PyObject *watchdogResult; do {watchdogResult = PyStackless_RunWatchdog(20000000);if (!watchdogResult)PyFlushError("PumpPython::Watchdog");Py_XDECREF(watchdogresult); } while (!watchdogResult);

  • blue.pyos.synchroSynchro:Provides Thread-like tasklet utilities:Sleep(ms)Yield()BeNice()

  • blue.pyos.synchro cont.Sleep: A python script makes the call blue.pyos.Sleep(200)C++ code runs:Main tasklet checksleeper = New Sleeper(); mSleepers.insert(sleeper); PyObject *r = PyChannel_Receive(sleeper->mChannel);Another tasklet runs

  • blue.pyos.synchro, ctd.Main tasklet in windows loop enters PyOS::Tick()mSleepers are examined for all that are due we do:mSleepers.remove(sleeper); PyChannel_Send(sleepers.mChannel, Py_NONE);Main tasklet is suspended (but runnable), sleeper runs.

  • Points to note:A tasklet goes to sleep by calling PyChannel_Receive() on a channel which has no pending sender.It will sleep there (block) until someone sendsTypically the main tasklet does this, doing PyChannel_Send() on a channel with a readerErgo: The main tasklet may not block

  • Socket ReceiveUse Windows asynchronous file APIProvide a synchronous python API. A python script calls Read().Tasklet may be blocked for a long time, (many frames) other tasklets continue running.Do this using channels.

  • Receive, cont.Python script runs: foo, bar = socket.Read()C code executes the request:Request *r = new Request(this); WSAReceive(mSocket, ); mServe->insert( r ); PyChannel_Receive(r->mChannel);Tasklet is suspended

  • Receive, cont.Socket server is ticked from main loopFor all requests that are marked completed, it transfers the data to the sleeping tasklets:PyObject *r = PyString_FromStringAndSize(req->mData, req->mDataLen); PyChannel_Send(req->mChannel, r); Py_DECREF(data); delete req;The sleeping tasklet wakes up, main tasklet is suspended (but runnable)

  • Receive completed



    PyChannel_Send(mReceiver, data);

    Cmodule::Work(PyObject *cb)

    Stack switch

    def TaskletMain():

    Stack switch

    def ProcessStuff:

    def Callback():

    Socket::Receive(PyObject *cb)

    d = PyChannel_Receive(mReceiver);

  • Main TaskletThe one running the windows loopCan be suspended, allowing other tasklets to runCan be blocked, as long as there is another tasklet to unblock it (dangerous)Is responsible for waking up Sleepers, Yielders, IO tasklets, etc. therefore cannot be one of themIs flagged as non-blockable (stackless.get_current().block_trap = True)

  • Channel magicChannels perform the stackless context switch.If there is a C stack in the call chain, it will magically swap the stacks.Your entire C stack (with C and python invocations) is whisked away and stored, to be replaced with a new one.This allows stackless to simulate cooperative multi-threading

  • Co-operative multitaskingContext is switched only at known points.In Stakcless, this is channel.send() and channel.receive()Also synchro.Yield(), synchro.Sleep(), BeNice(), socket and DB ops, etc.No unexpected context switchesAlmost no race conditionsProgram like you are single-threadedVery few exceptions.This extends to C state too!

  • TaskletsTasklets are cheapUsed liberally to reduce perceived lagUI events forked out to taskletsA click can have heavy consequences.Heavy logicDB AccessNetworks accessspecial rendering tasks forked out to tasklets.controlling an audio tracktasklet it outUse blue.pyos.synchro.BeNice() in large loops

  • Example: UI Event:Main tasklet receives window messages such as WM_CLICKTrinity invokes handler on UI elements or global handlerHandler tasklets out any action to allow main thread to continue immediately.

    def OnGlobalUp(self, *args): if not self or self.destroyed: return mo = eve.triapp.uilib.mouseOver if mo in self.children: class Action(xtriui.QuickDeco): def _OnClick(self, *args): pass

  • Thats allFor more info:http://www.ccpgames.comhttp://www.eve-online.comhttp://[email protected]