RodModule Class Library

T. [email protected]

ROD Software Workshop18-19 July 2002

RodModule Class LibraryRodModule Class Library

Tom MeyerIowa State [email protected]



ContextContext

RodModule is the class library providing driver routines for the ROD VME module.

ConsumerProcess 2

ConsumerProcess 2

Other Hardware

AncillaryManagers

ROD PrimList Managers

ROD PrimList Managers

ROD Module

VME Interface

ROD Module

VME Interface

ActionList Handler

CommandPackage

ROD C rateManagers

(one per crate)

CrateManager

ROD Module

VME Interface

TIM Module

VME Interface

Other VME

VME Interface

Other Hardware

Run Control

AncillaryManagers Data Manager

OtherConsumerProcesses

Real TimeMonitorProcess

User

RCC Process

SCT/PixelHost P rocess

ROD DA Q Overview :Con tro l and D ata F low

State Trans itionsand res pons es

Commandsand res pons es

Pr imitiv es Rep lies

Selec tedData ty pes

Selec tedData ty pes

Ev ent data (a ll ty pes )Pr imLis t Reply Lis t

Pr imL is t Rep ly Lis t/Ev ent data

Commands

Detector ConfigData Bases

DetectorControl (DDC )

Other Services:GUI (IUI)Messaging (MRS)Information D istributor (IS )E lectronic Logbook

Version 0.5T. Meyer, et al.

15 Feb 02

Key:ORANGE : DAQ-1GREEN: HOST PROCESSBLUE: RC C PROC ESSRED : CORBA (OR TCP/IP)YELLOW : CONSUMERBLACK: OTHER

.

.



Present StatusPresent Status

Spent 4 weeks in Berkeley in April, mostly wrestling with Linux demons.

Brought SBC back to Ames, got it working in a 6u crate. 9u Crate, ROD, and TIM arrived in Ames in early June. Spent 2 weeks trying to get 9u crate to work. Succeeded on last day

before shipping it back to LBNL. When I get ROD back, I am ready to begin debugging the RodModule

software.



VME Module Insertion KitVME Module Insertion Kit



Class DiagramClass Diagram

Vm eIn terface

long m _ifaceIdbool exceptionTrapping

getVmePort()deleteVmePort()...

1

V ersion 1 .611 Ju l 2002

T . M eye r

R O D M odu le C lasses

R odM odule

long m _num Slaveslong m _finBufferS izestring m _m asterIm ageNam estring m _slaveIm ageNam e[4]

void in itialize()void status()void loadMaster()sendP rim List()HP I AccessFP GA AccessRegis ter AccessTextBuff Access

R odP rim List

ulong m _indexUINT32* m _bufferlong m _bufferS ize

1

R ccIn terfaceN IV isaInte rface

V m eP ort

UINT32 m _baseAddressaddrMode m _addrModeint m _m apSize

1 ..n

V m eM odu le

ulong m _baseAddressVm eInterface & m _ourInterfaceVm ePort & m _m yVm ePort

void in itialize()=0void status()=0

1

1 ..n

1

readxx()writexx()readBlockxx()writeBlockxx()

1

R odP rim itive

0 ..n

bufferBuild()insert()clear()

long m _lengthlong m _indexlong m _idlong* m _body

RodO utL is t

long m _lengthUINT32* m _buffer

1

1



Methods Inherited from VmeModuleMethods Inherited from VmeModule initialize()

Configure all FPGA registers (bit 6 = 0x20) Read ROD serial number Initialize Master HPIC register Wait for MasterDSP init Read reply buffer addresses Initialize Slave HPIC registers Initialize state variables

reset() Reset all FPGAs and DSPs (bit 7 = 0x40) Clear IDRAM Reset all DSPs Reinitialize state variables

status() Dummy for now Soon: write to standard output Eventually: send status message to MRS Do we need different levels of status information? What information do we want?



VME AddressesVME Addresses

DSP Host Port Interface (HPI) Command Register (HPIC): Sets big-endian. Set once at start of program

and leave it alone. Address Register (HPIA): Address in DSP space to be accessed. Data Registers: Data port

o HPID_AUTO: Auto increments HPIA after accesso HPID_NOAUTO: Leaves HPIA unchanged

FPGA Status and Control Registers At present these are unused by RodModule. They access the FPGAs directly

for configuration, reset, status, halt, etc. FPGA_CONTROL_REG_REL_ADDR[8] FPGA_STATUS_REG_REL_ADDR[8]

RodM odule

Prog R esetM G R32/16

M asterDS P

RO D32 bits

32 b its 16 b its

16 b its

HPI



DSP AddressesDSP Addresses Primitive buffers

PRIM_BUFF_BASE PRIM_BUFF_SIZE

Reply buffers REPLY_BUFF_BASE REPLY_BUFF_SIZE

Text Buffers TEXT_BUFF_SIZE ERR_BUFF_BASE INFO_BUFF_BASE DIAG_BUFF_BASE XFER_BUFF_BASE

ROD Status Registers STATUS_REG[3]

VME Command Registers COMMAND_REG[2]



ConstructorConstructor

RodModule( unsigned long baseAddr, unsigned long mapSize, VmeInterface & ourInterface, long slot, long numSlaves);

There are no copy or assignment operators. The constructor creates a VmePort object and registers it with its

VmeInterface object.



Shared Header FilesShared Header Files The C-code DSP programs and the TestStand program use shared

header files to specify addresses and bit assignments. We would like to use these same headers for C++.

We want to specify these details in only one place and propagate the information.

In C, these are set with #define statements. C++ can use C headers, but it is more in keeping with C++ principles

to declare the constants as const unsigned long variables. This lets the compiler catch any attempts to change them, and it does not expose the code to global text substitutions from #define statements.

I have wrapped the C headers with C++ header files that expose only data of interest to RodModule and use const unsigned long values. This also isolates C++ code from changes in C headers, and hides C-only parts of the header.



VME Address WrapperVME Address Wrapper//File: RodVmeAddresses.h#ifndef SCTPIXELROD_RODVMEADDRESSES_H #define SCTPIXELROD_RODVMEADDRESSES_H/*!* RodVmeAddresses.h is a wrapper around the testBench C

code header filevmeAddressMap.h. This wrapper adapts things to appear

more like C++ and todeclare things const so the compiler can catch attempts to

change them in thecode.*/namespace SctPixelRod {

#include "../VmeInterface/processor.h"#include "vmeAddressMap.h"

// Give some shorter aliases to the HPI registersconst unsigned long HPIC =

HPI_CONTROL_REG_REL_ADDR;const unsigned long HPIA =

HPI_ADDRESS_REG_REL_ADDR;const unsigned long HPID_AUTO =

HPI_DATA_REG_WITH_AUTOINC_REL_ADDR;const unsigned long HPID_NOAUTO =

HPI_DATA_REG_WITHOUT_AUTOINC_REL_ADDR;

//FPGA Program/Reset Manager - 32bit size registers - valid only low 8 bits!

const unsigned long FPGA_CONTROL_REG_REL_ADDR[8] = {

FPGA_CONTROL_REG_0_REL_ADDR, // fpga cnfg control reg

[ . . .] FPGA_CONTROL_REG_7_REL_ADDR // };const unsigned long MAX_HPID_WORD_ELEMENTS =

0x3FFFE; // 32b words - 20 bits available for address autoincrement => hwil

const unsigned long FPGA_STATUS_REG_REL_ADDR[8] = {

FPGA_STATUS_REG_0_REL_ADDR, // fpga cnfg status reg

[ . . .] FPGA_STATUS_REG_7_REL_ADDR // flash read data

reg}; } // End namespace SctPixelRod#endif // SCTPIXELROD_RODVMEADDRESSES_H



DSP Address WrapperDSP Address Wrapper//File: RodDspAddresses.h#ifndef SCTPIXELROD_RODDSPADDRESSES_H #define SCTPIXELROD_RODDSPADDRESSES_H/*!* RodDspAddresses.h is a wrapper around the testBench C

code header filememoryPartitions.h. This wrapper adapts things to appear

more like C++ and todeclare things const so the compiler can catch attempts to

change them in thecode.*/

namespace SctPixelRod {#include "../VmeInterface/processor.h"#include "memoryPartitions.h"// Use only the names in this file in C++ code.// Give some shorter aliases to the primitive list and reply

list buffersconst unsigned long PRIM_BUFF_BASE =

HOST_PRM_BFR_BASE;const unsigned long PRIM_BUFF_SIZE =

HOST_PRM_BFR_SZ;const unsigned long REPLY_BUFF_BASE =

HOST_REP_BFR_BASE;const unsigned long REPLY_BUFF_SIZE =

HOST_REP_BFR_SZ;

// Give aliases to text buffersconst unsigned long TEXT_BUFF_SIZE = TXT_BFR_SZ;const unsigned long ERR_BUFF_BASE =

ERR_BFR_BASE;const unsigned long INFO_BUFF_BASE =

INFO_BFR_BASE;const unsigned long DIAG_BUFF_BASE =

DIAG_BFR_BASE;const unsigned long XFER_BUFF_BASE =

XFER_BFR_BASE;

// Give aliases to VmeCommand and RodStatus registersconst unsigned long STATUS_REG[3] =

{STATUS_REG_0, STATUS_REG_1, STATUS_REG_2};

const unsigned long COMMAND_REG[2] = {COMMAND_REG_0, COMMAND_REG_1};

} // End namespace SctPixelRod

#endif // SCTPIXELROD_RODDSPADDRESSES_H



Register Bits WrapperRegister Bits Wrapper//File: RodRegisterBits.h#ifndef SCTPIXELROD_RODREGISTERBITS_H #define SCTPIXELROD_RODREGISTERBITS_H/*!* RodRegisterBits.h is a wrapper around the testBench

C code header filecomRegDfns.h. This wrapper adapts things to appear

more like C++ and todeclare things const so the compiler can catch attempts

to change them in thecode.*/

namespace SctPixelRod {

#include "../VmeInterface/processor.h"#include "comRegDfns.h"

// Give aliases to the text buffer bits in the VmeCommand and Status registers

// Use only these names in C++ code.

// StatusRegister[0] bitsconst unsigned long OUTLIST_READY =

SR_OUT_LIST_RDY;

const unsigned long TEXT_BUFF_NOT_EMPTY[4] = { SR_TXT_BUFF_NE(0), // ERR Buffer SR_TXT_BUFF_NE(1), // INFO Buffer SR_TXT_BUFF_NE(2), // DIAG Buffer SR_TXT_BUFF_NE(3) // XFER Buffer};const unsigned long SR_TEXT_BIT_MASK[4] =

{1<<SR_TXT_BUFF_NE(0), 1<<SR_TXT_BUFF_NE(1),

1<<SR_TXT_BUFF_NE(2), 1<<SR_TXT_BUFF_NE(3)};

const unsigned long SR_TEXT_MASK = SR_TEXT_BIT_MASK[0] | SR_TEXT_BIT_MASK[1] |

SR_TEXT_BIT_MASK[2] | SR_TEXT_BIT_MASK[3];

// VmeCommandRegister[0] bitsconst unsigned long TEXT_BUFF_READ_REQ[4] = { CR_TXT_BUFF_RR(0), // ERR Buffer CR_TXT_BUFF_RR(1), // INFO Buffer CR_TXT_BUFF_RR(2), // DIAG Buffer CR_TXT_BUFF_RR(3) // XFER Buffer};} // End namespace SctPixelRod#endif // SCTPIXELROD_RODREGISTERBITS_H



RodModule: Data MembersRodModule: Data MembersBoard Params: long m_slot; //! Slot number in VME crate unsigned long m_serialNumber; //! Board serial number VmePort* m_myVmePort; //! VME Port handle long m_numSlaves; //! Number of slave DSPs on this ROD long m_finBufferSize; //! Size of file input buffer, in bytes. Default=4096 std::string m_masterImageName; //! string holding name of master image file std::string m_slaveImageName[4]; //! string array holding names of slave image filesRegister copies: unsigned long m_vmeCommandReg[2]; //! Cached copy of VME command registers unsigned long m_rodStatusReg[3]; //! Cached copy of ROD Status registersPrimitive handling: RodOutList* m_myOutList; //! A pointer to the array holding a reply buffer PrimState m_myPrimState; //! State variable for sending primitive listsText buffer handling: TextBuffState m_myTextState; //! State variable for reading text buffers TEXT_BUFFER_TYPE m_textType; //! Text buffer type struct TXTBUFFER m_txtBuffer; //! A struct holding text buffer info from txtbuffer.h char m_textData[TEXT_BUFF_SIZE]; //! A local buffer to hold the text data unsigned long* m_textBuff[N_TXT_BUFFS]; //! Array of pointers to the text buffers in HPI space



RodModule: HPI Access MethodsRodModule: HPI Access Methods //! Load a 32-bit value into an HPI register. void hpiLoad(const unsigned long hpiReg, const unsigned long hpiValue) throw (VmeException &);

//! Fetch a 32-bit value from an HPI register. unsigned long hpiFetch(const unsigned long hpiReg) throw (VmeException &);

//! Read a buffer of wordCount 32b words from HPI void hpiRead(const unsigned long dspAddr, unsigned long buffer[], const long wordCount) throw (VmeException &, HpiException &);

//! Write a buffer of wordCount 32b words to HPI void hpiWrite(const unsigned long dspAddr, unsigned long *buffer, long wordCount) throw (VmeException &, HpiException &);

//! Write a buffer of wordCount 32b words to a slave DSP memory address void writeSlaveMem(long slaveNum, unsigned long dspAddr, unsigned long buffer[], long wordCount) throw (VmeException &, HpiException &);

//! Read a buffer of wordCount 32b words to a slave DSP memory address void readSlaveMem(long slaveNum, unsigned long dspAddr, unsigned long buffer[], long wordCount)

throw (VmeException &, HpiException &);



RodModule: VME Access MethodsRodModule: VME Access Methods

//! Load a 32-bit value into a 32-bit status or VME command register. void regLoad(unsigned long regAddr, unsigned long regValue) throw (VmeException &);

//! Fetch a 32-bit value from a 32-bit status or VME command register. unsigned long regFetch(unsigned long regAddr) throw (VmeException &);

//! Reset the Master DSP via FPGA Control Reg 2 void resetMasterDsp(); //! Reset a Slave DSP via FPGA Control Reg 2 void resetSlaveDsp(long slaveNumber);



RodModule: UtilitiesRodModule: Utilities

//! Calculate a checksum (bit-wise XOR) long checkSum(const unsigned long *sourceArray, const long wordCount); //! Go to sleep for a specified number of milliseconds void sleep(const double milliSecs);

These methods may be moved to a RodUtilities class in the future.



RodPrimitive ClassRodPrimitive Class

A simple class to hold Primitive objects Data members:

long m_length long m_index long m_id long *m_body

Methods: Only the usual data accessor methods (“get” and “put”).

Body A

PrimList Format

Version 2.313 Apr 2000

List Length

List Index

Checksum

Prim Length A

Num Prim s

Prim Length B

Body B

Prim Index A

Prim Id A

Prim Id B

Prim Index B

List Length



RodPrimList ClassRodPrimList Class Derived from the C++ Standard Template

Library (STL) list class. This gives us almost all the functionality we need for free.

Data members: unsigned long m_index unsigned long *m_buffer unsigned long m_listLength

Methods (other than data accessors) unsigned long numWords() unsigned long checkSum() void bufferBuild() void clear()

Body A

PrimList Format

Version 2.313 Apr 2000

List Length

List Index

Checksum

Prim Length A

Num Prim s

Prim Length B

Body B

Prim Index A

Prim Id A

Prim Id B

Prim Index B

List Length



RodOutList ClassRodOutList Class

Contains reply lists generated by the MDSP in response to an associated primitive list.

Structure is the same as a PrimList Data members:

long m_length unsigned long* m_body

Methods: unsigned long* getBody() long getLength()

RodModule makes no attempt to understand the reply list, it merely passes it to a user thread for further handling.



Example of Building and Sending a PrimListExample of Building and Sending a PrimList

This is a test program I will use to debug RodModule. In real RODDAQ, this functionality will be spread between processes

and threads on the host and RCC machines.



Sequence DiagramSequence Diagram

CommandThread

BkgndThread Primitive PrimList RodModule

UserCommand

Create Primitive

Add Primitive

Send PrimList

Iterate

write HPIC and HPIA

send inList

primHandler()read dspAck

if (dspAck=0 and LOADED):set inListRdy=1

STATE=IDLE

setState(LOADED)

if (dspAck=1 and EXECUTING):read outListRdy

Return(LOADED)

Clear PrimList

Delete PrimitiveIterate

setState(EXECUTING)

Primitive List:User Command to ROD

Version 1.6T. Meyer29 May 2002

OutList

return(EXECUTING)

primHandler()

if (outListRdy=1):fetch outList

setState(WAITING)

if (outListRdy=1)construct OutList

return(WAITING)

outListRdy

processoutlist

if (Outlist) delete OutListdelete OutList

Reply touser

1

2

3

4a

4b

5

6

7

8

9

10

ROD

setState(IDLE)

getOutList

return(IDLE)

read dspAck



#include <iostream>#include <ctype.h>#include "RodModule.h"#include "../commonWithDsp/primParams.h"using namespace SctPixelRod;#include "../VmeInterface/RCCVmeInterface.h"

int main() {int i,j;unsigned short value;short buf[1000];const unsigned long baseAddress13=0x0d000000; // Base address for slot 13const unsigned long mapSize=0x10000; const long slot13=13; const long numSlaves=4; RodPrimList primList(1); // Create VME interface RCCVmeInterface *vme1 = new

RCCVmeInterface();

RodModuleTest.cxx (1)RodModuleTest.cxx (1) // Create RodModule and initialize it RodModule* rod0 = new

RodModule(baseAddress13, mapSize, *vme1, slot13, numSlaves);

rod0->initialize();

// Create and Send a simple ECHO primitive long echoData[2] = {0xDEADF00D,0}; RodPrimitive* echo = new RodPrimitive(13, 1,

ECHO, echoData); primList.insert(primList.begin(), *echo); primList.bufferBuild(); rod0->sendPrimList(&primList);

// Wait for ROD to begin executing and then wait for it to finish executing

do {} while (rod0->primHandler() != PRIM_EXECUTING);

do {} while (rod0->primHandler() != PRIM_WAITING);



RodModuleTest.cxx (2)RodModuleTest.cxx (2)// Retrieve output buffer RodOutList* outList = rod0->getOutList(); // Print results (User processing of outList) UINT32 outLength = UINT32(outList-

>getLength()); unsigned long* outBody = outList->getBody(); UINT32 outIndex = UINT32(outBody[1]); UINT32 outNumPrims = outBody[2]; cout << "outLength = " << outLength << ",

outIndex = " << outIndex << ", OutNumPrims = " << outNumPrims << '\n';

UINT32 primLength = outBody[3]; UINT32 primIndex = outBody[4]; UINT32 primId = outBody[5]; cout << "primLength = ", primLength, ",

primIndex = ", primIndex, ", primId = ", primId << '\n';

cout << "ECHO data: "; for (int i=0; i=primLength-3; i++) cout <<

outBody[i+6];

// Copy to local buffer UINT32* myOutBody = new

UINT32[outLength]; for (i=0; i<outLength; i++) { myOutBody[i] = outBody[i]; }; // Clean up: clear primList, delete primitive, delete

the outList primList.clear(); delete echo; delete myOutBody; rod0->deleteOutList(); delete vme1;

return 0; }



Text BuffersText Buffers Text buffers are more useful for debugging than for data taking. If a DSP CPU crashes, we cannot execute primitives. But the on-board

extended memory interface (EMIF) circuitry still lets us read the SDRAM contents.

Text Buffers are blocks of SDRAM memory to which we can write ASCII records with traceback information, or simple diagnostic info for later analysis. (Recall debugging FORTRAN with print statements.)

Even if DSP is fine, it may be useful to pass ASCII data into a message stream (MRS) in ATLAS.

There are four text buffers defined for the Master DSP Err: for reporting fatal software error tracebacks Info: for non-priority information messages Diag: for diagnostic messages, “print” to it like we used to print out

FORTRAN debug messages. Xfer: A temporary MDSP buffer to relay text buffers from the Slave DSPs



Text Buffer HandlingText Buffer Handling

User TextThread

BkgndThread RodModule

read xxxBufferNotEmpty

set xxxBufferReadRequest

textHandler()

read buffer header

read buffer contents

STATE=IDLE

setState(RQ_SET)

clear xxxBufferReadRequest

Return(RQ_SET)

clearTxtBuffer()

Iterate

setState(READOUT)

Text Buffers:User Thread to ROD

Version 1.3T. Meyer5 Jun 2002

return(READOUT)

setState(IDLE)

bufferReady

processbuffer

1

2

3

4

5

6

7

8

9

ROD

getTxtBuffer(buffer, length, type))

textHandler()read xxxBufferNotEmpty

"1"

"0"

buffer



Future WorkFuture Work Debug what is already written

using RodModuleText (“If it compiles, publish!” – anon, from another collaboration)

Interface to DAQ-1 Front End Calibration program Add Posix threads Strengthen weak areas

status() FPGA operations exceptions bullet proofing

Phase O ne

D AQ -1Run Control

C om m andLayer

R O D M oduleD river

C++ VM E Interface

C VM E Interface

U ser

VM E M odules

VM E calls

RO D Prim itives Replies

StateT ransitions R eplies

Analysis T askand D isp lay

D ata U ser



DocumentationDocumentation

User’s Guide: will eventually be derived from this and similar presentations. Paper and web versions.

Technical reference: will be generated automatically using Doxygen. Users can generate it for themselves simply, and output can be in HTML, LaTex, and PostScript formats. Doxygen uses special comments embedded in the code along with a knowledge of C++ file structure to produce good quality documentation.

RodModule Class Library

Documents

Transcript of RodModule Class Library