ROD Software Workshop18-19 July 2002
RodModule Class LibraryRodModule Class Library
Tom MeyerIowa State [email protected]
ROD Software Workshop18-19 July 2002
ContextContext
RodModule is the class library providing driver routines for the ROD VME module.
ConsumerProcess 2
ConsumerProcess 2
Other Hardware
AncillaryManagers
ROD PrimList Managers
ROD PrimList Managers
ROD Module
VME Interface
ROD Module
VME Interface
ActionList Handler
CommandPackage
ROD C rateManagers
(one per crate)
CrateManager
ROD Module
VME Interface
TIM Module
VME Interface
Other VME
VME Interface
Other Hardware
Run Control
AncillaryManagers Data Manager
OtherConsumerProcesses
Real TimeMonitorProcess
User
RCC Process
SCT/PixelHost P rocess
ROD DA Q Overview :Con tro l and D ata F low
State Trans itionsand res pons es
Commandsand res pons es
Pr imitiv es Rep lies
Selec tedData ty pes
Selec tedData ty pes
Ev ent data (a ll ty pes )Pr imLis t Reply Lis t
Pr imL is t Rep ly Lis t/Ev ent data
Commands
Detector ConfigData Bases
DetectorControl (DDC )
Other Services:GUI (IUI)Messaging (MRS)Information D istributor (IS )E lectronic Logbook
Version 0.5T. Meyer, et al.
15 Feb 02
Key:ORANGE : DAQ-1GREEN: HOST PROCESSBLUE: RC C PROC ESSRED : CORBA (OR TCP/IP)YELLOW : CONSUMERBLACK: OTHER
.
.
ROD Software Workshop18-19 July 2002
Present StatusPresent Status
Spent 4 weeks in Berkeley in April, mostly wrestling with Linux demons.
Brought SBC back to Ames, got it working in a 6u crate. 9u Crate, ROD, and TIM arrived in Ames in early June. Spent 2 weeks trying to get 9u crate to work. Succeeded on last day
before shipping it back to LBNL. When I get ROD back, I am ready to begin debugging the RodModule
software.
ROD Software Workshop18-19 July 2002
ROD Software Workshop18-19 July 2002
VME Module Insertion KitVME Module Insertion Kit
ROD Software Workshop18-19 July 2002
Class DiagramClass Diagram
Vm eIn terface
long m _ifaceIdbool exceptionTrapping
getVmePort()deleteVmePort()...
1
V ersion 1 .611 Ju l 2002
T . M eye r
R O D M odu le C lasses
R odM odule
long m _num Slaveslong m _finBufferS izestring m _m asterIm ageNam estring m _slaveIm ageNam e[4]
void in itialize()void status()void loadMaster()sendP rim List()HP I AccessFP GA AccessRegis ter AccessTextBuff Access
R odP rim List
ulong m _indexUINT32* m _bufferlong m _bufferS ize
1
R ccIn terfaceN IV isaInte rface
V m eP ort
UINT32 m _baseAddressaddrMode m _addrModeint m _m apSize
1 ..n
V m eM odu le
ulong m _baseAddressVm eInterface & m _ourInterfaceVm ePort & m _m yVm ePort
void in itialize()=0void status()=0
1
1 ..n
1
readxx()writexx()readBlockxx()writeBlockxx()
1
R odP rim itive
0 ..n
bufferBuild()insert()clear()
long m _lengthlong m _indexlong m _idlong* m _body
RodO utL is t
long m _lengthUINT32* m _buffer
1
1
ROD Software Workshop18-19 July 2002
Methods Inherited from VmeModuleMethods Inherited from VmeModule initialize()
Configure all FPGA registers (bit 6 = 0x20) Read ROD serial number Initialize Master HPIC register Wait for MasterDSP init Read reply buffer addresses Initialize Slave HPIC registers Initialize state variables
reset() Reset all FPGAs and DSPs (bit 7 = 0x40) Clear IDRAM Reset all DSPs Reinitialize state variables
status() Dummy for now Soon: write to standard output Eventually: send status message to MRS Do we need different levels of status information? What information do we want?
ROD Software Workshop18-19 July 2002
VME AddressesVME Addresses
DSP Host Port Interface (HPI) Command Register (HPIC): Sets big-endian. Set once at start of program
and leave it alone. Address Register (HPIA): Address in DSP space to be accessed. Data Registers: Data port
o HPID_AUTO: Auto increments HPIA after accesso HPID_NOAUTO: Leaves HPIA unchanged
FPGA Status and Control Registers At present these are unused by RodModule. They access the FPGAs directly
for configuration, reset, status, halt, etc. FPGA_CONTROL_REG_REL_ADDR[8] FPGA_STATUS_REG_REL_ADDR[8]
RodM odule
Prog R esetM G R32/16
M asterDS P
RO D32 bits
32 b its 16 b its
16 b its
HPI
ROD Software Workshop18-19 July 2002
DSP AddressesDSP Addresses Primitive buffers
PRIM_BUFF_BASE PRIM_BUFF_SIZE
Reply buffers REPLY_BUFF_BASE REPLY_BUFF_SIZE
Text Buffers TEXT_BUFF_SIZE ERR_BUFF_BASE INFO_BUFF_BASE DIAG_BUFF_BASE XFER_BUFF_BASE
ROD Status Registers STATUS_REG[3]
VME Command Registers COMMAND_REG[2]
ROD Software Workshop18-19 July 2002
ConstructorConstructor
RodModule( unsigned long baseAddr, unsigned long mapSize, VmeInterface & ourInterface, long slot, long numSlaves);
There are no copy or assignment operators. The constructor creates a VmePort object and registers it with its
VmeInterface object.
ROD Software Workshop18-19 July 2002
Shared Header FilesShared Header Files The C-code DSP programs and the TestStand program use shared
header files to specify addresses and bit assignments. We would like to use these same headers for C++.
We want to specify these details in only one place and propagate the information.
In C, these are set with #define statements. C++ can use C headers, but it is more in keeping with C++ principles
to declare the constants as const unsigned long variables. This lets the compiler catch any attempts to change them, and it does not expose the code to global text substitutions from #define statements.
I have wrapped the C headers with C++ header files that expose only data of interest to RodModule and use const unsigned long values. This also isolates C++ code from changes in C headers, and hides C-only parts of the header.
ROD Software Workshop18-19 July 2002
VME Address WrapperVME Address Wrapper//File: RodVmeAddresses.h#ifndef SCTPIXELROD_RODVMEADDRESSES_H #define SCTPIXELROD_RODVMEADDRESSES_H/*!* RodVmeAddresses.h is a wrapper around the testBench C
code header filevmeAddressMap.h. This wrapper adapts things to appear
more like C++ and todeclare things const so the compiler can catch attempts to
change them in thecode.*/namespace SctPixelRod {
#include "../VmeInterface/processor.h"#include "vmeAddressMap.h"
// Give some shorter aliases to the HPI registersconst unsigned long HPIC =
HPI_CONTROL_REG_REL_ADDR;const unsigned long HPIA =
HPI_ADDRESS_REG_REL_ADDR;const unsigned long HPID_AUTO =
HPI_DATA_REG_WITH_AUTOINC_REL_ADDR;const unsigned long HPID_NOAUTO =
HPI_DATA_REG_WITHOUT_AUTOINC_REL_ADDR;
//FPGA Program/Reset Manager - 32bit size registers - valid only low 8 bits!
const unsigned long FPGA_CONTROL_REG_REL_ADDR[8] = {
FPGA_CONTROL_REG_0_REL_ADDR, // fpga cnfg control reg
[ . . .] FPGA_CONTROL_REG_7_REL_ADDR // };const unsigned long MAX_HPID_WORD_ELEMENTS =
0x3FFFE; // 32b words - 20 bits available for address autoincrement => hwil
const unsigned long FPGA_STATUS_REG_REL_ADDR[8] = {
FPGA_STATUS_REG_0_REL_ADDR, // fpga cnfg status reg
[ . . .] FPGA_STATUS_REG_7_REL_ADDR // flash read data
reg}; } // End namespace SctPixelRod#endif // SCTPIXELROD_RODVMEADDRESSES_H
ROD Software Workshop18-19 July 2002
DSP Address WrapperDSP Address Wrapper//File: RodDspAddresses.h#ifndef SCTPIXELROD_RODDSPADDRESSES_H #define SCTPIXELROD_RODDSPADDRESSES_H/*!* RodDspAddresses.h is a wrapper around the testBench C
code header filememoryPartitions.h. This wrapper adapts things to appear
more like C++ and todeclare things const so the compiler can catch attempts to
change them in thecode.*/
namespace SctPixelRod {#include "../VmeInterface/processor.h"#include "memoryPartitions.h"// Use only the names in this file in C++ code.// Give some shorter aliases to the primitive list and reply
list buffersconst unsigned long PRIM_BUFF_BASE =
HOST_PRM_BFR_BASE;const unsigned long PRIM_BUFF_SIZE =
HOST_PRM_BFR_SZ;const unsigned long REPLY_BUFF_BASE =
HOST_REP_BFR_BASE;const unsigned long REPLY_BUFF_SIZE =
HOST_REP_BFR_SZ;
// Give aliases to text buffersconst unsigned long TEXT_BUFF_SIZE = TXT_BFR_SZ;const unsigned long ERR_BUFF_BASE =
ERR_BFR_BASE;const unsigned long INFO_BUFF_BASE =
INFO_BFR_BASE;const unsigned long DIAG_BUFF_BASE =
DIAG_BFR_BASE;const unsigned long XFER_BUFF_BASE =
XFER_BFR_BASE;
// Give aliases to VmeCommand and RodStatus registersconst unsigned long STATUS_REG[3] =
{STATUS_REG_0, STATUS_REG_1, STATUS_REG_2};
const unsigned long COMMAND_REG[2] = {COMMAND_REG_0, COMMAND_REG_1};
} // End namespace SctPixelRod
#endif // SCTPIXELROD_RODDSPADDRESSES_H
ROD Software Workshop18-19 July 2002
Register Bits WrapperRegister Bits Wrapper//File: RodRegisterBits.h#ifndef SCTPIXELROD_RODREGISTERBITS_H #define SCTPIXELROD_RODREGISTERBITS_H/*!* RodRegisterBits.h is a wrapper around the testBench
C code header filecomRegDfns.h. This wrapper adapts things to appear
more like C++ and todeclare things const so the compiler can catch attempts
to change them in thecode.*/
namespace SctPixelRod {
#include "../VmeInterface/processor.h"#include "comRegDfns.h"
// Give aliases to the text buffer bits in the VmeCommand and Status registers
// Use only these names in C++ code.
// StatusRegister[0] bitsconst unsigned long OUTLIST_READY =
SR_OUT_LIST_RDY;
const unsigned long TEXT_BUFF_NOT_EMPTY[4] = { SR_TXT_BUFF_NE(0), // ERR Buffer SR_TXT_BUFF_NE(1), // INFO Buffer SR_TXT_BUFF_NE(2), // DIAG Buffer SR_TXT_BUFF_NE(3) // XFER Buffer};const unsigned long SR_TEXT_BIT_MASK[4] =
{1<<SR_TXT_BUFF_NE(0), 1<<SR_TXT_BUFF_NE(1),
1<<SR_TXT_BUFF_NE(2), 1<<SR_TXT_BUFF_NE(3)};
const unsigned long SR_TEXT_MASK = SR_TEXT_BIT_MASK[0] | SR_TEXT_BIT_MASK[1] |
SR_TEXT_BIT_MASK[2] | SR_TEXT_BIT_MASK[3];
// VmeCommandRegister[0] bitsconst unsigned long TEXT_BUFF_READ_REQ[4] = { CR_TXT_BUFF_RR(0), // ERR Buffer CR_TXT_BUFF_RR(1), // INFO Buffer CR_TXT_BUFF_RR(2), // DIAG Buffer CR_TXT_BUFF_RR(3) // XFER Buffer};} // End namespace SctPixelRod#endif // SCTPIXELROD_RODREGISTERBITS_H
ROD Software Workshop18-19 July 2002
RodModule: Data MembersRodModule: Data MembersBoard Params: long m_slot; //! Slot number in VME crate unsigned long m_serialNumber; //! Board serial number VmePort* m_myVmePort; //! VME Port handle long m_numSlaves; //! Number of slave DSPs on this ROD long m_finBufferSize; //! Size of file input buffer, in bytes. Default=4096 std::string m_masterImageName; //! string holding name of master image file std::string m_slaveImageName[4]; //! string array holding names of slave image filesRegister copies: unsigned long m_vmeCommandReg[2]; //! Cached copy of VME command registers unsigned long m_rodStatusReg[3]; //! Cached copy of ROD Status registersPrimitive handling: RodOutList* m_myOutList; //! A pointer to the array holding a reply buffer PrimState m_myPrimState; //! State variable for sending primitive listsText buffer handling: TextBuffState m_myTextState; //! State variable for reading text buffers TEXT_BUFFER_TYPE m_textType; //! Text buffer type struct TXTBUFFER m_txtBuffer; //! A struct holding text buffer info from txtbuffer.h char m_textData[TEXT_BUFF_SIZE]; //! A local buffer to hold the text data unsigned long* m_textBuff[N_TXT_BUFFS]; //! Array of pointers to the text buffers in HPI space
ROD Software Workshop18-19 July 2002
RodModule: HPI Access MethodsRodModule: HPI Access Methods //! Load a 32-bit value into an HPI register. void hpiLoad(const unsigned long hpiReg, const unsigned long hpiValue) throw (VmeException &);
//! Fetch a 32-bit value from an HPI register. unsigned long hpiFetch(const unsigned long hpiReg) throw (VmeException &);
//! Read a buffer of wordCount 32b words from HPI void hpiRead(const unsigned long dspAddr, unsigned long buffer[], const long wordCount) throw (VmeException &, HpiException &);
//! Write a buffer of wordCount 32b words to HPI void hpiWrite(const unsigned long dspAddr, unsigned long *buffer, long wordCount) throw (VmeException &, HpiException &);
//! Write a buffer of wordCount 32b words to a slave DSP memory address void writeSlaveMem(long slaveNum, unsigned long dspAddr, unsigned long buffer[], long wordCount) throw (VmeException &, HpiException &);
//! Read a buffer of wordCount 32b words to a slave DSP memory address void readSlaveMem(long slaveNum, unsigned long dspAddr, unsigned long buffer[], long wordCount)
throw (VmeException &, HpiException &);
ROD Software Workshop18-19 July 2002
RodModule: VME Access MethodsRodModule: VME Access Methods
//! Load a 32-bit value into a 32-bit status or VME command register. void regLoad(unsigned long regAddr, unsigned long regValue) throw (VmeException &);
//! Fetch a 32-bit value from a 32-bit status or VME command register. unsigned long regFetch(unsigned long regAddr) throw (VmeException &);
//! Reset the Master DSP via FPGA Control Reg 2 void resetMasterDsp(); //! Reset a Slave DSP via FPGA Control Reg 2 void resetSlaveDsp(long slaveNumber);
ROD Software Workshop18-19 July 2002
RodModule: UtilitiesRodModule: Utilities
//! Calculate a checksum (bit-wise XOR) long checkSum(const unsigned long *sourceArray, const long wordCount); //! Go to sleep for a specified number of milliseconds void sleep(const double milliSecs);
These methods may be moved to a RodUtilities class in the future.
ROD Software Workshop18-19 July 2002
RodPrimitive ClassRodPrimitive Class
A simple class to hold Primitive objects Data members:
long m_length long m_index long m_id long *m_body
Methods: Only the usual data accessor methods (“get” and “put”).
Body A
PrimList Format
Version 2.313 Apr 2000
List Length
List Index
Checksum
Prim Length A
Num Prim s
Prim Length B
Body B
Prim Index A
Prim Id A
Prim Id B
Prim Index B
List Length
ROD Software Workshop18-19 July 2002
RodPrimList ClassRodPrimList Class Derived from the C++ Standard Template
Library (STL) list class. This gives us almost all the functionality we need for free.
Data members: unsigned long m_index unsigned long *m_buffer unsigned long m_listLength
Methods (other than data accessors) unsigned long numWords() unsigned long checkSum() void bufferBuild() void clear()
Body A
PrimList Format
Version 2.313 Apr 2000
List Length
List Index
Checksum
Prim Length A
Num Prim s
Prim Length B
Body B
Prim Index A
Prim Id A
Prim Id B
Prim Index B
List Length
ROD Software Workshop18-19 July 2002
RodOutList ClassRodOutList Class
Contains reply lists generated by the MDSP in response to an associated primitive list.
Structure is the same as a PrimList Data members:
long m_length unsigned long* m_body
Methods: unsigned long* getBody() long getLength()
RodModule makes no attempt to understand the reply list, it merely passes it to a user thread for further handling.
ROD Software Workshop18-19 July 2002
Example of Building and Sending a PrimListExample of Building and Sending a PrimList
This is a test program I will use to debug RodModule. In real RODDAQ, this functionality will be spread between processes
and threads on the host and RCC machines.
ROD Software Workshop18-19 July 2002
Sequence DiagramSequence Diagram
CommandThread
BkgndThread Primitive PrimList RodModule
UserCommand
Create Primitive
Add Primitive
Send PrimList
Iterate
write HPIC and HPIA
send inList
primHandler()read dspAck
if (dspAck=0 and LOADED):set inListRdy=1
STATE=IDLE
setState(LOADED)
if (dspAck=1 and EXECUTING):read outListRdy
Return(LOADED)
Clear PrimList
Delete PrimitiveIterate
setState(EXECUTING)
Primitive List:User Command to ROD
Version 1.6T. Meyer29 May 2002
OutList
return(EXECUTING)
primHandler()
if (outListRdy=1):fetch outList
setState(WAITING)
if (outListRdy=1)construct OutList
return(WAITING)
outListRdy
processoutlist
if (Outlist) delete OutListdelete OutList
Reply touser
1
2
3
4a
4b
5
6
7
8
9
10
ROD
setState(IDLE)
getOutList
return(IDLE)
read dspAck
ROD Software Workshop18-19 July 2002
#include <iostream>#include <ctype.h>#include "RodModule.h"#include "../commonWithDsp/primParams.h"using namespace SctPixelRod;#include "../VmeInterface/RCCVmeInterface.h"
int main() {int i,j;unsigned short value;short buf[1000];const unsigned long baseAddress13=0x0d000000; // Base address for slot 13const unsigned long mapSize=0x10000; const long slot13=13; const long numSlaves=4; RodPrimList primList(1); // Create VME interface RCCVmeInterface *vme1 = new
RCCVmeInterface();
RodModuleTest.cxx (1)RodModuleTest.cxx (1) // Create RodModule and initialize it RodModule* rod0 = new
RodModule(baseAddress13, mapSize, *vme1, slot13, numSlaves);
rod0->initialize();
// Create and Send a simple ECHO primitive long echoData[2] = {0xDEADF00D,0}; RodPrimitive* echo = new RodPrimitive(13, 1,
ECHO, echoData); primList.insert(primList.begin(), *echo); primList.bufferBuild(); rod0->sendPrimList(&primList);
// Wait for ROD to begin executing and then wait for it to finish executing
do {} while (rod0->primHandler() != PRIM_EXECUTING);
do {} while (rod0->primHandler() != PRIM_WAITING);
ROD Software Workshop18-19 July 2002
RodModuleTest.cxx (2)RodModuleTest.cxx (2)// Retrieve output buffer RodOutList* outList = rod0->getOutList(); // Print results (User processing of outList) UINT32 outLength = UINT32(outList-
>getLength()); unsigned long* outBody = outList->getBody(); UINT32 outIndex = UINT32(outBody[1]); UINT32 outNumPrims = outBody[2]; cout << "outLength = " << outLength << ",
outIndex = " << outIndex << ", OutNumPrims = " << outNumPrims << '\n';
UINT32 primLength = outBody[3]; UINT32 primIndex = outBody[4]; UINT32 primId = outBody[5]; cout << "primLength = ", primLength, ",
primIndex = ", primIndex, ", primId = ", primId << '\n';
cout << "ECHO data: "; for (int i=0; i=primLength-3; i++) cout <<
outBody[i+6];
// Copy to local buffer UINT32* myOutBody = new
UINT32[outLength]; for (i=0; i<outLength; i++) { myOutBody[i] = outBody[i]; }; // Clean up: clear primList, delete primitive, delete
the outList primList.clear(); delete echo; delete myOutBody; rod0->deleteOutList(); delete vme1;
return 0; }
ROD Software Workshop18-19 July 2002
Text BuffersText Buffers Text buffers are more useful for debugging than for data taking. If a DSP CPU crashes, we cannot execute primitives. But the on-board
extended memory interface (EMIF) circuitry still lets us read the SDRAM contents.
Text Buffers are blocks of SDRAM memory to which we can write ASCII records with traceback information, or simple diagnostic info for later analysis. (Recall debugging FORTRAN with print statements.)
Even if DSP is fine, it may be useful to pass ASCII data into a message stream (MRS) in ATLAS.
There are four text buffers defined for the Master DSP Err: for reporting fatal software error tracebacks Info: for non-priority information messages Diag: for diagnostic messages, “print” to it like we used to print out
FORTRAN debug messages. Xfer: A temporary MDSP buffer to relay text buffers from the Slave DSPs
ROD Software Workshop18-19 July 2002
Text Buffer HandlingText Buffer Handling
User TextThread
BkgndThread RodModule
read xxxBufferNotEmpty
set xxxBufferReadRequest
textHandler()
read buffer header
read buffer contents
STATE=IDLE
setState(RQ_SET)
clear xxxBufferReadRequest
Return(RQ_SET)
clearTxtBuffer()
Iterate
setState(READOUT)
Text Buffers:User Thread to ROD
Version 1.3T. Meyer5 Jun 2002
return(READOUT)
setState(IDLE)
bufferReady
processbuffer
1
2
3
4
5
6
7
8
9
ROD
getTxtBuffer(buffer, length, type))
textHandler()read xxxBufferNotEmpty
"1"
"0"
buffer
ROD Software Workshop18-19 July 2002
Future WorkFuture Work Debug what is already written
using RodModuleText (“If it compiles, publish!” – anon, from another collaboration)
Interface to DAQ-1 Front End Calibration program Add Posix threads Strengthen weak areas
status() FPGA operations exceptions bullet proofing
Phase O ne
D AQ -1Run Control
C om m andLayer
R O D M oduleD river
C++ VM E Interface
C VM E Interface
U ser
VM E M odules
VM E calls
RO D Prim itives Replies
StateT ransitions R eplies
Analysis T askand D isp lay
D ata U ser
ROD Software Workshop18-19 July 2002
DocumentationDocumentation
User’s Guide: will eventually be derived from this and similar presentations. Paper and web versions.
Technical reference: will be generated automatically using Doxygen. Users can generate it for themselves simply, and output can be in HTML, LaTex, and PostScript formats. Doxygen uses special comments embedded in the code along with a knowledge of C++ file structure to produce good quality documentation.
Top Related