Intel Linux VME CPU in Hall A

Hall A meetingOctober 15th 2013

Alexandre Camsonne

Outline

Single board computers Hall A servers implementationDifferences with VxWorksRead out codeAdapting codeAvailable codesPerformanceTo do listConclusion

Single board computersCore i7 single board computer2.53 GHz4 GB RAMFull Linux PC on a boardVME320

XVB601/XVB602 Compact Flash

XVR14 On board Flash

Hall A server implementationDHCP server

PXE/TFTP and NFS server Accessible from adaql1 in /root/diskless/

If need to add a new CPU : need to know MAC address and add to DHCP and create directory for system files on CHAFS

host intelha7 { filename "linux-install/pxelinux.0"; # File location relative to /tftpboot/ next-server 129.57.99.58; # TFTP server hardware ethernet 00:20:fa:fa:fa:fa; # Client MAC XVR14 fixed-address 129.57.255.255; # Client IP }

Difference with vxWorks CPUsBoot using PXE or onboard driveCentos 5.8 distribution

No boot parameters, use DHCP to assign IP address

Little Endian Use C directly instead of CRLUse virtual memory

Readout codeCODA library : jvmeReadout is C program with includes and link

to JVME

Supported and developped by Bryan Moffithttps://coda.jlab.org/wiki/index.php/Intel-

based_VME_Controllers/u/site/coda/3.0/linuxvme/u/site/coda/3.0/linuxvme/examples

Adapting programsvxWorks Linux Notes

STATUS sysBusToLocalAdrs (int adrsSpace, char *busAdrs, char

**pLocalAdrs);

int vmeBusToLocalAdrs(int vmeAdrsSpace, char *vmeBusAdrs, char **pLocalAdrs);

int vxMemProbe(char *addr, int mode, int size, char *rval);

int vmeMemProbe(char *addr, int size, char *rval); Linux, mode is read only

STATUS intConnect (VOIDFUNCPTR *vector, VOIDFUNCPTR routine, int

parameter);

int vmeIntConnect(unsigned int vector, unsigned int level, VOIDFUNCPTR routine, unsigned int arg);

int intDisconnect(int vector) int vmeIntDisconnect(unsigned int level); vector != level

int sysIntEnable(int); NONE

int sysIntDisable(int); NONE

int intLock(); NONE

void intUnlock(int key); NONE

STATUS usrVmeDmaConfig(unsigned int addrType, unsigned int dataType,

unsigned int sstMode);

int vmeDmaConfig(unsigned int addrType, unsigned int dataType, unsigned int sstMode);

STATUS sysVmeDmaSend (UINT32 locAdrs, UINT32 vmeAdrs, int size,

BOOL toVme);

int vmeDmaSend(unsigned int locAdrs, unsigned int vmeAdrs, int size); Linux, always from VME

int sysVmeDmaDone(int pcnt, int pflag); int vmeDmaDone();

vxWorks: returns size - (#bytes transferred)

Linux: returns (#bytes transferred)

https://coda.jlab.org/wiki/index.php/VxWorks_to_Linux_HOWTO

Readout code ( vme_list.c )/************************************************************************* * * vme_list.c - Library of routines for readout and buffering of * events using a JLAB Trigger Interface (TI) with * a Linux VME controller. * */

/* Event Buffer definitions */#define MAX_EVENT_POOL 400#define MAX_EVENT_LENGTH 1024*10 /* Size in Bytes */

/* Define Interrupt source and address */#define TIR_SOURCE#define TIR_ADDR 0x0ed0/* TIR_MODE: 0 : interrupt on trigger, 1 : interrupt from Trigger Supervisor signal 2 : polling for trigger 3 : polling for Trigger Supervisor signal */#define TIR_MODE 2

#include "linuxvme_list.c" /* source required for CODA */#include "usrstrutils.c" /* helper routines to pass data from mSQL to ROC */

/* Globals to be filled by usrstrutils */int string1=0, string2=0; /* defined when string is present in CODA database */int ps1=0,ps2=0,ps3=0,ps4=0,ps5=0,ps6=0,ps7=0,ps8=0; /* defined in "ffile" */

/* function prototype */void rocTrigger(int arg);

voidrocDownload(){

/* Setup Address and data modes for DMA transfers * * vmeDmaConfig(addrType, dataType, sstMode); * * addrType = 0 (A16) 1 (A24) 2 (A32) * dataType = 0 (D16) 1 (D32) 2 (BLK32) 3 (MBLK) 4 (2eVME) 5 (2eSST) * sstMode = 0 (SST160) 1 (SST267) 2 (SST320) */ vmeDmaConfig(2,5,1);

/* Initialize usrstrutils */ init_strings(); string1 = getflag("string"); string2 = getflag("string2");

printf("usrstrutils configuration:\n"); printf("\tstring1 = %d\n\tstring2 = %d\n",string1,string2);

printf("rocDownload: User Download Executed\n");

}

voidrocPrestart(){ unsigned short iflag; int stat;

/* Check ffile for changes (usrstrutils) Useful for items that may change without re-download (e.g. prescale factors) */ init_strings(); ps1 = getint("ps1"); printf("usrstrutils configuration:\n"); printf("\tps1 = %d\n",ps1);

/* Program/Init VME Modules Here */

printf("rocPrestart: User Prestart Executed\n");

}

voidrocGo(){ /* Enable modules, if needed, here */

/* Interrupts/Polling enabled after conclusion of rocGo() */}

voidrocEnd(){

printf("rocEnd: Ended after %d events\n",tirGetIntCount()); }

voidrocTrigger(int arg){ int ii;

/* Example: Raise the 0th (1<<0) and 2nd (1<<2) output level on the TI */ tirIntOutput(1<<0 | 1<<2);

/* Insert some data here - Make sure bytes are ordered little-endian (LSWAP)*/ *dma_dabufp++ = LSWAP(0xda000022); for(ii=0; ii<20; ii++) { *dma_dabufp++ = LSWAP(ii); } *dma_dabufp++ = LSWAP(0xda0000ff);

/* Drop all output levels on the TI */ tirIntOutput(0);

}

Readout code ( vme_list.c ) /* Initialize usrstrutils */ init_strings(); string1 = getflag("string"); string2 = getflag("string2");

printf("usrstrutils configuration:\n"); printf("\tstring1 = %d\n\tstring2 = %d\n",string1,string2);

printf("rocDownload: User Download Executed\n");

}

voidrocPrestart(){ unsigned short iflag; int stat;

/* Check ffile for changes (usrstrutils) Useful for items that may change without re-download (e.g. prescale factors) */ init_strings(); ps1 = getint("ps1"); printf("usrstrutils configuration:\n"); printf("\tps1 = %d\n",ps1);

/* Program/Init VME Modules Here */

printf("rocPrestart: User Prestart Executed\n");

}

voidrocGo(){ /* Enable modules, if needed, here */

/* Interrupts/Polling enabled after conclusion of rocGo() */}

voidrocEnd(){

printf("rocEnd: Ended after %d events\n",tirGetIntCount()); }

voidrocTrigger(int arg){ int ii;

/* Example: Raise the 0th (1<<0) and 2nd (1<<2) output level on the TI */ tirIntOutput(1<<0 | 1<<2);

/* Insert some data here - Make sure bytes are ordered little-endian (LSWAP)*/ *dma_dabufp++ = LSWAP(0xda000022); for(ii=0; ii<20; ii++) { *dma_dabufp++ = LSWAP(ii); } *dma_dabufp++ = LSWAP(0xda0000ff);

/* Drop all output levels on the TI */ tirIntOutput(0);

}

Linux VME drivers and readout

JLAB boards : FADC250, F1,TI, TSCAEN V792, V775, V1190Fastbus SFI 1877, 1881 Scaler SIS3801MPD library ( C++ based )

PerformancesMost gain when transferring a lot of data

Should benefit DVCS and experiment using FADC

Up to 115 Mbytes/s, close to Gigabit ethernet line speed

Twice faster than vxWorks CPU thanks to multicore

To do listAdd adaq and a-onl directory

Automatic start of ROC in startcoda , ROC not started automatically for debugging

DVCS port of ARS and trigger libraries and debug

Test INFN APV25 Readout with CODA

Setup serial port and output to portserver

ConclusionNew Intel VME CPU deployed

Different way to boot

Server infrastructure in place

Readout available for most boards

Board about twice more faster for about same price as Power PC