Software

This chapter describes Linux software package for control (SlowControl) and data acquision, sitcp-dump (DC) and Run44 (Pulse). The software package is distributed at the Open-It page for Kalliope after user registration.

Environment for Development
There are a few PC environment employed. Since the Kalliope software (SlowControl and sitcp-dump) are using only the standard functions in c and c++, it should run in most of the linux environment, including MacOSX and Windows WSL2 (ubuntu). The DAQ programs (sitcp-dump and Run44) depends on CERN ROOT, which sometimes causes compatibility issues in gcc versions (but solvable). The Pulse-mode DAQ program is heavily dependent on ROOT and tend to have more problems porting to other platforms.

  • Scientific Linux 6.x -> 7.x (at J-PARC MLF: mostly Run44 and SlowControl)
  • gcc version 4.4.7 -> 4.8.5
  • MacOSX 14.x (mostly SlowControl and sitcp-dump, and sometimes Run44)
  • gcc version 4.2.1 (Apple LLVM 10.0.1)
  • Ubuntu 20.04.3 LTS on Microsoft Windows version 10.0.19042.1237 (mostly SlowControl and sitcp-dump)
  • gcc version 9.3.0

SlowControl

This software package takes care of Kalliope settings by UDP connections of SiTCP communication. Most of the programs writes or reads the RBCP registers on Kalliope firmware. (Refer to Firmware section for RBCP register map.) Here is the list of programs:

WRITE_IP_ADDRESS
Sets the address for user IP mode to a SiTCP device at 192.168.10.16.
dcdelay DELAY
Sets the DELAY parameter.
kc_ip CMD PARAM
Triggers On-chip command CMD with parameter PARAM.
Most important one is CMD=1 and PARAM=N, which starts serialized transfer from DACDataN. Other functions are described in Firmware section.
pc2012_ip IP N parameter-file
Writes DAC parameters to DACDataN for VOLUME2012 at IP address.
pc2014_ip IP N parameter-file
Writes DAC parameters to DACDataN for VOLUME2014 at IP address.
pcfgati_ip IP N parameter-file
Writes DAC parameters to DACDataN for FGATI at IP address.
read-registers_ip IP
Reads RBCP register map and shows.

The following are the legacy commands for NIM-in trigger-driven SPI functions of Pulse-mode firmware.

triggered-dump
Reads RBCP register map for NIM-in trigger driven SPI.
tw N VALUE
Writes DELAY (N=1) or GAP (N=2) at 192.168.10.16.

DAC format for Volume2012, 2014 and FGATI

The parameter-file given to pc201x_ip or pcfgati_ip command is a text file with a list of DAC values, one line per channel. The examples are

./SlowControl/VOLUME2012AllON.txt
./SlowControl/VOLUME2014AllON.txt
./SlowControl/FGATI.txt

The meaning of DAC values are dependent on ASICs, and are different. In the following, the symbol [] is either a Hexadecimal number (4 bits) or a byte (8 bits), and {} is a bit.

Volume2012 0x[0][ThDAC][AmpDAC1][AmpDAC2][BiasDAC][CTRL] ([]=4 bits)
[ThDAC] Threshold level of discriminator. Unconventional order, because [ThDAC]={4}{3}{2}{1} is interpreted in ASIC as {not 1=sign}{2}{3}{4}: Low <-- 0xE,6,A,2,C,4,8,0,1,9,5,D,3,B,7,F--> High
[AmpDAC1] Transistor bias for Amplifier1. Recommended to use a small number (e.g. 0x4).
Usual order: Low <-- 0x1,2,3,4,5,6,7,8,9,A,B,C,D,E,F --> High
[AmpDAC2] Transistor bias for Amplifier2. Recommended to use a large number (e.g. 0xC).
Usual order: Low <-- 0x1,2,3,4,5,6,7,8,9,A,B,C,D,E,F --> High
[BiasDAC] HV bias control in subtraction??.. Usual order: Low <-- 0x1,2,3,4,5,6,7,8,9,A,B,C,D,E,F --> High
[CTRL] [CTRL]={BiasDACen}{Gain100en}{Aout}{Dout} bits.
  • {BiasDACen} 1=enable/0=disable BiasDAC control.
  • {Gain100} 1=x100 / 0=x10 gain.
  • {Aout} 1=enable/0=disable Ananlog out.
  • {Dout} 1=enable/0=disable Digital out.
  • Volume2014 0x[ThDAC][VRSDAC][BiasDAC][CTRL] ([]=8 bits)
    DAC convention for all parameters Trancated to 6 bits with {not 6=sign}{5}{4}{3}{2}{1}.
    Low <-- 0x1F,1E,1D,1C,1B,1A,19,18,17,16,15,14,13,12,11,10,0F,0E,0D,0C,0B,0A,09,08,07,06,05,04,03,02,01,00,20,21,22,23,24,25,26,27,28,29,2A,2B,2C,2D,2E,2F,30,31,32,33,34,35,36,37,38,39,3A,3B,3C,3D,3E,3F --> High
    [ThDAC] Determines the threshold level of discriminator.
    [VRSDAC] Determines the Pole-zero cancellation VRS value.
    [BiasDAC] Determines the HV control in subtraction??.
    [CTRL] [CTRL]={0}{0}{Amon}{BiasMon}{OFFSET}{POL}{FAST}{Dout} bits.
  • {Amon} 1=enable/0=disable Analog monitor.
  • {BiasMon} 1=enable/0=disable VDETREF monirotr at VDETREFMON.
  • {OFFSET} 1=enable/0=disable VOFF internal level.
  • {POL} 1=enables/0=disable positive ?? leading edge for Dout ???.
  • {FAST} 1=enable/0=disable DC level check at Dout.
  • {Dout} 1=enable/0=disable Dout.
  • note Since the DACDataN on RBCP register is 3-byte x 32 channel, the 4x 4-byte DAC/channel in the parameter file is truncated to 4x 6-bits and concatinated to 24 bits = 3 bytes.
    The read-register command returns DACs not looking like the parameter-file contents.
    FGATI 0x[ThDAC] ([]=8 bits)
    [ThDAC] Trancated to 6 bits with {not 6=sign}{5}{4}{3}{2}{1}.
    Low <-- 0x1F,1E,1D,1C,1B,1A,19,18,17,16,15,14,13,12,11,10,0F,0E,0D,0C,0B,0A,09,08,07,06,05,04,03,02,01,00,20,21,22,23,24,25,26,27,28,29,2A,2B,2C,2D,2E,2F,30,31,32,33,34,35,36,37,38,39,3A,3B,3C,3D,3E,3F --> High

    The minimum command (and a good example) to setup a Kalliope module for operation is:

    $./SlowControl/pc2012_ip 192.168.10.x 1 VOLUME2012AllON.txt , or 
    $./SlowControl/pc2014_ip 192.168.10.x 1 VOLUME2014AllON.txt , or
    $./SlowControl/pcfgati_ip 192.168.10.x 1 FGATI.txt
    
    $./SlowControl/kc_ip 192.168.10.x 1 1
    

    DAQ program sitcp-dump (for DC and Pulse)

    The simplest DAQ program sitcp-dump reads SiTCP TCP data from one Kalliope module (or any type of SiTCP module) and writes into a file. It is in the sitcp-dump package with the following commands generated by make.

    decode-gatenet GATENET-time
    decodes GATENET time into the UNIX time.
    kalliope-config IP
    sets GATENET time from the PC clock to Kalliope at IP address. For DC-mode firmware only.
    nhit
    count rate monitor during the run. (This program causes a compilation issue in MacOSX, with pthread_create. Not yet resolved.)
    offlinedecoder rawdata-file
    a user-developed program to decode the data, analyze and creates histograms etc.
    sitcp-dump [--nim_tdc] [--datadir DIRECTORY] IP:port
    --nim_tdc: accepts pause/resume command. NIM-TDC works as the trigger distribution master,
    with its NIM OUT3 coincidenced with the NIM-in trigger signals of all other Kalliope modules in the experiment.
    --datadir: specifies the directory, where the data is written.
    sitcp-dump is the DAQ program, which accepts TCP data from IP:port SiTCP device (port=24 usually), and waits controling commands from another terminal (at port 2222).

    Running the DAQ program (sitcp-dump)

    on a terminal run the following, and this terminal goes into a wait.

    $./sitcp-dump --datadir /data 192.168.10.x:24
    

    Open another terminal which controls the runs. The above terminal responds accordingly.

    $ telnet localhost 2222
    start N            (to start a run with run number N)
    connection closed.
    
    $ telnet localhost 2222
    stop               (to stop a run)
    connection closed.
    
    $ telnet localhost 2222
    pause              (to pause a run,   ??? I am not sure how NIM-TDC is identified ????)
    connection closed.
    
    $ telnet localhost 2222
    resume             (to resume a run,  ??? I am not sure how NIM-TDC is identified ????)
    connection closed.
    

    In one experiment, sitcp-dump must be issued to all the Kalliope modules involved in separate terminals. The rawdata files are created as many as the number of devices under /data directory, with the sub-directory and filenames ./MSE[runnno]__YYYYMMDD/MSE[runno]_[IP].rawdata .

    The rawdata structure

    The rawdata structure written by sitcp-dump is the TCP data received from the Kalliope firmware. Refer to the Firmware section for the detailed TCP data structure. The file is writen in little endian (lower bytes first), and may be difficult to read in xxd dump. It is best to dump in 4-byte unit, using od. For example:

    $ od -t x4 MSE000001_192.168.10.16.rawdata
    0000000          5c000040        c5e9b208        7fff000a        00ad7ad5
    0000020          00000000        01000000        ffaa0000        00000000
    0000040          02010000        03002bb5        04002bd5        02010001
    0000060          02010002        02010003        02010004        02010005
    ...
    0002320          0201012a        0201012b        0201012c        0201012d
    0002340          0201012e        ff550000        00030000        5c000040
    0002360          c5fe0833        7fff000a        00d359c5        00000000
    0002400          01000001        ffaa0000        00000100        02010000
    0002420          02010001        02010002        02010003        02010004
    0002440          02010005        02010006        02010007        02010008
    ...
    

    This sample shows only one hit (03/04 at ~0x2bb5 = 11189 ns) from the TDCstart.

    DAQ program Run44 (for pulsed muon at MLF)

    The DAQ programs running at MLF muon experiments are complicated, after expansions and quick fixes over ten years. (The current Run44 distribution was created in 2012.) The Linux operating system was upgraded from Scientific Linux 6.x to 7.x in year 2020, which caused some issues with gcc and glibc compatibilities with ROOT. All these has been solved by the end of year 2020. The DAQ program has the common structure which covers both the µSR and X-ray measurements. However, for X-ray measurements, the on-line analysis and display has been upgraded, and the program has branched out.

    S1-, D1-, U1A-, CYCLOPS- µSR and D2-X-ray

    S1 and D1 experimental areas have twin general-purpose µSR spectrometers with 40 Kalliope units (with Volume2014) installed, in addition to one NIM-TDC as the trigger master. U1A- and CYCLOPS spectrometer has the same structure, but with 16 and 94 Kalliope units, respectively. D2-X-ray spectrometer has Niki A3N00 modules as the main devices (VME), with NIM-TDC for the trigger master. The DAQ structure is in common; we take an example on S1-µSR spectrometer. (For D1-µSR spectrometer, simply change ms1 --> md1.)

    The DAQ-PC structure with local disks for S1-µSR spectrometer was

    ms1-ctr: ---------(NFS mount)--------- ms1-daq:
    /home/daq/                             /data/
    ├── DaqComponents/src/Run44/           └── edb/
    └── monitorparams.cc            
    /arc                       
    ├── edb/
    ├── root/
    └── msr/
    

    where ms1-daq takes data, and ms1-ctr analyzes (OfflineDecoder + r2mud) to make histograms. Disks were NFS mounted to the pair PC. The CPU load of the DAQ was found not too large (~10-25% of single CPU), and nowadays, single DAQ-PC is taking and analyzing the data to remove the complications related to NFS mount. There is another PC named ms1-dev, which is the IROHA device server for sample environment devices, but it is out of the scope of this document.

    All the DAQ program for Kalliope is stored in ms1-ctr:/home/daq/DaqComponents/Run44. The important parameter files are:

    module_numbers.h
    defines the number of hardware modules for COPPER-LITE (obsolete), Kalliope, NIM-TDC, GATENET and Niki-3N00.
    For Niki-3N00, the parameter USE_46PACKET determines whether 46 packet is used (TDC limited to 24 bit ~ 80ms in 5ns resolution) or 42/44 packet pair is used with TDC full 40 bit length.
    module_ip.h
    defines the IP address of the active modules defined above.
    KALLIOPE_TDC.map
    defines the coincidence pair of positron telescopes, and the FBUDLR histograms which the coincidence pair contributes. An example is:
    S1-Specrometer CounterMap (#Total+pair(D1/S1:+16), IP:Inner-Outter, T0NS, Map                      (comment line)
    1280+16                                                   (Total number of channels +  default ch-pair increment)
    1: 0-16,8500,BD    (#1 module, ch0 is paired to ch16, t0=8500, contributes to BD histogram)
    1: 1-17,8500,BD
    ...
    2:15-31,8500,BD    (#2 module, ch15 is paired to ch31, t0=8500, contributes to BD histogram)
    3:all,8500,BLD     (#3 module, all channels are paired to +16 counterpart, t0=8500, contributes to BLD histogram)
    
    NIM_TDC.map
    defines the mapping and on-line analysis, but may be obsolete. An example is:
    NIM_TDC test HistogramMap                                                                          (comment line)
    32*1                                                                              (Total 32 ch input of one kind)
    0N:
    1N: 0,1.0 > C                                                            (Histogram C is defined as =0+1.0x(1N) )
    ...
    
    NIKI_3N00.map
    defines the mapping, and on-line analysis, but may be obsolete. An example is:
    Niki VME test HistogramMap                                                                         (comment line)
    32*2                                                          (Total 32 ch with Time and Analog for 16 ch inputs)
    0T:
    0A:
    1T:
    1A: 0,1.0 > C                                                            (Histogram C is defined as =0+1.0x(1A) )
    /1T=x(0,20) > S                                     (Histogram S is defined as 1A with cut by 1T in (0.00,20.0) )
    ...
    

    The analyzed histograms (e.g. Histogram C and S in the above) appear in the ROOT file: /dev/shm/monitor_status.root .

    Compile

    Once the module_numbers.h and module_ip.h are defined, the package has to be recompiled by make.
    The map files are read at the start of the DAQ program, so recompiling is not necessary.

    Running the DAQ program (Run44)

    on a terminal, run the following and the window has to be left as is.

    $ ./Run44/worker_x11
    

    The low-level control of the DAQ program is, opening another terminal and issue telnet commands to port 2222. The above terminal responds accordingly.

    $ telnet localhost 2222
    start N            (to start a run with run number N)
    connection closed.
    
    $ telnet localhost 2222
    stop               (to stop a run)
    connection closed.
    
    $ telnet localhost 2222
    pause              (to pause a run)
    connection closed.
    
    $ telnet localhost 2222
    resume             (to resume a run)
    connection closed.
    

    The list-mode data, called edb (event data binary?) files are created under /data/edb directory, with the sub-directory and filenames MUSE[runnno]__YYYYMMDD/MUSE[runno]_01_001_Edn.edb, with the Edn (edition number) increments in every 1GB size.

    Analysis (Conversion) of the data for µSR

    • OfflineDecoder.cc is the program to analyze the edb data offline. It creates ROOT file after taking coincidences on the detector pairs. If no ROOTfilename is specified, Coincidence.root is greated. (Accelerator) frame is the NIM-in trigger ID.
    $ ./Run44/OfflineDecoder [-o ROOTfilename] [-g Coincidence in ns] [-s StartFrame] [-c EndFrame ] EDBfilename
    
    
    • r2mud.c is the program (developed by Takashi Ito) which converts the ROOT histogram to MUD (µSR histogram data).
    $ ./r2mud/r2mud CounterMapFile ROOTfile [deadtime_ns_F deadtime_ns_B deadtime_rho_F deadtime_rho_B] 
    
    

    Analysis (Conversion) of the data for X-ray

    • must be filled by someone...

    On-line analysis behavior has been altered for X-ray version of Run44, such as Nhit window contents: Run44Display.h, Run44Decoder.h and Run44Decoder.cc has been modified.

    The edb data structure

    The DAQ program Run44 decodes the TCP data (described in Firmware section) and writes out data in a decoded format called as edb. The decode of the TCP data is done in GathererDecoder.h, and the format of edb is defined in MLFdatum.h. The edb is in 64 bit (8 byte) unit in big endian. xxd dump reads well.

    Pulse-mode EDB data. 
    10-11 events (NIM-TDC) for each module
    MSB                                                      LSB 
     | 0x10 | 0x00 | 0x0000 |     TriggerID[31:0]             |  
     | 0x11 | 0x00 | 0x0000 | 0x000000 |"00000" | NIM-in[2:0] |  
     | 0x10 | 0x00 | 0x0000 |     Length[31:0]                |  
     | 0x10 | 0x00 | 0x0000 |     TDC-start[31:0]             |  
     | 0x10 | 0x00 | 0x0000 |     FooterValue[31:0]           |  
    { repetition of hits
    22 event                                                           
     | 0x22 | 0x00 | Module[15:0] | Channel[15:0] | TDC[15:0] |
    }
    
    20 events (Kalliope) for each module
    MSB                                                      LSB 
     | 0x20 | 0x00 | 0x0000 |     TriggerID[31:0]             |  
     | 0x20 | 0x00 | 0x0000 |  Keyword[31:0] (8ns counter)    |  
     | 0x20 | 0x00 | 0x0000 |     Length[31:0]                |  
     | 0x20 | 0x00 | 0x0000 |     TDC-start[31:0]             |  
     | 0x20 | 0x00 | 0x0000 |     FooterValue[31:0]           |  
     | 0x20 | 0x00 | 0x0000 |     ModuleID[31:0]              |  
    { repetition of hits
    22 event                                                           
     | 0x22 | 0x00 | Module[15:0] | Channel[15:0] | TDC[15:0] |
    }
    
    40 events (Niki) for each VME crate controler
    MSB                                                      LSB 
     | 0x40 | 0x00 | 0x0000 |     TriggerID[31:0]             |  
     | 0x40 | 0x00 | 0x0000 | Keyword[31:0] (Niki parameter)  |  
     | 0x40 | 0x00 | 0x0000 |     Length[31:0]                |  
    { repetition of hits 
    46 event (when USE_46PACKET = 1)                            Reso[7:0] indexes to [5,10,20,100,200,500,1000] (ns).
     | 0x46 | Reso[7:0] | ModCh[7:0] |  TDC[23:0] | ADC[15:0] | 
    or pair of 42-44 event                                      ModCh[7:0] = 0x(module number)(channel number).
     | 0x42 | Reso[7:0] | ModCh[7:0] |        TDC[39:0]       | 
     | 0x44 |    0x00   | ModCh[7:0] |  0x000000  | ADC[15:0] |
    }
    
    GATENET-time
    MSB                                                      LSB
     | 0x5c |           GATENET time (56bit)                  |  
    

    The format for TDC-start[31:0] is (also described in Firmware section)

    TDC-start[31:0]=
    MSB                                           LSB
     |MultiStartError| "001" |0x000 |  TDC[15:0]   |     Normally: 0x1000 xxxx, Error: 0x9000 xxxx
    

    The format for FooterValue[31:0] is (also described in Firmware section)

    FooterValue[31:0]=
    MSB                                           LSB
     | 0x00 | "00000" | txBuffFull | "11" | 0x0000 |     Normally: 0x0003 0000, Error: 0x0007 0000
    

    The format for Keyword[31:0] (Niki parameter) is

    Keyword[31:0]=
    MSB                                                        LSB
     |0x00| NikiModule[7:0] | NikiMCSR[7:0] | NikiGAINMode[7:0] |
    
     NikiModule[7:0]   =0x01(A3100), 0x02(A3200), 0x03(A3300), 0x04(A3400)
     NikiMCSR[7:0]     ={EMP, FUL, TILSEL, PHMOD, "0", TB2, TB1, TB0} bits
     NikiGainMode[7:0] ={"0", CG2,    CG1,   CG0, "0", LM2, LM1, LM0} bits