------17/12 Jan2021 preparation start new directory with HT -jan21 -htools from hg clone from his home directory hg clone ssh://land@lxir123/lynx.landexp/202006_test/htools or public repo: https://hg.osdn.jp/view/htools/htools -nurdlib git clone /u/htoernqv/repos/nurdlib.git had to select the proper branch (origin/master) -r3bfuser git clone /u/htoernqv/repos/r3bfuser.git -trloii git clone /u/johansso/trloii.git Compile: make, then make BUILD_MODE=release (mbs) for nurdlib make fw___build for trloii and tridi (both needed by nurdlib) The m_read_meb are links for the r3bfuser directory. Same for all rios --- Current working directory for LAND: /u/land/land.exp/202006_test ... scripts directory of interest Issue with environement variables: env.csh present in sec_jan2021 csh2sh.sh to convert env.csh to env.sh, has to be done manually Now issue is rio in prm need the environement variables due to new nurdlib configuration solve in adding the config default path inside the f_user.c ==> MBS seems to execute shell command direcly when x is used so no way to play with the environment variables Tags instead of groups now in nurdlib!!! ------18/12 DAQ runs with current trigger, 9600 Hz up to 10000 Hz, single VME access is 460 ns ==> 200 words in a sub event means ~100 us dead time ---- Tag are used instead of groups, when a module is defined in a tag it checks which trigger triggers said tag, Tirgger comes, increment tag in nurdlib, and check all module counters in all tags expecting increas only for tags defined for said trigger. Runs in single event mode without issue When comming in multi event mode: ==> at 100 kHz works fine (trig 0 is multi event and trig 1 readout trigger) ==> at 1 MHz crashes, so decision to try to work on the bussy, by modifyinh vulom.ctrl and puting 5 us busy length (cause 2.5us mas bor busy_len was not long enough) ==> did not make it work the v1290 was getting lost in initialisation loops ==> back to 100 kHz free trigger, test of blt in the v830, nurdlib modified to assure size of output buffer don t go nuts, but reading with blt makes sometimes miss a channel==> issue with Linux+caen ..... solution to move back to lynx for the crates with caen. let see in 2021!!! ------18/01/2021 MR_TOF setup of th mr_tof crate with Christine and Hans T worked as planned minus few modifications in Go4 -v820 does not have an event counter so code modified to fake one to not get nurdlib angry - event buffer check was in reversed logic -v1190 was checking lsb before setting it, so things were not good. -- we use lsb of 200 ps for the 100 us gate -cvt should be at least 100 us (see delay in the trloii) - no signal tested yet [] to understand why 25 Hz become 56ish in the input of the vulom ECL -----09/02/2021 and before -able to run the MEB for the FRS crates but without BLT mode -scalers being the SIS ones or the CAEN ones do not accept BLT mode -switching BLT on and off is just a notion of main.cfg -ran the traveling music, with old firmware, in the crate -second QDC added in the frs-user crate -virgo issues solved with mounting of data on profi and frs-phys -first files taken for unpacker -vftx + mtdc + v1290 + tac for all plastics -vulom was the issue on the preparation of trav music, new code simpler to read -----10/02/2021 - two new word appeared at the beginign of each readout subevent from nurdlib, see packing in packingwords.txt ---- 12/02/2021 - configuration of the DAQ done, possiblity to run either 2 crates at S2 independantely or in concert with the 3 others - vulom confiugred for point above - strange problem with QDC in the FRS main user crate, first 16 channels ahve spurious bit encoding, same with different QDC from LAND... and same if slot 11 or 12 used, no clear why. - the sis3820 used in the tof crate seems to have the third adressing nob not working --- 14/02/2021 - change in main cfg the order we read the adc in the daq, problem follows the QDC, meaning no clue, corrupted bus??? - I add the three QDC in the crate, the old one from FRS and the new one from LAND have the same issue. LAND one is in Geo12 and the FRS one in Geo 19. ==> LAND 7,8,9 too highg 15,26 too low for example ==> FRS 16,1 too low 7,8,9 too high... - try them in the user crate - I moved the FRS one to the Geo18 of the user crate, same problem, cha 1,6,7,8 have spurious piedestal same as other...i -X86L-93 :transport : 110: 601e406e 600f406d 601f405c 6400125f babababa -X86L-93 :transport : 115: 92002000 900043b8 9010401b 900144a0 90114019 -X86L-93 :transport : 120: 90024423 90124031 90034610 90134029 9004468f -X86L-93 :transport : 125: 90144a42 9005464f 90154148 9006457d 901644a5 -X86L-93 :transport : 130: 90075f58 90174699 90084628 9018451c 900946bf -X86L-93 :transport : 135: 90194515 900a46e4 901a4585 900b4530 901b4626 -X86L-93 :transport : 140: 900c45e2 901c46c4 900d44dd 901d45f8 900e47e4 -X86L-93 :transport : 145: 901e456b 900f471b 901f44e3 9400125f -test with something connected to it in 20 to 27: -X86L-93 :transport : 105: 600c5fff 601c4063 600d5fff 601d4063 600e4073 -X86L-93 :transport : 110: 601e406e 600f406d 601f405d 6407b849 babababa -X86L-93 :transport : 115: 92002000 900043b6 9010401b 900144a0 90114019 -X86L-93 :transport : 120: 90024424 90124032 9003460e 90134029 9004468d -X86L-93 :transport : 125: 90144a42 9005464c 9015414d 90064586 90164048 -X86L-93 :transport : 130: 90075f5a 9017405e 90084631 901843f8 900946c7 -X86L-93 :transport : 135: 901940dd 900a46ec 901a4397 900b4531 901b403e -X86L-93 :transport : 140: 900c45e3 901c46c3 900d44df 901d45f8 900e47e3 -X86L-93 :transport : 145: 901e456d 900f4719 901f44e3 9407b849 -- now 50 ohms on the low 4 to 13 -X86L-93 :transport : 120: 90114019 90024423 90124031 9003460f 90134028 -X86L-93 :transport : 125: 90044620 90144a42 90054384 90154140 90064575 -X86L-93 :transport : 130: 901644a3 90075f24 90174697 90084622 9018451a -X86L-93 :transport : 135: 900946b6 90194515 900a46de 901a4585 900b40d1 -X86L-93 :transport : 140: 901b4623 900c45e1 901c46c4 900d44de 901d45f7 -X86L-93 :transport : 145: 900e47e2 901e456c 900f4719 901f44e2 9409bd4b ==> still isssue with channel 5,6,7,8 -inconclusive, continue with the two QDC in the main crate -we change the way the QDC are connected --15/02 18:15 daq crashed due to r4l-40 TPC[3]=CAEN_V785 module error ==> not first time tried to reboot twice, problem keeps occuring every few thousands of triggers happened around 8am this morning, around 12 today and now since 17h Power cycle crate did not solve the issue, still " -R4L-40 :read_meb :2021-02-15,17:31:01:ERRR: had readout error, ret=0x4, trigger=1, prev=1 (f_user.c:779) -R4L-40 :read_meb :2021-02-15,17:31:04:ERRR: Data timeout (1s). (module/caen_v7nn/caen_v7nn.c:543) -R4L-40 :read_meb :2021-02-15,17:31:04:ERRR: TPC[3]=CAEN_V785 readout error=0x00000004, dumping data: (crate/crate.c:1791) -R4L-40 :read_meb :2021-02-15,17:31:04:INFO: ---[ Dump begin ]--- -R4L-40 :read_meb :2021-02-15,17:31:04:INFO: Start=0x3067ee0c Bytes=0=0x0 -R4L-40 :read_meb :2021-02-15,17:31:04:INFO: ---[ Dump end ]--- -R4L-40 :read_meb :2021-02-15,17:31:04:ERRR: TPC: readout failed! (crate/crate.c:1333) -R4L-40 :read_meb :2021-02-15,17:31:04:ERRR: TPC: had problems, re-initializing. (crate/crate.c:1377) " Need to desactivate the ADC... Remove the first onei 0x00040000: -R4L-40 :read_meb :2021-02-15,18:01:57:ERRR: had readout error, ret=0x4, trigger=1, prev=1 (f_user.c:779) -R4L-40 :read_meb :2021-02-15,18:02:01:ERRR: Data timeout (1s). (module/caen_v7nn/caen_v7nn.c:543) -R4L-40 :read_meb :2021-02-15,18:02:01:ERRR: TPC[2]=CAEN_V785 readout error=0x00000004, dumping data: (crate/crate.c:1791) -R4L-40 :read_meb :2021-02-15,18:02:01:INFO: ---[ Dump begin ]--- -R4L-40 :read_meb :2021-02-15,18:02:01:INFO: Start=0x30836a64 Bytes=0=0x0 -R4L-40 :read_meb :2021-02-15,18:02:01:INFO: ---[ Dump end ]--- -R4L-40 :read_meb :2021-02-15,18:02:01:ERRR: TPC: readout failed! (crate/crate.c:1333) -R4L-40 :read_meb :2021-02-15,18:02:01:ERRR: TPC: had problems, re-initializing. (crate/crate.c:1377) Remove the second one: 0x00030000 Since completly stable ==> maybe change the second one comments on r4l-13 test, as soon daq is stopped without real trigger it does not recover and a soft reboot of the crate is needed. Wait for sc21 or 22 to be in the beam line to setup Back to r4l-40, started crashing again due to the ADC, so we remove the last surviving ADC from the crate... we leave the barrier in in case to not confuse Go4 --16/02 connect to vulom r4l-13 the sc22l and sc22r 12:53: Running all 5 nodes in x86l-93all. Removed MTDC32 in vme_tof, its event counting is all over the place. Could not find differences in the regs of the trav-mus mdpp16 between the old and new nurdlibs, let's try the new one again with settings from sec_s468/TEST/main.cfg. If it doesn't work, we'll have to hack in the default dir in the f_user. --17/02 Traveling music worked, but one channel was missing and the resolution is not the best. mhtdc will be given to Hans at first oportunity For the time being the piedestals are not removed ==>currently 12us @ 18:07 ==> 6 us in the morning ==> we try 14 it is back so we decide to put 20 us to keep it in We had to remove the vetar cause of readout error due to dt Since we increased the gate we have a readout error in the music once in a while it could be cvt too short. --18/02 The Traveling music diseaper from the data at a moment --20/02 DAQ quicly restarted, permits to get the traveling music data backi --29/03 Learn today we need the fission crate (was not planned) and the mr-tof crate (was not planned), work until 1am with Hans to get the grey rabbit time stamping running from mh and not s2. Works at the end, some reshufling of vulom positions in main.cfg, new directory /sev_s511mar21 --05/03 Switch to despec experiment, new directory, /des_s452mar21 Go from grey rabit to white rabit time stamping Andrew shows time correlation works between FRS and DESPEC Issue with trigger 11 showing in data without reason (?) like a pulser on or off spill In the night go4 start to crash while reasonably stable before claim wron geo in proc id 30 --06/03 Christine remove the scaler without logging anything cause it had changed its geo in the data. Seems to be the reason for the crash of go4. Call Stephane at 10 PM to investigate, the ADC in proc id 30 switched from 3 to 2 in geo, not clear why, decide to run the night like that. --07/03 Forsenic analysis ==> from s511 data Event 150947 Type/Subtype 10 1 Size 1332 Trigger 1 SubEv ProcID 10 Type/Subtype 10 1 Size 20 Ctrl 20 Subcrate 0 00000100 03e17862 04e11a8c 05e1ca55 06e10001 SubEv ProcID 10 Type/Subtype 12 1 Size 336 Ctrl 20 Subcrate 0 SubEv ProcID 10 Type/Subtype 36 3600 Size 20 Ctrl 20 Subcrate 0 SubEv ProcID 35 Type/Subtype 12 1 Size 100 Ctrl 20 Subcrate 0 SubEv ProcID 30 Type/Subtype 12 1 Size 444 Ctrl 20 Subcrate 0 80000000 60410b0f babababa 0480497a 0000f74b 08000000 10022058 18000000 2025d6f6 2800005c 30000002 38000021 400000a5 480000a5 5008949c 58000000 60000000 6802497b 70000000 78000000 80000000 88000000 90000000 98000000 a0000000 a8000000 b0000000 b8000000 c0000000 c8000000 d0000000 d8000000 e0000000 e8000000 f0000000 f8000000 4a000200 480e49da 480f4f2e 4c02497a 42000100 40165fff 4402497a 1a002000 18004042 181040d7 18014063 18114056 18024084 18124093 1803405e 1813408a 18044045 181440c3 1805405c 1815408a 18064063 18164092 18074066 18174080 18084087 181840cf 18094055 1819408e 180a4060 181a40e3 180b4063 181b408a 180c406e 181c40c3 180d4075 181d4089 180e4094 181e407e 180f4068 181f408e 1c02497a 62002000 6000404a 60104061 6001406b 6011405b 60024065 60124068 60034072 60134045 6004405f 60144078 6005405d 60154056 6006406f 60164078 6007406a 6017405c 60084079 60184058 6009405b 60194067 600a4073 601a4068 600b4071 601b406c 600c4069 601c4064 600d4071 601d4064 600e4076 601e406f 600f4071 601f405e 6402497a SubEv ProcID 20 Type/Subtype 12 1 Size 160 Ctrl 21 Subcrate 0 SubEv ProcID 45 Type/Subtype 12 1 Size 72 Ctrl 20 Subcrate 0 SubEv ProcID 40 Type/Subtype 12 1 Size 76 Ctrl 20 Subcrate 0 Event 150948 Type/Subtype 10 1 Size 1420 Trigger 1 SubEv ProcID 10 Type/Subtype 10 1 Size 20 Ctrl 20 Subcrate 0 SubEv ProcID 10 Type/Subtype 12 1 Size 336 Ctrl 20 Subcrate 0 SubEv ProcID 10 Type/Subtype 36 3600 Size 20 Ctrl 20 Subcrate 0 SubEv ProcID 35 Type/Subtype 12 1 Size 100 Ctrl 20 Subcrate 0 SubEv ProcID 30 Type/Subtype 12 1 Size 444 Ctrl 20 Subcrate 0 80000000 60410b0f babababa 0480497b 0000f756 08000000 10022067 18000000 2025d800 2800005c 30000002 38000021 400000a5 480000a5 5008949c 58000000 60000000 6802497c 70000000 78000000 80000000 88000000 90000000 98000000 a0000000 a8000000 b0000000 b8000000 c0000000 c8000000 d0000000 d8000000 e0000000 e8000000 f0000000 f8000000 4a000200 480c4572 480f4fff 4c02497b 42000100 40165fff 4402497b 1a002000 18004040 181040d7 18014064 18114056 18024084 18124093 1803405c 1813408a 18044047 181440c4 1805405c 1815408a 18064062 18164092 18074067 18174081 1808408d 181840d0 18094054 1819408e 180a4064 181a40e4 180b4063 181b408a 180c4070 181c40c4 180d4074 181d4089 180e4092 181e407f 180f4068 181f408f 1c02497b 62002000 6000404a 60104061 6001406b 6011405b 60024065 60124068 60034072 60134045 60044060 60144078 6005405d 60154056 6006406e 60164078 6007406a 6017405c 60084078 60184058 6009405b 60194066 600a4073 601a4067 600b4071 601b406c 600c4069 601c4063 600d4071 601d4063 600e4076 601e406e 600f4071 601f405e 6402497b SubEv ProcID 20 Type/Subtype 12 1 Size 256 Ctrl 21 Subcrate 0 SubEv ProcID 45 Type/Subtype 12 1 Size 72 Ctrl 20 Subcrate 0 SubEv ProcID 40 Type/Subtype 12 1 Size 68 Ctrl 20 Subcrate 0 *** From first data writen in s452 ************* Event 150947 Type/Subtype 10 1 Size 1332 Trigger 1 SubEv ProcID 10 Type/Subtype 10 1 Size 20 Ctrl 20 Subcrate 0 SubEv ProcID 10 Type/Subtype 12 1 Size 336 Ctrl 20 Subcrate 0 SubEv ProcID 10 Type/Subtype 36 3600 Size 20 Ctrl 20 Subcrate 0 SubEv ProcID 35 Type/Subtype 12 1 Size 100 Ctrl 20 Subcrate 0 SubEv ProcID 30 Type/Subtype 12 1 Size 444 Ctrl 20 Subcrate 0 80000000 60410b0f babababa 0480497a 0000f74b 08000000 10022058 18000000 2025d6f6 2800005c 30000002 38000021 400000a5 480000a5 5008949c 58000000 60000000 6802497b 70000000 78000000 80000000 88000000 90000000 98000000 a0000000 a8000000 b0000000 b8000000 c0000000 c8000000 d0000000 d8000000 e0000000 e8000000 f0000000 f8000000 4a000200 480e49da 480f4f2e 4c02497a 42000100 40165fff 4402497a 1a002000 18004042 181040d7 18014063 18114056 18024084 18124093 1803405e 1813408a 18044045 181440c3 1805405c 1815408a 18064063 18164092 18074066 18174080 18084087 181840cf 18094055 1819408e 180a4060 181a40e3 180b4063 181b408a 180c406e 181c40c3 180d4075 181d4089 180e4094 181e407e 180f4068 181f408e 1c02497a 62002000 6000404a 60104061 6001406b 6011405b 60024065 60124068 60034072 60134045 6004405f 60144078 6005405d 60154056 6006406f 60164078 6007406a 6017405c 60084079 60184058 6009405b 60194067 600a4073 601a4068 600b4071 601b406c 600c4069 601c4064 600d4071 601d4064 600e4076 601e406f 600f4071 601f405e 6402497a SubEv ProcID 20 Type/Subtype 12 1 Size 160 Ctrl 21 Subcrate 0 SubEv ProcID 45 Type/Subtype 12 1 Size 72 Ctrl 20 Subcrate 0 SubEv ProcID 40 Type/Subtype 12 1 Size 76 Ctrl 20 Subcrate 0 Event 150948 Type/Subtype 10 1 Size 1420 Trigger 1 SubEv ProcID 10 Type/Subtype 10 1 Size 20 Ctrl 20 Subcrate 0 SubEv ProcID 10 Type/Subtype 12 1 Size 336 Ctrl 20 Subcrate 0 SubEv ProcID 10 Type/Subtype 36 3600 Size 20 Ctrl 20 Subcrate 0 SubEv ProcID 35 Type/Subtype 12 1 Size 100 Ctrl 20 Subcrate 0 SubEv ProcID 30 Type/Subtype 12 1 Size 444 Ctrl 20 Subcrate 0 80000000 60410b0f babababa 0480497b 0000f756 08000000 10022067 18000000 2025d800 2800005c 30000002 38000021 400000a5 480000a5 5008949c 58000000 60000000 6802497c 70000000 78000000 80000000 88000000 90000000 98000000 a0000000 a8000000 b0000000 b8000000 c0000000 c8000000 d0000000 d8000000 e0000000 e8000000 f0000000 f8000000 4a000200 480c4572 480f4fff 4c02497b 42000100 40165fff 4402497b 1a002000 18004040 181040d7 18014064 18114056 18024084 18124093 1803405c 1813408a 18044047 181440c4 1805405c 1815408a 18064062 18164092 18074067 18174081 1808408d 181840d0 18094054 1819408e 180a4064 181a40e4 180b4063 181b408a 180c4070 181c40c4 180d4074 181d4089 180e4092 181e407f 180f4068 181f408f 1c02497b 62002000 6000404a 60104061 6001406b 6011405b 60024065 60124068 60034072 60134045 60044060 60144078 6005405d 60154056 6006406e 60164078 6007406a 6017405c 60084078 60184058 6009405b 60194066 600a4073 601a4067 600b4071 601b406c 600c4069 601c4063 600d4071 601d4063 600e4076 601e406e 600f4071 601f405e 6402497b SubEv ProcID 20 Type/Subtype 12 1 Size 256 Ctrl 21 Subcrate 0 SubEv ProcID 45 Type/Subtype 12 1 Size 72 Ctrl 20 Subcrate 0 SubEv ProcID 40 Type/Subtype 12 1 Size 68 Ctrl 20 Subcrate 0 *************** clearly the scaler did not switch its GEO adress from 3 to 1 5bits geo, 1bit set 1,2 dummy, 6 nwords, 2 bit trigger (00is external), 0x10022067 means geo2, 1missing (bit26), 0 words, vme trigger!!! before was 0x0480497b means geo0, 1, dummy0, 32 words, vme trigger But two words before and 0x048 is present so there was nothing For the file s452_07_implantation_s4_0021 scaler data seems normal again and everything good ==> after scaler removed, ADC change geo to 2 from 3... ==> and TDC comes systematically with an event which shall not be here SubEv ProcID 30 Type/Subtype 12 1 Size 304 Ctrl 20 Subcrate 0 80000000 6043f938 babababa 4a000000 4c01d5de 42000100 40165fff 4401d5de 12002000 10004045 101040d5 10014065 10114055 10024085 10124094 10034063 So my understanding, but NO information is given: error on scalers came at night, something was done at 19:22 ==> 19:25 runs==> 0021 it is fine ==>20:52 they close file and take decision to remove the scaler ==>21:09 start again ==> now the ADC is Geo2 ==> normal nurdlib phenomenom After discussion with Christine she confirms this course of events The ADC has no PAUX but a PAUX set to 11... We will try with the scaler again: ==> put back everything in it seems again normal... and I reconected the two channels from the v775 which are really noisy all is gud again Conclusion it always works, the issue was in the unpacker: presence of event 10,11,12,13 triggers without readout trigger, which meant nothing in the subevent, despite this it was unpacked and error message at not finding babababa was not existing. -----------------------08/03 and 09/03 ADC investigation ==> some discussion on loss of data on proc 30 ADC happeneds, it turn out it was again an error in the unpacker, the daq worked fine ----------------------15/03 switch to S455 r4l-13 for the r3b people, they do what they want we run system in mh alone, normal trigger box new directories sec_s530 for Jianwei to start working ----------------------16/03 prepare s530 new place s530 for the experiment -- tried to run with new nurlib did not work, needed to go back to r3bfuser from 2020... november ---------------------18/03 tried two new QDC in the fission crate, issue maybe cause not real gate generated ---------------------19/03 Learnt no need for the QDC of yesterday Added a v785 to the traveling music crate: PAUX 19 ==> realize the ones of vme_user have autopedestal!!! Confiugred the vulom of trav crate to have the proper self trigger and generate an ADC gate from LEMO_OUT(2) -------------------20/03 Seems the autopeds are present sice we use nurdlib, most probably did not do anything before... It is by default "false" in the cfg file of Nurdlib.... Hans does not know what for... decide to switch all off. I move to the new directory, here clean up vulom, make sure we do not play with MEB Should work from now on, some channels are cleaned and pedestals set configure full DAQ for 5 crate system, MH, fission and traveling music ----------------- 14/04 Few realization, we ran with multi event enabeled at 1 for most (except s530 and s455) experiments this year ==> for s460 we keep it to be like before in DESPEC ==> the channel accepted trigger is at 15 in vme_user in the scaler ==> the geo adress changed in between for s2 crate, main.cfg changed accordingly ==> no news on the online