Thanasi taking notes ## Feb 10 work ### `run6` - 1:56 pm TM1 on - 1:58 pm TM2/exp on, can `ssh` - 1:59 pm running `formatter` - 1:59 pm stopping, will run `formatter_isohk` instead - 2 pm running running `formatter_isohk` - 2:05 or so lost HK, didn't get Timepix on. ### `run7` - X axis vibe. - Power cycled to get HK back. - 2:08 pm started running `formatter_isohk` - Didn't make it, switched to `formatter` - Made it through this vibe without formatter or HK cutting out. - Saved pcap. Not much CMOS data stored. ### `run8` - Y axis vibe. - Formatter still running (continuously) after last vibe - Formatter hung (Housekeeping-related) around 2:45 pm - Started up `formatter_nohk` to shutdown CMOSes and CdTe - 2:50 pm Power cycled - 2:52 pm starting `formatter` (not `_nohk`) - 3:01 pm starting vibe - Saved pcap. Not much CMOS data stored. - 3:15 pm powered off subsystems via uplink, leaving Formatter + HK running. 1.5 A. ### `run9` - ~~Z axis random~~ - 3:24 pm started `formatter` (not `_nohk`) - Started GSE, getting packets - Stopped 3:27 pm, will power cycle - Stopped again 3:32 pm and power cycled - Stopped again 3:37 pm and power cycled ### `run10` - ~~Z axis random~~ - 3:39 pm starting `formatter` (not `_isohk` or `_nohk`) - 3:45 pm all subsystems on, taking data. - 3:45 pm `formatter` froze, no more hk. - 3:50 pm `formatter` froze again, switched to `_nohk`, but that failed to pull data from SPMU-001 - 3:52 pm power cycled, started `formatter` again - 3:54 pm turned on CMOS 1, see data come in - 3:55 pm turned on CMOS 2, see data come in - 3:56 pm turned on CdTes and DE - 3:57 pm turned on SAAS and Timepix, but current 3.8 A. - 3:58 pm `formatter` stopped due to HK. Started `_nohk` just to test, but no SPMU-001 data. - 4:03 pm another power cycle. `formatter` runs and get HK data to ground. - 4:06 pm CMOSes on, DE on. - 4:08 pm CdTes on - 4:09 pm Timepix on - 4:10 pm SAAS on - 4:11 pm CdTes data taking started up, CMOSes too. - Confirm pedestals, confirm CMOSes nonzero packets - 4:20 pm scrubbing Z axis to Monday or something. Fix problem over weekend. - 4:25 pm shutting down CMOSes, CdTes, Timepix, and SAAS. Current 1.5 A after. Formatter still running. - 4:48 pm formatter still running. - 5:07 pm still running - 5:30 pm still running ### `run12` - Formatter still running from before - 5:33 pm starting up subsystems - 5:35 pm all on, 4 A - 5:37 pm started CdTe data collection - 5:42 pm started CMOS data collection - 5:45 pm restarting GSE - 5:49 pm stopped taking data for all subsystems and powered them off. - 5:51 pm Formatter froze! - 5:56 pm building one that: - Omits UDP uplink commands in `Circle` - Tries a local variable swap buffer instead of a TransportLayer property for `tcp_local_receive_swap`. - 6:02 pm, new `formatter_isohk` has been built. - Can ping both SPMU and Housekeeping. - `formatter_isohk` hangs when trying to connect (I think to SPMU) - Same for `formatter`. - In Wireshark (laptop), seeing retransmissions to 192.168.1.100 from 192.168.1.8 - 6:13 pm reboot (in shell) Formatter. Still no luck starting up `_isohk` as expected. ### `run13` - 6:19 pm power cycled Formatter. Back in and running `_isohk`. - See HK data coming down. - 6:21 pm powered on CMOSes - See blank frames come down. - Housekeeping was `ABANDON`'d shortly after. - 6:24 pm stopped formatter - Can ping SPMU, not hk. - Started the `_nohk` version successfully, get CMOS data - Trying uplink commands to `stop_to_start` CMOSes - Data gets to ground fine from CMOSes. - Try getting rid of non-RTD requests from HK. - 6:28 pm shut down both CMOSes. - 6:30 pm power cycled Formatter. ### `run14` - 6:33 pm building `formatter_rtdhk` that only queries RTD data from Housekeeping board (not power ADC or `intro`) and `ABANDON`s HK system on read failure (after closing socket). - 6:42 pm running `formatter_rtdhk` - Getting data down. - 6:45 pm CMOSes on, blank frames coming down. Turning on DE. - HK un`ABANDON`ed - 6:46 pm started CdTes, still see HK - 6:47 pm turning on SAAS and Timepix - 3.3 A total Exp current - Re-sent CdTe and SAAS and Timepix power on commands - Now at 3.95 A. Good. - Still getting HK - 6:49 pm starting CdTe data collection - Full mode, 60 V bias, see write pointers moving, see some pedestal - 6:51 pm starting CMOS data collection - HK still un`ABANDON`ed - CMOS frames looking good in Wireshark - 6:52 pm stopping CdTe data collection - 6:53 pm stopping CMOS data collection - and shut down - HK still un`ABANDON`ed - 6:55 pm powered off all subsystems, at 1.55 A. ### `run15` Repeat of `run14` with Exp power cycle in between. - 6:57 pm running `formatter_rtdhk` - 6:58 pm get HK down, powering on CMOS 1 - 6:59 pm powering CMOS 2 on - Blank frames from the CMOSes in Wireshark - 7:01 pm powered on DE, the CdTes too close together. Always got bad write pointer length from DE. Power cycled DE and error cleared. - 7:02 pm all subsystems on, 4 A. - DE write pointer problem is coming back intermittently. - 7:04 pm sent CMOSes `stop_to_start` , data looks good in Wireshark - 7:05 pm started CdTe data collection, pedestals noisy but fine. - 7:05 pm still getting HK data. - 7:06 pm stopped and shutdown CMOSes - 7:07 pm stopped DE - 7:08 pm powered off all subsystems, current 1.56 A. - Stopping `formatter_rtdhk` . - HK still un`ABANDON`ed - Power cycled Formatter ### `run16` New build of `formatter_nointrohk`, which queries power ADC and RTDs, but not the introspection fields (clock, errors, state). - 7:18 pm running `formatter_nointrohk` - Getting data down - 7:19 pm powered on CMOS 1 and 2, very close to each other. May cause SPMU problem. - Get blank frames from both down - 7:21 pm not seeing any ADC data come out of Formatter. Stopped. Rebuilding. - Power cycle after. - 7:49 pm running `formatter_nointrohk` again. - Getting HK data - 7:50 pm CMOSes on - HK data stops, stopped running Formatter. - 7:52 pm started `_nohk` - Send `enable_double_commands` and `shutdown` to CMOS 1, got no response - Sent `stop_to_start`, no response - Still see blank data coming out of CMOS 1. Still no response to command. - 7:54 pm power cycling >[!info] So is the problem [AD7490](https://www.analog.com/media/en/technical-documentation/data-sheets/ad7490.pdf)? ### `run17` - Started `formatter_rtdhk` at 7:57 pm - Getting just RTD data - 7:58 pm both CMOSes on, getting blank frames down - 8:00 pm DE and CdTes on - DE showing oddness (bad write pointer lengths) again. - 8:01 pm all systems on, 3.9 A - 8:02 pm started CdTe data collection, see noisy pedestals come down. - 8:03 pm started CMOS data collection - Raw frames look good in Wireshark - 8:04 pm set CdTe bias to zero, changed to sparse mode, started taking data again - 8:04 pm went back to full mode and started taking data, write pointers were too slow - 8:05 pm stopped CdTe data taking - 8:06 pm stopped and shutdown CMOSes - 8:07 pm shutting down all subsystems. 1.57 A. - 8:08 pm power cycled. ### `run18` - Started `formatter_nointrohk` at 8:13 pm - Getting RTD and ADC data - 8:15 pm powering on CMOS 1 and 2 - Seeing blank packets - 8:16 pm powering on CdTes - Still getting HK data - 8:17 pm all subsystems on, 4 A - 8:18 pm started CdTe data taking, full mode, 60 V bias - 8:19 pm starting CMOS data taking - Looks good in Wireshark - Still getting HK data - 8:20 pm setting CdTe 0 V bias, restarting observe - 8:21 pm stopping CdTe data taking - HK had stopped - Timeout was to talk to AD7490. - 8:25 pm stopped Formatter. Copying out data. ## Feb 12 work ### `run1` - Z-axis random and 0.5 g sine sweep - 8:15 am exp power on, can `ssh` formatter - 8:17 am running `formatter_rtdhk` - Getting downlink - 8:19 am powering on CMOSes - Confirm blank frames are downlinked - 8:20 am powering on DE - 8:21 am powered on canisters - Still getting data - 8:22 am Timepix and SAAS on, 3.9 A total - 8:23 am DE starting data taking, bias 60 V - 8:25 am pedestals looking fine - 8:25 am starting CMOSes - Raw data looks good in Wireshark - 8:32 am things look fine on the ground, moving exp to vibe table. - 8:52 am data still looks fine, bolting to z-axis table. - **9:06 am finished z-random** - Still getting downlink, things look fine - **9:13 am starting 0.5 g sine sweep** - Held at T–10 - 9:19 am power cycled exp ### `run2` - 9:21 am starting `formatter_rtdhk` - 9:27 am CdTe and CMOS data looks good - 9:28:29 am starting half-g sine in T–120 s - 9:30:29 am starting half-g sine - Data looks good - Housekeeping abandoned - From Wireshark, last HK packet was received 9:28:58 am, ~1.5 minutes before start of test. - Looked fine throughout, eventually shut down subsystems and powered off. ### `run3` - Vibe table found issue with accelerometers during sine sweep. Repeating 0.5 g sine survey. - 1:32 pm starting formatter, confirm HK data - 1:33 pm CMOSes on, confirm blank frames - 1:36 pm HK ABANDONed, maybe ran wrong formatter. Power cycling. - 1:40 pm same problem, with `formatter_rtdhk`. Can still ping HK board, will try restarting software. Saw big dropout of detector HK data when CMOS was turning on. - 1:42 pm can restart `formatter_rtdhk` successfully. But HK abandons very soon after. In Circle.cpp, was still requesting response from AD7490, just not factoring it into the `any_zero` variable. Get rid of the request. - On second thought, this was probably just because the last executable I built used those settings (`formatter_nointrohk`), which *did* touch the ADC. - 1:46 pm building `formatter_rtdhk` that **DOESN'T** make the request to AD7490 at all. - 2:06 pm frozen again (with newly built `formatter_justrtdhk`) - 2:13 pm built `formatter_nordhk` which only exposes the HK system for commanding, not reading. Also locks up. Later found this version was built incorrectly, and *was* taking data from RTDs. - 2:32 pm running `formatter_justrtdhk` again, all systems on, CdTe and CMOS looks good - 2:36:00 pm starting sine survey - 2:38:26 pm end of test - Still getting everything - Post test, `stop` ed CdTe and CMOS data taking. Still getting HK data. - 2:53 pm still getting data - 3:03 pm still getting data (including HK) ### `run4` - Sine test. - 3:07:01 pm started new GSE recording (and a new pcapng around the same time). - 3:11 pm commanded CdTes and CMOSes to start collecting. - Don't see CdTes moving, stepped through whole startup process again. - 3:18 pm or so HK dropped. Seems related to commanding. - 3:26:19 pm testing in T–120 s - Getting data (CMOS + CdTe), no HK (as expected) - 3:28:19 pm starting test - 3:31 pm test done, commanding CdTe stop - 3:32 pm shutdown CMOSes. Exp power off.