Feb 3 2024 x-ray alignment test.
Thanasi taking notes
## Contents
- CdTe data taken `run1` to `run19`.
- CMOS 1 data taken `run20` to `run22`.

The `run*` folders contain a mix of information depending on the test. There may be:
- Screenshots.
- A `gse/` folder. This contains data logged on the GSE by the ["old logger"](https://github.com/foxsi/foxsi-4matter/blob/main/util/logudp.py).
- A `de/` folder. This contains data recorded by the CdTe DE locally. 
- A folder formatted `day-month-year_hour_minute_second/`. This contains data logged on the GSE by the ["new logger"](https://github.com/foxsi/GSE-FOXSI-4/blob/main/FoGSE/listening.py), which *is unreliable!* Don't trust the data in this folder!
- Various `.txt` files of terminals, for debugging work later.
## Setup
- 9 am moved Exp to high bay, set up facing x-ray generator plate. Setting up power supply.
	- Exp chassis grounded to building. Probes ok.
- 9:05 am connecting LN2 cooler harnessing to Exp
- 9:07 am setting up 
- 9:30 am connected power, Kris verified
- 9:34 am turned on power to Exp, current normal ~0.3 A
- 9:34 am can `ssh` to Formatter
- 9:35 am can `ping` Housekeeping and SPMU-001 from Formatter
- 9:35 am turned on Timepix for connectivity check
	- Savannah can `ssh` to Pi
- 9:38 am startup DE
	- Can `ssh`, `htop` shows `CdTe_DE` running
- 9:40 am turn on CMOS 1
	- Can `ssh`, see normal stuff running (`rmap_slave`, `control`, `housekeeping`).
- 9:42 am turn on CMOS 2
	- Can `ssh`, see normal stuff running (`rmap_slave`, `control`, `housekeeping`).
- 9:44 am turn on all canisters
## Data collection
### `run1`
**To verify Formatter data collection from all systems.**

- Config:
	- Formatter software from yesterday afternoon. Will become `v0.1.3`.
- 9:49 am starting Formatter software:
```bash
./bin/formatter --verbose --config foxsi4-commands/systems.json
```
- 9:50 am no Timepix ping
- Get 4 canisters pingback
- Get data downlinked from all systems (including Timepix)
- 9:56 am sending CMOS 1 `stop_to_start`
	- See data in Wireshark
- 9:57 am sending CMOS 2 `stop_to_start`
- 9:58 am stopping GSE GUI, switching to `logudp.py`
- 10:01 am stopping `logudp.py`
- Pedestals look good.
- Got data from Timepix, despite no pingback initially.
- CMOSs get images, but no CMOS 1 PC data displays. Not sure why.
- Power 2.7 A at end.
- 10:10 am sending `stop` to CMOS 1
- 10:11 am sending `stop` to CMOS 2
- 10:11 am sending broadcast `stop` to CdTes
- 10:12 am sending `stop_readout` to CdTes
- 10:13 am sending `de_end`
- 10:14 am powering off CdTes
	- Write pointers not advancing for a while now
- Take from DE:
	- `foxsi4_cdte_20230524_055404_det*`
	- `CdTeDE_Log_20230524_054222.log`
	- `CdTeDE_COUT_Log_20230524_054222.log`
- 10:17 am power off CMOS 2 (not shutdown)
	- Still get data from CMOS 1. And DE.
- 10:18 am enable double commands to CMOS 1
	- Still get data from CMOS 1 and DE
- 10:19 am shutdown CMOS 1
	- Now no data from DE
- 10:20 am stop Formatter
- 10:21 am can still operate Housekeeping from GSE
	- Power off CMOS 1
- 10:21 am pulled data out of DE and shut off
### `run2`
**To verify Formatter data collection from all systems.**

- Removing CMOS 1 and 2 housekeeping sending from Formatter software, removed CdTe 1 housekeeping sending from Formatter software, rebuilt on Pi.
- 11:53 am power consumption up to 0.4 A just Formatter while building
- 11:58 am turned on all CdTe
	- Current 1.2 A
	- Can `ssh` DE
- 11:59 am turned on all CMOS
	- Current 1.9 A
	- Can `ssh` CMOS 1. `rmap_slave` and `control` running, no `house_keeping`.
	- Cannot `ssh` CMOS 2.
- 12:05 pm 
	- Can `ssh` CMOS 2. Same tasks running as CMOS 1. Disk full. Shimizu san explains this is why no housekeeping task. She will remove the data now.
- 12:09 pm
	- Cannot `ssh` CMOS 1. Pings back, but not at first.
	- Cable issue on Ethernet cable from CMOS 1 to GSE switch. Wiggled until it connects. 
	- Now can `ssh` CMOS 1. Disk full. Same tasks running as before.
- 12:16 pm start Formatter
	- Get weird pingback `0x44 44 22 22` from CdTe, but all canisters turned on.
		- Thanasi has seen this value from CdTe DE before for canister pingback. Not sure if it has special meaning.
- 12:19 pm send `enable_double_commands` and `remove_all_data` to CMOS 1.
	- Does not take (checked with `df` shell command inside CMOS 1).
	- Now getting no more Spacewire data from anybody. Also CdTe write pointers were bad, not sure for how long. Should check Formatter tape.
- 12:20 pm stopping Formatter.
- 12:23 cannot start Formatter software again (freeze during connection to SPMU). Power cycle Formatter.
- 12:24 pm start DE again. Canister status still `0x44 44 22 22`. Restart DE. 
- 12:27 pm power cycle all canisters.
- 12:30 pm start Formatter again.
	- Canisters ping ok
	- Can `ssh` CMOS 2, `df` 45%.
- Sent CMOS 2 `stop_to_start`
- 12:31 pm started GSE, looks good, longer integration time for CdTe would be nice. SEE NO CMOS 2 data
	- Only 1 frame logged for CMOS 2 PC, nothing else. Not running CMOS 1.
	- This is issue with the logger embedded in the GSE.
- 12:36 pm stopped GSE
- 12:38 pm running `logudp.py` for CMOS 2
	- QL looks nice, no PC image displayed with `demo_foxsi_dets*`.
	- 12:41 took screenshot, stopped screen recording.
### `run3`
**Biasing CdTe.**
- 12:56 pm focal plane temperature -20 ºC
- 12:59 pm set `broadcast_can_stop` 
	- Then set 60V bias broadcast
	- Confirmed in Formatter tape
- 1:01 pm set 200V bias broadcast
	- Confirmed in Formatter tape
- 1:03 pm stopping GSE. Going to swap to old logger.
## Getting GSE plugged into old logger
- Made these changes to `gse.py`:
```
f0 = AllCdTeView("/Users/foxsi/Documents/FOXSI/foxsi-4matter/log/gse/cdte1.log",
"/Users/foxsi/Documents/FOXSI/foxsi-4matter/log/gse/cdte2.log",
"/Users/foxsi/Documents/FOXSI/foxsi-4matter/log/gse/cdte3.log",
"/Users/foxsi/Documents/FOXSI/foxsi-4matter/log/gse/cdte4.log")

f1 = AllCMOSView("/Users/foxsi/Documents/FOXSI/foxsi-4matter/log/gse/cmos1_pc.log"),
"/Users/foxsi/Documents/FOXSI/foxsi-4matter/log/gse/cmos1_ql.log",
/Users/foxsi/Documents/FOXSI/foxsi-4matter/log/gse/cmos2_pc.log",
/Users/foxsi/Documents/FOXSI/foxsi-4matter/log/gse/cmos2_ql.log"
```
### `run4`
**Trying to take pedestal from CdTe**

- 1:13 pm sent broadcast `can_start`
	- No write pointer movement
	- Sent again 1:15
	- Power is 2 A
- 1:21 pm stopped log for this run with old logger
#### `run5`
**Actually taking pedestal for CdTe.**

- 1:22 pm starting GSE, try sending broadcast `start` to canisters again.
- Try sending broadcast 0V to canisters several times, no evidence in DE. Get confirmation in Formatter:
```
managing cdte1 system
in sync_send_buffer_commands_to_system()
command is not for frame read
got command for system. Sending...
in sync_send_command_to_system(), sending 000000000000000000000023 0102 fe 01 7d 02 00000103 fe 0000 00 00000000 00000c b5 3c3c0101030300003c3c3c3c 55 
expecting reply of length 20 B...
0x00 00 00 00 00 00 00 00 00 00 00 08 fe 01 3d 00 fe 00 00 aa 
got reply to command: 0x000000000000000000000008fe013d00fe0000aa
```
- Tried 0V setting again.
- 1:30 pm sent DE end.
- 1:32 pm sending DE power off, then back on.
- 1:34 pm can get back into DE.
	- Sent `idle`
	- Sent `init`
	- Sent `standby`
	- Broadcast `hv0v`
- 1:37 pm Broadcast `hv60v`
- 1:38 pm broadcast `hv200v`
- 1:39 pm set `de_obs`, `f4` status changes to 3
	- Broadcast `start`
- 1:40 stopped GSE, start old logger, see data come in
	- Saved to `run5`
### `run6`
**Taking CdTe pedestal with less LN2 flow.**

- Turned down LN2, taking more data.
- It's good enough!!
### `run7`
**Running new GSE, same Exp configuration as `run6`.**

- X-rays position 2
	- 30 kV, 1.01 mA
	- -20 ºC
- No change since earlier to LN2 valve
- Adding Kris's new `main` GSE to `~/Desktop/`
	- Wow Kris!
- 2:16 taking data
- 2:19 stopping
- 2:21 pm pulling DE data out
	- `data/foxsi4_cdte_20230524_0805*`
- 2:25 pm sent DE `broadcast_stop`
- 2:26 pm sent DE `readout_stop`
- 2:29 pm `enable_double_commands` to CMOS 2
- 2:30 pm `shutdown_cmos` to CMOS 2
	- CdTe cuts out shortly after.
- 2:34 need to restart formatter. Will set 0V bias, will need to ramp back up.
- 2:34 Formatter back up. Starting software.
	- All canisters ping OK.
### `run8`
**Setting CdTe sparse mode.**

- Same x-ray and cooling setup as `run7`
- No CMOS now
- 2:38 pm commanding broadcast 200 V
- 2:39 pm commanding sparse readout mode
	- Then command broadcast_start_observe
- 2:41 pm starting log
- 2:58 pm stopping log
- Lindsay adjusts LN2 valve
### `run9`
**Reduced LN2 flow compared to `run8`.**

- Lower LN2 rate, same x-ray setup as `run7`.
- 3 pm starting log
- GSE looks good
- 3:03 pm shut off LN2.
- 3:09 pm focal plane temperature -15.3 ºC
- DE recording to `foxsi4_cdte_20230524_08480*`
- 3:12 pm stopping logging.
### `run10`
**Trying less x-ray current.**

- X-ray generator down to 0.503 mA, 30 kV (still position 2).
- Restarted LN2 at some point.
- Focal plane at -16 ºC.
- 3:13 pm starting logging, restarted GSE
- Not making clear image in GSE.
- 3:17 stopped logging.
### `run11`
**"Messing around" with x-ray setup.**

- Focal plane at -20 ºC
- 3:19 taking data
- 3:24 pm re-sent bias 200V.
- 3:32 pm stopped logging.
### `run12`
**Trying more x-ray current.**

- X-ray generator
	- 30 kV, 2.0 mA
- Temp -20.2 ºC
- `foxsi4_cdte_20230524_091309_det1_0000000003`
- `foxsi4_cdte_20230524_085828_det2_0000000003`
- `foxsi4_cdte_20230524_085937_det4_0000000003`
- `foxsi4_cdte_20230524_092124_det3_0000000003`
- `foxsi4_cdte_20230524_092129_det1_0000000003`
- Formatter stopped when sending `0x01 f0`.
	- To Housekeeping. HK data not sent for a while.

- Wrote `python` thing to set bias to 0 V safely. Doesn't seem to work. 
- 4:38 pm cutting power from power supply.
- 4:39 pm
	- Can get back in Formatter
	- Can ping Housekeeping board
### `run13`
**Moving x-ray generator to position 3.**

- X-ray generator to position 3
- 5 pm turning on all CdTe
	- 1.2 A
- Focal plane -20 ºC
- 5:01 pm turning on x-rays
	- Check logbook for currents
- DE log: `CdTeDE_COUT_Log_20230524_101722.log`
- 5:02 pm starting Formatter
	- 4 detectors pingback ok
	- Recording to `foxsi4_cdte_20230524_101950*`
- 5:04 pm starting logging
	- Stopped soon after to raising bias
- 5:12 pm starting logging
- 5:14 pm DE logging to `foxsi4_cdte_20230524_102824*`
- 5:16 pm stopped logging, grabbed DE data out
### `run14`
**More data, same configuration as `run13`.**

- Same condition as `run13`
- 5:19 pm logging again
- 5:24 pm stopping logging
- Same filenames as last DE log, but more data in them: `foxsi4_cdte_20230524_102824*`
	- Grabbed.
### `run15`
**Moved x-ray generator to position 4.**

- 5:32 pm X-ray generator to position 4
	- Still aligned
- Exp power: 1.3 A
- Focal plane -20 ºC
- DE still logging to last file
- 5:37 pm x-rays on
- 5:38 pm Housekeeping not responding, eventually causes Formatter loop to stop. 
	- Can ping HK now
>[!warning] Need to prevent HK TCP disconnect from stopping main loop.
>[Tracking here](https://github.com/foxsi/foxsi-4matter/issues/62)
>#### Temporary fix:
>- Thanasi removing housekeeping system from loop.
>- [x] Make housekeeping IP `127.0.0.1` in `systems.json`
>- [x] No hk in `main`
>- [x] No hk in `Circle`
>- [x] TransportLayer retains `*housekeeping*` methods but will not open, bind, or connect `local_tcp_housekeeping_sock`. And interface methods should never be called from Circle.
>- Changes made to clone of `foxsi-4matter` `main` branch located on GSE at: `~/Documents/FOXSI/foxsi-4matter_nohk`

>[!info] No control of housekeeping board until full power cycle!

>[!info] Main loop will enter without setting DE bias or readout mode!

- 6:20 while doing this, DE recording:
	- `foxsi4_cdte_20230524_112050_det1_0000000002`
	- `foxsi4_cdte_20230524_112031_det3_0000000002`

### `run16`
**Again in position 4, this time running without accessing Housekeeping system:**
```
./bin/formatter_nohk --verbose --config foxsi4-commands/systems.json
```

- 6:24 ramping voltage to x-ray generator, no current yet.
- 6:24 pm starting Formatter. Seems to work. See CdTe coming through (good pingback canisters as well).
- 6:25 pm 1 mA, 30 kV
	- Cooler -20 ºC
- 6:26 pm starting logging
- 6:27 pm started GSE, looks like bias is 60V
- 6:30 pm set bias and sparse mode
- DE recording to `foxsi4_cdte_20230524_11472*`
- 6:38 pm stopped taking data
	- Transferring DE data.
### `run17`
**Same condition, just a new recording.**

- DE still on `foxsi4_cdte_20230524_11472*`
- Circle is there!
### `run18`
**Moving x-ray generator to position 5.**

- Moving to position 5
- Focal plane at -20 ºC
- 7:21 pm turning on x-ray
- 7:22 pm 30 kV, 1 mA
	- And taking data to file
- 7:27 pm CdTe DE taking data
	- dets 2-4 still on `foxsi4_cdte_20230524_11472*`
	- det 1 rolled over to `foxsi4_cdte_20230524_124029_det1_0000000004`
- 7:32 pm data looks good!
	- Pulling out rest of data from DE
- 7:33 stopping logging
### `run19`
**Taking full readout mode pedestal, no x-rays.**

- Going to take full readout mode pedestal. Bias still at 200 V. X-rays off.
- 7:35 pm `set_can_broadcast_stop`, then `set_can_broadcast_readout_stop`.
- 7:36 pm `set_can_broadcast_full`, then `set_can_broadcast_start`.
- 7:37 pm logging on GSE
	- DE logging to `foxsi4_cdte_20230524_125240_det*`.
- 7:40 pm turned off lights in high bay
	- Turned off when DE files this many bytes in them:
```
det1: 8301764 B
det2: 4925424 B
det3: 9612964 B
det4: 2893064 B
```
- 7:42 pm stopped GSE logging.
- 7:48 pm sent all stopping commands to DE (later clarify with Minami san the correct sequence), then broadcast 0 V.
## Setup CMOS
- 7:51 pm power cycled experiment
- 7:52 pm can `ssh` formatter, turned on both CMOSes
	- Experiment current ~1.0 A
	- Could use Housekeeping system again for power on/off commanding.
- 8:05 pm Shimizu san updated ratio of PC to QL images taken to prioritize QL data downlink for alignment.
	- For both CMOS.
- Going to run Formatter without housekeeping:
```
./bin/formatter_nohk --verbose --config foxsi4-commands/systems.json
```
### `run20`
**Setting up CMOS for checks.**

- 8:08 pm starting Formatter 
- 8:11 pm sent CMOS 1 `stop_to_start`.
- 8:12 pm sent CMOS 2 `stop_to_start`.
- 8:14 pm all downlink data (in Wireshark) is zeros. 
	- Formatter running fine, packets logging on GSE
	- Saw `stop_to_start` commands land correctly in Formatter, got SpaceWire confirmation replies.
- 8:16 pm stopping logging on GSE, all zeros. Stopping GUI.
	- Issue is in CMOS software, an exposure setting was constrained by Shimafuji software. Shimizu san editing.
- 8:28 pm CMOSes exposure settings fixed (they are running already)
- 8:29 pm getting data
- 8:36 pm data was old ^^^. Sent `stop_to_start` to both CMOSes again. Can see data in GSE.
	- Only the non-old data was saved to `run20`.
- 8:37 pm stopped GSE and logging. Going to start x-rays.
### `run21`
**Moved x-rays to position 0, taking data with CMOSes.**

- X-rays in position 0
	- 1 mA, 30 kV
- Focal plane -20 ºC
- 8:40 pm started logging on ground
	- Can't make out image
- 8:45 pm stopping logging
- 8:47 pm send `cmos_stop_to_start` to both CMOS for next exposure.
### `run22`
**Same setup, x-ray generator MAX CURRENT.**
- X-rays same position 0
	- **10** mA, 30 kV.
	- Started ramping up current 8:51 pm
- -20 ºC focal plane
- 8:53 pm at x-ray generator current 9.96 mA, 30 kV.
	- Logging data
- 9:04 pm stopped logging on GSE.
- 9:15 or so sent `stop` commands to both CMOSes. 
- 9:20 or so x-ray generator not working. Letting it rest. 
	- Stuck at 0.09 mA, able to ramp voltage to 20 kV. Not sure when this happened.
- Nominal downlink datarate (through Formatter) from`run22` is ~1.3 Mbps. 
	- This is comparable to downlink rates seen with only CdTe, and all CMOS and CdTe.
- 11:10 pm formatter still running, pulling data from CMOSes!
### `end`
- 11:12 pm Shimizu san shutdown CMOS 1 from `ssh` (not uplink command).
	- Shortly after, stopped receiving data from CMOS 2.
- 11:13 pm Shimizu san shutdown CMOS 2 same way.
- 11:15 pm powered off CMOSes. Experiment total current 0.3 A (nominal).
- 11:20 pm bringing focal plane temperature back up.