Duplicated nights with ignorenonwear=TRUE

[Luca-Menghini]

Hello! Thank you so much for developing and maintaining this amazing package!

I'm writing as I'm observing an issue when I use the GGIR shell function with 4-night 3-axis acceleration data.
Briefly, my data were recorded with a wrist-worn device whose memory capacity is about 40 hours, so to collect 4 nights (Mon-Thu) I need to replace it with a new device on Wed afternoon (with a gap in the recording between Wed morning and Wed afternoon, but no gaps within nights) and I recollect the second device on Fry morning. So for each subject I get 2 acc data files (for the 2 devices) named "subjID-deviceID_acc.csv".

Here is the shell function with the settings that I'm using:
GGIR(
datadir="mydata_dir", outputdir="myoutput_dir",
rmc.firstrow.acc = 2, rmc.col.acc = 2:4, rmc.col.time = 1, rmc.sf = 64,
idloc=7, # ID location in the file name: 7 = string preceding "-" because my files are named "subjID-deviceID_acc.csv"
cosinor = TRUE,
ignorenonwear = FALSE,
includenightcrit = 10, # set to 10 rather than 16 because sometimes the device memory gets full in the early morning
loglocation = "mysleeplog_location",
save_ms5raw_format = "csv",
part5_agg2_60seconds = TRUE
)

Issue:
Since we recently implemented an algorithm that detects non-wear periods directly in the acc data and removes the associated acc dataset rows before using GGIR, we changed the ignorenonwear argument from TRUE (default) to FALSE.
Following that change, I realized that for most subjects (but not all of them!) nights 3 and 4 are duplicated in the GGIR output files as if the first device recorded 4 nights rather than 2.
Specifically, for these subjects I see:

• doubled nights 3 and 4 in the part4_nightsummary_sleep_full.csv file
• n_nights_acc = 2 for each device in the part4_summary_sleep_full.csv file (which is ok), n_nights_sleeplog = 2 for the second device (still ok), but n_nights_sleeplog = 4 for the first device (not ok)
• 4 rather than 2 nights associated with the first device in the meta/ms4.out folder

Importantly, if I remove the duplicates from part4_nightsummary_sleep_full.csv files (i.e., by only keeping the first encoded row for each subject-by-night pair), I noticed that the resulting sleep onset and wake-up times show a larger difference from sleep log times than what would have been shown if I kept the second duplicate rather than the first one (i.e., as if the second night showing the same subject-by-night value was "more accurate" than the second one). I generally don't trust subjects' recall of bedtimes, but most of my subjects use clock alarms, so the self-reported wake-up times should be accurate.

I'm attaching some screenshots with examples:
GGIR_issue_duplicated-nights_ignorenonwearTRUE.pdf

To solve this issue, I've tried changing several settings (e.g., recoding the sleeplog at the device level with idloc=1, joining the two acc.csv files in a single file, adding ".csv" after each ID in the sleeplog file, etc.) until I came back to ignorenonwear=TRUE. This completely solved the issue of duplicated nights (now I get zero duplicates), BUT I noticed that the retained nights 3 and 4 are the ones showing the largest differences from sleeplog-based times in those subjects that previously presented the issue of duplicated nights, whereas the "more accurate" duplicated nights are no longer there.

Any idea of what is happening here, i.e., why should the argument ignorenonwear=FALSE result in duplicated nights?
And particularly, what is making the retained nights under the ignorenonwear=TRUE less accurate than the excluded duplicates?

Thank you in advance!
Luca

[vincentvanhees]

1. Do you also get duplicated nights when you process the files individually?
2. Did you consider appending the files with parameter maxrecordinginterval? For that to work both files need to have the same participant ID.

[Luca-Menghini]

Thanks for your reply!

1. Yes, but only with the full 4-night sleeplog, not if I use a sleeplog with just nights 1-2
2. No, but I'm not sure about the requirement to have the same participant ID. If idloc = 7 identifies the ID as the first character string before the - symbol, then I should be ok with my files being named "subjID-deviceID_acc.csv", right?

Anyway, it seems that I've solved my problem, although I'm not sure about what happened.
I forgot to mention that we use the first device once again on Friday afternoon to record heart rate data for about 10 minutes. This caused our pre-processing script to export a single file with a first main chunk of acc data from Mon afternoon (device activation) to Wed morning (full device memory) followed by a second small chunk from Fry afternoon, with a time gap >48 hours in between.
My intuition is that ignorenonwear = FALSE treats those time gaps as valid acc data. Indeed, changing the pre-processing script to remove all 10-min final chunks resulted in zero duplicated cases even with ignorenonwear = FALSE.
However, I don't understand:

1. Why this only happened for some subjects (about 30/55) and not for all of them
2. How it is possible that the duplicated nights did not show TST = SPD.
   Indeed, if ignorenonwear = FALSE processes the time gap as valid data, then I would expect the data were inputed with a constant angle = 0 (i.e., only inactive time with no WASO), but the example at p. 2 with subject TG6TS shows that this was not the case.

This brings me to a potentially more general issue with the argument idloc = 7.
Indeed, I noticed that in most cases the part4_nightsummary_sleep_full.csv values under the guider_onset_ts and the guider_wakeup_ts are identical across devices within the same subject (i.e., n1 of d1 = n1 of d2, n2 of d1 = n2 of d2) despite the different times reported in the sleep logs. Since the guider should depend on the sleep log, I feel that maybe something is not working in the match of subject and night between the data and the sleep log, and I'm worried I'm doing something wrong.

Below I'm attaching an example of subject with large sleep timing variations, where it is particularly evident that nights 3-4 (Wed and Thu) are missidentified based on the guider derived from nights 1-2 rather than night 3-4. It also shows how the output changes when I use a sleep log that processes the files individually (as you suggested) by setting idloc = 1 (default), but it does not seem to solve the problem (it's like the sleeplog doesn't get linked to the data):
GGIR_issue_idloc7.pdf

What am I missing?
Thank you!

[Luca-Menghini]

Dear Vincent,
I finally solved the issue by recoding the sleep log at the filename level (subjID-deviceID_acc.csv with 2 nights per subject-by-device pair) rather than at the subject ID level (with 4 nights per subject), and by simultaneously setting idloc = 1.
Now each subject-by-device pair is treated as a different subject, and the guider values correspond to the times specified in the sleep log for each night. Then, I load the output data and combine those belonging to the same subject, hoping this does not introduce any bias.

To sum up:

• I solved the problem of the duplicated nights by removing the last chunk of ACC data that was recorded with the first device 2 days after the main recording was interrupted (see response above)
• I solved the inconsistency between sleep log and guider times by processing the files individually

Thank you for your hint on processing the files individually!
Note that this might be an issue with the idloc = 7 setting, but maybe I'm not getting how to use the parameter maxrecordinginterval that you mentioned above.

Best
Luca