Acoustic Feature Extraction in Pipeline

[rtmcardle]

There is an upcoming need for generation/extraction of acoustic features of subject speech, largely defined by the GeMAPS set. I've an initial implementation prepared as a stand-alone Python module and am hoping to begin integration into the pipeline. The implementation leverages opensmile to load a file by path and extract the features to Pandas frame. My expectation is that this extraction can be added into the pipeline as a runner following 21_fetch_audio.py, which I understand to decrypt the audio.

@dheshanm does this sound like the right step of the pipeline for this feature extraction?
Are there additional considerations around decryption that will be relevant here?
Dr. Wolff mentioned the AV server as potentially relevant, but I understand/expect that to be just for serving to the front end; let me know if it should play a role in this functionality as well.

[dheshanm]

@rtmcardle ,Yes the runner for opensmile extraction should follow 21_fetch_audio.py. The expected flow is:

• 21_fetch_audio.py requests that the audio be made available.
• Depending on the study, this is handled by importer.py or decryption.py
  • For ProNET, we do complete copies to a local filesystem with a privileged helper. Data is mounted and accessible only as privileged user.
  • For PRESCIENT, we use symlinks, to standardize filenames and directory structure, without using up space.
  • For other studies at MGB, we have to decrypt encrypted files (not relevant for ampscz branch)
• the new opensmile runner can then process the available file and record the run and its output in the database.
• If the extracted features are approved as de-identified, their exported output should go under the GENERAL directory in PHOENIX, via exporter.py.
  • Only data classified as GENERAL may leave the originating research network to DPACC. Raw audio in PROTECTED must remain within that network. I believe GeMAPS output meets this de-identification requirements.

Regarding the AV server, I believe Dr. Wolff was referring to Yale’s AWS EC2 instance, where the opensmile feature extraction runner would execute, rather than the dpinterview-web service (which runs in the same server/EC2 instance). If users need visibility into extraction status or results, we could separately add that to dpinterview-web.