This course project is built upon the emg2qwerty work from Meta. The first section of this README provides some guidance for working with the repo and contains a running list of FAQs. Note that the rest of the README is from the original repo and we encourage you to take a look at their work.
This experiment investigates how decoding performance changes as the number of electrode channels decreases. We randomly keep K channels per band (out of 16) and zero out the rest so that the model architecture does not need to change.
File to edit
emg2qwerty/transforms.py
Add the following transform:
class RandomChannelSubset(Transform[torch.Tensor, torch.Tensor]):
"""
Randomly keep K EMG channels per band and zero out the rest.
Keeps tensor shape unchanged.
Input shape: (T, bands=2, channels=16)
"""
def __init__(self, k: int):
self.k = k
def __call__(self, x: torch.Tensor) -> torch.Tensor:
T, B, C = x.shape
keep = torch.randperm(C)[: self.k]
mask = torch.zeros_like(x)
mask[:, :, keep] = x[:, :, keep]
return maskFile to edit
configs/transforms/transforms.log_spectrogram(_aug).yaml
Add the transform definition:
random_channel_subset:
_target_: emg2qwerty.transforms.RandomChannelSubset
k: 8 <--- the hyperparameter we changeAdd the transform right after to_tensor.
transforms:
train:
- ${to_tensor}
- ${random_channel_subset} # NEW
- ${raw_noise}
- ${raw_timemask}
- ${band_rotation}
- ${temporal_jitter}
- ${logspec}
- ${specaug}
- ${logspec_norm}val:
- ${to_tensor}
- ${random_channel_subset} # NEW
- ${logspec}
- ${logspec_norm}Change the parameter:
random_channel_subset.k
| k (channels per band) | total electrodes |
|---|---|
| 1 | 2 |
| 2 | 4 |
| 4 | 8 |
| 8 | 16 |
| 16 | 32 (baseline) |
Example:
k: 4This keeps 4 channels per band (8 electrodes total).
This experiment investigates how the amount of training data affects decoding performance.
configs/user/single_user.yaml
Modify the list:
dataset.train
Only change training sessions.
Do not modify val or test sessions.
Create subsets of the dataset.train list.
| Data fraction | # sessions | Example |
|---|---|---|
| 25% | 4 | first 4 sessions |
| 50% | 8 | first 8 sessions |
| 75% | 12 | first 12 sessions |
| 100%(baseline) | 16 | all sessions |
Edit the train list to include only the first 8 sessions:
dataset:
train:
- session1
- session2
- session3
- session4
- session5
- session6
- session7
- session8
Leave val and test unchanged.
To investigate how the sEMG sampling rate affects decoding performance (CER), modify the hop_length parameter in the spectrogram transform.
configs/transforms/log_spectrogram.yaml (or any transform yaml we investigate upon, like log_spectrogram_aug.yaml)
Locate the logspec section:
logspec:
_target_: emg2qwerty.transforms.LogSpectrogram
n_fft: 64
hop_length: 16
Change the hop_length value to control the effective sampling rate.
| hop_length | Effective Rate |
|---|---|
| 1 | 2000 Hz |
| 2 | 1000 Hz |
| 4 | 500 Hz |
| 8 | 250 Hz |
| 16 | 125 Hz (baseline) |
| 32 | 62.5 Hz |
The relationship is:
effective_rate = 2000 / hop_length
because the original EMG is sampled at 2000 Hz.
To test 500 Hz, change:
hop_length: 4
Then retrain the model and record the resulting CER. Repeat for the different hop lengths to produce a sampling rate vs CER plot.
Last updated 2/13/2025
- Read through the Project Guidelines to ensure that you have a clear understanding of what we expect
- Familiarize yourself with the prediction task and get a high-level understanding of their base architecture (it would be beneficial to read about CTC loss)
- Get comfortable with the codebase
lightning.py+modules.py- where most of your model architecture development will take placedata.py- defines PyTorch dataset (likely will not need to touch this much)transforms.py- implement more data transforms and other preprocessing techniquesconfig/*.yaml- modify model hyperparameters and PyTorch Lightning training configuration- Q: How do we update these configuration files? A: Note the structure of YAML files include basic key-value pairs (i.e.
<key>: <value>) and hierarchical structure. So, for instance, if we wanted to update themlp_featureshyperparameter of theTDSConvCTCModule, we would change the value at line 5 ofconfig/model/tds_conv_ctc.yaml(undermodule). Read more details here. - Q: Where do we configure data splitting? A: Refer to
config/user/single_user.yaml. Be careful with your edits, so that you don't accidentally move the test data into your training set.
- Q: How do we update these configuration files? A: Note the structure of YAML files include basic key-value pairs (i.e.
[ Paper ] [ Dataset ] [ Blog ] [ BibTeX ]
A dataset of surface electromyography (sEMG) recordings while touch typing on a QWERTY keyboard with ground-truth, benchmarks and baselines.
# Install [git-lfs](https://git-lfs.github.com/) (for pretrained checkpoints)
git lfs install
# Clone the repo, setup environment, and install local package
git clone git@github.com:joe-lin-tech/emg2qwerty.git ~/emg2qwerty
cd ~/emg2qwerty
conda env create -f environment.yml
conda activate emg2qwerty
pip install -e .
# Download the dataset, extract, and symlink to ~/emg2qwerty/data
cd ~ && wget https://fb-ctrl-oss.s3.amazonaws.com/emg2qwerty/emg2qwerty-data-2021-08.tar.gz
tar -xvzf emg2qwerty-data-2021-08.tar.gz
ln -s ~/emg2qwerty-data-2021-08 ~/emg2qwerty/dataThe dataset consists of 1,136 files in total - 1,135 session files spanning 108 users and 346 hours of recording, and one metadata.csv file. Each session file is in a simple HDF5 format and includes the left and right sEMG signal data, prompted text, keylogger ground-truth, and their corresponding timestamps. emg2qwerty.data.EMGSessionData offers a programmatic read-only interface into the HDF5 session files.
To load the metadata.csv file and print dataset statistics,
python scripts/print_dataset_stats.py
To re-generate data splits,
python scripts/generate_splits.pyThe following figure visualizes the dataset splits for training, validation and testing of generic and personalized user models. Refer to the paper for details of the benchmark setup and data splits.
To re-format data in EEG BIDS format,
python scripts/convert_to_bids.pyGeneric user model:
python -m emg2qwerty.train \
user=generic \
trainer.accelerator=gpu trainer.devices=8 \
--multirunPersonalized user models:
python -m emg2qwerty.train \
user="single_user" \
trainer.accelerator=gpu trainer.devices=1If you are using a Slurm cluster, include "cluster=slurm" override in the argument list of above commands to pick up config/cluster/slurm.yaml. This overrides the Hydra Launcher to use Submitit plugin. Refer to Hydra documentation for the list of available launcher plugins if you are not using a Slurm cluster.
Greedy decoding:
python -m emg2qwerty.train \
user="glob(user*)" \
checkpoint="${HOME}/emg2qwerty/models/personalized-finetuned/\${user}.ckpt" \
train=False trainer.accelerator=cpu \
decoder=ctc_greedy \
hydra.launcher.mem_gb=64 \
--multirunBeam-search decoding with 6-gram character-level language model:
python -m emg2qwerty.train \
user="glob(user*)" \
checkpoint="${HOME}/emg2qwerty/models/personalized-finetuned/\${user}.ckpt" \
train=False trainer.accelerator=cpu \
decoder=ctc_beam \
hydra.launcher.mem_gb=64 \
--multirunThe 6-gram character-level language model, used by the first-pass beam-search decoder above, is generated from WikiText-103 raw dataset, and built using KenLM. The LM is available under models/lm/, both in the binary format, and the human-readable ARPA format. These can be regenerated as follows:
- Build kenlm from source: https://github.com/kpu/kenlm#compiling
- Run
./scripts/lm/build_char_lm.sh <ngram_order>
emg2qwerty is CC-BY-NC-4.0 licensed, as found in the LICENSE file.
@misc{sivakumar2024emg2qwertylargedatasetbaselines,
title={emg2qwerty: A Large Dataset with Baselines for Touch Typing using Surface Electromyography},
author={Viswanath Sivakumar and Jeffrey Seely and Alan Du and Sean R Bittner and Adam Berenzweig and Anuoluwapo Bolarinwa and Alexandre Gramfort and Michael I Mandel},
year={2024},
eprint={2410.20081},
archivePrefix={arXiv},
primaryClass={cs.LG},
url={https://arxiv.org/abs/2410.20081},
}