Issue 1460 export results as parquet format by samuelafolabi · Pull Request #1487 · wadpac/GGIR

samuelafolabi · 2026-03-27T00:47:26Z

Related to #1460

This PR adds a new parameter save_dashboard_parquet to GGIR that allows users to export a consolidated, web dashboard-ready Parquet file from the GGIR output CSVs.

When save_dashboard_parquet = TRUE, GGIR writes results/ggir_results.parquet after all requested parts have completed
Attaches Parquet key-value metadata recording the variable dictionary, activity threshold configuration, and accelerometer metric used
Cleans all column names to be SQL-friendly (lowercase, no special characters) for direct use with DuckDB or similar query engines
Find web dashboard repo and live link here- https://github.com/samuelafolabi/ggir-web-dashboard

Checklist before merging:

…ue_1460_export_results_as_parquet_format # Please enter a commit message to explain why this merge is necessary, # especially if it merges an updated upstream into a topic branch. # # Lines starting with '#' will be ignored, and an empty message aborts # the commit.

vincentvanhees · 2026-04-01T12:36:11Z

Thanks @samuelafolabi looks good, I will review it tomorrow.
I just made some updates to the main branch, could you update your branch with those changes? I do not expect them to interfere with your work, but it may be good to have it all up to date.

vincentvanhees

Code looks great Samuel, only a few comments below. My main concern is an error message I get when I test it on one of my files (Axivity cwa).

I am only using GGIR defaults parameter values and save_dashboard_parquet = TRUE.

GGIR(
  datadir = datadir,
  outputdir = outputdir,
  overwrite = TRUE,
  mode = 1:5,
  do.report = c(2, 4, 5),
  save_dashboard_parquet  = TRUE
)

which results in error

Error: Invalid: Can only convert data frames to Struct type

I think the error originates around line 174 in write_parquet.R, where the nested epoch-level time series are attached as a list-column. From that point onward arrow::arrow_table(consolidated) is no longer possible.

samuelafolabi · 2026-04-02T14:43:57Z

Code looks great Samuel, only a few comments below. My main concern is an error message I get when I test it on one of my files (Axivity cwa).

I am only using GGIR defaults parameter values and save_dashboard_parquet = TRUE.
GGIR(
  datadir = datadir,
  outputdir = outputdir,
  overwrite = TRUE,
  mode = 1:5,
  do.report = c(2, 4, 5),
  save_dashboard_parquet  = TRUE
)
which results in error

Error: Invalid: Can only convert data frames to Struct type

I think the error originates around line 174 in write_parquet.R, where the nested epoch-level time series are attached as a list-column. From that point onward arrow::arrow_table(consolidated) is no longer possible.

@vincentvanhees Thank you for reveiwing.
I'll check this out and fix as soon as I can. I was only able to test with the test file I created using the create_test_file function before I made a PR.

Is it possible you share some files with me so I could rigorously test before tagging you to the PR again?

Thanks once again.

samuelafolabi · 2026-04-02T14:54:33Z

Thanks @samuelafolabi looks good, I will review it tomorrow. I just made some updates to the main branch, could you update your branch with those changes? I do not expect them to interfere with your work, but it may be good to have it all up to date.

Yea, thanks for the heads-up. I saw the recent commits. I'll pull them in before I make these fixes.

vincentvanhees · 2026-04-02T15:21:24Z

Is it possible you share some files

No problem, I have sent you an email with the link.

Improve dashboard parquet generation by hardening epoch merge behavior and filename-to-ID matching for edge-case recordings, preventing Arrow struct conversion failures. Add and refine .Rd docs for parquet export helpers (including author info and dashboard privacy-focused usage note), and align documentation style with GGIR conventions.

…_export_results_as_parquet_format

samuelafolabi · 2026-04-05T22:54:25Z

90fbb43#diff-d2e3eb08f9cb1813199166ff82920b03c3c7759ad946fa9f43bd7cdcc06bca94

@vincentvanhees

vincentvanhees

Thanks, all looks better now. A few more comments.

Please also update the NEWS.md file in the root of the repository. As you will see there is a block for GGIR version 3.3-? at the top of that file to reflect the upcoming release. Please add a line to this block. For example:

- Functionality added to save all key output to one parquet file per person. #1460

where the #1460 is my way of tracking the GitHub item an update corresponded to.

vincentvanhees · 2026-04-15T14:38:11Z

+  # ---------------------------------------------------------------
+  # Build Arrow table and attach Parquet key-value metadata
+  # ---------------------------------------------------------------
+  parquet_path = paste0(results_dir, "/ggir_epochs.parquet")


GGIR users expect to be able to easily navigate the results folder while searching for the main GGIR output files. This will be complicated if all individual parquet files are stored in the root of this folder.
Please create a new subdirectory and store the parquet files in there, e.g. results/parquet.

Okay. Duely noted.

vincentvanhees · 2026-04-15T14:44:10Z

+  metadatadir = "path/to/output_run/output_test_run",
+  params_output = list(),
+  params_general = list(desiredtz = "UTC", acc.metric = "ENMO"),
+  params_phyact = list(part6_threshold_combi = "WW_L40M100V400_T5A5"),


This is not a plausible setting, please replace by:
part6_threshold_combi = "40_100_400"

vincentvanhees · 2026-04-15T14:46:48Z

+write_epoch_parquet(
+  metadatadir = "path/to/output_run/output_test_run",
+  params_output = list(),
+  params_general = list(desiredtz = "UTC", acc.metric = "ENMO"),


tz = "UTC" is not meaningful for GGIR. Data is always collected in a specific timezone with specific DST conditions. Therefore, GGIR expects a specific timezone in order to understand how to account for DST. Time is always expressed in local time and never in UTC time (Greenwich winter time).
Instead, use tz = "" as example, which uses the timezone where the data is processed.

Okay. Thanks for the clarification! I was actually wondering about what happens when a participant moves between two or more places with different timezones. This is clearer now.

People moving between timezones is not accounted for at the moment, but GGIR does account for DST within the same time zone.

Okay. Thank you!

vincentvanhees · 2026-04-15T14:48:48Z

+}
+\arguments{
+  \item{metadatadir}{
+    Directory that holds a folder 'meta' and inside this a folder 'basic'


folder basic is not used, should this be replaced by ms5.outraw?

vincentvanhees · 2026-04-15T14:49:04Z

+\arguments{
+  \item{metadatadir}{
+    Directory that holds a folder 'meta' and inside this a folder 'basic'
+    which contains the milestone data produced by \link{g.part1}. The folder structure


Should \link{g.part1} be \link{g.part5}? Same comment for next sentence.

vincentvanhees · 2026-04-15T15:07:10Z

"Related to #1460" in the first message should probably be "Fixes #1460" because that will make sure the issue is auto-closed once the PR is merged.

samuelafolabi · 2026-04-15T15:17:22Z

"Related to #1460" in the first message should probably be "Fixes #1460" because that will make sure the issue is auto-closed once the PR is merged.

Oh yes! I had plans of reading the comment section of the issue over again after the PR is merged. I assumed it would help reinforce my understanding of things and avoid making similar mistakes in future. Then the issue could be manually closed after that. @vincentvanhees

samuelafolabi added 9 commits February 8, 2026 14:30

Add Parquet export for dashboard-ready GGIR results (wadpac#1460)

d8c05a0

Implement nested GGIR Parquet export with per-day epoch time series

794e162

pulls from main

9121d0a

add run script

51c0a6d

remove created test files

4d86622

add pr requirement files

724d056

update param length in load_param test to accommodate save_parquet param

86adbe4

make parquet filename reflect participant ID

7754afe

vincentvanhees requested changes Apr 2, 2026

View reviewed changes

Comment thread R/write_parquet.R

Comment thread vignettes/GGIRParameters.Rmd Outdated

samuelafolabi added 2 commits April 5, 2026 22:49

Merge branch 'main' of https://github.com/wadpac/GGIR into issue_1460…

be304f5

…_export_results_as_parquet_format

samuelafolabi requested a review from vincentvanhees April 5, 2026 22:54

vincentvanhees requested changes Apr 15, 2026

View reviewed changes

Conversation

samuelafolabi commented Mar 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist before merging:

Uh oh!

vincentvanhees commented Apr 1, 2026

Uh oh!

vincentvanhees left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

samuelafolabi commented Apr 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

samuelafolabi commented Apr 2, 2026

Uh oh!

vincentvanhees commented Apr 2, 2026

Uh oh!

samuelafolabi commented Apr 5, 2026

Uh oh!

vincentvanhees left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vincentvanhees commented Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

samuelafolabi commented Apr 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

samuelafolabi commented Mar 27, 2026 •

edited

Loading

samuelafolabi commented Apr 2, 2026 •

edited

Loading

vincentvanhees commented Apr 15, 2026 •

edited

Loading