-
Notifications
You must be signed in to change notification settings - Fork 11
Add test for precision conversion with decl_hdf5
#672
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
Yushan-Wang
wants to merge
82
commits into
pdidev:main
Choose a base branch
from
Yushan-Wang:precision-conversion-with-hdf5
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
82 commits
Select commit
Hold shift + click to select a range
6c022f9
add fix for hdf5 2.0.0
Yushan-Wang 4e88161
Fix MPI condition for HDF5 parallel support
Yushan-Wang b342921
add test for precision conversion
Yushan-Wang 28e3989
Add 'd2s_precision' to ALL_TEST_NAMES
Yushan-Wang 3c39885
Remove parallel HDF5 version check
Yushan-Wang dfa8769
Fix HDF5 parallel check in FindNetCDF.cmake
Yushan-Wang 94997ce
Refactor HDF5 version checks in CMakeLists
Yushan-Wang b980a9d
fix indent
Yushan-Wang bcdccf2
Merge branch 'precision-conversion-with-hdf5' of https://github.com/Y…
Yushan-Wang 2774595
update decl_hdf5/CHANGELOG.md
Yushan-Wang 9e51956
add cmake fix for version less than 4.3
Yushan-Wang 071dff9
Add version checks for HDF5 in CMakeLists.txt
Yushan-Wang 6c5ceaa
Typo [skip ci]
Yushan-Wang 1c5c227
Merge branch 'patch-hdf5-2.0.0' of https://github.com/Yushan-Wang/pdi…
Yushan-Wang 2c2c58b
add fix for hdf5 2.0.0
Yushan-Wang ba0aed4
Fix MPI condition for HDF5 parallel support
Yushan-Wang 5150802
add cmake fix for version less than 4.3
Yushan-Wang 1dcb3c2
Typo [skip ci]
Yushan-Wang b01bb1b
Add version checks for HDF5 in CMakeLists.txt
Yushan-Wang bd034c8
Remove redundant CMAKE_MODULE_PATH entry
Yushan-Wang 3c5f4bc
Merge branch 'patch-hdf5-2.0.0' of https://github.com/Yushan-Wang/pdi…
Yushan-Wang 106bfbf
remove 4.1 cmake patch
Yushan-Wang eeab5c1
Update CMakeLists.txt
Yushan-Wang 6a33909
Update plugins/decl_hdf5/CMakeLists.txt
Yushan-Wang 7ac0058
Update plugins/decl_netcdf/CMakeLists.txt
Yushan-Wang ff9f88b
Apply suggestions from code review
jbigot 73cd0fc
Apply suggestion from @jbigot
jbigot 8678b69
Apply suggestions from code review
jbigot 7a863c4
Apply suggestions from code review
jbigot 169bdad
Fix compatibility with cmake pre 3.29
jbigot f3d585a
Fix compatibility with cmake pre 3.25
jbigot 6d26bee
Merge branch 'main' into patch-hdf5-2.0.0
Yushan-Wang c9817dd
update changelog
Yushan-Wang f7794cc
Merge branch 'main' into patch-hdf5-2.0.0
Yushan-Wang fe561e8
Merge branch 'main' into patch-hdf5-2.0.0
Yushan-Wang b43a1c7
update changelogs and spack.yaml
Yushan-Wang bf2cdc1
Merge branch 'patch-hdf5-2.0.0' of https://github.com/Yushan-Wang/pdi…
Yushan-Wang b842376
add fix for hdf5 2.0.0
Yushan-Wang fc361bb
Fix MPI condition for HDF5 parallel support
Yushan-Wang 7b288e9
add test for precision conversion
Yushan-Wang 97edfe9
fix indent
Yushan-Wang adc2e8d
Add 'd2s_precision' to ALL_TEST_NAMES
Yushan-Wang f78960d
Remove parallel HDF5 version check
Yushan-Wang 6d24ef8
Fix HDF5 parallel check in FindNetCDF.cmake
Yushan-Wang 21df9af
Refactor HDF5 version checks in CMakeLists
Yushan-Wang 121fa7b
update decl_hdf5/CHANGELOG.md
Yushan-Wang 8ecb3a5
Merge branch 'precision-conversion-with-hdf5' of https://github.com/Y…
Yushan-Wang 595124d
Merge branch 'patch-hdf5-2.0.0' into precision-conversion-with-hdf5
Yushan-Wang 4db4f55
update precision conversion test using new testing.h
Yushan-Wang a83ab3b
Merge branch 'main' into precision-conversion-with-hdf5
Yushan-Wang 450cdf8
Update precision conversion test documentation and remove unused incl…
Yushan-Wang 0e1f68c
Merge branch 'precision-conversion-with-hdf5' of https://github.com/Y…
Yushan-Wang 6ab160c
Remove unused includes from decl_hdf5_precision_conversion_test.cxx
Yushan-Wang f12d601
Apply suggestions from code review
Yushan-Wang 20a8ab1
compare the results using hdf5 read.
Yushan-Wang 5e52643
Refactor precision conversion tests to use template functions for com…
Yushan-Wang dcfdf07
Merge branch 'main' into precision-conversion-with-hdf5
Yushan-Wang cec1507
Apply suggestions from code review
Yushan-Wang ddedac5
fix some typo and add gtest style d2d and f2d checking
Yushan-Wang 4180b15
remain: int2double checking
Yushan-Wang d493808
indent
Yushan-Wang fd37e5a
add status checking for hdf5 read
Yushan-Wang 5cdd046
one possible way to compare int and double
Yushan-Wang 8a01504
fix: correct lambda parameter types in int to double comparison
Yushan-Wang d372511
attempt for fixing CI faileur with spack oldest
Yushan-Wang f60cb25
indent
Yushan-Wang 7102952
refactor: update HDF5 precision conversion test in a more compact way
Yushan-Wang 5dd30bb
skip precision conversion test when clang15
Yushan-Wang 717cbc6
avoid compiling precision conversion test with clang15
Yushan-Wang 4416895
skip only numerical comparaison with clang15
Yushan-Wang 54ac0f0
empty line at eof
Yushan-Wang 3dc95ad
Merge branch 'main' into precision-conversion-with-hdf5
Yushan-Wang 0a3f3fa
update upon review
Yushan-Wang de7c30d
update upon review
Yushan-Wang f996b6c
update upon review
Yushan-Wang 71c1805
Merge branch 'main' into precision-conversion-with-hdf5
Yushan-Wang dc32279
indent
Yushan-Wang 18f8009
Merge branch 'precision-conversion-with-hdf5' of https://github.com/Y…
Yushan-Wang 0e1f995
Refactor HDF5 test setup: move precision conversion test into decl_hd…
Yushan-Wang 3bf59d2
Remove unnecessary blank line in CMakeLists.txt
Yushan-Wang 7c775a4
Fix formatting in error message
Yushan-Wang 507c45f
fix indent
Yushan-Wang File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
|
|
@@ -28,6 +28,8 @@ | |||||
| #include <numeric> | ||||||
| #include <ranges> | ||||||
|
|
||||||
| #include <hdf5.h> | ||||||
|
|
||||||
| #include <pdi/testing.h> | ||||||
|
|
||||||
| using PDI::make_random; | ||||||
|
|
@@ -845,3 +847,126 @@ logging: trace | |||||
| ); | ||||||
| PDI_expose("array_data", array_data.data(), PDI_OUT); | ||||||
| } | ||||||
|
|
||||||
| /* Precision conversion with decl_hdf5 | ||||||
| * data in double precision | ||||||
| * file dataset in double, float, and int | ||||||
| */ | ||||||
| TEST_F(DeclHdf5, PrecisionConversion) | ||||||
| { | ||||||
| InitPdi(PC_parse_string(R"==( | ||||||
| logging: trace | ||||||
| metadata: | ||||||
| N: int | ||||||
| data: | ||||||
| array: {type: array, size: [$N, $N], subtype: double} | ||||||
| plugins: | ||||||
| decl_hdf5: | ||||||
| - file: d2d_test.h5 | ||||||
| datasets: | ||||||
| double_ds: {type: array, size: [$N, $N], subtype: double} | ||||||
| write: | ||||||
| array: | ||||||
| dataset: double_ds | ||||||
| - file: d2f_test.h5 | ||||||
| datasets: | ||||||
| float_ds: {type: array, size: [$N, $N], subtype: float} | ||||||
| write: | ||||||
| array: | ||||||
| dataset: float_ds | ||||||
| - file: d2i_test.h5 | ||||||
| datasets: | ||||||
| int_ds: {type: array, size: [$N, $N], subtype: int} | ||||||
| write: | ||||||
| array: | ||||||
| dataset: int_ds | ||||||
| )==")); | ||||||
|
|
||||||
| EXPECT_FALSE(std::filesystem::exists("d2d_test.h5")); | ||||||
| EXPECT_FALSE(std::filesystem::exists("d2f_test.h5")); | ||||||
| EXPECT_FALSE(std::filesystem::exists("d2i_test.h5")); | ||||||
|
|
||||||
| static constexpr size_t N = 100; | ||||||
| PDI_expose("N", &N, PDI_OUT); | ||||||
|
|
||||||
| auto const test_array = make_a<std::array<std::array<double, N>, N>>(); | ||||||
| PDI_expose("array", test_array.data(), PDI_OUT); | ||||||
|
|
||||||
| ASSERT_TRUE(std::filesystem::exists("d2d_test.h5")); | ||||||
| ASSERT_TRUE(std::filesystem::exists("d2f_test.h5")); | ||||||
| ASSERT_TRUE(std::filesystem::exists("d2i_test.h5")); | ||||||
|
|
||||||
| // read double precision dataset and compare | ||||||
| hid_t file_id = H5Fopen("d2d_test.h5", H5F_ACC_RDONLY, H5P_DEFAULT); | ||||||
| hid_t dataset_id = H5Dopen2(file_id, "/double_ds", H5P_DEFAULT); | ||||||
| hid_t type_id = H5Dget_type(dataset_id); | ||||||
| ASSERT_GE(file_id, 0); | ||||||
| ASSERT_GE(dataset_id, 0); | ||||||
| ASSERT_GE(type_id, 0); | ||||||
|
|
||||||
| EXPECT_GT(H5Tequal(type_id, H5T_IEEE_F64LE), 0); | ||||||
| std::array<std::array<double, N>, N> read_double_array; | ||||||
|
|
||||||
| herr_t status = H5Dread(dataset_id, H5T_NATIVE_DOUBLE, H5S_ALL, H5S_ALL, H5P_DEFAULT, read_double_array.data()->data()); | ||||||
| ASSERT_GE(status, 0); | ||||||
|
|
||||||
| EXPECT_EQ(test_array, read_double_array); | ||||||
|
|
||||||
| H5Tclose(type_id); | ||||||
| H5Dclose(dataset_id); | ||||||
| H5Fclose(file_id); | ||||||
|
|
||||||
| // read simple precision dataset and compare | ||||||
| file_id = H5Fopen("d2f_test.h5", H5F_ACC_RDONLY, H5P_DEFAULT); | ||||||
| dataset_id = H5Dopen2(file_id, "/float_ds", H5P_DEFAULT); | ||||||
| type_id = H5Dget_type(dataset_id); | ||||||
| ASSERT_GE(file_id, 0); | ||||||
| ASSERT_GE(dataset_id, 0); | ||||||
| ASSERT_GE(type_id, 0); | ||||||
|
|
||||||
| EXPECT_GT(H5Tequal(type_id, H5T_IEEE_F32LE), 0); | ||||||
| std::array< std::array<float, N>, N > read_float_array; | ||||||
|
|
||||||
| status = H5Dread(dataset_id, H5T_NATIVE_FLOAT, H5S_ALL, H5S_ALL, H5P_DEFAULT, read_float_array.data()->data()); | ||||||
| ASSERT_GE(status, 0); | ||||||
|
|
||||||
| #if defined(__clang__) && (__clang_major__ == 15) | ||||||
| // Skipping the numerical comparaison because Clang 15 has known issues with ranges. | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
| #else | ||||||
| EXPECT_THAT(read_float_array, testing::ElementsAreArray(test_array | std::views::transform([](std::array<double, N> const & aref) { | ||||||
| return testing::Pointwise(testing::FloatEq(), aref); | ||||||
| }))); | ||||||
| #endif | ||||||
|
|
||||||
| H5Tclose(type_id); | ||||||
| H5Dclose(dataset_id); | ||||||
| H5Fclose(file_id); | ||||||
|
|
||||||
| // read integer dataset and compare | ||||||
| file_id = H5Fopen("d2i_test.h5", H5F_ACC_RDONLY, H5P_DEFAULT); | ||||||
| dataset_id = H5Dopen2(file_id, "/int_ds", H5P_DEFAULT); | ||||||
| type_id = H5Dget_type(dataset_id); | ||||||
| ASSERT_GE(file_id, 0); | ||||||
| ASSERT_GE(dataset_id, 0); | ||||||
| ASSERT_GE(type_id, 0); | ||||||
|
|
||||||
| EXPECT_GT(H5Tequal(type_id, H5T_STD_I32LE), 0); | ||||||
| std::array< std::array<int, N>, N > read_int_array; | ||||||
|
|
||||||
| status = H5Dread(dataset_id, H5T_NATIVE_INT, H5S_ALL, H5S_ALL, H5P_DEFAULT, read_int_array.data()->data()); | ||||||
| ASSERT_GE(status, 0); | ||||||
|
|
||||||
| #if defined(__clang__) && (__clang_major__ == 15) | ||||||
| // Skipping the numerical comparaison because Clang 15 has known issues with ranges. | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
| #else | ||||||
| EXPECT_THAT(read_int_array, testing::ElementsAreArray(test_array | std::views::transform([](std::array<double, N> const & aref) { | ||||||
| return testing::ElementsAreArray(aref | std::views::transform([](double const & ref) { | ||||||
| return static_cast<int>(ref); | ||||||
| })); | ||||||
| }))); | ||||||
| #endif | ||||||
|
|
||||||
| H5Tclose(type_id); | ||||||
| H5Dclose(dataset_id); | ||||||
| H5Fclose(file_id); | ||||||
| } | ||||||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.