Skip to content

Miscellaneous improvements for 2026-W03#58

Merged
khaeru merged 17 commits into
mainfrom
enh/2026-w03
Apr 28, 2026
Merged

Miscellaneous improvements for 2026-W03#58
khaeru merged 17 commits into
mainfrom
enh/2026-w03

Conversation

@khaeru
Copy link
Copy Markdown
Contributor

@khaeru khaeru commented Jan 15, 2026

  • Add new CLI command tdc check-record.
    This command retrieves the metadata for a given record from the TDC CKAN API and checks/prints some properties of the record. For example:

    $ tdc check-record registered-vehicles-in-tanzania
    - https://portal.transport-data.org/@ministry of works and transport of tanzania/registered-vehicles-in-tanzania
    - Title: 'Registered Vehicles in Tanzania'
    - Category: tdc_formatted
    
    - Number of files by extension: 1 .csv, 2 .pdf, 1 .xlsx
    - Number of data files: 2
    - Number of possible SDMX-CSV files: 1
    
    Criteria for a TDC Formatted record:
    - At least one file in CSV format: True
    - Correct category assigned: True
    - CSV file(s) are in SDMX-CSV format (not implemented yet): True
    - Overall: YES
    
    Criteria for a TDC Harmonized record—all of the above, plus:
    - Correct category assigned: False
    - Overall: NO
    • Retrieve and cache the data file(s).
    • Check the contents of data file(s).
  • Filter/ignore the warning due to Replace pkg_resources with importlib.metadata ckan/ckanapi#218. The upstream PR was merged, but the package has not been released since 2024, so the warning still appears.

How to review

  • Try running tdc check-record for some known record IDs.
    • Report whether the outputs appear correct.

PR checklist

  • Checks all ✅
  • Update documentation
  • Update doc/whatsnew.rst

- Convert "resources" collection to instances of Resource.
  - Adjust ckan_package_to_mdr() to match.
- Add portal_url() method.
- Add type hints for known members/attributes.
khaeru added a commit that referenced this pull request Apr 28, 2026
@codecov
Copy link
Copy Markdown

codecov Bot commented Apr 28, 2026

Codecov Report

❌ Patch coverage is 94.48276% with 16 lines in your changes missing coverage. Please review.
✅ Project coverage is 95.85%. Comparing base (dc12c7a) to head (438982f).
⚠️ Report is 18 commits behind head on main.

Files with missing lines Patch % Lines
transport_data/util/sdmx.py 80.48% 8 Missing ⚠️
transport_data/cli/check_record.py 91.83% 4 Missing ⚠️
transport_data/util/ckan.py 89.18% 4 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main      #58      +/-   ##
==========================================
- Coverage   96.15%   95.85%   -0.30%     
==========================================
  Files          67       69       +2     
  Lines        3195     3353     +158     
==========================================
+ Hits         3072     3214     +142     
- Misses        123      139      +16     
Files with missing lines Coverage Δ
transport_data/__init__.py 100.00% <100.00%> (ø)
transport_data/ato/__init__.py 99.02% <100.00%> (+0.01%) ⬆️
transport_data/cli/__init__.py 85.71% <100.00%> (-11.69%) ⬇️
transport_data/cli/check_file.py 100.00% <100.00%> (ø)
transport_data/estat/__init__.py 89.28% <100.00%> (+1.28%) ⬆️
transport_data/iamc/__init__.py 97.87% <100.00%> (+0.07%) ⬆️
transport_data/ipcc/__init__.py 100.00% <100.00%> (ø)
transport_data/iso/__init__.py 100.00% <100.00%> (ø)
transport_data/itdp/__init__.py 100.00% <100.00%> (ø)
transport_data/jrc/__init__.py 93.47% <100.00%> (+0.07%) ⬆️
... and 12 more
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

khaeru added 5 commits April 28, 2026 23:03
- Use sub-paths from registry in .is_available() call.
- Handle ConnectionError/HTTPSConnectionPool max retries exceeded.
  This may be caused by repeated queries to incorrect URLs.
khaeru added a commit that referenced this pull request Apr 28, 2026
khaeru added a commit that referenced this pull request Apr 28, 2026
@khaeru
Copy link
Copy Markdown
Contributor Author

khaeru commented Apr 28, 2026

Will merge here despite the codecov/patch check failure. Testing the still missing lines requires more extensive features, which can be added in a subsequent PR.

@khaeru khaeru merged commit 504ca78 into main Apr 28, 2026
19 of 20 checks passed
@khaeru khaeru deleted the enh/2026-w03 branch April 28, 2026 21:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enh New feature or request

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant