mkdocs. Mapping tables apis. Fix #1032 #1043
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,29 +1,55 @@ | ||
| When analyzing a Codeberg repository, SOMEF uses the [Codeberg API](https://codeberg.org/api/v1/swagger) | ||
| (`GET /api/v1/repos/{owner}/{repo}`) to retrieve metadata. The table below shows how Codeberg API | ||
| fields map to SOMEF categories: | ||
|
|
||
| | SOMEF category | Codeberg API field | Notes | | ||
| |---|---|---| | ||
| | `code_repository` | `html_url` | | | ||
| | `owner` | `owner.login` | | | ||
| | `full_name` | `full_name` | Format: `{owner}/{repo}` | | ||
| | `name` | `name` | | | ||
| | `description` | `description` | | | ||
| | `date_created` | `created_at` | | | ||
| | `date_updated` | `updated_at` | | | ||
| | `stargazers_count` | `stars_count` | Called `stars_count` in Codeberg | | ||
| | `forks_count` | `forks_count` | | | ||
| | `homepage` | `website` | Called `website` in Codeberg, `homepage` in GitHub | | ||
| | `download_url` | *(built from URL)* | `https://codeberg.org/{owner}/{repo}/releases` | | ||
| | `keywords` | `topics` | | | ||
| | `issue_tracker` | *(built from URL)* | `{html_url}/issues` | | ||
| | `license` | *(content API)* | *1* | | ||
| | `programming_languages` | `languages_url` | Additional GET request to the languages endpoint, returns byte counts per language | | ||
| | `releases` | `/repos/{owner}/{repo}/releases` | Additional GET request, mapped via `release_codeberg_crosswalk_table` | | ||
|
|
||
| For releases, the field mapping is identical to GitHub. The only differences are that Codeberg | ||
| uses `attachments` instead of `assets` for release files, and it does not provide | ||
| `author.type` (`AGENT_TYPE`) for release authors. | ||
|
|
||
|
|
||
| --------------- | ||
|
|
||
| *1* | ||
| Extracted by fetching the LICENSE file via `GET /api/v1/repos/{owner}/{repo}/contents/{filename}` (tries `LICENSE`, `LICENSE.md`, `LICENCE`, `COPYING`). The content is base64-decoded and analyzed with `detect_license_spdx()` to obtain the SPDX identifier, name and URL. | ||
|
Collaborator
There was a problem hiding this comment. This should not be this way. We do the inspection of the files offline, so this behavior should not happen if the API does not support licenses. |
||
|
|
||
| ### Archive download | ||
|
|
||
| Repository archives are downloaded from: | ||
| `https://codeberg.org/{owner}/{repo}/archive/{branch}.zip` | ||
|
|
||
| Codeberg archive URLs typically include a `Content-Length` header, so the size limit check | ||
| can be performed before downloading. | ||
|
|
||
| ### Enrichment via CODEOWNERS | ||
|
|
||
| When `--reconcile_authors` (`-ra`) is enabled, SOMEF fetches additional user details | ||
| (full name, email) from `GET https://codeberg.org/api/v1/users/{username}` for the | ||
| repository owner and for each CODEOWNERS entry. | ||
|
|
||
| ### Limitations | ||
|
|
||
| - **Rate limits**: Unauthenticated requests are limited to 60 requests/hour. Authenticated | ||
| requests (via `codeberg-token`) have higher limits. Create a token at | ||
| `https://codeberg.org/user/settings/applications`. | ||
| - **License detection**: Codeberg does not provide a `license` field in the repository | ||
|
Collaborator
There was a problem hiding this comment. Per above, this should be removed |
||
| API response. SOMEF falls back to fetching LICENSE/COPYING files directly via the | ||
| content API. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,54 @@ | ||
| When analyzing a GitHub repository, SOMEF uses the [GitHub REST API](https://docs.github.com/en/rest/repos/repos) | ||
| (`GET /repos/{owner}/{repo}`) to retrieve metadata. The table below shows how GitHub API | ||
| fields map to SOMEF categories: | ||
|
|
||
| | SOMEF category | GitHub API field | Notes | | ||
| |---|---|---| | ||
| | `code_repository` | `html_url` | | | ||
| | `owner` | `owner.login` | `agent_type` is extracted from `owner.type` (User or Organization) | | ||
| | `date_created` | `created_at` | | | ||
| | `date_updated` | `updated_at` | | | ||
| | `license` | `license` | Nested object with `spdx_id`, `name`, `url` | | ||
| | `description` | `description` | | | ||
| | `name` | `name` | | | ||
| | `full_name` | `full_name` | Format: `{owner}/{repo}` | | ||
| | `issue_tracker` | `issues_url` | The `{/number}` suffix is stripped | | ||
| | `forks_url` | `forks_url` | | | ||
| | `stargazers_count` | `stargazers_count` | | | ||
| | `keywords` | `topics` | | | ||
| | `forks_count` | `forks_count` | | | ||
| | `homepage` | `homepage` | | | ||
| | `programming_languages` | `languages` | Additional GET to `/repos/{owner}/{repo}/languages`. Returns a dictionary with byte counts per language | | ||
| | `releases` | `/repos/{owner}/{repo}/releases` | Paginated results, mapped via `release_crosswalk_table` | | ||
| | `download_url` | *(constructed)* | Built as `https://github.com/{owner}/{repo}/releases` | | ||
|
|
||
| ### Archive download | ||
|
|
||
| SOMEF downloads the repository archive from `https://github.com/{owner}/{repo}/archive/{ref}.zip`. | ||
| GitHub archive URLs do not include a `Content-Length` header, so SOMEF uses a streaming check: | ||
| it reads the response in 1 MB chunks and aborts if the total exceeds the configured size limit | ||
| (see `--download-limit`). | ||
|
|
||
| If the ref name is ambiguous (a branch and a tag share the same name), GitHub returns | ||
| HTTP 300. SOMEF handles this by trying the following fallback URLs in order: | ||
|
|
||
| 1. `https://github.com/{owner}/{repo}/archive/{ref}.zip` (short form) | ||
| 2. `https://github.com/{owner}/{repo}/archive/refs/heads/{ref}.zip` (explicit branch) | ||
| 3. `https://github.com/{owner}/{repo}/archive/refs/tags/{ref}.zip` (explicit tag) | ||
| 4. `https://github.com/{owner}/{repo}/archive/main.zip` (legacy rename fallback) | ||
|
|
||
| ### Limitations | ||
|
|
||
| - **No Content-Length**: GitHub archive downloads lack a `Content-Length` header, so the | ||
| size limit check relies on streaming (reading 1 MB chunks until the limit is exceeded). | ||
| - **Rate limits**: Unauthenticated requests are limited to 60 requests/hour. Authenticated | ||
|
Collaborator
There was a problem hiding this comment. We already address this in the config, so it's not a limitation |
||
| requests (via `github-token`) are limited to 5,000 requests/hour. | ||
| - **Private repositories**: SOMEF cannot access private repositories without a valid token. | ||
| - **Asset download count**: GitHub provides `download_count` for release assets; other | ||
| providers may not offer this field. | ||
|
Collaborator
There was a problem hiding this comment. This last thing, I am not sure we need it |
||
|
|
||
| ### Enrichment via CODEOWNERS | ||
|
|
||
| When `--reconcile_authors` (`-ra`) is enabled, SOMEF fetches additional user details | ||
| (name, company, email) from `GET https://api.github.com/users/{username}` for the | ||
| repository owner and for each CODEOWNERS entry. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,57 @@ | ||
| When analyzing a GitLab repository, SOMEF uses the [GitLab REST API](https://docs.gitlab.com/ee/api/projects.html) | ||
| (`GET /api/v4/projects/{project_id}`) to retrieve metadata. The table below shows how GitLab API | ||
| fields map to SOMEF categories: | ||
|
|
||
| | SOMEF category | GitLab API field | Notes | | ||
| |---|---|---| | ||
| | `code_repository` | *(built from URL)* | `https://{host}/{project_path}/` | | ||
| | `download_url` | *(built from URL)* | `https://{host}/{project_path}/-/branches` | | ||
| | `defaultBranch` | `default_branch` | Also checks `defaultBranch` (legacy key) | | ||
| | `description` | `description` | | | ||
| | `name` | `name` | | | ||
| | `full_name` | `path_with_namespace` | | | ||
| | `owner` | `owner.username` | `agent_type` deduced from `namespace.kind` ("group" → Organization, "user" → Person) | | ||
| | `date_created` | `created_at` | | | ||
| | `date_updated` | `last_activity_at` | | | ||
| | `issue_tracker` | *(built from URL)* | `https://{host}/{project_path}/-/issues` | | ||
| | `license` | `license` | `name` and `url` with SPDX detection. Falls back to fetching raw LICENSE file | | ||
| | `keywords` | `topics` | | | ||
| | `stargazers_count` | `star_count` | Called `star_count` in GitLab | | ||
| | `forks_count` | `forks_count` | | | ||
| | `programming_languages` | `languages` | Only language names (keys), no byte counts | | ||
| | `readme_url` | `readme_url` | | | ||
| | `releases` | `/projects/{id}/releases` | Paginated via `X-Next-Page`, mapped via `release_gitlab_crosswalk_table` | | ||
|
|
||
|
|
||
| ### Self-hosted GitLab instances | ||
|
|
||
| SOMEF supports self-hosted GitLab instances (e.g., `gitlab.in2p3.fr`). The project path is | ||
| URL-encoded and used to query the instance's API at `https://{host}/api/v4/projects/{encoded_path}`. | ||
| SOMEF detects self-hosted instances by checking if the URL contains `gitlab.com`. | ||
|
|
||
| ### Archive download | ||
|
|
||
| Repository archives are downloaded from: | ||
| `https://{host}/{project_path}/-/archive/{branch}/{repo_name}-{branch}.zip` | ||
|
|
||
| GitLab archive URLs typically include a `Content-Length` header, so the size limit check | ||
| can be performed before downloading. | ||
|
|
||
| ### Enrichment via CODEOWNERS | ||
|
|
||
| When `--reconcile_authors` (`-ra`) is enabled, SOMEF fetches additional user details | ||
| (name, organization, public email) from `GET {server_url}/api/v4/users?username={username}` | ||
| for the repository owner and for each CODEOWNERS entry. Self-hosted instances use their | ||
| own API endpoint; `gitlab.com` uses the standard `https://gitlab.com/api/v4/users`. | ||
|
|
||
| ### Limitations | ||
|
|
||
| - **Self-hosted detection**: SOMEF detects self-hosted instances by checking for `gitlab.com` | ||
| in the URL. If a self-hosted instance uses a custom domain without a GitLab API endpoint, | ||
| detection may fail. | ||
| - **Rate limits**: GitLab.com unauthenticated requests are limited to 600 requests/hour. | ||
|
Collaborator
There was a problem hiding this comment. This is already addressed. Remove |
||
| Authenticated requests (via `gitlab-token`)have higher limits depending on the token type. Self-hosted | ||
| instances have their own rate limits. | ||
| - **Programming languages**: GitLab returns only language names (no byte counts). | ||
|
Collaborator
There was a problem hiding this comment. This is not a problem. Remove |
||
| - **Release assets**: GitLab releases include sources (tar.gz, zip) and links, but do not | ||
| provide `download_count` or `content_size` like GitHub. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| # OpenAIRE and Zenodo API Mapping | ||
|
|
||
| When the enrichment flag (`-e` / `--enrichment`) is enabled, SOMEF integrates with the **OpenAIRE** and **Zenodo** APIs to fetch project context, funding background, and ecosystem identifiers. | ||
|
|
||
| ## Funding Enrichment (OpenAIRE) | ||
|
|
||
| These fields are injected into the **Funding** category when a matching grant or project is reconciled via OpenAIRE keywords or grant identifiers. | ||
|
|
||
| | SOMEF Property | Source API Field | Description | | ||
| | :--- | :--- | :--- | | ||
| | `project_code` | `code` | The official project or grant code assigned by the funding body. | | ||
| | `project_title` | `title` | The full title of the funded research project. | | ||
| | `project_acronym` | `acronym` | The official acronym of the project. | | ||
| | `grant_id` | `callidentifier` | Unique identifier for the specific call or grant agreement. | | ||
|
|
||
| ## Identifier Enrichment (OpenAIRE & Zenodo) | ||
|
|
||
| When processing repository identifiers or DOIs found in citations, SOMEF resolves them against OpenAIRE to construct an explore landing page link. Additionally, if the DOI belongs to Zenodo, SOMEF queries the Zenodo API to extract its corresponding Software Heritage identifier (`swhid`). | ||
|
|
||
| | SOMEF Property | Source API / Function | Description | | ||
| | :--- | :--- | :--- | | ||
| | `openaire_id` | `get_openaire_id(doi)` | Direct link to the OpenAIRE explore page dedicated to this artifact. | | ||
| | `swhid` | `get_zenodo_swhid(doi)` | The Software Heritage Identifier (SWHID) associated with a Zenodo record. | |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| # OpenAlex API Mapping | ||
|
|
||
| When the enrichment flag (`-e` / `--enrichment`) is enabled, SOMEF integrates with the **OpenAlex API** to resolve and complete scholarly publication data and missing author identities. | ||
|
|
||
| ## Citation Enrichment | ||
| If OpenAIRE fails to resolve a publication's DOI, SOMEF queries OpenAlex as a fallback to fetch the publication's unique ID. | ||
|
|
||
| | SOMEF Property | Source API Function | Description | | ||
| | :--- | :--- | :--- | | ||
| | `openalex_id` | `get_openalex_id(doi)` | The unique OpenAlex URL identifier assigned to the publication. | | ||
|
|
||
| ## Author/Contributor Enrichment | ||
| For extracted authors or contributors who lack a structured identifier, SOMEF searches OpenAlex by name to retrieve their professional ORCID. | ||
|
|
||
| | SOMEF Property | Source API Function | Description | | ||
| | :--- | :--- | :--- | | ||
| | `identifier` | `search_openalex_author(name)` | The author's verified ORCID URL (used as fallback when missing in local files). | |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The last comment, I don't know who is targeted to. Please remove