Alternative GitHub actions test suite runner on Juwels Booster

[okrochak]

Ideally, the status of the runner should be monitored with itwinai commands.

The output of SLURM job is parsed via the runner and can be read on GitHub actions log directly, which is in accordance to open-source principles in contrast to currently employed Dagger container runner which requires explicit access to read the error logs.

[matbun]

It is not true that with Dagger you cannot see the logs. It is actually very well designed and they can be accessed from here: https://dagger.cloud/itwinai

If needed, I can add you using your github email

[okrochak]

It is not true that with Dagger you cannot see the logs. It is actually very well designed and they can be accessed from here: https://dagger.cloud/itwinai

If needed, I can add you using your github email

Where did I say that you cannot read the logs? I've said:
"requires explicit access to read the error logs", and this is true.

With my proposed implementation, all logs will be visible directly on GitHub for anyone to see. And I won't need to ask anyone to make an account and give them permission

[matbun]

It is not true that with Dagger you cannot see the logs. It is actually very well designed and they can be accessed from here: https://dagger.cloud/itwinai
If needed, I can add you using your github email

Where did I say that you cannot read the logs? I've said: "requires explicit access to read the error logs", and this is true.

With my proposed implementation, all logs will be visible directly on GitHub for anyone to see. And I won't need to ask anyone to make an account and give them permission

The logs are visible via Dagger as well. Example: https://github.com/interTwin-eu/itwinai/actions/runs/16986210686/job/48155574503

Could you elaborate more on how the solution proposed by Dagger is not "in accordance to open-source principles"?

[okrochak]

@matbun this should be ready to merge now. I think the cleanest way to have this alternative workflow "juwels-runner" which will only be triggered for selected branches

[matbun]

@matbun this should be ready to merge now. I think the cleanest way to have this alternative workflow "juwels-runner" which will only be triggered for selected branches

Considering that this workflow would be triggered only on juwels-runner branch, is there a reason to have it on main (or on other branches) as well?

[okrochak]

@matbun this should be ready to merge now. I think the cleanest way to have this alternative workflow "juwels-runner" which will only be triggered for selected branches

Considering that this workflow would be triggered only on juwels-runner branch, is there a reason to have it on main (or on other branches) as well?

I'd like to have this within the main branch for the following case:

Let's say that someone clones the repo and makes a feature branch from main. They develop some use-case which might require use-case tests that go beyond what dagger container can offer in terms of memory / CPU resources. These tests don't need to be part of main CI/CD per se, but they would still be important for use-case validation. Then one would simply change the branch in juwels-runner workflow to check if these HPC tests also pass.

P.S. if you want to discuss this further, let's chat on Slack

[r-sarma]

Looks good to me!

[matbun]

Hi @okrochak , for transparency I will paste below the message I sent you on Slack on the 4th Sep 2025, which didn't receive an answer (apart from a greeting :) )

Hi Alex, I saw your comment on GH, but maybe it is easier to communicate here, as you suggested.

Regarding the point on future use cases, potentially needing the file as an example, I would say that the plan is not to have new use cases code integrated directly in the itwinai repo. This was something we did in the first half of interTwin, but the suggested way to go for the use cases now would be working on separate plugin repos. We will discuss with the use cases a plan to move their code out of the itwinai repo, if they haven't done it yet. But after the last review of interTwin on November.
The goal is to separate the core library from the use cases code, making it more modular and easy to maintain.

Although the GH action workflow file you created is not on the main branch, it is part of the repository. Anyone cloning the repo will be able to access it as easily as it was on the main branch -- developers can just "git checkout" it from your branch. So I don't see it as a problem to have it on a separate branch. The rationale behind is to keep the main branch as clean as possible. In this case, this file does not really belong to main in that it would be anyways ignored (by the CI).

Moreover, this pilot/PoC is not mature enough to be pushed to main, imo. Not because the code is not working, but because the current solution is hardcoded on the current workspace. In other words, it is not easily portable for other users not being part of intertwin. Also, some scripts are not part of the repo and are only on JSC (e.g., /p/project1/intertwin/juwels-selfhosted-runner/RUNNER/container.sh ).

During the project I also developed some PoCs that I left in separate branches because they were not mature enough to be pushed to main. You could also consider adding this workflow file to the wiki, as a reference for developers

[okrochak]

Hi @matbun, as this is a somewhat complicated issue IMO, I've sent you an e-mail to matteo.buninocern.ch. It goes beyond this MR but is related to it.