add docs and script to fetch rucio dataset files

[garciagenrique]

Summary

This PR adds a bash script, to be run on VEGA, that fetches all the files from a RUCIO dataset. The script assumes that the full dataset is present on the VEGA RSE, otherwise it would be necessary to run a replication rule first.

The script creates a simlink per file, allowing the user to access the dataset without searching nor interacting with the rucio file structure.

---

Related issue : #156

Co-authored-by: Giovanni Guerrieri giovanni.guerrieri@cern.ch

[matbun]

It would be nice to have an example with a dataset already on the interTwin data lake, e.g., using the CERN use case dataset.
I am a bit unsure on these aspects:

• usage of .txt file formats: is this a constraint? If not, would it be possible to make it more general?
• Does this script allow to copy a whole dataset in one shot ?

Also, before merging I would kindly ask to fix linter problems

[matbun]

Thinking to it twice, I would suggest the following improvements:

First, incorporating the shell script logic into itwinai would simplify the life of the users. Instead of copying and executing the script, they could call it from within Python. We could also add it to the itwinai CLI. Example:

itwinai get-rucio-dataset <SCOPE:DataSet> <output_file> <output_symlink_dir>

This would prevent the users to manage yet another script, allowing us to ship with itwinai always the latest version of it.
If you think that this is doable, could you implement a function under src/itwinai/rucio.py, please? I will take care of integrating it into the itwinai CLI.

Second, the tutorial on how to get data from Rucio should come after the explanation on how to properly setup the Rucio client. Thus, I have created another tutorial folder under tutorials/data-lake/01-configure-rucio. I know you have some info on that, so could you please add a bit of documentation in the README file?

[garciagenrique]

Thinking to it twice, I would suggest the following improvements:

First, incorporating the shell script logic into itwinai would simplify the life of the users. Instead of copying and executing the script, they could call it from within Python. We could also add it to the itwinai CLI. Example:

itwinai get-rucio-dataset <SCOPE:DataSet> <output_file> <output_symlink_dir>

What about just doing a itwin get-rucio-dataset and manage internally the <output_file> and the <output_symlink_dir> ? It will be basically doing a (more or less) os.lisdir(output_file).
I can implement it of course

Second, the tutorial on how to get data from Rucio should come after the explanation on how to properly setup the Rucio client. Thus, I have created another tutorial folder under tutorials/data-lake/01-configure-rucio. I know you have some info on that, so could you please add a bit of documentation in the README file?

I have improved the README of this PR, I will add a more detailed tutorial tomorrow on the pointed directory.

[matbun]

What about just doing a itwin get-rucio-dataset and manage internally the <output_file> and the <output_symlink_dir> ? It will be basically doing a (more or less) os.lisdir(output_file). I can implement it of course

Indeed, the simpler the better! I would still leave an optional argument to specify the output folder, to avoid over constraining the users to cd in the target folder before executing the command.

I have improved the README of this PR, I will add a more detailed tutorial tomorrow on the pointed directory.

Thanks!

[matbun]

Hi @garciagenrique could we resume this conversation ?