Custom driver support for KubeflowV2DagRunner

[jeongukjae]

System information

• TFX Version (you are using): 1.14
• Environment in which you plan to use the feature (e.g., Local
  (Linux/MacOS/Windows), Interactive Notebook, Google Cloud, etc..): Vertex AI
• Are you willing to contribute it (Yes/No): yes

Describe the feature and the current behavior/state.

Currently, KubeflowV2DagRunner does not support custom driver class. It only supports file based example gen driver which is located in

https://github.com/tensorflow/tfx/blob/master/tfx/orchestration/kubeflow/v2/file_based_example_gen/driver.py

Will this change the current API? How?

Not expected to change the API.
But this will change following codes.

• https://github.com/tensorflow/tfx/tree/master/tfx/orchestration/kubeflow/v2/file_based_example_gen
• https://github.com/tensorflow/tfx/blob/master/tfx/orchestration/kubeflow/v2/step_builder.py

Who will benefit with this feature?

Developers who want to leverage the tfx concepts more widely.

Do you have a workaround or are completely blocked by this? :

Blocked by this now.

Name of your Organization (Optional)

Any Other info.

[singhniraj08]

@jeongukjae, Since you mentioned you are willing to contribute for this feature, I will request you to go ahead a create a PR for feature implementation. Thank you for the contributions.

[github-actions]

This issue has been marked stale because it has no recent activity since 7 days. It will be closed if no further activity occurs. Thank you.

[jeongukjae]

do not close this

[pritamdodeja]

@jeongukjae can you please describe what you're trying to accomplish? I have successfully implemented components which are at the beginning of the pipeline that didn't use file based example gen. To your point about KubeflowV2DagRunner, is your pipeline executable by LocalDagRunner? It's possible there's some translation issue related to ML Metadata etc., let me know what your situation is, and we can figure out how to solve it.

[bharatjetti]

Hi @jeongukjae ,
Thank you so much for filing this Feature request! We truly appreciate your support. At the moment, this request is not prioritized and mark it for community contributions. If you would like to contribute on a fix, please feel free to fork the repository and submit a pull request in accordance with the TensorFlow Extended CONTRIBUTION GUIDELINES and link this issue in your PR to ensure the maintainers can easily review and collaborate.
To avoid duplicate work, please leave a comment here if you decide to take it on!

Thank you again for your interest and we look forward to seeing your contributions!