Provide a single command to create_or_update the pipeline in CLI

[jiyongjung0]

System information

• All
• Are you willing to contribute it (Yes/No): Yes

Describe the feature and the current behavior/state.
Currently pipeline create command can be used when there is no pipeline, and update command can be used when there *is an existing pipeline. It would be better to have just one command that can be used in CI/CD to create/update pipelines.

For example,

tfx pipeline create --overwrite ...

or

tfx pipeline update --missing_ok ...  (We can support --missing_ok for the `delete` command.)

Will this change the current API? How?
No.

Who will benefit with this feature?
Mostly users with CI/CD flow which doesn't have human intervention.

Do you have a workaround or are completely blocked by this? :
We can use pipeline list and examine the output but it is quite cumbersome and need a conditional branch.

[SaschaHeyer]

Hi @jiyongjung0

thanks so much for your efforts on this topic.
I prefer tfx pipeline create --overwrite it feels more natural and easier to understand

[jiyongjung0]

@SaschaHeyer
Thank you for the feedback. May I ask one more thing? Kubeflow Pipelines has a concept of pipeline version, and when we do tfx pipeline update, we actually create a new version of the pipeline instead of deleting existing one. If we delete a pipeline it might not displayed well in Kubeflow Pipelines web UI.

However, overwrite feels like we should delete the existing pipeline to me, but it might not a best fit for repeated CI/CD jobs. Do you think that is it ok to update instead of deleting the existing pipeline?

[SaschaHeyer]

Yes you're right updating make more sense. We should keep the concept of versions

I've had the same issue as Sascha. For my situation, I have a small number of pipelines, each with a large, growing number of versions.

The tfx pipeline update --missing_ok is most appealing for us because we are typically attempting to create a new version in a CICD build context that would not necessarily have the local pipeline_args.json snippet. (As a workaround we run the create step manually, add the initial JSON to the repo, and reuse that in CICD setup by setting KUBEFLOW_HOME.)

[FatimahAdwan]

Hi, I would love to contribute to this, pls let me know soon if it is possible

[jiyongjung0]

I think that both of create --overwrite and update --missing_ok should work in the same way, so it is a matter of taste or understandability.

Contributions are always welcome! Please send a PR with a reference to this issue. Thanks!

@jiyongjung0, thanks for the clarification. From my first readI thought pipeline update --missing_ok would create a new version of the pipeline even when the local pipeline metadata is "missing" (e.g., when you're running the update in a different context than you ran the original create).

[FatimahAdwan]

@jiyongjung0 alright then!. thanks

[bharatjetti]

Hi @jiyongjung0 ,
Thank you so much for filing this Feature request! We truly appreciate your support. At the moment, this request is not prioritized and mark it for community contributions. If you would like to contribute on a fix, please feel free to fork the repository and submit a pull request in accordance with the TensorFlow Extended CONTRIBUTION GUIDELINES and link this issue in your PR to ensure the maintainers can easily review and collaborate.
To avoid duplicate work, please leave a comment here if you decide to take it on!

Thank you again for your interest and we look forward to seeing your contributions!