Add job submission to Flink runner.#15
Add job submission to Flink runner.#15axelmagn wants to merge 2 commits intobsidhom:flink-portable-runnerfrom
Conversation
Adds a JobService implementation to java-fn-execution that should be generalizable across runners that implement the new JobInvoker and JobInvocation interfaces.
This implements job invocation wrappers for the Flink runner, as well as a standalone driver for a JobService daemon. Together these complete the job submission story for the portability framework on the Flink runner.
| /** | ||
| * Invocation of a Flink Job via {@link FlinkRunner}. | ||
| */ | ||
| public class FlinkJobInvocation implements JobInvocation { |
There was a problem hiding this comment.
Just throwing out some ideas:
I managed to make some changes to the Flink ClusterClient interface. This should make it easier to implement the methods of this: https://github.com/apache/flink/blob/10d52f268db3eda7ee1511ea30afb9a982644148/flink-clients/src/main/java/org/apache/flink/client/program/ClusterClient.java. Notable are:
submitJob(JobGraph, ClassLoader), which is non-blocking if you allsetDetached(true)beforecancel(JobID)getJobStatus(JobID)
A JobGraph will have a random JobID, you can get it from the JobGraph via getJobID(). For streaming you can get the JobGraph via StreamExecutionEnvironment.getStreamGraph().getJobGraph(). For batch it's a bit more involved but a starting point is this: https://github.com/apache/flink/blob/10d52f268db3eda7ee1511ea30afb9a982644148/flink-clients/src/main/java/org/apache/flink/client/program/ClusterClient.java#L890.
In general those APIs are somewhat internal and they are also somewhat interesting... 😉
|
Please update and fix the merge conflict. |
Wrap all unknown coders in length-prefixed byte array coders
3 participants
This includes both a generic framework for job submission via gRPC, as well as a concrete implementation for the Flink runner.