-
Notifications
You must be signed in to change notification settings - Fork 4
ci: Add GitHub workflows to build and publish coordinator runtime, worker runtime, and worker development environment container images. #26
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 28 commits
4f23c7b
48dff13
9bd9207
4997a02
d74b885
4815351
e7fe20c
fc2cc18
cc2c510
60dd19d
a534bec
9e17bc0
aeb4e48
dbb28fa
1e69681
4ca014c
fec665a
20a7054
283b751
64983f0
a6ebbaa
5ca0420
fa06d23
d120672
4c90aad
3b76c12
b55aeda
74787c7
313a648
47b9c10
ceaa809
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
@@ -46,5 +46,71 @@ jobs: | |||||||||||||||||
| run: | | ||||||||||||||||||
| export MAVEN_OPTS="${MAVEN_INSTALL_OPTS}" | ||||||||||||||||||
| ./mvnw install -B -V -T 1C -DskipTests -Dmaven.javadoc.skip=true --no-transfer-progress -P ci -pl '!presto-test-coverage,!:presto-docs' | ||||||||||||||||||
| - name: Upload presto-server | ||||||||||||||||||
| if: matrix.java == '8.0.442' | ||||||||||||||||||
| uses: actions/upload-artifact@v4 | ||||||||||||||||||
| with: | ||||||||||||||||||
| name: presto-server | ||||||||||||||||||
| path: presto-server/target/presto-server-0.293.tar.gz | ||||||||||||||||||
| if-no-files-found: "error" | ||||||||||||||||||
| retention-days: 1 | ||||||||||||||||||
| - name: Upload presto-cli | ||||||||||||||||||
| if: matrix.java == '8.0.442' | ||||||||||||||||||
| uses: actions/upload-artifact@v4 | ||||||||||||||||||
| with: | ||||||||||||||||||
| name: presto-cli | ||||||||||||||||||
| path: presto-cli/target/presto-cli-0.293-executable.jar | ||||||||||||||||||
|
anlowee marked this conversation as resolved.
Outdated
|
||||||||||||||||||
| if-no-files-found: "error" | ||||||||||||||||||
| retention-days: 1 | ||||||||||||||||||
| - name: Clean Maven Output | ||||||||||||||||||
| run: ./mvnw clean -pl '!:presto-server,!:presto-cli,!presto-test-coverage' | ||||||||||||||||||
| presto-coordinator-image: | ||||||||||||||||||
| name: presto-coordinator-image | ||||||||||||||||||
| needs: maven-checks | ||||||||||||||||||
| runs-on: ubuntu-22.04 | ||||||||||||||||||
| steps: | ||||||||||||||||||
| - uses: "actions/checkout@v4" | ||||||||||||||||||
| with: | ||||||||||||||||||
| submodules: "recursive" | ||||||||||||||||||
|
|
||||||||||||||||||
| - name: Download presto-server | ||||||||||||||||||
| uses: actions/download-artifact@v4 | ||||||||||||||||||
| with: | ||||||||||||||||||
| name: presto-server | ||||||||||||||||||
| path: ./docker | ||||||||||||||||||
|
|
||||||||||||||||||
| - name: Download presto-cli | ||||||||||||||||||
| uses: actions/download-artifact@v4 | ||||||||||||||||||
| with: | ||||||||||||||||||
| name: presto-cli | ||||||||||||||||||
| path: ./docker | ||||||||||||||||||
|
|
||||||||||||||||||
| - name: "Login to image registry" | ||||||||||||||||||
| uses: "docker/login-action@v3" | ||||||||||||||||||
| with: | ||||||||||||||||||
| registry: ghcr.io | ||||||||||||||||||
| username: ${{github.actor}} | ||||||||||||||||||
| password: ${{secrets.GITHUB_TOKEN}} | ||||||||||||||||||
|
|
||||||||||||||||||
| - name: "Set up container image metadata" | ||||||||||||||||||
| id: "meta" | ||||||||||||||||||
| uses: "docker/metadata-action@v5" | ||||||||||||||||||
| with: | ||||||||||||||||||
| images: >- | ||||||||||||||||||
| ghcr.io/${{github.repository}}/coordinator | ||||||||||||||||||
| tags: |- | ||||||||||||||||||
| type=raw,value=dev | ||||||||||||||||||
|
|
||||||||||||||||||
| - name: "Build and push" | ||||||||||||||||||
| uses: "docker/build-push-action@v6" | ||||||||||||||||||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🛠️ Refactor suggestion Possible GHCR push failure with mixed-case repo name
🤖 Prompt for AI Agents |
||||||||||||||||||
| with: | ||||||||||||||||||
| build-args: |- | ||||||||||||||||||
| JMX_PROMETHEUS_JAVA_AGENT_VERSION=0.20.0 | ||||||||||||||||||
| PRESTO_VERSION=0.293 | ||||||||||||||||||
| context: "./docker/" | ||||||||||||||||||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🛠️ Refactor suggestion Build args repeat the hard-coded version
- PRESTO_VERSION=0.293
+ PRESTO_VERSION=${{ env.PRESTO_VERSION }}📝 Committable suggestion
Suggested change
🤖 Prompt for AI Agents |
||||||||||||||||||
| file: "./docker/Dockerfile" | ||||||||||||||||||
| push: >- | ||||||||||||||||||
| ${{github.event_name != 'pull_request' | ||||||||||||||||||
| && github.ref == 'refs/heads/release-0.293-clp-connector'}} | ||||||||||||||||||
| tags: "${{steps.meta.outputs.tags}}" | ||||||||||||||||||
|
Comment on lines
+104
to
+114
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🧹 Nitpick (assertive) Consider skipping the heavy Docker build on pull-requests 🤖 Prompt for AI Agents |
||||||||||||||||||
| labels: "${{steps.meta.outputs.labels}}" | ||||||||||||||||||
| Original file line number | Diff line number | Diff line change | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| @@ -0,0 +1,72 @@ | ||||||||||||||||||
| name: prestissimo-worker-images-build | ||||||||||||||||||
|
|
||||||||||||||||||
| on: | ||||||||||||||||||
| pull_request: | ||||||||||||||||||
| push: | ||||||||||||||||||
|
|
||||||||||||||||||
| jobs: | ||||||||||||||||||
| prestissimo-worker-images-build: | ||||||||||||||||||
| name: prestissimo-worker-images-build | ||||||||||||||||||
| runs-on: ubuntu-22.04 | ||||||||||||||||||
| steps: | ||||||||||||||||||
| - uses: "actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683" | ||||||||||||||||||
| with: | ||||||||||||||||||
| submodules: "recursive" | ||||||||||||||||||
|
|
||||||||||||||||||
| - name: "Login to image registry" | ||||||||||||||||||
| uses: "docker/login-action@74a5d142397b4f367a81961eba4e8cd7edddf772" | ||||||||||||||||||
| with: | ||||||||||||||||||
|
Comment on lines
+12
to
+18
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🧹 Nitpick (assertive) Mixed pinning style: commit SHAs here, version tags elsewhere This workflow pins actions to SHAs while 🤖 Prompt for AI Agents |
||||||||||||||||||
| registry: ghcr.io | ||||||||||||||||||
| username: ${{github.actor}} | ||||||||||||||||||
| password: ${{secrets.GITHUB_TOKEN}} | ||||||||||||||||||
|
|
||||||||||||||||||
| - name: "Set up metadata for dependency image" | ||||||||||||||||||
| id: "metadata-deps-image" | ||||||||||||||||||
| uses: "docker/metadata-action@902fa8ec7d6ecbf8d84d538b9b233a880e428804" | ||||||||||||||||||
| with: | ||||||||||||||||||
| images: >- | ||||||||||||||||||
| ghcr.io/${{github.repository}}/prestissimo-worker-dev-env | ||||||||||||||||||
| tags: |- | ||||||||||||||||||
| type=raw,value=dev | ||||||||||||||||||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Upper-case repository names break GHCR pushes Convert - ghcr.io/${{github.repository}}/prestissimo-worker-dev-env
+ ghcr.io/${{ github.repository_owner }}/prestissimo-worker-dev-env📝 Committable suggestion
Suggested change
🤖 Prompt for AI Agents |
||||||||||||||||||
|
|
||||||||||||||||||
| - name: "Build and push dependency image" | ||||||||||||||||||
| uses: "docker/build-push-action@471d1dc4e07e5cdedd4c2171150001c434f0b7a4" | ||||||||||||||||||
| with: | ||||||||||||||||||
| context: "./presto-native-execution/" | ||||||||||||||||||
| file: "./presto-native-execution/scripts/dockerfiles/ubuntu-22.04-dependency.dockerfile" | ||||||||||||||||||
| push: >- | ||||||||||||||||||
| ${{github.event_name != 'pull_request' | ||||||||||||||||||
| && github.ref == 'refs/heads/release-0.293-clp-connector'}} | ||||||||||||||||||
| tags: "${{steps.metadata-deps-image.outputs.tags}}" | ||||||||||||||||||
| labels: "${{steps.metadata-deps-image.outputs.labels}}" | ||||||||||||||||||
|
|
||||||||||||||||||
| - name: "Set up metadata for runtime image" | ||||||||||||||||||
| id: "metadata-runtime-image" | ||||||||||||||||||
| uses: "docker/metadata-action@902fa8ec7d6ecbf8d84d538b9b233a880e428804" | ||||||||||||||||||
| with: | ||||||||||||||||||
| images: >- | ||||||||||||||||||
| ghcr.io/${{github.repository}}/prestissimo-worker | ||||||||||||||||||
| tags: |- | ||||||||||||||||||
| type=raw,value=dev | ||||||||||||||||||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🛠️ Refactor suggestion Same lowercase issue for the runtime image path Apply the same fix as above to keep both images consistent and push-able. 🤖 Prompt for AI Agents |
||||||||||||||||||
|
|
||||||||||||||||||
| - name: Get number of cores | ||||||||||||||||||
| id: get-cores | ||||||||||||||||||
| run: |- | ||||||||||||||||||
| echo "num_cores=$(nproc)" >> $GITHUB_OUTPUT | ||||||||||||||||||
|
|
||||||||||||||||||
| - name: "Build and push runtime image" | ||||||||||||||||||
| uses: "docker/build-push-action@471d1dc4e07e5cdedd4c2171150001c434f0b7a4" | ||||||||||||||||||
| with: | ||||||||||||||||||
| build-args: |- | ||||||||||||||||||
| BASE_IMAGE=ubuntu:22.04 | ||||||||||||||||||
| DEPENDENCY_IMAGE=${{steps.metadata-deps-image.outputs.tags}} | ||||||||||||||||||
| EXTRA_CMAKE_FLAGS=-DPRESTO_ENABLE_TESTING=OFF -DPRESTO_ENABLE_PARQUET=ON -DPRESTO_ENABLE_S3=ON | ||||||||||||||||||
| NUM_THREADS=${{steps.get-cores.outputs.num_cores}} | ||||||||||||||||||
| OSNAME=ubuntu | ||||||||||||||||||
| context: "./presto-native-execution/" | ||||||||||||||||||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
- DEPENDENCY_IMAGE=${{steps.metadata-deps-image.outputs.tags}}
+ DEPENDENCY_IMAGE=${{ steps.metadata-deps-image.outputs.tags }}
+ # If multiple tags are present, keep only the first one
+ DEPENDENCY_IMAGE=${DEPENDENCY_IMAGE%%$'\n'*}or parse it in a separate step and set an output.
🤖 Prompt for AI Agents |
||||||||||||||||||
| file: "./presto-native-execution/scripts/dockerfiles/prestissimo-runtime.dockerfile" | ||||||||||||||||||
| push: >- | ||||||||||||||||||
| ${{github.event_name != 'pull_request' | ||||||||||||||||||
| && github.ref == 'refs/heads/release-0.293-clp-connector'}} | ||||||||||||||||||
| tags: "${{steps.metadata-runtime-image.outputs.tags}}" | ||||||||||||||||||
| labels: "${{steps.metadata-runtime-image.outputs.labels}}" | ||||||||||||||||||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -39,6 +39,15 @@ FROM ${BASE_IMAGE} | |
| ENV BUILD_BASE_DIR=_build | ||
| ENV BUILD_DIR="" | ||
|
|
||
| # Temporary fix for https://github.com/prestodb/presto/issues/25531 | ||
| # TODO: Update this code when there's a proper fix | ||
| RUN apt-get update && \ | ||
| DEBIAN_FRONTEND=noninteractive apt-get install -y tzdata && \ | ||
| ln -snf /usr/share/zoneinfo/America/Toronto /etc/localtime && \ | ||
| echo "America/New_York" > /etc/timezone && \ | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. What error do we get if we don't have these lines?
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. prestodb#25531 |
||
| apt-get clean && \ | ||
| rm -rf /var/lib/apt/lists/* | ||
|
|
||
| COPY --chmod=0775 --from=prestissimo-image /prestissimo/${BUILD_BASE_DIR}/${BUILD_DIR}/presto_cpp/main/presto_server /usr/bin/ | ||
| COPY --chmod=0775 --from=prestissimo-image /runtime-libraries/* /usr/lib64/prestissimo-libs/ | ||
| COPY --chmod=0755 ./etc /opt/presto-server/etc | ||
|
|
||
Uh oh!
There was an error while loading. Please reload this page.