-
Notifications
You must be signed in to change notification settings - Fork 0
Deno 2, Hash-Blobs, Fuseki Update #34
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
retog
wants to merge
91
commits into
master
Choose a base branch
from
deno2
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from 83 commits
Commits
Show all changes
91 commits
Select commit
Hold shift + click to select a range
42defa2
move old stuff out of the way
nleanba 3c7dcd8
defined all routes
nleanba 5d13636
fmt
nleanba 315819c
added GET /version
nleanba b11dee5
docker framework
nleanba 08ff7a3
setup devcontainer
nleanba f08e7f1
added zip restore
nleanba 1ed70b6
added cors headers
nleanba d208e9e
added cors headers
nleanba 21700f8
Merge branch 'deno' of github.com:tridoc/tridoc-backend into deno
nleanba ffce65f
added count
nleanba 4c45866
added count
nleanba 9a72686
added GET /doc/:id
nleanba c1c0b30
added GET /doc/:id/comment
nleanba 1c5795c
added POST /doc
nleanba 36ec78c
added filename to GET /doc
nleanba 1ef5b30
added delete
nleanba 4728531
redactored to fusekiFetch/Update instead of fetch
nleanba 1d41598
added POST /doc/:id/comment
nleanba a9828f3
added GET /doc/:id/thumb
nleanba 6b67780
added GET /doc/:id/title
nleanba c5576ef
added GET /doc/:id/tag, /doc/:id/meta
nleanba 4187119
added GET /raw/rdf
nleanba 520a4d0
added GET /raw/tgz
nleanba 8151fe3
added GET /raw/zip
nleanba 6050e11
rm broken
nleanba 93bfaa6
added GET /tag/:tagLabel
nleanba b0384de
added POST /doc/:id/tag
nleanba 690595b
added PUT /doc/:id/title
nleanba 0479d8d
added POST /tag
nleanba 876f0ae
added DELETE /tag/label and /doc/:id/tag/:label
nleanba aa92beb
refined HTTP response codes
nleanba 8c4d889
implemented DELETE /doc/:id/title
nleanba 30e234e
added message about file permissions as temp fix
nleanba b875206
Added correct content-type for all JSON responses
nleanba 9d17a03
Removed deadlock
nleanba 7aa1e21
tidying up
retog 70168fb
version
retog 354d237
typo
retog 74215d8
sonnet 4
retog 070df3d
Pending changes exported from your codespace
retog 0d22487
builds/starts
retog be1d662
allow pings
retog 4510135
bash
retog 0088070
permission on deno-dir
retog c302591
peesist bash history on host
retog 3e7cbc4
avoiding ownership issuer
retog 813c55a
obsolete
retog 8757236
ensure dir exists
retog de2ffc5
deno lock
retog 7764dd3
removed obsolete
retog 767f2a8
dev one in .devcontaIner
retog 20cec7f
removed duplication
retog 668f805
fixed hist conf
retog 4c6cc3d
responding as expected by cockpit
retog 1647370
responses expected by cockpit
retog b1b4d61
addressed lint errors
retog c84a413
fixed table
retog 7e0d2c0
clarified
retog b1ec0c6
Allow to PUT graph
retog 73ee2ea
get orphaned
retog 5b4f612
comment obsolete
retog 19b1090
turtle default + cleanup
retog ee464b1
tmp - permission
retog 7eefc42
create orphaned archive in tmp
retog 25b8afa
hash based blob storage #33
retog 40b9075
removing extension before comparing with hash #33
retog 854f0b2
procps and deno 2.4.5
retog 6a5e9d4
code duplication / fixes
retog 56752b0
permissions / no-prompt
retog de5834f
flat file structure
retog 28442c5
obsolete
retog f2db2a3
fixed thumbnails
retog 8936219
adapted container paths
retog ebe62ae
fixed indent
retog 008c5f3
removing legacy files after migration
retog ec31e39
fixed OCR
retog 5e8825e
cleanupAllEmptyLegacyDirectories
retog 4bbdcd5
updated fuseki, uniondefaultgraph not working
retog c718452
specifying graph rather than using uniondefault
retog 64eff5a
using fuseki-base:5.4.0
retog ef799e0
removed `.env`
retog c56ee55
/migrate changed to POST as the operation changes the state of the se…
retog d1bcbe7
remove unused datecheck function from doc.ts
retog 91878be
formated (deno fmt)
retog cf814e2
validate tag label presence in deleteTag and getDocs functions
retog fd59d51
Refactor: extract pdfsandwich OCR to helper runPdfsandwich
retog a53fca4
Remove unstable flag from Deno settings in VSCode configuration
retog 356a082
removed variable as this wronly suggests configurability while in fac…
retog fa8edb1
Refactor: remove unused _getDir function and update getPath for legac…
retog 4134f2c
setting OCR_LANG
retog File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,55 @@ | ||
| // For format details, see https://aka.ms/devcontainer.json. For config options, see the README at: | ||
| // https://github.com/microsoft/vscode-dev-containers/tree/v0.245.2/containers/docker-existing-docker-compose | ||
| // If you want to run as a non-root user in the container, see .devcontainer/docker-compose.yml. | ||
| { | ||
| "name": "Tridoc Backend Development", | ||
|
|
||
| // Use the independent dev container docker-compose configuration | ||
| "dockerComposeFile": "docker-compose.yml", | ||
|
|
||
| "containerEnv": { | ||
| "TRIDOC_PWD": "pw123", | ||
| "OCR_LANG": "deu" | ||
| }, | ||
|
|
||
| // The 'service' property is the name of the service for the container that VS Code should | ||
| // use. Update this value and .devcontainer/docker-compose.yml to the real service name. | ||
| "service": "tridoc", | ||
|
|
||
| // The optional 'workspaceFolder' property is the path VS Code should open by default when | ||
| // connected. This is typically a file mount in .devcontainer/docker-compose.yml | ||
| "workspaceFolder": "/usr/src/app", | ||
|
|
||
| // Use 'forwardPorts' to make a list of ports inside the container available locally. | ||
| "forwardPorts": [8000, 8001], | ||
|
|
||
| // Start the fuseki service when the dev container starts | ||
| "runServices": ["fuseki"], | ||
|
|
||
| // Uncomment the next line if you want to keep your containers running after VS Code shuts down. | ||
| "shutdownAction": "stopCompose", | ||
|
|
||
| // Post-create command to set up the development environment | ||
| "postCreateCommand": "bash .devcontainer/setup-dev.sh", | ||
|
|
||
| // Connect as the deno user | ||
| "remoteUser": "deno", | ||
|
|
||
| "customizations": { | ||
| "vscode": { | ||
| "extensions": [ | ||
| "denoland.vscode-deno" | ||
| ], | ||
| "settings": { | ||
| "deno.enable": true, | ||
| "deno.lint": true, | ||
| "terminal.integrated.defaultProfile.linux": "bash", | ||
| "terminal.integrated.profiles.linux": { | ||
| "bash": { | ||
| "path": "/bin/bash" | ||
| } | ||
| } | ||
| } | ||
| } | ||
| } | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,47 @@ | ||
| services: | ||
| # Development environment for tridoc-backend | ||
| tridoc: | ||
| build: | ||
| context: .. | ||
| dockerfile: Dockerfile | ||
| user: deno | ||
| volumes: | ||
| # Mount the entire workspace for development | ||
| - ..:/usr/src/app:cached | ||
| # Mount blobs directory separately if needed | ||
| - ../blobs:/usr/src/app/blobs | ||
| ports: | ||
| - "8000:8000" | ||
| depends_on: | ||
| fuseki: | ||
| condition: service_healthy | ||
| environment: | ||
| TRIDOC_PWD: "${TRIDOC_PWD:-pw123}" | ||
| OCR_LANG: "${OCR_LANG:-deu}" | ||
| # Keep container running for development | ||
| command: ["sleep", "infinity"] | ||
| networks: | ||
| - tridoc-dev | ||
|
|
||
| # Fuseki service accessible as 'fuseki' hostname | ||
| fuseki: | ||
| image: "linkedsolutions/fuseki-base:5.4.0" | ||
| environment: | ||
| ADMIN_PASSWORD: "${FUSEKI_PWD:-pw123}" | ||
| ports: | ||
| - "8001:3030" # Expose for development access | ||
| volumes: | ||
| - ./fuseki-data:/fuseki/base | ||
| - ../config-tdb.ttl:/config.ttl | ||
| healthcheck: | ||
| test: ["CMD", "curl", "-fsS", "http://localhost:3030/$/ping"] | ||
| interval: 5s | ||
| timeout: 3s | ||
| retries: 30 | ||
| start_period: 10s | ||
| networks: | ||
| - tridoc-dev | ||
|
|
||
| networks: | ||
| tridoc-dev: | ||
| driver: bridge |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| #!/bin/bash | ||
|
|
||
| echo "Setting up Tridoc Backend development environment..." | ||
|
|
||
| # Ensure dataset exists using shared script (waits for Fuseki internally) | ||
| if [ -f "./database-create.sh" ]; then | ||
| bash ./database-create.sh 3DOC || echo "(setup-dev) Dataset ensure script exited with non-zero status; continuing." | ||
| else | ||
| echo "(setup-dev) WARNING: database-create.sh not found; skipping dataset ensure." | ||
| fi | ||
|
|
||
| # Cache Deno dependencies if deps.ts exists | ||
| if [ -f "src/deps.ts" ]; then | ||
| echo "Caching Deno dependencies..." | ||
| deno cache src/deps.ts | ||
| fi | ||
|
|
||
| echo "Dataset bootstrap (if needed) complete." | ||
|
|
||
| echo "Development environment setup complete!" | ||
| echo "" | ||
| echo "You can now run the Tridoc backend with:" | ||
| echo "deno run --watch --allow-net --allow-read=blobs,rdf.ttl --allow-write=blobs,rdf.ttl --allow-run --allow-env=TRIDOC_PWD,FUSEKI_PWD,OCR_LANG src/main.ts" | ||
| echo "" | ||
| echo "Fuseki is available at:" | ||
| echo "- Internal: http://fuseki:3030" | ||
| echo "- External: http://localhost:8001" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,3 +1,4 @@ | ||
| old | ||
| blobs | ||
| fuseki-base | ||
| node_modules |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,65 +1,12 @@ | ||
| # Logs | ||
| logs | ||
| *.log | ||
| npm-debug.log* | ||
| yarn-debug.log* | ||
| yarn-error.log* | ||
|
|
||
| # Runtime data | ||
| pids | ||
| *.pid | ||
| *.seed | ||
| *.pid.lock | ||
|
|
||
| # Directory for instrumented libs generated by jscoverage/JSCover | ||
| lib-cov | ||
|
|
||
| # Coverage directory used by tools like istanbul | ||
| coverage | ||
|
|
||
| # nyc test coverage | ||
| .nyc_output | ||
|
|
||
| # Grunt intermediate storage (http://gruntjs.com/creating-plugins#storing-task-files) | ||
| .grunt | ||
|
|
||
| # Bower dependency directory (https://bower.io/) | ||
| bower_components | ||
|
|
||
| # node-waf configuration | ||
| .lock-wscript | ||
|
|
||
| # Compiled binary addons (https://nodejs.org/api/addons.html) | ||
| build/Release | ||
|
|
||
| # Dependency directories | ||
| node_modules/ | ||
| jspm_packages/ | ||
|
|
||
| # TypeScript v1 declaration files | ||
| typings/ | ||
|
|
||
| # Optional npm cache directory | ||
| .npm | ||
|
|
||
| # Optional eslint cache | ||
| .eslintcache | ||
|
|
||
| # Optional REPL history | ||
| .node_repl_history | ||
|
|
||
| # Output of 'npm pack' | ||
| *.tgz | ||
|
|
||
| # Yarn Integrity file | ||
| .yarn-integrity | ||
|
|
||
| # dotenv environment variables file | ||
| .env | ||
|
|
||
| # next.js build output | ||
| .next | ||
|
|
||
| node_modules | ||
| blobs | ||
|
|
||
| fuseki-base | ||
| fuseki-data | ||
| .devcontainer/.bash_history | ||
| .bash_history | ||
| .deno-dir |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,15 +1,28 @@ | ||
| { | ||
| // Use IntelliSense to learn about possible attributes. | ||
| // Hover to view descriptions of existing attributes. | ||
| // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387 | ||
| "version": "0.2.0", | ||
| "configurations": [ | ||
| { | ||
| "name": "Launch Tridoc Backend", | ||
| "type": "node", | ||
| "request": "launch", | ||
| "name": "Launch Program", | ||
| "program": "${workspaceFolder}/lib/server", | ||
| "env": {"TRIDOC_PWD": "tridoc"} | ||
| "program": "${workspaceFolder}/src/main.ts", | ||
| "runtimeExecutable": "deno", | ||
| "runtimeArgs": [ | ||
| "run", | ||
| "--watch", | ||
| "--no-prompt", | ||
| "--allow-net", | ||
| "--allow-read=blobs,rdf.ttl,/tmp", | ||
| "--allow-write=blobs,rdf.ttl,/tmp", | ||
| "--allow-run", | ||
| "--allow-env=FUSEKI_PWD,TRIDOC_PWD,OCR_LANG, TMPDIR" | ||
| ], | ||
| "attachSimplePort": 9229, | ||
| "env": { | ||
| "TRIDOC_PWD": "pw123", | ||
| "OCR_LANG": "deu" | ||
| }, | ||
| "console": "integratedTerminal" | ||
| } | ||
| ] | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,3 +1,5 @@ | ||
| { | ||
| "npm.packageManager": "yarn" | ||
| "deno.enable": true, | ||
| "deno.lint": true, | ||
| "deno.unstable": true | ||
|
retog marked this conversation as resolved.
Outdated
|
||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,77 @@ | ||
| { | ||
| "version": "2.0.0", | ||
| "tasks": [ | ||
| { | ||
| "label": "Start Tridoc Backend", | ||
| "type": "shell", | ||
| "command": "deno", | ||
| "args": [ | ||
| "run", | ||
| "--watch", | ||
| "--no-prompt", | ||
| "--allow-net", | ||
| "--allow-read=blobs,rdf.ttl,/tmp", | ||
| "--allow-write=blobs,rdf.ttl,/tmp", | ||
| "--allow-run", | ||
| "--allow-env=TRIDOC_PWD,OCR_LANG,FUSEKI_PWD", | ||
| "src/main.ts" | ||
| ], | ||
| "group": { | ||
| "kind": "build", | ||
| "isDefault": true | ||
| }, | ||
| "presentation": { | ||
| "echo": true, | ||
| "reveal": "always", | ||
| "focus": false, | ||
| "panel": "new" | ||
| }, | ||
| "problemMatcher": [], | ||
| "options": { | ||
| "env": { | ||
| "TRIDOC_PWD": "pw123", | ||
| "OCR_LANG": "deu" | ||
| } | ||
| } | ||
| }, | ||
| { | ||
| "label": "Cache Dependencies", | ||
| "type": "shell", | ||
| "command": "deno", | ||
| "args": ["cache", "src/deps.ts"], | ||
| "group": "build", | ||
| "presentation": { | ||
| "echo": true, | ||
| "reveal": "always", | ||
| "focus": false, | ||
| "panel": "shared" | ||
| } | ||
| }, | ||
| { | ||
| "label": "Format Code", | ||
| "type": "shell", | ||
| "command": "deno", | ||
| "args": ["fmt"], | ||
| "group": "build", | ||
| "presentation": { | ||
| "echo": true, | ||
| "reveal": "silent", | ||
| "focus": false, | ||
| "panel": "shared" | ||
| } | ||
| }, | ||
| { | ||
| "label": "Lint Code", | ||
| "type": "shell", | ||
| "command": "deno", | ||
| "args": ["lint"], | ||
| "group": "test", | ||
| "presentation": { | ||
| "echo": true, | ||
| "reveal": "always", | ||
| "focus": false, | ||
| "panel": "shared" | ||
| } | ||
| } | ||
| ] | ||
| } |
Empty file.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why not have FUSKEI_PWD here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A change would require recreating all containers of the multi-container dev container. A normal rebuild / full rebuild leaves the fuseki contatiner untouched. @nleanba, do you know how to do that? Because we should instruct users on what needs to be done for changes to have effect (or else not present the option to change).