Refactor AI

.vscode/ltex.dictionary.en-US.txt

@@ -9,3 +9,11 @@ OpenFastTrace
 OpenRewrite
 Temurin
 jpackage
+impl
+utest
+dsn
+summarizator
+HuggingFace
+vLLM
+Ollama
+de-facto

CHANGELOG.md

@@ -11,6 +11,7 @@ Note that this project **does not** adhere to [Semantic Versioning](https://semv
 
 ### Added
 
+- Added support for selecting answer engines and summarization algorithms, allowing users to change the underlying AI behavior.
 - We fixed a glitch with the sidepane divider position on startup. [#15394](https://github.com/JabRef/jabref/issues/15394)
 - We added a label to the Group dropdown in the Import Dialog. [#15567](https://github.com/JabRef/jabref/issues/15567)
 - We added a related work text extractor, which finds and inserts the related work text into bib entries from references in the texts. [#9840](https://github.com/JabRef/jabref/issues/9840)
@@ -34,6 +35,7 @@ Note that this project **does not** adhere to [Semantic Versioning](https://semv
 
 ### Changed
 
+- We refactored and enhanced the code of AI features.
 - We replaced deprecated Gemini Models from the AI chat model selection and with current ones. [#15398](https://github.com/JabRef/jabref/issues/15398)
 - We changed CSL reference format by adding citation type at the end.  [#15370](https://github.com/JabRef/jabref/issues/15370) [#15434](https://github.com/JabRef/jabref/issues/15434)
 - We changed the groups filter field to use a filter icon. [#15402](https://github.com/JabRef/jabref/issues/15402)

build.gradle.kts

@@ -51,6 +51,18 @@ requirementTracing {
             "jabsrv/src/test/java"
         )
     )
+
+	filteredArtifactTypes =
+        listOf(
+            "impl",
+            "utest",
+            "model",
+            "guard",
+            "pp",
+            "feat",
+            "req"
+		)
+
     // TODO: Short Tag Importer: https://github.com/itsallcode/openfasttrace-gradle#configuring-the-short-tag-importer
 }
 

docs/code-howtos/ai.md

@@ -4,29 +4,101 @@ parent: Code Howtos
 
 # AI
 
-The AI feature of JabRef is built on [LangChain4j](https://github.com/langchain4j/langchain4j) and [Deep Java Library](https://djl.ai/).
+The JabRef has next AI features:
+
+- Chatting with entries,
+- Chatting with groups,
+- Summarization of entries,
+- Parsing of plain citations using LLMs
+- Extracting "References" section from PDFs with the help of LLMs.
+
+The features are built on [LangChain4j](https://github.com/langchain4j/langchain4j) and [Deep Java Library](https://djl.ai/).
 
 ## Architectural Decisions
 
 See [ADR-0037](../decisions/0037-rag-architecture-implementation.md) for the decision regarding the RAG infrastructure.
 
-## Feature "Chat with PDF(s)"
+The [ADR-0032](../decisions/0032-store-chats-in-local-user-folder.md) and [ADR-0033](../decisions/0033-store-chats-in-mvstore.md) are important ones, because they explain the decisions regarding the storage of AI artifacts (summaries, chat histories, embeddings, etc.).
+
+## Requirements
+
+See [the requirements page of AI features](../requirements/ai.md). 
+
+## Features
+
+### Feature "Chat with PDF(s)"
+
+The interface with all of the features (chat history, regeneration, follow up questions, etc.) is implemented in the class [org.jabref.gui.ai.chat.AiChatView]. From there, one will find preferences and other required infrastructure.
 
-This is implemented mainly in the class [org.jabref.logic.ai.chatting.AiChatLogic].
-From there, one will find preferences and other required infrastructure.
+The RAG entry point is located in [org.jabref.logic.ai.chatting.tasks.GenerateRagResponseTask].
 
-## Feature "Summarize PDF(s)"
+### Feature "Summarize PDF(s)"
 
-This is implemented in the class [org.jabref.logic.ai.summarization.GenerateSummaryTask].
+This is implemented in the class [org.jabref.logic.ai.summarization.tasks.GenerateSummaryTask].
 
-## Feature "BibTeX from Reference Text"
+### Feature "BibTeX from Reference Text"
 
 The general interface is [org.jabref.logic.importer.plaincitation.PlainCitationParser].
 The class implementing it using AI is [org.jabref.logic.importer.plaincitation.LlmPlainCitationParser].
 
-## Feature "Reference Extractor"
+### Feature "Reference Extractor"
 
 Extracts the list of references (Section ["References"](../glossary/references.md)) from the last page of the PDF to a List of BibEntry.
 
 The general interface is [org.jabref.logic.importer.fileformat.pdf.BibliographyFromPdfImporter].
 The class implementing it using AI is [org.jabref.logic.importer.plaincitation.LlmPlainCitationParser].
+
+## Code organization
+
+As every JabRef feature, AI is divided into 3 layers: GUI, logic, and model. Inside the `logic` package the AI code is split by feature (each feature has its own package).
+
+The GUI code strongly follows [MVVM pattern](./javafx.md). Though, the GUI code is a bit complicated as:
+
+1. Most of the core GUI components (chat and summary components) are designed as a state machine. Typical states include: loading, presenting the result, error, etc.
+2. These core GUI components are also made that way so it would be possible to rebind them to another `BibEntry`. For the details, take a look at the section [How to add a new AI feature](## How to add a new AI feature).
+
+## Internal model (v2)
+
+There are 3 core models in the AI features:
+
+1. Chat history.
+2. Summaries.
+3. Embeddings.
+4. Fully ingested documents.
+
+The code strictly follows the repository pattern, where an interface is created to access the internal storage for the purpose of abstraction. At the moment of writing, all of these models are implemented by using the [`MVStore`](https://www.h2database.com/html/mvstore.html). For the details of this decisions take a look at the [ADR 0033](./../0033-store-chats-in-mvstore.md). A helper class was made `MVStoreBase` so that it would be possible to use an in-memory `MVStore` in case there are some errors while opening on-disk storage.
+
+A note needs to be made for embeddings: the embeddings storage is also implementing the internal LangChain4j interface for embeddings so that it could be used in LangChain4j algorithms. Additionally, there is a "fully ingested" repository, which simply contains a "list" of files that were fully ingested. This helps with checking if a file needs to be ingested or not, as there is no 1 to 1 correspondense with embeddings to file (which is many to one).
+
+Because JabRef is not build around one global database, but rather it is a `.bib` file editor, a problem of identifying a `BibEntry` arose and it was solved in a somewhat complicated way:
+
+- In order to uniquely identify a library, an "AI library ID" was introduced (as a metadata field), which is just a UUID. An alternative would be to use the library path, but if the library moves, the path changes, but AI library ID is not.
+- In order to uniquely identify an entry, the citation key is used, but only if it is non-empty and unique.
+- In some cases (that arise potentially often), the conditions above are not met (for example, a library is not saved - it does not have a path, or an entry does not have a citation key), however user is actively working on an entry. In this case the AI features have an *in-memory cache layer*. So whenever a chat or a summary is created for an entry, it is firstly interacted with the in-memory storage layer. The cache is flushed to the on-disk storage at the close of the JabRef.
+- In order to uniquely identify a file, we use the file hash. An alternative would be to use the file path, but the file could be moved, or defined by a relative path. This is also useful when several libraries cite the same paper, and instead of ingesting 
+
+## [OLD] Internal model (v1)
+
+The model v1 differs from v2 by:
+
+1. Fields of the chat messages and summaries were differently organized in the `MVStore`.
+2. A `LinkedFile#getLink()` was used to identify a file.
+
+To migrate from v1 to v2, the classes `ChatHistoryMigrationV1` and `SummariesMigrationV2` were made.
+
+## How to add a new AI feature
+
+This section describes the standard pattern used for AI features. If should follow a similar plan:
+
+1. Define the model of the artifact of your feature (for example, for summarization it is an AI summary, for chatting they are chat messages and chat history).
+2. Define a repository interface (e.g. `SummaryRepository`, `ChatHistoryRepository`) and implement an `MVStore` implementation using the [org.jabref.logic.ai.util.MVStoreBase].
+3. Define a logic class in the `logic` package: either a task (e.g. `GenerateSummaryTask` or a utility class for performing an AI feature. It is recommended to make it "without side-effects" (it does not change or write anything in the system). Firstly, this will help in testing the class, and, secondly, the storage is typically hanlded in *in-memory cache* layer, that will be discussed next.
+4. Make an in-memory cache storage layer for your feature that has a RAM map between a `BibEntry` (or a group, or some other object that your artifact is linked to) and your model. Sometimes this can be omitted (for example, embeddings do not have the in-memory cache and always use a repository), but generally it is made in order to always have access to the AI feature even if some precondition is not satisfied (for example, storing chat history and summmaries requires that there is a database path and a non-empty unique citation key, but in-memory layer allows to work with them as is). At the close of JabRef (or a library) the in-memory cache layer will check the preconditions and only then write the data to the repository.
+5. Make a `TaskAggregator` class. This is needed in order to be able to switch a component between entries and to deduplicate the tasks. So whenever you want to generate the artifact of your feature, you need to always communicate to the `TaskAggregator` class which will either create a new task or give you an already running one. The `TaskAggregator` also connects the results to the in-memory cache.
+
+The next points are targeted to the GUI of the feature:
+
+1. Design a component using the MVVM pattern. You need to write the interface in the FXML, then write a controller `Ai<Feature>View` and a view-model `Ai<Feature>ViewModel`. 
+2. A typical AI component will be a state machine: first and foremost, check if the AI features are enabled in JabRef (which equals to accepting a privacy policy of AI features). If not, then you must ensure that you component does nothing. To show the privacy policy banner, there is a dedicated component [org.jabref.gui.ai.AiPrivacyNoticeView]. The next states typically envolve checking some preconditions (for example, you can not summarize an entry, if it does not have linked files), and the final is the working state. You might find the [org.jabref.gui.util.BindingsHelper#bindEnum] useful.
+3. The entry editor tabs are designed to be switchable (rebound to some other `BibEntry`), so you can have an `entryProperty` and whenver it is changed, the state machine of the component is rerun.
+4. When you read an artifact for an entry (or a group, or other entity that is linked to your AI feature), the look-up should be made in 3 steps: look into the repository, look in to the in-memory cache, and only then contact the `TaskAggregator` to start a new generation task.

docs/decisions/0033-store-chats-in-mvstore.md

@@ -4,6 +4,8 @@ parent: Decision Records
 ---
 
 # Store Chats in MVStore
+<!-- dsn->req~ai.summarization.general.storage~1 -->
+<!-- dsn->req~ai.chat.entries.history-storage~1 -->
 
 ## Context and Problem Statement
 
@@ -51,3 +53,7 @@ Chosen option: "MVStore", because it is simple and memory-efficient.
 * Good, because we have the full control
 * Bad, because involves writing our own language and parser
 * Bad, because we need to implement optimizations found in databases on our own (storing some data in RAM, other on disk)
+
+## More information
+
+For the same logic, the summaries are stored in MVStore.

docs/decisions/0036-use-textarea-for-chat-content.md → docs/decisions/0036-use-markdown-for-chat-content.md

@@ -3,7 +3,7 @@ nav_order: 0036
 parent: Decision Records
 ---
 
-# Use `TextArea` for Chat Message Content
+# Use Markdown rendering for Chat Message Content
 
 ## Context and Problem Statement
 
@@ -25,10 +25,8 @@ This decision record concerns the UI component that is used for rendering the co
 
 ## Decision Outcome
 
-Chosen option: "Use `TextArea`".
-All other options require more time to implement.
-Some of the options do not support text selection and copying,
-which for now we value more than Markdown rendering.
+Chosen option: (modified) "Use a Markdown parser and convert AST nodes to JavaFX TextFlow elements".
+In JabRef there is a component `SelectableTextFlow` which allows to create a formatted text and to select it. This makes possible to use a Markdown parser that converts the content into JavaFX nodes and adds the feature selecting the text.
 
 ## Pros and Cons of the Options
 

docs/decisions/0056-embedding-implementation-approach.md

@@ -0,0 +1,77 @@
+---
+nav_order: 0056
+parent: Decision Records
+---
+# Implementation of embeddings in JabRef
+
+<!-- dsn->feat~ai.answer-engines.embeddings-search~1 -->
+
+## Context and Problem Statement
+
+JabRef needs to implement embedding models to perform Retrieval-Augmented Generation (RAG) by generating embeddings for chunks of papers. The AI ecosystem in Java is not as diverse or developed as it is in Python, which limits the available tools for this task. 
+
+We need to decide how to design this integration to balance ease of development with the constraints of a desktop application.
+
+The features that we need are:
+
+1. Ability to find models.
+2. Ability to download models.
+3. Ability to execute inference on models.
+
+## Decision Drivers
+
+* The approach should not require additional setup from the user side
+* It should be cross-platform
+* It should support a wide variety of model architectures
+* It should have an easy-to-use API
+* The request that embedding libraries make should be known and controlled
+* We should know how and where some library downloads and stores models
+
+## Considered Options
+
+* Custom implementation
+* Use a mix of custom implementation and an inference library
+* Use a comprehensive library
+
+## Decision Outcome
+
+Chosen option: "Use a comprehensive library", because it is the standard approach in software engineering to rely on specialized libraries for complex tasks rather than re-implementing them. It allows us to delegate the heavy lifting of model management and inference to a dedicated tool.
+
+### Consequences
+
+* Good, because it reduces the maintenance burden on the JabRef team
+* Good, because we do not have to implement complex inference algorithms ourselves
+* Bad, because we are dependent on the external library for updates and maintenance
+* Bad, because it is not easy to find such a library in Java ecosystem
+
+## Pros and Cons of the Options
+
+### Custom implementation
+
+* Good, because it will just work (no strange issues with PyTorch or ONNX, etc.)
+* Bad, because it is a lot of work
+* Bad, because for each architecture of embedding models we would have to write code
+
+### Use a mix of custom implementation and an inference library
+
+In this approach we manually implement the features 1 and 2, while relying on an inference library for 3.
+
+* Good, because complex computations are delegated to another library
+* Good, because we are in full control of data (network rquests, model storage)
+* Bad, because an inference library might be too scientifically centered with a complex API
+* Bad, because we have to write code to find the models, which is not easy
+* Bad, because the inference engine might not provide every tool needed
+
+### Use a comprehensive library
+
+* Good, because this doesn't require custom code
+* Good, because this is the right approach from software engineering POV
+* Good, because it implements all features
+* Good, because we delegate responsibilities to another library
+* Neutral, because there is a small number of libraries for Java for these tasks
+* Bad, because it might make untraceable requests to network
+* Bad, because the storage of the models might be not customizable
+
+## More information
+
+This ADR is highly related to [ADR 0037 - RAG Architecture Implementation](./0037-rag-architecture-implementation.md).