Skip to content

⚡ Bolt: Resolve cloning latency with batched citation insertions #211

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
aicoder2009 wants to merge 1 commit into
base: main
Choose a base branch
from
Open

⚡ Bolt: Resolve cloning latency with batched citation insertions #211

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .jules/bolt.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,7 @@
## 2024-05-18 - Concurrent Fetching with Owner IDs in Share Links
**Learning:** In the share system, endpoints processing a share code typically suffer from waterfall latency by first loading target entity details (like a project) and then querying its associated elements (like project lists) using the entity's owner ID (`userId`). However, the `shareLink` object already contains the `userId` field (representing the owner's ID).
**Action:** Always leverage the existing owner's ID within `shareLink` to bypass sequential dependencies and fetch parent entities (like projects) concurrently with their child elements (like user lists) using `Promise.all`.

## 2024-05-19 - Batch Operations for DynamoDB N+1 Insert Latency
**Learning:** During iterative data duplication tasks like cloning projects or lists, mapping over array insertions using `Promise.all` triggers a severe N+1 database request bottleneck.
**Action:** Always prefer `BatchWriteCommand` over iteration. By chunking sequential array insertions into groups of 25 items (DynamoDB's batch limit) via custom functions like `batchAddCitations`, we drastically reduce I/O overhead, optimizing write-speed and scaling capacity safely.
34 changes: 4 additions & 30 deletions src/app/api/share/[code]/clone/route.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ import {
createList,
createProject,
addCitation,
batchAddCitations,
reorderCitations,
} from "@/lib/db";
import { parseShareSegment } from "@/lib/share-utils";
Expand Down Expand Up @@ -111,21 +112,7 @@ export async function POST(

const newList = await createList(userId, originalList.name, undefined, originalList.description);

const newCitations = await Promise.all(
citations.map((c) =>
addCitation(
newList.id,
c.fields,
c.style,
c.formattedText,
c.formattedHtml,
c.tags,
c.notes,
c.quotes,
c.readingStatus
)
)
);
const newCitations = await batchAddCitations(newList.id, citations);

if (citations.some((c) => c.sortOrder !== undefined)) {
await reorderCitations(newList.id, newCitations.map((c) => c.id));
Expand Down Expand Up @@ -155,21 +142,8 @@ export async function POST(
originalLists.map(async (list) => {
const citations = await getListCitations(list.id);
const newList = await createList(userId, list.name, newProject.id, list.description);
const newCitations = await Promise.all(
citations.map((c) =>
addCitation(
newList.id,
c.fields,
c.style,
c.formattedText,
c.formattedHtml,
c.tags,
c.notes,
c.quotes,
c.readingStatus
)
)
);
const newCitations = await batchAddCitations(newList.id, citations);

if (citations.some((c) => c.sortOrder !== undefined)) {
await reorderCitations(newList.id, newCitations.map((c) => c.id));
}
Expand Down
84 changes: 83 additions & 1 deletion src/lib/db/dynamodb.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -288,11 +288,10 @@
if (citations.length > 0) {
// DynamoDB allows a maximum of 25 items per BatchWriteItem request
const BATCH_SIZE = 25;
const batches = [];

for (let i = 0; i < citations.length; i += BATCH_SIZE) {
const batch = citations.slice(i, i + BATCH_SIZE);
batches.push(

Check failure on line 294 in src/lib/db/dynamodb.ts

View workflow job for this annotation

GitHub Actions / Lint, typecheck, test, build 

Cannot find name 'batches'. Did you mean 'batch'?
docClient.send(
new BatchWriteCommand({
RequestItems: {
Expand All @@ -310,7 +309,7 @@
);
}

await Promise.all(batches);

Check failure on line 312 in src/lib/db/dynamodb.ts

View workflow job for this annotation

GitHub Actions / Lint, typecheck, test, build 

Cannot find name 'batches'.
}

// Revoke any share links pointing at this list so they don't outlive the target.
Expand Down Expand Up @@ -479,6 +478,89 @@

// ============ CITATIONS ============

export async function batchAddCitations(
listId: string,
citationsData: {
fields: CitationFields;
style: CitationStyle;
formattedText: string;
formattedHtml: string;
tags?: string[];
notes?: string;
quotes?: CitationQuote[];
readingStatus?: ReadingStatus;
}[]
): Promise<Citation[]> {
const now = new Date().toISOString();

const citations: Citation[] = citationsData.map((data) => ({
id: generateId(),
listId,
fields: data.fields,
style: data.style,
formattedText: data.formattedText,
formattedHtml: data.formattedHtml,
tags: data.tags,
notes: data.notes,
quotes: data.quotes,
readingStatus: data.readingStatus,
createdAt: now,
updatedAt: now,
}));

if (citations.length > 0) {
// DynamoDB allows a maximum of 25 items per BatchWriteItem request
const BATCH_SIZE = 25;

for (let i = 0; i < citations.length; i += BATCH_SIZE) {
const batch = citations.slice(i, i + BATCH_SIZE);

let requestItems = {
[TABLE_NAME]: batch.map((citation) => ({
PutRequest: {
Item: {
PK: keys.list(listId),
SK: keys.citation(citation.id),
GSI1PK: keys.citation(citation.id),
GSI1SK: keys.list(listId),
...citation,
entityType: "CITATION",
},
},
})),
};

let retries = 0;
const MAX_RETRIES = 3;

while (Object.keys(requestItems).length > 0 && retries <= MAX_RETRIES) {
const response = await docClient.send(
new BatchWriteCommand({
RequestItems: requestItems,
})
);

if (
response.UnprocessedItems &&
Object.keys(response.UnprocessedItems).length > 0
) {
// eslint-disable-next-line @typescript-eslint/no-explicit-any
requestItems = response.UnprocessedItems as any;
retries++;
if (retries <= MAX_RETRIES) {
// Exponential backoff
await new Promise((resolve) => setTimeout(resolve, Math.pow(2, retries) * 100));
}
} else {
break;
}
}
}
}
Comment on lines +511 to +559

@coderabbitai coderabbitai Bot Jul 4, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Inspect relevant symbols and call sites
ast-grep outline src/lib/db/dynamodb.ts --view expanded || true
printf '\n--- batchAddCitations references ---\n'
rg -n "batchAddCitations|reorderCitations|BatchWriteCommand|UnprocessedItems|MAX_RETRIES|BATCH_SIZE" src/lib/db/dynamodb.ts src -g '!**/node_modules/**' || true

printf '\n--- relevant section ---\n'
sed -n '460,590p' src/lib/db/dynamodb.ts

Repository: aicoder2009/opencitation

Length of output: 11308

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Inspect any tests or docs around batch write retry behavior
rg -n "exceeded max retries|unprocessed items|batchAddCitations|BatchWriteItem|BatchWriteCommand" . -g '!**/node_modules/**' -g '!**/dist/**' || true

Repository: aicoder2009/opencitation

Length of output: 1259

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Check whether this file is used in a path where partial success would be observable
rg -n "reorderCitations\\(|clone.*citation|batchAddCitations\\(" src -g '!**/node_modules/**' || true

Repository: aicoder2009/opencitation

Length of output: 1137

Throw when retry exhaustion leaves unprocessed citation writes.

If UnprocessedItems is still non-empty after the last retry, the loop exits and the remaining citation writes are silently dropped. batchAddCitations still returns the full citations array, so callers can’t tell that only part of the batch persisted. Throw on retry exhaustion (or return only persisted items) instead.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/db/dynamodb.ts` around lines 511 - 559, The retry loop in
batchAddCitations silently drops citation writes when BatchWriteCommand still
returns UnprocessedItems after the last retry, so update the logic around the
requestItems/retries loop to detect retry exhaustion and fail explicitly. After
the final retry, if requestItems still contains items, throw an error (or
otherwise surface partial persistence) instead of breaking out and letting the
function return all citations as if they were saved. Use the existing
batchAddCitations and BatchWriteCommand flow to locate the fix.


return citations;
}

export async function addCitation(
listId: string,
fields: CitationFields,
Expand Down
1 change: 1 addition & 0 deletions src/lib/db/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ export const {getProjectLists} = db;

// ============ CITATIONS ============
export const {addCitation} = db;
export const {batchAddCitations} = db;
export const {getCitation} = db;
export const {getListCitations} = db;
export const {updateCitation} = db;
Expand Down
37 changes: 37 additions & 0 deletions src/lib/db/local-store.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -223,6 +223,43 @@ export async function getProjectLists(userId: string, projectId: string): Promis

// ============ CITATIONS ============

export async function batchAddCitations(
listId: string,
citationsData: {
fields: CitationFields;
style: CitationStyle;
formattedText: string;
formattedHtml: string;
tags?: string[];
notes?: string;
quotes?: CitationQuote[];
readingStatus?: ReadingStatus;
}[]
): Promise<Citation[]> {
const now = new Date().toISOString();

const citations: Citation[] = citationsData.map((data) => ({
id: generateId(),
listId,
fields: data.fields,
style: data.style,
formattedText: data.formattedText,
formattedHtml: data.formattedHtml,
tags: data.tags,
notes: data.notes,
quotes: data.quotes,
readingStatus: data.readingStatus,
createdAt: now,
updatedAt: now,
}));

citations.forEach(citation => {
store.citations.set(`${listId}:${citation.id}`, citation);
});

return citations;
}

export async function addCitation(
listId: string,
fields: CitationFields,
Expand Down
Loading