Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/afraid-mangos-follow.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@exactly/infra": minor
---

🧱 setup credit queue worker
5 changes: 5 additions & 0 deletions .changeset/blue-bottles-wave.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@exactly/server": patch
---

✨ automate firewall allow after kyc approval
5 changes: 5 additions & 0 deletions .changeset/bumpy-regions-read.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@exactly/infra": minor
---

🧱 setup poke queue worker
5 changes: 5 additions & 0 deletions .changeset/free-lamps-pump.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@exactly/infra": minor
---

🧱 setup allow queue worker
5 changes: 5 additions & 0 deletions .changeset/proud-tools-sneeze.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@exactly/server": patch
---

✨ process poke in worker
5 changes: 5 additions & 0 deletions .changeset/tender-foxes-feel.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@exactly/server": patch
---

✨ process auto-credit in worker
1 change: 1 addition & 0 deletions infra/Pulumi.base.yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
encryptionsalt: v1::v1:AAAAAAAAAAAAAAAb:rYjVO7/uF2+Qv1LhWq5c2S0ZR4rZaQ==
config:
gcp:project: eexxxaa
exa:pokeMinimum: 1
exa:subscribeTimeout: 900s
1 change: 1 addition & 0 deletions infra/Pulumi.production.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,5 +2,6 @@ encryptionsalt: v1::v1:AAAAAAAAAAAAAAAb:rYjVO7/uF2+Qv1LhWq5c2S0ZR4rZaQ==
config:
gcp:project: eexxxaa
exa:domain: web.exactly.app
exa:pokeMinimum: 1
exa:subscribeTimeout: 900s
exa:whatsappPhoneNumberId: "1287864854409817"
3 changes: 3 additions & 0 deletions infra/utils/modules.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,14 @@ export default define({
},
},
workers: {
allow: { signers: ["allower"] },

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Provision firewall authority for the allower signer

In an infra deployment this creates a new allower KMS account, and server/workers/allow/worker.ts uses that EOA to call Firewall.allow directly, but a repo-wide search finds no on-chain role grant or other provisioning for its address. Consequently Persona approvals enqueue transactions from an unauthorized signer, so allow jobs exhaust their retries and approved accounts remain blocked; reuse the already-authorized account or add an explicit firewall grant during deployment.

Useful? React with 👍 / 👎.

chat: {
env: { WHATSAPP_PHONE_NUMBER_ID: "whatsappPhoneNumberId" },
secrets: ["anthropic-api-key", "whatsapp-access-token"],
},
credit: { secrets: ["onesignal-api-key", "postgres-url"] },
hook: { secrets: ["panda-api-key", "postgres-url"], shared: ["panda-api-url"] },
poke: { secrets: ["onesignal-api-key", "segment-write-key"], signers: ["poker"] },

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Use an authorized keeper for poke transactions

In an infra-deployed worker this creates and loads a distinct poker KMS account, but the deployed plugin grants KEEPER_ROLE only to acct("keeper") in contracts/script/ExaPlugin.s.sol, while poke and pokeETH are explicitly mapped to keeper runtime validation in contracts/src/ExaPlugin.sol; a repo-wide search finds no role grant for the new poker address. Consequently every main-chain poke sent by this worker reverts Unauthorized and exhausts its retries, so the worker must reuse the authorized keeper or provision an on-chain grant for the KMS address.

Useful? React with 👍 / 👎.

Comment on lines +11 to +18

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Provision gas for the new transaction signers

In a fresh infra rollout, these entries cause infra/meta.ts to create new allower and poker KMS keys, but the infrastructure only grants IAM access and never funds their derived addresses with native gas. Both workers submit EIP-1559 transactions through exaSend, so even after the separate on-chain authorization issues are fixed, every allow transaction and account deployment/poke from these zero-balance signers fails with insufficient funds; reuse a funded signer or add an explicit funding/provisioning step before enabling the worker pools.

Useful? React with 👍 / 👎.

refund: {
secrets: ["panda-api-key", "onesignal-api-key", "postgres-url", "sardine-api-key", "segment-write-key"],
shared: ["panda-api-url", "sardine-api-url"],
Expand Down
22 changes: 12 additions & 10 deletions server/api/card.ts
Original file line number Diff line number Diff line change
Expand Up @@ -40,10 +40,10 @@ import { BASE_PRODUCT_ID, PLATINUM_PRODUCT_ID, SIGNATURE_PRODUCT_ID } from "@exa
import { Address, Base64URL, Hex } from "@exactly/common/validation";

import { cards, credentials } from "../database/schema";
import { autoCredit } from "../utils/panda";
import publicClient from "../utils/publicClient";
import ServiceError from "../utils/ServiceError";
import validatorHook from "../utils/validatorHook";
import { name as creditName } from "../workers/credit/job";

import type * as schema from "../database/schema";
import type { Auth } from "../middleware/auth";
Expand All @@ -53,6 +53,7 @@ import type createPersona from "../utils/persona";
import type createSardine from "../utils/sardine";
import type createSegment from "../utils/segment";
import type createWalletExtension from "../utils/walletExtension";
import type createCredit from "../workers/credit/queue";
import type { NodePgDatabase } from "drizzle-orm/node-postgres";

const CardResponse = object({
Expand Down Expand Up @@ -150,6 +151,7 @@ const Scopes = picklist(["provisioning", "siwe", "webauthn"]);

export default function route({
auth,
credit,
database,
panda,
pax,
Expand All @@ -159,6 +161,7 @@ export default function route({
walletExtension,
}: {
auth: Auth;
credit: ReturnType<typeof createCredit>;
database: NodePgDatabase<typeof schema>;
panda: ReturnType<typeof createPanda>;
pax: ReturnType<typeof createPax>;
Expand Down Expand Up @@ -644,15 +647,14 @@ This endpoint only accepts Wallet Extension bearer access. It does not accept \`
}
});

let mode = 0;
try {
if (await autoCredit(account)) mode = 1;
} catch (error) {
captureException(error);
}
await database
.insert(cards)
.values([{ id: card.id, credentialId, lastFour: card.last4, mode, productId }]);
await database.insert(cards).values([{ id: card.id, credentialId, lastFour: card.last4, productId }]);
await credit.enqueue(account).catch((error: unknown) =>

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve credit work when enqueueing fails

When an already-funded account creates its card while Redis or BullMQ is temporarily unavailable, this catch returns success after storing the card in debit mode but discards the only credit evaluation scheduled by card creation. The earlier deposit's activity job may have completed before the card existed, and no later event is guaranteed to enqueue another evaluation, so an otherwise eligible card can remain in debit mode indefinitely; persist or retry this work without failing card issuance.

Useful? React with 👍 / 👎.

captureException(error, {
level: "error",
tags: { queue: creditName, job: creditName },
extra: { account },
}),
);
Comment on lines +650 to +657

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | 🏗️ Heavy lift

Make credit publication durable.

If credit.enqueue rejects, this handler only captures the error and returns card creation success. The card is already persisted. A later request returns "already created" at Line 601 and does not publish another job. An eligible account can then remain in debit mode after a transient Redis outage.

Persist an outbox record in the same transaction as the card insert. Dispatch and retry that record until the credit queue accepts it. Update the queue-failure test to verify eventual dispatch.

segment.track({
event: "CardIssued",
userId: account,
Expand Down
5 changes: 4 additions & 1 deletion server/api/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -27,13 +27,15 @@ import type createManteca from "../utils/ramps/manteca";
import type createSardine from "../utils/sardine";
import type createSegment from "../utils/segment";
import type createWalletExtension from "../utils/walletExtension";
import type createCredit from "../workers/credit/queue";
import type createSubscribe from "../workers/subscribe/queue";
import type { NodePgDatabase } from "drizzle-orm/node-postgres";
import type { Redis } from "ioredis";

export default function api({
authSecret,
bridge,
credit,
database,
intercom,
manteca,
Expand All @@ -48,6 +50,7 @@ export default function api({
}: {
authSecret: string;
bridge: ReturnType<typeof createBridge>;
credit: ReturnType<typeof createCredit>;
database: NodePgDatabase<typeof schema>;
intercom: ReturnType<typeof createIntercom>;
manteca: ReturnType<typeof createManteca>;
Expand Down Expand Up @@ -77,7 +80,7 @@ export default function api({
authentication({ authSecret, createCredential: credential, database, intercom, redis, walletExtension }),
)
.route("/activity", activity({ auth, database }))
.route("/card", card({ auth, database, panda, pax, persona, sardine, segment, walletExtension }))
.route("/card", card({ auth, credit, database, panda, pax, persona, sardine, segment, walletExtension }))
.route("/kyc", kyc({ auth, database, panda, persona }))
.route("/passkey", passkey({ auth, database })) // eslint-disable-line @typescript-eslint/no-deprecated -- // TODO remove
.route("/pax", paxRoute({ auth, database, pax }))
Expand Down
187 changes: 18 additions & 169 deletions server/hooks/activity.ts
Original file line number Diff line number Diff line change
@@ -1,52 +1,34 @@
import { vValidator } from "@hono/valibot-validator";
import { SPAN_STATUS_ERROR, SPAN_STATUS_OK } from "@sentry/core";
import {
captureException,
continueTrace,
getActiveSpan,
getTraceData,
SEMANTIC_ATTRIBUTE_SENTRY_OP,
setContext,
setTag,
setUser,
startSpan,
withScope,
} from "@sentry/node";
import createDebug from "debug";
import { eq, inArray } from "drizzle-orm";
import { inArray } from "drizzle-orm";
import { Hono } from "hono";
import { validator } from "hono/validator";
import * as v from "valibot";
import { bytesToBigInt, hexToBigInt, withRetry, type LocalAccount } from "viem";
import { bytesToHex, hexToBigInt } from "viem";
import { anvil } from "viem/chains";

import exaChain, {
auditorAbi,
exaAccountFactoryAbi,
exaPluginAbi,
exaPreviewerAbi,
exaPreviewerAddress,
marketAbi,
upgradeableModularAccountAbi,
wethAddress,
} from "@exactly/common/generated/chain";
import exaChain, { exaPreviewerAbi, exaPreviewerAddress, wethAddress } from "@exactly/common/generated/chain";
import { Address, Hash, Hex } from "@exactly/common/validation";

import { cards, credentials } from "../database/schema";
import { credentials } from "../database/schema";
import t, { f } from "../i18n";
import { activityNetworks, activityUrl, NETWORKS } from "../utils/alchemy";
import decodePublicKey from "../utils/decodePublicKey";
import { autoCredit } from "../utils/panda";
import publicClient from "../utils/publicClient";
import revertFingerprint from "../utils/revertFingerprint";
import validatorHook from "../utils/validatorHook";
import verifySignature from "../utils/verifySignature";
import createWallet from "../utils/wallet";

import type * as schema from "../database/schema";
import type createAlchemy from "../utils/alchemy";
import type createOnesignal from "../utils/onesignal";
import type createSegment from "../utils/segment";
import type createPoke from "../workers/poke/queue";
import type { NodePgDatabase } from "drizzle-orm/node-postgres";
import type { Redis } from "ioredis";

Expand All @@ -59,17 +41,15 @@ Object.assign(debug, { inspectOpts: { depth: undefined } });
export default function hook({
alchemy,
database,
executor,
onesignal,
poke,
redis,
segment,
}: {
alchemy: ReturnType<typeof createAlchemy>;
database: NodePgDatabase<typeof schema>;
executor: LocalAccount;
onesignal: ReturnType<typeof createOnesignal>;
poke: ReturnType<typeof createPoke>;
redis: Redis;
segment: ReturnType<typeof createSegment>;
}) {
const networks = activityNetworks();
let entries = new Map<string, { network: string; signingKey: string }>();
Expand Down Expand Up @@ -204,150 +184,19 @@ export default function hook({
pokes.set(account, { publicKey, factory, source, assets: new Set([asset]) });
}
}
const { "sentry-trace": sentryTrace, baggage } = getTraceData();
const wallet = createWallet(executor, chain);
Promise.allSettled(
[...pokes].map(([account, { publicKey, factory, source, assets }]) =>
continueTrace({ sentryTrace, baggage }, () =>
withScope((scope) =>
startSpan(
{ name: "account activity", op: "exa.activity", attributes: { account }, forceTransaction: true },
async (span) => {
scope.setUser({ id: account });
const isDeployed = !!(await wallet.getCode({ address: account }));
scope.setTag("exa.new", !isDeployed);
if (!isDeployed) {
try {
await wallet.exaSend(
{ name: "create account", op: "exa.account", attributes: { account } },
{
address: factory,
functionName: "createAccount",
args: [0n, [decodePublicKey(publicKey, bytesToBigInt)]],
abi: exaAccountFactoryAbi,
},
chain.id === exaChain.id ? undefined : { fees: "auto" },
);
segment.track({ event: "AccountFunded", userId: account, properties: { source } });
} catch (error: unknown) {
span.setStatus({ code: SPAN_STATUS_ERROR, message: "account_failed" });
throw error;
}
}
if (chain.id !== exaChain.id) {
span.setStatus({ code: SPAN_STATUS_OK });
return;
}
if (assets.has(ETH)) assets.delete(WETH);
const results = await Promise.allSettled(
[...assets]
.filter((asset) => marketsByAsset.has(asset) || asset === ETH)
.map(async (asset) =>
withRetry(
() =>
wallet
.exaSend(
{ name: "poke account", op: "exa.poke", attributes: { account, asset } },
{
address: account,
abi: [...exaPluginAbi, ...upgradeableModularAccountAbi, ...auditorAbi, ...marketAbi],
...(asset === ETH
? { functionName: "pokeETH" }
: {
functionName: "poke",
args: [marketsByAsset.get(asset)!], // eslint-disable-line @typescript-eslint/no-non-null-assertion
}),
},
{ ignore: ["NoBalance()"] },
)
.then((receipt) => {
if (receipt) return receipt;
throw new Error("NoBalance()");
}),
{
delay: 2000,
retryCount: 5,
shouldRetry: ({ error }) => {
if (error instanceof Error && error.message === "NoBalance()") return true;
withScope((captureScope) => {
captureScope.setUser({ id: account });
captureException(error, { level: "error", fingerprint: revertFingerprint(error) });
});
return true;
},
},
),
),
);
for (const result of results) {
if (result.status === "fulfilled") continue;
if (result.reason instanceof Error && result.reason.message === "NoBalance()") {
withScope((captureScope) => {
captureScope.setUser({ id: account });
captureScope.addEventProcessor((event) => {
if (event.exception?.values?.[0]) event.exception.values[0].type = "NoBalance";
return event;
});
captureException(result.reason, {
level: "warning",
fingerprint: ["{{ default }}", "NoBalance"],
});
});
continue;
}
span.setStatus({ code: SPAN_STATUS_ERROR, message: "poke_failed" });
throw result.reason;
}
autoCredit(account)
.then(async (auto) => {
span.setAttribute("exa.autoCredit", auto);
if (!auto) return;
const credential = await database.query.credentials.findFirst({
where: eq(credentials.account, account),
columns: {},
with: {
cards: {
columns: { id: true, mode: true },
where: inArray(cards.status, ["ACTIVE", "FROZEN"]),
},
},
});
const card = credential?.cards[0];
if (!card) return;
span.setAttribute("exa.card", card.id);
if (card.mode !== 0) return;
await database.update(cards).set({ mode: 1 }).where(eq(cards.id, card.id));
span.setAttribute("exa.mode", 1);
onesignal
.sendPushNotification({
userId: account,
headings: t("Card mode changed"),
contents: t("Credit mode activated"),
})
.catch((error: unknown) => captureException(error));
})
.catch((error: unknown) => captureException(error));
span.setStatus({ code: SPAN_STATUS_OK });
},
),
),
).catch((error: unknown) => {
withScope((scope) => {
scope.setUser({ id: account });
captureException(error, { level: "error", fingerprint: revertFingerprint(error) });
});
throw error;
await Promise.all(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Defer notifications until queue publication succeeds

When poke.enqueue rejects after the notification promise above has started, this awaited publication returns a 500 even though the user may already have received the funds notification. A normal webhook redelivery then executes the notification path again, so a transient Redis failure can send duplicate push notifications; publish the durable job before sending the notification or otherwise deduplicate that side effect. .agents/rules/server.mdL75-L76

Useful? React with 👍 / 👎.

[...pokes].map(([account, { assets, factory, publicKey, source }]) =>
poke.enqueue({
account,
assets: [...assets],
chainId: chain.id,
factory,
origin: "activity",
publicKey: bytesToHex(publicKey),
source,
}),
),
)
.then((results) => {
getActiveSpan()?.setStatus(
results.every((result) => result.status === "fulfilled")
? { code: SPAN_STATUS_OK }
: { code: SPAN_STATUS_ERROR, message: "activity_failed" },
);
})
.catch((error: unknown) => captureException(error));
);
Comment on lines +187 to +199

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win

Awaiting the enqueue makes webhook retries resend notifications.

A rejected poke.enqueue now returns 500, so Alchemy retries the webhook. The poke job dedupes through its job id, but the push notification sent earlier in the handler does not. Each retry delivers another "Funds received" message for the same transfer. Pass a deterministic idempotencyKey to onesignal.sendPushNotification (for example, the transfer hash plus asset) so retries do not duplicate user notifications.

return c.json({});
},
);
Expand Down
Loading