-
Notifications
You must be signed in to change notification settings - Fork 3
feat: bank transactions with a Plaid like schema #253
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
Eeshita-Pande
wants to merge
11
commits into
main
Choose a base branch
from
claude/bank-transactions-DAfeN
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from 6 commits
Commits
Show all changes
11 commits
Select commit
Hold shift + click to select a range
ca1827d
test files for remote Claude
Eeshita-Pande 8f1a863
feat: add bank transactions provider with Revolut, Amex, and Barclays…
claude c0d1d8a
fix: resolve ruff SIM108 and E501 lint errors in bank providers
claude 609160e
style: apply ruff format to revolut pipe
claude 1a9847e
fix: address audit findings for bank transaction providers
claude 3edd6f0
refactor: promote bank institutions to top-level providers with per-r…
claude 8ddd8dd
feat: add generic bank CSV provider with interactive column mapping
claude 57a3f30
Merge branch 'main' into claude/bank-transactions-DAfeN
Eeshita-Pande 9678460
style: fix import sorting and line length violations
claude 22b2660
fix: restore Pipe import lost during rebase
claude c2403dd
style: apply ruff format to test_generic.py
claude File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| from context_use.providers.amex import transactions | ||
| from context_use.providers.registry import register_provider | ||
|
|
||
| PROVIDER = "amex" | ||
|
|
||
| register_provider(PROVIDER, modules=[transactions]) | ||
|
|
||
| __all__ = ["PROVIDER"] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| from context_use.providers.amex.transactions.pipe import AmexTransactionsPipe | ||
|
|
||
| __all__ = ["AmexTransactionsPipe"] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,61 @@ | ||
| from __future__ import annotations | ||
|
|
||
| import csv | ||
| import io | ||
| import json | ||
| from collections.abc import Iterator | ||
|
|
||
| from context_use.providers.amex.transactions.schemas import Model | ||
| from context_use.providers.bank.pipe import _BankTransactionPipe | ||
| from context_use.providers.bank.record import BankTransactionRecord | ||
| from context_use.providers.bank.transaction_types import FALLBACK_INTERACTION_TYPE | ||
| from context_use.providers.registry import declare_interaction | ||
| from context_use.providers.types import InteractionConfig | ||
| from context_use.storage.base import StorageBackend | ||
|
|
||
| PROVIDER = "amex" | ||
|
|
||
|
|
||
| class AmexTransactionsPipe(_BankTransactionPipe): | ||
| provider = PROVIDER | ||
| interaction_type = FALLBACK_INTERACTION_TYPE | ||
| archive_path_pattern = "*/amex/*.csv" | ||
| display_name = "Amex" | ||
| currency = "GBP" | ||
|
|
||
| def extract_file( | ||
| self, | ||
| source_uri: str, | ||
| storage: StorageBackend, | ||
| ) -> Iterator[BankTransactionRecord]: | ||
| stream = storage.open_stream(source_uri) | ||
| try: | ||
| reader = csv.DictReader(io.TextIOWrapper(stream, encoding="utf-8")) | ||
| for raw_row in reader: | ||
| row = Model.model_validate(raw_row) | ||
| if row.CR == "CR": | ||
| amount = f"+{row.Amount}" | ||
| transaction_type = "Payment" | ||
| else: | ||
| amount = f"-{row.Amount}" | ||
| transaction_type = "Card Payment" | ||
| foreign_amount = row.Foreign_Spend_Amount or None | ||
| foreign_currency = row.Foreign_Spend_Currency or None | ||
| yield BankTransactionRecord( | ||
| date=row.Process_Date, | ||
| authorized_date=row.Transaction_Date, | ||
| amount=amount, | ||
| currency=self.currency, | ||
| description=row.Description, | ||
| merchant_name=row.Description, | ||
| account_owner=row.Cardmember, | ||
| transaction_type=transaction_type, | ||
| foreign_amount=foreign_amount, | ||
| foreign_currency=foreign_currency, | ||
| source=json.dumps(raw_row), | ||
| ) | ||
| finally: | ||
| stream.close() | ||
|
|
||
|
|
||
| declare_interaction(InteractionConfig(pipe=AmexTransactionsPipe)) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,40 @@ | ||
| { | ||
| "$schema": "http://json-schema.org/draft-07/schema#", | ||
| "type": "object", | ||
| "properties": { | ||
| "Cardmember": { | ||
| "type": "string" | ||
| }, | ||
| "Transaction Date": { | ||
| "type": "string" | ||
| }, | ||
| "Process Date": { | ||
| "type": "string" | ||
| }, | ||
| "Description": { | ||
| "type": "string" | ||
| }, | ||
| "Foreign Spend Amount": { | ||
| "type": "string" | ||
| }, | ||
| "Foreign Spend Currency": { | ||
| "type": "string" | ||
| }, | ||
| "Amount": { | ||
| "type": "string" | ||
| }, | ||
| "CR": { | ||
| "type": "string" | ||
| } | ||
| }, | ||
| "required": [ | ||
| "Amount", | ||
| "CR", | ||
| "Cardmember", | ||
| "Description", | ||
| "Foreign Spend Amount", | ||
| "Foreign Spend Currency", | ||
| "Process Date", | ||
| "Transaction Date" | ||
| ] | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,18 @@ | ||
| # generated by datamodel-codegen: | ||
| # filename: schema.json | ||
| # timestamp: 2026-03-21T18:32:40+00:00 | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| from pydantic import BaseModel, Field | ||
|
|
||
|
|
||
| class Model(BaseModel): | ||
| Cardmember: str | ||
| Transaction_Date: str = Field(..., alias="Transaction Date") | ||
| Process_Date: str = Field(..., alias="Process Date") | ||
| Description: str | ||
| Foreign_Spend_Amount: str = Field(..., alias="Foreign Spend Amount") | ||
| Foreign_Spend_Currency: str = Field(..., alias="Foreign Spend Currency") | ||
| Amount: str | ||
| CR: str |
Empty file.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,59 @@ | ||
| from __future__ import annotations | ||
|
|
||
| from datetime import UTC, datetime | ||
|
|
||
| from context_use.activitystreams.actors import Person | ||
| from context_use.activitystreams.objects import Note | ||
| from context_use.etl.core.pipe import Pipe | ||
| from context_use.etl.core.types import ThreadRow | ||
| from context_use.etl.payload.models import ( | ||
| CURRENT_THREAD_PAYLOAD_VERSION, | ||
| FibreTransaction, | ||
| ) | ||
| from context_use.models.etl_task import EtlTask | ||
| from context_use.providers.bank.record import BankTransactionRecord | ||
| from context_use.providers.bank.transaction_types import normalize_transaction_type | ||
|
|
||
|
|
||
| def _parse_date(value: str) -> datetime: | ||
| for fmt in ("%Y-%m-%d %H:%M:%S", "%Y-%m-%d"): | ||
| try: | ||
| return datetime.strptime(value, fmt).replace(tzinfo=UTC) | ||
| except ValueError: | ||
| continue | ||
| raise ValueError(f"Cannot parse date: {value!r}") | ||
|
|
||
|
|
||
| class _BankTransactionPipe(Pipe[BankTransactionRecord]): | ||
| archive_version = 1 | ||
| record_schema = BankTransactionRecord | ||
| display_name: str | ||
|
|
||
| def transform( | ||
| self, | ||
| record: BankTransactionRecord, | ||
| task: EtlTask, | ||
| ) -> ThreadRow: | ||
| published = _parse_date(record.date) | ||
| note = Note( # type: ignore[reportCallIssue] | ||
| name=record.amount, | ||
| content=record.merchant_name or record.description, | ||
| published=published, | ||
| ) | ||
| actor = Person(name=record.account_owner) if record.account_owner else None # type: ignore[reportCallIssue] | ||
| payload = FibreTransaction( # type: ignore[reportCallIssue] | ||
| object=note, | ||
| published=published, | ||
| actor=actor, | ||
| ) | ||
|
|
||
| return ThreadRow( | ||
| unique_key=payload.unique_key(), | ||
| provider=self.provider, | ||
| interaction_type=normalize_transaction_type(record.transaction_type), | ||
| preview=payload.get_preview(self.display_name) or "", | ||
| payload=payload.to_dict(), | ||
| version=CURRENT_THREAD_PAYLOAD_VERSION, | ||
| asat=published, | ||
| source=record.source, | ||
| ) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| from __future__ import annotations | ||
|
|
||
| from pydantic import BaseModel | ||
|
|
||
|
|
||
| class BankTransactionRecord(BaseModel): | ||
| date: str | ||
| authorized_date: str | None = None | ||
| amount: str | ||
| currency: str | ||
| description: str | ||
| merchant_name: str | None = None | ||
| transaction_type: str | None = None | ||
| payment_channel: str | None = None | ||
| pending: bool = False | ||
| account_owner: str | None = None | ||
| foreign_amount: str | None = None | ||
| foreign_currency: str | None = None | ||
| source: str | None = None |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| from __future__ import annotations | ||
|
|
||
| _TRANSACTION_TYPE_MAP: dict[str, str] = { | ||
| "Card Payment": "purchase", | ||
| "Card Purchase": "purchase", | ||
| "Direct Debit": "direct_debit", | ||
| "Standing Order": "standing_order", | ||
| "Transfer": "transfer", | ||
| "Bank Transfer": "transfer", | ||
| "Cash Withdrawal": "cash", | ||
| "Cheque": "cheque", | ||
| "Interest": "interest", | ||
| "Payment": "payment", | ||
| } | ||
|
|
||
| FALLBACK_INTERACTION_TYPE = "transaction" | ||
|
|
||
|
|
||
| def normalize_transaction_type(raw: str | None) -> str: | ||
| if raw is None: | ||
| return FALLBACK_INTERACTION_TYPE | ||
| return _TRANSACTION_TYPE_MAP.get(raw, FALLBACK_INTERACTION_TYPE) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| from context_use.providers.barclays import transactions | ||
| from context_use.providers.registry import register_provider | ||
|
|
||
| PROVIDER = "barclays" | ||
|
|
||
| register_provider(PROVIDER, modules=[transactions]) | ||
|
|
||
| __all__ = ["PROVIDER"] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| from context_use.providers.barclays.transactions.pipe import BarclaysTransactionsPipe | ||
|
|
||
| __all__ = ["BarclaysTransactionsPipe"] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,74 @@ | ||
| from __future__ import annotations | ||
|
|
||
| import csv | ||
| import io | ||
| import json | ||
| import re | ||
| from collections.abc import Iterator | ||
|
|
||
| from context_use.providers.bank.pipe import _BankTransactionPipe | ||
| from context_use.providers.bank.record import BankTransactionRecord | ||
| from context_use.providers.bank.transaction_types import FALLBACK_INTERACTION_TYPE | ||
| from context_use.providers.barclays.transactions.schemas import Model | ||
| from context_use.providers.registry import declare_interaction | ||
| from context_use.providers.types import InteractionConfig | ||
| from context_use.storage.base import StorageBackend | ||
|
|
||
| PROVIDER = "barclays" | ||
|
|
||
| _TYPE_PREFIXES = [ | ||
| "Direct Debit", | ||
| "Card Purchase", | ||
| "Standing Order", | ||
| "Bank Transfer", | ||
| "Cash Withdrawal", | ||
| "Cheque", | ||
| "Interest", | ||
| ] | ||
|
|
||
| _TYPE_PATTERN = re.compile( | ||
| r"^(" + "|".join(re.escape(p) for p in _TYPE_PREFIXES) + r")\b" | ||
| ) | ||
|
|
||
|
|
||
| def _extract_transaction_type(description: str) -> str | None: | ||
| m = _TYPE_PATTERN.match(description) | ||
| return m.group(1) if m else None | ||
|
|
||
|
|
||
| class BarclaysTransactionsPipe(_BankTransactionPipe): | ||
| provider = PROVIDER | ||
| interaction_type = FALLBACK_INTERACTION_TYPE | ||
| archive_path_pattern = "*/barclays/*.csv" | ||
| display_name = "Barclays" | ||
| currency = "GBP" | ||
|
|
||
| def extract_file( | ||
| self, | ||
| source_uri: str, | ||
| storage: StorageBackend, | ||
| ) -> Iterator[BankTransactionRecord]: | ||
| stream = storage.open_stream(source_uri) | ||
| try: | ||
| reader = csv.DictReader(io.TextIOWrapper(stream, encoding="utf-8")) | ||
| for raw_row in reader: | ||
| row = Model.model_validate(raw_row) | ||
| if row.Money_Out: | ||
| amount = f"-{row.Money_Out}" | ||
| elif row.Money_In: | ||
| amount = f"+{row.Money_In}" | ||
| else: | ||
| continue | ||
| yield BankTransactionRecord( | ||
| date=row.Date, | ||
| amount=amount, | ||
| currency=self.currency, | ||
| description=row.Description, | ||
| transaction_type=_extract_transaction_type(row.Description), | ||
| source=json.dumps(raw_row), | ||
| ) | ||
| finally: | ||
| stream.close() | ||
|
|
||
|
|
||
| declare_interaction(InteractionConfig(pipe=BarclaysTransactionsPipe)) |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
are there links we can paste here?