Skip to content

feat(cpex-scraper): add file logger for better debugging and tracing #4376

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
faizkhairi wants to merge 3 commits into
base: master
Choose a base branch
from
Open

feat(cpex-scraper): add file logger for better debugging and tracing #4376

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
8964243
feat(cpex-scraper): add file logger for better debugging and tracing
faizkhairi Mar 24, 2026
b1610aa
fix: correct log path and await stream flush
faizkhairi Mar 24, 2026
2ab9d26
Merge branch 'master' into feat/cpex-scraper-file-logger
jloh02 May 18, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 10 additions & 2 deletions scrapers/cpex-scraper/src/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
import fs from 'node:fs';
import path from 'node:path';

import { createFileLogger } from './logger';
import { scrapeCPEx, type ScraperEnv } from './scraper';

const ACADEMIC_YEAR = '2026/27';
Expand All @@ -12,10 +13,17 @@ const threshold = 1500;
const envPath = path.join(__dirname, '../../env.json');
const env = JSON.parse(fs.readFileSync(envPath, 'utf8')) as ScraperEnv;

const logger = createFileLogger();

scrapeCPEx({
academicYear: ACADEMIC_YEAR,
env,
logger,
threshold,
}).catch((error) => {
console.error(`Failed to scrape: ${error}`);
}).then(async () => {
await logger.close();
}).catch(async (error) => {
logger.log(`Failed to scrape: ${error}`);
await logger.close();
process.exitCode = 1;
});
Comment on lines +23 to 29
Copy link
Copy Markdown
Contributor

@greptile-apps greptile-apps Bot May 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Double-close on stream failure, wrong exit code

If logger.close() in the .then() block rejects (e.g. the write stream was already destroyed due to a disk error), the .catch() handler fires with that stream error as error. This causes three problems at once: logger.log(...) writes to the already-errored stream (silently dropped), logger.close() is called a second time on a destroyed stream (which can reject again with no further handler → unhandled promise rejection), and process.exitCode = 1 is set even though the scrape itself succeeded.

The standard fix is to use .finally() so close() runs exactly once regardless of outcome.

Prompt To Fix With AI 
This is a comment left during a code review.
Path: scrapers/cpex-scraper/src/index.ts
Line: 23-29

Comment:
**Double-close on stream failure, wrong exit code**

If `logger.close()` in the `.then()` block rejects (e.g. the write stream was already destroyed due to a disk error), the `.catch()` handler fires with that stream error as `error`. This causes three problems at once: `logger.log(...)` writes to the already-errored stream (silently dropped), `logger.close()` is called a second time on a destroyed stream (which can reject again with no further handler → unhandled promise rejection), and `process.exitCode = 1` is set even though the scrape itself succeeded.

The standard fix is to use `.finally()` so `close()` runs exactly once regardless of outcome.

How can I resolve this? If you propose a fix, please make it concise.

67 changes: 67 additions & 0 deletions scrapers/cpex-scraper/src/logger.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
import fs from 'node:fs';
import path from 'node:path';

// At runtime, __dirname is build/src/ (compiled from src/).
// Two levels up reaches the scraper root: build/src/ -> build/ -> scrapers/cpex-scraper/
const LOG_DIR = path.join(__dirname, '../../logs');

function pad2(n: number): string {
return n < 10 ? `0${n}` : String(n);
}

function formatTimestamp(date: Date): string {
return (
date.getFullYear().toString() +
'-' +
pad2(date.getMonth() + 1) +
'-' +
pad2(date.getDate()) +
'.' +
pad2(date.getHours()) +
'-' +
pad2(date.getMinutes()) +
'-' +
pad2(date.getSeconds())
);
}

export type FileLogger = Pick<Console, 'log'> & {
close: () => Promise<void>;
};

/**
* Creates a logger that writes timestamped lines to both stdout and a log file.
* The log file is written to `scrapers/cpex-scraper/logs/cpex-YYYY-MM-DD.HH-mm-ss.log`.
*
* Satisfies the `Pick<Console, 'log'>` interface used by the CPEx scraper.
*/
export function createFileLogger(now: Date = new Date()): FileLogger {
if (!fs.existsSync(LOG_DIR)) {
fs.mkdirSync(LOG_DIR, { recursive: true });
}
Comment on lines +39 to +41
Copy link
Copy Markdown
Contributor

@greptile-apps greptile-apps Bot May 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 The existsSync guard is redundant here because mkdirSync with { recursive: true } is a no-op when the directory already exists — it never throws in that case. The extra check adds a TOCTOU window (another process could remove the directory between the check and the mkdir) without buying anything.

Suggested change
if (!fs.existsSync(LOG_DIR)) {
fs.mkdirSync(LOG_DIR, { recursive: true });
}
fs.mkdirSync(LOG_DIR, { recursive: true });
Prompt To Fix With AI 
This is a comment left during a code review.
Path: scrapers/cpex-scraper/src/logger.ts
Line: 39-41

Comment:
The `existsSync` guard is redundant here because `mkdirSync` with `{ recursive: true }` is a no-op when the directory already exists — it never throws in that case. The extra check adds a TOCTOU window (another process could remove the directory between the check and the mkdir) without buying anything.

```suggestion
  fs.mkdirSync(LOG_DIR, { recursive: true });
```

How can I resolve this? If you propose a fix, please make it concise.


const filename = `cpex-${formatTimestamp(now)}.log`;
const logPath = path.join(LOG_DIR, filename);
const stream = fs.createWriteStream(logPath, { flags: 'a' });

stream.on('error', (err) => {
console.error(`[FileLogger] write stream error: ${err.message}`);
});

function log(...args: Array<unknown>): void {
const timestamp = new Date().toISOString();
const message = args.map((a) => (typeof a === 'string' ? a : String(a))).join(' ');
const line = `[${timestamp}] ${message}`;

console.log(line);
stream.write(`${line}\n`);
}

function close(): Promise<void> {
return new Promise((resolve, reject) => {
stream.end((err?: Error | null) => (err ? reject(err) : resolve()));
});
}

return { close, log };
}