Refactor: Simplify scraper and parser

.github/workflows/ci.yml

@@ -0,0 +1,55 @@
+name: CI
+
+on:
+  push:
+    branches: [main, master]
+  pull_request:
+    branches: [main, master]
+
+env:
+  CARGO_TERM_COLOR: always
+
+jobs:
+  test:
+    name: Test
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Install Rust toolchain
+        run: |
+          rustup update stable
+          rustup default stable
+
+      - name: Run tests
+        run: cargo test --verbose
+
+  stub-check:
+    name: Verify stub.lua
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Install Rust toolchain
+        run: |
+          rustup update stable
+          rustup default stable
+
+      - name: Generate stub.lua
+        run: cargo run --release > generated_stub.lua
+
+      - name: Compare with committed stub.lua
+        run: |
+          if diff -q stub.lua generated_stub.lua > /dev/null; then
+            echo "✓ stub.lua matches generated output"
+          else
+            echo "✗ stub.lua does not match generated output!"
+            echo ""
+            echo "Differences:"
+            diff stub.lua generated_stub.lua || true
+            echo ""
+            echo "Please regenerate stub.lua by running: cargo run > stub.lua"
+            exit 1
+          fi

Cargo.toml

@@ -6,15 +6,12 @@ version = "0.2.5"
 # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
 
 [dependencies]
-
 clap = { version = "4.5.37", features = ["derive", "string"] }
+pest = "2"
+pest_derive = "2"
 regex = "1.11.1"
 reqwest = { version = "0.12.15", features = ["blocking"] }
 scraper = "0.23.1"
 serde = { version = "1.0.219", features = ["derive"] }
-
-# preserve_order uses "IndexMap" instead of "BTreeMap" preserving order after parsing.
-pest = "2.7.9"
-pest_derive = "2.7.9"
 textwrap = "0.16.2"
 toml = { version = "0.8.21", features = ["preserve_order"] }

DESIGN.md

@@ -5,23 +5,25 @@
 tldr: This is an overly complicated tool with the following parts:
 
 1. DSL (domain specific language) for Lua typing information (LUARS)
-2. PEG Grammar for LUARS using the [pest.rs](https://pest.rs) crate (see: [luars.pest](src/luars.pest))
-3. Manually documented Types for the PlaydateSDK expressed in LUARS (see: [playdate.luars](playdate.luars))
-4. A web scraper which scrapes the PlaydateSDK documentation using the [scraper](https://crates.io/crates/scraper) crate.
-5. A rust which combines the scraped documentation and types and generates LuaLS compatible Type Annotations with documentation and types for the entire SDK.
+2. Manually documented Types for the PlaydateSDK expressed in LUARS (see: [playdate.luars](playdate.luars))
+3. Grammar/Parser for LUARS using [pest.rs](https://pest.rs) crate (see: [luars.rs](./src/luars.rs))
+4. A web scraper which scrapes the PlaydateSDK HTML documentation using the [scraper](https://crates.io/crates/scraper) crate (see: [scraper.rs](./src/scraper.rs))
+5. Rust which combines the scraped documentation and types and generates LuaLS compatible Type Annotations with documentation and types for the entire SDK.
 
 ## No seriously, what the actual fuck?
 
 1. I've been coding Lua for Playdate.
-2. The PlaydateSDK has no types and leverages Lua dynamic nature for variadic params and returns.
-3. There's a pretty good Language server for Lua which created a standard for Lua typing via comments (LuaCATS)
-4. I've been learning rust.
-6. I wrote a scraper for the PlaydateSDK docs (in rust)
-7. For each function I generated a TOML skeleton and manually determined parameter and return types.
-8. TOML turned out to overly verbose, fragile and unsearchable for ~1000 functions and ~3500 type definitions (~15K lines).
-9. So I came up with a format for function signatures which only requires one line per function.
-10. I learned PEG and wrote a parser for function signature format I came up with.
-11. Iterate. Iterate. Iterate.
+1. The PlaydateSDK has no types and leverages Lua dynamic nature for variadic params and returns.
+1. There's a pretty good Language server for Lua (lua-language-server) with a standard for Lua typing via comments (LuaCATS)
+1. I've been learning rust.
+1. I wrote a scraper for the PlaydateSDK docs (in rust)
+1. For each function I generated a TOML skeleton and manually determined parameter and return types.
+1. TOML turned out to overly verbose, fragile and unsearchable for ~1000 functions and ~3500 type definitions (~15K lines).
+1. So I came up with a format for function signatures which only requires one line per function.
+1. I learned PEG and wrote a parser for function signature format I came up with using [pest.rs](https://pest.rs).
+1. Iterate. Iterate. Iterate.
+1. Years pass
+1. I had Claude Code rewrite my LUARS parser and simplify my scraper.
 
 Now we can do static code analysis, type checking and autocomplete in VSCode and other IDEs
 that support the LuaLS Language Server's LUACATS style type annotation comments.
@@ -37,4 +39,4 @@ Local is for table types used by the SDK.
 
 Fun is functions (and parameters, parameter types, returns, return types and optionality of each).
 
-See the [LUARS pest.rs PEG Grammar](src/luars.pest) and [Playdate.luars](playdate.luars) for more.
+See [Playdate.luars](playdate.luars) for examples.

README.md

@@ -12,7 +12,7 @@ This repository contains the Playdate Lua API specification
 and tools for generating Lua Comment And Type System (LUACats)
 compliant Lua comments for use with the Lua Language Server
 ([sumneko.lua](https://marketplace.visualstudio.com/items?itemName=sumneko.lua))
-in VSCode, NeoVIM, etc.
+in VSCode, Zed, NeoVIM, etc.
 
 These tools can optionally integrate the offical
 Playdate Docs into the generated output by scraping the API

src/args.rs

@@ -68,13 +68,14 @@ fn fetch_url(url: String) -> String {
 }
 
 /// Retrieves the contents of the docs (from file or url)
-pub fn fetch_docs(args: Args) -> String {
-    match args.url {
-        Some(url) => fetch_url(url),
-        _ => fetch_file(&args.path.clone().unwrap()),
+pub fn fetch_docs(args: &Args) -> String {
+    match &args.url {
+        Some(url) => fetch_url(url.clone()),
+        None => fetch_file(args.path.as_ref().unwrap()),
     }
 }
 
-pub fn setup() -> Args {
+/// Parse command line arguments
+pub fn parse() -> Args {
     Args::parse()
 }