Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 26 additions & 1 deletion src/app/functions/[slug]/page.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -118,6 +118,29 @@ function renderRemarks(remarks: string) {
while (i < lines.length) {
const line = lines[i];

if (line.startsWith("### ")) {
elements.push(<h3 key={key++} dangerouslySetInnerHTML={{ __html: parseInlineMarkdown(line.slice(4)) }} />);
i++;
continue;
}

if (line.startsWith("```")) {
const lang = line.slice(3).trim();
const codeLines: string[] = [];
i++;
while (i < lines.length && !lines[i].startsWith("```")) {
codeLines.push(lines[i]);
i++;
}
i++; // skip closing ```
elements.push(
<pre key={key++} className={lang ? `language-${lang}` : undefined}>
<code>{codeLines.join("\n")}</code>
</pre>
);
continue;
}

if (line.startsWith("|")) {
const tableLines: string[] = [];
while (i < lines.length && lines[i].startsWith("|")) {
Expand Down Expand Up @@ -169,7 +192,9 @@ function renderRemarks(remarks: string) {
i < lines.length &&
lines[i].trim() !== "" &&
!lines[i].startsWith("|") &&
!lines[i].startsWith("- ")
!lines[i].startsWith("- ") &&
!lines[i].startsWith("### ") &&
!lines[i].startsWith("```")
) {
paraLines.push(lines[i]);
i++;
Expand Down
109 changes: 107 additions & 2 deletions src/content/functions/web-contents.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -45,11 +45,16 @@ The optional `options` record supports the following fields:
| `Timeout` | duration | Override the default request timeout. e.g., `#duration(0, 0, 30, 0)` for 30 seconds. |
| `ManualStatusHandling` | list | A list of HTTP status codes (e.g., `{404, 500}`) that should not raise an error, allowing you to handle them in M code. |
| `IsRetry` | logical | Set to `true` to ignore any cached response and re-fetch from the server. |
| `ExcludedFromCacheKey` | list | Header names that should not be included in the cache key (useful for volatile auth headers). |
| `ManualCredentials` | logical | Set to `true` to bypass Power Query's credential prompt when you handle authentication yourself via `Headers`. |
| `ApiKeyName` | text | Name of the API key parameter when using the Web API credential type. |

### Important notes

- `Web.Contents` **breaks query folding** — any transformations applied after it run entirely in the M engine.
- In the Power BI Service, the base URL must be a **static string literal** for data source credentials to bind correctly. Use `RelativePath` and `Query` for dynamic parts.
- In the Power BI Service, the base URL must be a **static string literal** for data source credentials to bind correctly. Use `RelativePath` and `Query` for dynamic parts. Power Query automatically inserts a `/` between the base URL and `RelativePath` — don't include a leading slash yourself.
- Adding any `Content` field — even an empty binary — converts the request from GET to POST. Use `Uri.BuildQueryString` to build form-encoded POST bodies from a record instead of manually concatenating strings.
- Queries that combine a POST request (e.g. fetching an OAuth token) and a GET request in the same query may fail to refresh in the Power BI Service due to dynamic data source detection. Consider using Dataflows, a Data Gateway with a custom connector, or splitting the token fetch into a separate query.
- For paginated APIs, combine `Web.Contents` with `List.Generate` or a recursive function to fetch multiple pages.

### Typical usage patterns
Expand Down Expand Up @@ -96,4 +101,104 @@ in

## Examples

Since `Web.Contents` requires network access, live output examples are not shown. See the Remarks section above for typical usage patterns.
### Example 1: Fetch JSON from a REST API

Download the Products sample table from the pqm.guide API and parse it into a table.

```powerquery
let
Response = Web.Contents("https://pqm.guide/api/tables/Products"),
Parsed = Json.Document(Response),
AsTable = Table.FromRecords(Parsed[rows])
in
AsTable
```

<!--output
{"columns":[{"name":"ProductID","type":"number"},{"name":"ProductName","type":"text"},{"name":"Category","type":"text"},{"name":"Price","type":"number"},{"name":"InStock","type":"logical"}],"rows":[{"ProductID":1,"ProductName":"Widget A","Category":"Widgets","Price":25.00,"InStock":true},{"ProductID":2,"ProductName":"Gadget B","Category":"Gadgets","Price":50.00,"InStock":true},{"ProductID":3,"ProductName":"Widget C","Category":"Widgets","Price":15.00,"InStock":false},{"ProductID":4,"ProductName":"Gadget D","Category":"Gadgets","Price":75.00,"InStock":true},{"ProductID":5,"ProductName":"Thingamajig E","Category":"Misc","Price":120.00,"InStock":false}]}
-->

<!--steps
{
"steps": [
{
"name": "Response",
"description": "Calls the pqm.guide REST API and returns the raw HTTP response as a binary value.",
"output": {"columns":[{"name":"Value","type":"text"}],"rows":[{"Value":"Binary.FromText(...)"}]}
},
{
"name": "Parsed",
"description": "Parses the binary response as JSON, producing an M record with columns and rows fields.",
"output": {"columns":[{"name":"Field","type":"text"},{"name":"Value","type":"text"}],"rows":[{"Field":"columns","Value":"[List]"},{"Field":"rows","Value":"[List]"}]}
},
{
"name": "Result",
"description": "The final output — a five-row table built from the rows field of the parsed JSON using Table.FromRecords.",
"output": {"columns":[{"name":"ProductID","type":"number"},{"name":"ProductName","type":"text"},{"name":"Category","type":"text"},{"name":"Price","type":"number"},{"name":"InStock","type":"logical"}],"rows":[{"ProductID":1,"ProductName":"Widget A","Category":"Widgets","Price":25.00,"InStock":true},{"ProductID":2,"ProductName":"Gadget B","Category":"Gadgets","Price":50.00,"InStock":true},{"ProductID":3,"ProductName":"Widget C","Category":"Widgets","Price":15.00,"InStock":false},{"ProductID":4,"ProductName":"Gadget D","Category":"Gadgets","Price":75.00,"InStock":true},{"ProductID":5,"ProductName":"Thingamajig E","Category":"Misc","Price":120.00,"InStock":false}]}
}
]
}
-->

### Example 2: Use RelativePath for Power BI Service compatibility

Split the URL into a static base and a dynamic RelativePath so the query can refresh in the Power BI Service.

```powerquery
let
TableName = "Sales",
Response = Web.Contents("https://pqm.guide", [
RelativePath = "api/tables/" & TableName
]),
Parsed = Json.Document(Response),
AsTable = Table.FromRecords(Parsed[rows])
in
AsTable
```

<!--output
{"columns":[{"name":"OrderID","type":"number"},{"name":"CustomerName","type":"text"},{"name":"Product","type":"text"},{"name":"Category","type":"text"},{"name":"UnitPrice","type":"number"},{"name":"Quantity","type":"number"},{"name":"OrderDate","type":"date"},{"name":"Region","type":"text"}],"rows":[{"OrderID":1,"CustomerName":"Alice","Product":"Widget A","Category":"Widgets","UnitPrice":25.00,"Quantity":4,"OrderDate":"2024-01-15","Region":"East"},{"OrderID":2,"CustomerName":"Bob","Product":"Gadget B","Category":"Gadgets","UnitPrice":50.00,"Quantity":2,"OrderDate":"2024-01-18","Region":"West"},{"OrderID":3,"CustomerName":"Charlie","Product":"Widget C","Category":"Widgets","UnitPrice":15.00,"Quantity":10,"OrderDate":"2024-02-01","Region":"East"},{"OrderID":4,"CustomerName":"Alice","Product":"Gadget D","Category":"Gadgets","UnitPrice":75.00,"Quantity":1,"OrderDate":"2024-02-10","Region":"North"},{"OrderID":5,"CustomerName":"Diana","Product":"Widget A","Category":"Widgets","UnitPrice":25.00,"Quantity":6,"OrderDate":"2024-03-05","Region":"West"},{"OrderID":6,"CustomerName":"Bob","Product":"Thingamajig E","Category":"Misc","UnitPrice":120.00,"Quantity":1,"OrderDate":"2024-03-12","Region":"East"},{"OrderID":7,"CustomerName":"Charlie","Product":"Gadget B","Category":"Gadgets","UnitPrice":50.00,"Quantity":3,"OrderDate":"2024-04-01","Region":"West"},{"OrderID":8,"CustomerName":"Diana","Product":"Widget C","Category":"Widgets","UnitPrice":15.00,"Quantity":8,"OrderDate":"2024-04-15","Region":"North"}]}
-->

### Example 3: Handle a 404 with ManualStatusHandling

Attempt to fetch a table that doesn't exist and handle the 404 gracefully instead of raising an error.

```powerquery
let
Response = Web.Contents("https://pqm.guide/api/tables/NonExistent", [
ManualStatusHandling = {404}
]),
Status = Value.Metadata(Response)[Response.Status],
Result =
if Status = 404 then #table({"Status", "Message"}, {{404, "Table not found"}})
else Table.FromRecords(Json.Document(Response)[rows])
in
Result
```

<!--output
{"columns":[{"name":"Status","type":"number"},{"name":"Message","type":"text"}],"rows":[{"Status":404,"Message":"Table not found"}]}
-->

<!--steps
{
"steps": [
{
"name": "Response",
"description": "Calls the API for a non-existent table. Because 404 is listed in ManualStatusHandling, Power Query returns the response instead of raising an error.",
"output": {"columns":[{"name":"Value","type":"text"}],"rows":[{"Value":"Binary.FromText(...)"}]}
},
{
"name": "Status",
"description": "Reads the HTTP status code from the response metadata. Returns 404 because the table does not exist.",
"output": {"columns":[{"name":"Value","type":"number"}],"rows":[{"Value":404}]}
},
{
"name": "Result",
"description": "The final output — since the status is 404, returns a one-row table with the status code and a user-friendly message instead of failing.",
"output": {"columns":[{"name":"Status","type":"number"},{"name":"Message","type":"text"}],"rows":[{"Status":404,"Message":"Table not found"}]}
}
]
}
-->
83 changes: 78 additions & 5 deletions src/content/patterns/api-authentication.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,13 @@ All authentication in `Web.Contents` flows through its second argument — an op
| `Headers` | record | HTTP request headers (API keys, Bearer tokens, Content-Type) |
| `Query` | record | Query string parameters (appended to the URL) |
| `Content` | binary | POST body |
| `RelativePath` | text | Path appended to the base URL — required for dynamic endpoints that refresh in the Power BI Service |
| `ManualStatusHandling` | list | HTTP status codes Power Query should return instead of raising an error |
| `ManualCredentials` | logical | Set `true` to bypass Power Query's credential prompt |
| `Timeout` | duration | Maximum time to wait for a response (e.g. `#duration(0, 0, 30, 0)` for 30 seconds) |
| `IsRetry` | logical | Set `true` to ignore any cached response and re-fetch from the server |
| `ExcludedFromCacheKey` | list | Header names that should not be included in the cache key (useful for volatile auth headers) |
| `ApiKeyName` | text | Name of the API key parameter when using the Web API credential type |

### API Key in a Header

Expand Down Expand Up @@ -52,12 +58,15 @@ APIs using OAuth return a short-lived access token you include as a Bearer heade
let
// Step 1: Request a token
TokenResponse = Json.Document(
Web.Contents("https://auth.example.com/oauth/token", [
Web.Contents("https://auth.example.com", [
RelativePath = "oauth/token",
Headers = [#"Content-Type" = "application/x-www-form-urlencoded"],
Content = Text.ToBinary(
"grant_type=client_credentials"
& "&client_id=YOUR_CLIENT_ID"
& "&client_secret=YOUR_CLIENT_SECRET"
Uri.BuildQueryString([
grant_type = "client_credentials",
client_id = "YOUR_CLIENT_ID",
client_secret = "YOUR_CLIENT_SECRET"
])
),
ManualCredentials = true
])
Expand All @@ -66,7 +75,8 @@ let

// Step 2: Use the token in the data request
Data = Json.Document(
Web.Contents("https://api.example.com/v1/records", [
Web.Contents("https://api.example.com", [
RelativePath = "v1/records",
Headers = [
Authorization = "Bearer " & AccessToken,
#"Content-Type" = "application/json"
Expand All @@ -78,6 +88,8 @@ in
Data
```

Note: adding any `Content` field — even an empty binary — converts the request from GET to POST. Use `Uri.BuildQueryString` to build form-encoded POST bodies from a record instead of manually concatenating `&`-separated strings.

### Query String Parameters

Pass parameters in the `Query` field rather than concatenating them into the URL string — this handles URL encoding automatically:
Expand All @@ -100,6 +112,62 @@ Web.Contents("https://api.example.com/search?q=power+query&limit=50")

The `Query` record values must be text — convert numbers with `Text.From(...)`.

### RelativePath — Dynamic Endpoints That Refresh

When a query refreshes in the Power BI Service, the engine evaluates the base URL at design time to determine which data-source credentials to apply. If you build the full URL dynamically (e.g. by concatenating a page number or resource ID), the service can't match it to a known data source and the refresh fails.

`RelativePath` solves this — the base URL stays static, and the dynamic portion goes into `RelativePath`:

```powerquery
// Good — base URL is static, dynamic path handled by RelativePath
Web.Contents("https://api.example.com", [
RelativePath = "v2/orders/" & Text.From(orderId),
Headers = [#"X-API-Key" = ApiKey],
ManualCredentials = true
])

// Avoid — dynamic URL breaks scheduled refresh in the Power BI Service
Web.Contents("https://api.example.com/v2/orders/" & Text.From(orderId), [
Headers = [#"X-API-Key" = ApiKey],
ManualCredentials = true
])
```

Power Query automatically inserts a `/` between the base URL and `RelativePath` — don't include a leading slash yourself, or you'll end up with a double `//` in the URL.

`RelativePath` is especially important in pagination loops where the path changes on every iteration.

### ManualStatusHandling — Graceful Error Responses

By default, Power Query raises an error for any non-2xx HTTP response. `ManualStatusHandling` accepts a list of status codes that should be returned as normal responses instead:

```powerquery
let
Response = Web.Contents("https://api.example.com/users/42", [
Headers = [#"X-API-Key" = ApiKey],
ManualStatusHandling = {400, 404, 429, 500},
ManualCredentials = true
]),

Status = Value.Metadata(Response)[Response.Status],

Result =
if Status = 200 then Json.Document(Response)
else if Status = 429 then error "Rate limited — retry later"
else if Status = 404 then null
else error "API returned HTTP " & Text.From(Status)
in
Result
```

Common use cases:

- **404** — return `null` instead of failing when a record doesn't exist
- **429** — detect rate limiting and surface a clear error message
- **500** — log the failure and continue processing other rows instead of aborting the entire query

Without `ManualStatusHandling`, any of these status codes would immediately halt the query with a generic `DataSource.Error`.

### Paginating Through Results

Combine authentication with pagination using a recursive or `List.Generate` approach:
Expand Down Expand Up @@ -162,3 +230,8 @@ in
- `ManualCredentials = true` bypasses the credential store, so the key must be embedded or read from a parameter/config query that is itself accessible during refresh.
- OAuth tokens are often short-lived. If the token expires between the token request and the data request (rare for scheduled refresh, possible for very large pulls), structure the query so the token fetch and data fetch happen in the same M evaluation.
- Power BI Service enforces privacy level rules — if your API and your config query have different privacy levels, M may refuse to combine them. Set both to the same privacy level, or set the dataset to `Ignore Privacy Levels` (only for trusted, internal data).
- Queries that combine a POST request (e.g. fetching an OAuth token) and a GET request (fetching data with that token) in the same query may fail to refresh in the Power BI Service because the engine detects a dynamic data source. Workarounds include using Dataflows, a Data Gateway with a custom connector, or splitting the token fetch into a separate query.

### Further Reading

- [APIs as Power BI Datasources](https://datameerkat.com/apis-as-power-bi-datasources) — Štěpán Rešl's walkthrough of connecting REST APIs to Power BI using `Web.Contents`, covering URL construction, authentication flows, and refresh limitations.
Loading