docs: correct unsourced and misattributed statistics across published blog posts - #802
Open
promptless[bot] wants to merge 2 commits into
Open
docs: correct unsourced and misattributed statistics across published blog posts#802promptless[bot] wants to merge 2 commits into
promptless[bot] wants to merge 2 commits into
Conversation
… blog posts Replaces statistics asserted more confidently than their sources support. Postman "68% ... top frustration" (7 files): no State of the API report supports 68%. Corrected to the real figures — 39% "inconsistent docs are the biggest roadblock" (2024) and 55% "struggling with inconsistent documentation" (2025). The underlying argument is well supported, so the figures were corrected rather than removed. 15-25% engineering capacity (6 spots): the GetDX attribution was real but the figure is an uncited assertion in an unbylined vendor blog post, not research. Reframed as an explicit DX vendor estimate with a direct link, replacing "Research consistently estimates" / "engineering surveys" and the circular internal links. Dropped "total", which DX does not use. DXI scope (developer-documentation-roi.mdx): the 13-minute figure belongs to the composite DXI score, where documentation is one of 14 drivers, not to a separately measured documentation-quality effect. Broken links (4 files): removed or repointed links to hidden: true posts, which generate no route and 404.
promptless
Bot
requested review from
adit-chandra,
frances720,
hawkeyexl,
mattlink and
prithvi2206
as code owners
July 29, 2026 17:19
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This was referenced Jul 30, 2026
![promptless[bot]](https://avatars.githubusercontent.com/in/979718?s=60&v=4){kind=small}
| That sequence plays out thousands of times across API ecosystems. The changelog announced the change, but the reference docs haven't caught up. The developer can't use either one to move forward. | ||
|
|
||
| The [Postman 2024 State of the API report](https://www.postman.com/state-of-api/2024), which surveyed over 5,600 developers and API professionals, found that 68% of developers cite outdated documentation as their top frustration when working with APIs. 39% say inconsistent documentation is the biggest onboarding roadblock. Those numbers don't reflect teams that never wrote documentation. They reflect teams whose documentation described a previous version of the product. | ||
| The [Postman 2024 State of the API report](https://www.postman.com/state-of-api/2024), which surveyed over 5,600 developers and API professionals, found that 39% of developers say inconsistent documentation is the biggest roadblock to API collaboration. That number doesn't reflect teams that never wrote documentation. It reflects teams whose documentation described a previous version of the product. |
Contributor
Author
There was a problem hiding this comment.
Postman 2024 State of the API Report, p.3: "58% of developers rely on internal documentation, but 39% say inconsistent docs are the biggest roadblock." Verified by direct PDF extraction. The source attaches no "onboarding" qualifier to this figure, so the corrected prose reads "the biggest roadblock to API collaboration" — matching the report's collaboration-challenges framing without narrowing the claim.
Source: https://voyager.postman.com/doc/postman-state-of-the-api-report-2024.pdf
| Documentation debt is the accumulated gap between what your docs say and what your product actually does, plus the sections where docs don't exist at all. Teams accumulate it passively by shipping without updating reference pages and releasing new features before the quickstart catches up. | ||
|
|
||
| The productivity cost inside your organization is real. Research consistently estimates that documentation problems consume 15 to 25% of total engineering capacity, as developers read source code instead of docs and ask Slack questions that accurate documentation would prevent. For a 100-person engineering team, that's the equivalent of 15 to 25 engineers whose time disappears into compensating for missing or inaccurate documentation. Annually, that translates to somewhere between $500,000 and $2 million in a mid-sized company. | ||
| The productivity cost inside your organization is real. DX, a developer-productivity vendor, [estimates that documentation problems consume 15 to 25% of engineering capacity](https://getdx.com/blog/developer-documentation/), as developers read source code instead of docs and ask Slack questions that accurate documentation would prevent. For a 100-person engineering team, that's the equivalent of 15 to 25 engineers whose time disappears into compensating for missing or inaccurate documentation. Annually, that translates to somewhere between $500,000 and $2 million in a mid-sized company. |
Contributor
Author
There was a problem hiding this comment.
DX blog post (Taylor Bruneaux, Analyst; updated Dec 9 2025): "This diagnostic typically reveals that documentation problems cost 15-25% of engineering capacity." Uncited vendor estimate from DX's own self-assessment diagnostic, no third-party research backing. Framing in doc as "DX ... estimates" is accurate.
| Documentation debt slows down the people trying to use your product. They don't show up in your standups. A developer who hits a broken quickstart doesn't file a support ticket and wait. Research consistently finds that [around 50% of developers abandon an API](https://userguiding.com/blog/user-onboarding-statistics) when documentation fails them, and the broken page stays up for the next developer who hits the same wall. | ||
|
|
||
| The scale of this is not subtle. [Postman's 2024 State of the API report](https://www.postman.com/state-of-api/2024) found that 68% of developers cite outdated documentation as their top frustration when working with APIs. 78% of development teams report challenges with outdated or insufficient documentation. 64% of developers spend four or more hours per week searching for project information that should already be accessible. | ||
| The scale of this is not subtle. [Postman's 2024 State of the API report](https://www.postman.com/state-of-api/2024) found that 39% of developers say inconsistent documentation is the biggest roadblock to API collaboration. 78% of development teams report challenges with outdated or insufficient documentation. 64% of developers spend four or more hours per week searching for project information that should already be accessible. |
Contributor
Author
There was a problem hiding this comment.
Postman 2024 State of the API Report, p.3: 39% figure re: inconsistent docs as "the biggest roadblock." The source attaches no "onboarding" qualifier, so the corrected prose reads "the biggest roadblock to API collaboration."
Source: https://voyager.postman.com/doc/postman-state-of-the-api-report-2024.pdf
| The result is that most teams discover documentation coverage gaps reactively. A developer opens a support ticket asking how to use a feature that was never documented. The question surfaces in a Slack channel. A post appears on a forum. By then, the gap has already caused friction for at least one person and probably for many more who silently gave up. | ||
|
|
||
| Research from GetDX estimates that documentation problems consume 15 to 25 percent of engineering capacity. That cost includes time developers spend searching for information and interrupting colleagues for answers. A significant share of that cost comes from features that shipped without documentation. | ||
| DX, a developer-productivity vendor, estimates that documentation problems consume [15 to 25 percent of engineering capacity](https://getdx.com/blog/developer-documentation/). That cost includes time developers spend searching for information and interrupting colleagues for answers. A significant share of that cost comes from features that shipped without documentation. |
Contributor
Author
There was a problem hiding this comment.
DX blog post: "This diagnostic typically reveals that documentation problems cost 15-25% of engineering capacity." Doc correctly frames this as "DX, a developer-productivity vendor, estimates" — no overclaiming.
| **Support volume.** Outdated API references and incorrect setup guides generate support tickets at scale. Each ticket consumes the customer's time, the support engineer's time, and often the context-switch overhead of a developer pulled in for escalation. | ||
|
|
||
| **Engineering interruptions.** Every time a developer answers a question that should be in the docs, they lose 15-20 minutes recovering their working context. Research consistently estimates this compounds to [15-25% of total engineering capacity](https://promptless.ai/blog/technical/documentation-drift-detection-problem) in teams with significant documentation debt. That's 15-25 engineers per 100-person team spending their time compensating for missing documentation instead of building. | ||
| **Engineering interruptions.** Every time a developer answers a question that should be in the docs, they lose 15-20 minutes recovering their working context. DX, a developer-productivity vendor, [estimates this compounds to 15-25% of engineering capacity](https://getdx.com/blog/developer-documentation/) in teams with significant documentation debt. That's 15-25 engineers per 100-person team spending their time compensating for missing documentation instead of building. |
Contributor
Author
There was a problem hiding this comment.
DX blog post: "This diagnostic typically reveals that documentation problems cost 15-25% of engineering capacity." Doc correctly frames this as DX's own vendor estimate.
| It's not malicious. It was accurate when it was written. But a parameter was renamed, an endpoint was deprecated, a flow was quietly restructured — and no one updated the page. The lie has been sitting there for weeks, maybe months, misleading developers who trust it. | ||
|
|
||
| This is documentation drift: the slow, continuous divergence between what your docs say and what your product actually does. According to a Postman survey, 68% of developers cite outdated documentation as their top frustration when working with APIs. A 2025 IEEE review of the field confirmed what practitioners already know: "Maintaining alignment between software code and specifications is a persistent challenge in the software development lifecycle." | ||
| This is documentation drift: the slow, continuous divergence between what your docs say and what your product actually does. According to [Postman's 2025 State of the API report](https://www.postman.com/state-of-api/), 55% of respondents report struggling with inconsistent documentation. A 2025 IEEE review of the field confirmed what practitioners already know: "Maintaining alignment between software code and specifications is a persistent challenge in the software development lifecycle." |
Contributor
Author
There was a problem hiding this comment.
Postman 2025 State of the API Report (7th annual, ~5,700 respondents), "Future outlook" section: "With 55% struggling with inconsistent documentation and 34% unable to find existing APIs..." URL confirmed to resolve (redirects to /state-of-api/2025/, HTTP 200) and contain this figure. The source does not name "developers" as the base — the survey covered developers, architects, and executives — so the corrected prose reads "55% of respondents."
| Even when teams have a dedicated documentation owner, the shape of the problem shifts rather than disappearing. Now the writer has to discover the change — by watching Slack, scanning pull requests, attending standups, or waiting for a support ticket that turns out to be an outdated doc. This is surveillance work: constant, manual, and impossible to sustain as codebases and teams scale. | ||
|
|
||
| The downstream costs are significant. A simple diagnostic — surveying developers on documentation quality, tracking Slack questions, measuring PR cycle time — typically finds that documentation problems consume **15–25% of total engineering capacity**. Not because people aren't writing, but because they're compensating: reading source code instead of reading docs, asking questions on Slack that docs should answer, debugging integration issues that trace back to an outdated spec. That's 15–25 engineers per 100-person team doing work that accurate documentation would eliminate. | ||
| The downstream costs are significant. DX, a developer-productivity vendor, recommends a simple diagnostic — surveying developers on documentation quality, tracking Slack questions, measuring PR cycle time — and [estimates it typically surfaces documentation problems consuming 15–25% of engineering capacity](https://getdx.com/blog/developer-documentation/). Not because people aren't writing, but because they're compensating: reading source code instead of reading docs, asking questions on Slack that docs should answer, debugging integration issues that trace back to an outdated spec. That's 15–25 engineers per 100-person team doing work that accurate documentation would eliminate. |
Contributor
Author
There was a problem hiding this comment.
DX blog post: "This diagnostic typically reveals that documentation problems cost 15-25% of engineering capacity." Doc's framing ("DX ... recommends a simple diagnostic ... and estimates it typically surfaces...") closely mirrors source phrasing and correctly attributes to DX as vendor estimate, not research.
| Stack Overflow's developer survey found that developers spend more than 30 minutes per day searching for solutions to technical problems. Separate research puts the figure higher, with half of developers losing roughly 10 hours per week sourcing basic information they need to do their jobs. They are senior-engineer hours paid at full rate, spent compensating for an information system that should have made the answer obvious. | ||
|
|
||
| Research on developer experience makes this concrete. The DXI framework from DX (formerly DX Data) measures documentation quality as its own dimension of developer experience. Their data finds that each 1-point improvement in documentation quality saves 13 minutes per developer per week. For a 100-person engineering team, a 5-point improvement translates to 5,000 hours per year, roughly $500,000 in recovered capacity at a $150k average salary. | ||
| Research on developer experience makes this concrete. The DXI framework from DX measures documentation quality as one of 14 drivers of developer experience. [DX's Developer Experience Index research](https://getdx.com/blog/guide-to-developer-experience-index/) finds that each one-point improvement in a team's overall DXI score saves 13 minutes per developer per week. For a 100-person engineering team, a 5-point improvement translates to 5,000 hours per year, roughly $500,000 in recovered capacity at a $150k average salary. |
Contributor
Author
There was a problem hiding this comment.
DX blog post confirms: "A single-point increase in the DXI score translates to saving 13 minutes per week per developer" and DXI is a composite of 14 drivers/dimensions, including Documentation. Doc's corrected framing ("DXI ... measures documentation quality as one of 14 drivers" and "each one-point improvement in a team's overall DXI score saves 13 minutes") accurately matches source scope.
Source: https://getdx.com/blog/guide-to-developer-experience-index/
| Research on developer experience makes this concrete. The DXI framework from DX measures documentation quality as one of 14 drivers of developer experience. [DX's Developer Experience Index research](https://getdx.com/blog/guide-to-developer-experience-index/) finds that each one-point improvement in a team's overall DXI score saves 13 minutes per developer per week. For a 100-person engineering team, a 5-point improvement translates to 5,000 hours per year, roughly $500,000 in recovered capacity at a $150k average salary. | ||
|
|
||
| The inverse calculation is equally useful. A team where documentation problems consume 15 to 25% of engineering capacity, a figure drawn from engineering surveys and [documented in our post on documentation drift](/blog/technical/documentation-drift-detection-problem), is effectively paying 15 to 25 engineers to compensate. Those engineers read source code instead of docs and ask Slack questions that a functioning information system would already answer. That is the cost denominator that rarely appears in a documentation business case. | ||
| The inverse calculation is equally useful. A team where documentation problems consume 15 to 25% of engineering capacity, [an estimate DX documents](https://getdx.com/blog/developer-documentation/) for teams that run this kind of self-assessment, is effectively paying 15 to 25 engineers to compensate. Those engineers read source code instead of docs and ask Slack questions that a functioning information system would already answer. That is the cost denominator that rarely appears in a documentation business case. |
Contributor
Author
There was a problem hiding this comment.
DX blog post: "This diagnostic typically reveals that documentation problems cost 15-25% of engineering capacity." Doc correctly frames as "an estimate DX documents ... for teams that run this kind of self-assessment," matching source's "this diagnostic typically reveals" framing. Same claim restated at line 79.
| This is called spec drift. [According to Kinde](https://www.kinde.com/learn/ai-for-software-engineering/ai-devops/spec-drift-the-hidden-problem-ai-can-help-fix/), it starts the moment a developer merges a route change without updating the spec. That's the default behavior on most teams, because there's no enforcement step that makes updating the spec mandatory before a PR merges. | ||
|
|
||
| [The Postman 2024 State of the API report](https://voyager.postman.com/doc/postman-state-of-the-api-report-2024.pdf), which surveyed over 5,600 developers, found that 68% of developers cite outdated documentation as their top frustration when working with APIs. Teams that have shipped interactive docs are not exempt from this. The interactivity doesn't solve the accuracy problem. It just adds a new surface on which inaccuracy shows up. | ||
| [The Postman 2024 State of the API report](https://voyager.postman.com/doc/postman-state-of-the-api-report-2024.pdf), which surveyed over 5,600 developers, found that 39% of developers say inconsistent documentation is the biggest roadblock to API collaboration. Teams that have shipped interactive docs are not exempt from this. The interactivity doesn't solve the accuracy problem. It just adds a new surface on which inaccuracy shows up. |
Contributor
Author
There was a problem hiding this comment.
Postman 2024 State of the API Report, p.3: 39% figure re: inconsistent docs as "the biggest roadblock." The source attaches no "onboarding" qualifier, so the corrected prose reads "the biggest roadblock to API collaboration."
Source: https://voyager.postman.com/doc/postman-state-of-the-api-report-2024.pdf
| The getting-started guide is where developers decide whether to keep going. Stripe benchmarks Time to First API Call under 90 seconds, and TTFC is the strongest leading indicator of developer activation. A single stale step can push TTFC from 90 seconds to 20 minutes. Many developers stop at that wall and don't file a support ticket explaining why. | ||
|
|
||
| The [Postman 2024 State of the API report](https://www.postman.com/state-of-api/2024) found that 68% of developers cite outdated documentation as their top frustration with APIs. Most of that frustration originates in quickstart and getting-started content, not reference pages. | ||
| The [Postman 2024 State of the API report](https://www.postman.com/state-of-api/2024) found that 39% of developers say inconsistent documentation is the biggest roadblock to API collaboration. That roadblock is concentrated in quickstart and getting-started content, not reference pages. |
Contributor
Author
There was a problem hiding this comment.
Postman 2024 State of the API Report, p.3: 39% figure re: inconsistent docs as "the biggest roadblock." The source attaches no "onboarding" qualifier, so the corrected prose reads "the biggest roadblock to API collaboration."
Source: https://voyager.postman.com/doc/postman-state-of-the-api-report-2024.pdf
| The draft that was accurate when it was written doesn't stay accurate because of how it was written. A tutorial for an authentication flow is correct until the auth flow changes. At that point, the quality of the original draft has no bearing on whether the tutorial is still right. | ||
|
|
||
| [Postman's 2024 State of the API report](https://www.postman.com/state-of-api/2024) found that 68% of developers cite outdated documentation as their top frustration. That number predates the current wave of AI-accelerated publishing. Teams publishing more content, without proportionally more capacity to [detect and fix drift](https://promptless.ai/blog/technical/documentation-drift-detection-problem), will not improve that statistic. | ||
| [Postman's 2024 State of the API report](https://www.postman.com/state-of-api/2024) found that 39% of developers say inconsistent documentation is the biggest roadblock to API collaboration. That number predates the current wave of AI-accelerated publishing. Teams publishing more content, without proportionally more capacity to [detect and fix drift](https://promptless.ai/blog/technical/documentation-drift-detection-problem), will not improve that statistic. |
Contributor
Author
There was a problem hiding this comment.
Postman 2024 State of the API Report, p.3: 39% figure re: inconsistent docs as "the biggest roadblock." The source attaches no "onboarding" qualifier, so the corrected prose reads "the biggest roadblock to API collaboration."
Source: https://voyager.postman.com/doc/postman-state-of-the-api-report-2024.pdf
…sourced-stats-blog-corpus # Conflicts: # src/content/blog/technical/documentation-coverage.mdx # src/content/blog/technical/documentation-debt-accrues-where-your-team-cant-see-it.mdx # src/content/blog/technical/documentation-debt.mdx # src/content/blog/technical/how-to-measure-developer-documentation-roi.mdx
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Open this suggestion in Promptless to view citations and reasoning process
Scope note: this originated while reviewing #801, but is deliberately scoped to pre-existing published content and does not overlap that PR. It touches none of #801's file and none of #800's files. #801 gets a review; these ten live posts need their own fix, because the inaccuracies below are public-facing today regardless of what happens to any draft in review.
Ten published blog posts carried statistics stated more confidently than their sources support. This corrects the figures against what the sources actually say rather than removing the underlying arguments, which are well supported.
Because the problem being fixed is exactly "figures asserted more confidently than their sourcing supports," here is the verified-versus-unverified split on everything touched.
Verified against primary sources
Postman — the "68% ... top frustration" figure is unsupported. It appeared in 7 files in near-identical wording. No State of the API report supports 68%. Figures read in full from the primary reports:
Corrected to 39% where the post already cited the 2024 report (keeping its existing link honest), and to 55% where the citation had no year at all. Every Postman citation now states a year and pairs it with that year's real figure.
Two wording details worth flagging, both caught in fact-check: the 2024 figure carries no "onboarding" qualifier in the source, so the corrected sentences read "the biggest roadblock to API collaboration" rather than narrowing it; and the 2025 figure's base is "respondents," not "developers," since that survey covered developers, architects, and executives.
DXI 13-minute figure — verified on getdx.com's DXI guide: it attaches to a one-point increase in the composite DXI score, where documentation is one of 14 drivers.
developer-documentation-roi.mdxhad narrowed it to documentation quality specifically. Restored to composite scope in both places it appears.Four broken links —
src/pages/blog/[...slug].astrobuilds routes viagetCollection('blog', ({ data }) => !data.hidden), so ahidden: truepost has no route and returns a hard 404. Confirmed no redirect covers them insrc/lib/generated/redirects.jsonorvercel.json, and every other surface (blog index,.mdendpoint, sitemap) filters hidden entries too. Removed, cut, or repointed to a published equivalent.Verified as an attribution, NOT as a research finding
The 15-25% engineering-capacity figure is a split verdict, and the fix reflects that rather than resolving it either way.
The GetDX attribution in
documentation-coverage.mdxis real — getdx.com/blog/developer-documentation/ carries the sentence verbatim. But the figure is not research: it is an uncited assertion in an unbylined vendor marketing post with no sample size or methodology, describing the predicted outcome of a two-week self-assessment the article tells the reader to run. It appears nowhere in DX's ~100-entry research archive, nor in the DevEx/SPACE/Core 4 papers or Abi Noda / Nicole Forsgren's published work, and exact-phrase searches surface zero third-party occurrences. No other credible source (DORA, Stack Overflow, McKinsey, IEEE/academic) supports it; adjacent findings exist but measure different things and are not substitutes.Per instruction, no replacement number was invented. Each of the six occurrences is now framed as an explicit vendor estimate with a direct link — replacing "Research consistently estimates," "Research from GetDX," and "engineering surveys," and dropping "total," a word DX does not use. The circular internal links (pointing at another Promptless post that also asserted the figure bare) are gone.
This is the item most in need of a human editorial call. The honest framing is now in place, but a reader who follows the link finds no backing. Whether to keep an attributed-but-unbacked vendor estimate, soften it further, or cut it is a positioning decision, not a factual one.
Left unchanged on purpose
$500K–$2Mannual range indocumentation-debt-accrues-where-your-team-cant-see-it.mdx(both thedescriptionand the body) is arithmetic derived from the 15-25% figure. Rewriting a post's headline claim is an editorial decision, so it is flagged rather than auto-corrected.documentation-debt-accrues(unattributed, but positioned inside a paragraph whose only citation is the Postman link, so they read as if Postman is the source); "around 50% of developers abandon an API" in the same file, sourced to a vendor listicle and using the same "Research consistently finds" pattern removed 8 lines above it; "half of developers losing roughly 10 hours per week" indeveloper-documentation-roi.mdx. Each is worth a follow-up.hidden: trueposts. One still contains the same "onboarding"-qualified 39% claim and the old "GetDX" spelling, but they generate no routes, so correcting them is zero-reader-impact scope expansion. They will need this same sweep if ever unhidden.Verification
npm run check(typecheck + MCP contract tests + build) andnpm run test:smoke(16/16) both pass. Theroute-manifest.jsondiff is exactly one line — the regenerateddescriptionfor the single edited frontmatter field. Note that runninggenerate:manifeston cleanmainproduces ~140 lines of unrelated pre-existing drift (staleordervalues and descriptions); that drift is deliberately excluded from this PR rather than swept in.Trigger Events