-
Notifications
You must be signed in to change notification settings - Fork 4
docs: correct unsourced and misattributed statistics across published blog posts #802
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from 1 commit
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -20,9 +20,9 @@ This is the wrong metric. Support ticket deflection is real, but it captures a s | |
|
|
||
| Stack Overflow's developer survey found that developers spend more than 30 minutes per day searching for solutions to technical problems. Separate research puts the figure higher, with half of developers losing roughly 10 hours per week sourcing basic information they need to do their jobs. They are senior-engineer hours paid at full rate, spent compensating for an information system that should have made the answer obvious. | ||
|
|
||
| Research on developer experience makes this concrete. The DXI framework from DX (formerly DX Data) measures documentation quality as its own dimension of developer experience. Their data finds that each 1-point improvement in documentation quality saves 13 minutes per developer per week. For a 100-person engineering team, a 5-point improvement translates to 5,000 hours per year, roughly $500,000 in recovered capacity at a $150k average salary. | ||
| Research on developer experience makes this concrete. The DXI framework from DX measures documentation quality as one of 14 drivers of developer experience. [DX's Developer Experience Index research](https://getdx.com/blog/guide-to-developer-experience-index/) finds that each one-point improvement in a team's overall DXI score saves 13 minutes per developer per week. For a 100-person engineering team, a 5-point improvement translates to 5,000 hours per year, roughly $500,000 in recovered capacity at a $150k average salary. | ||
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. DX blog post confirms: "A single-point increase in the DXI score translates to saving 13 minutes per week per developer" and DXI is a composite of 14 drivers/dimensions, including Documentation. Doc's corrected framing ("DXI ... measures documentation quality as one of 14 drivers" and "each one-point improvement in a team's overall DXI score saves 13 minutes") accurately matches source scope. Source: https://getdx.com/blog/guide-to-developer-experience-index/ |
||
|
|
||
| The inverse calculation is equally useful. A team where documentation problems consume 15 to 25% of engineering capacity, a figure drawn from engineering surveys and [documented in our post on documentation drift](/blog/technical/documentation-drift-detection-problem), is effectively paying 15 to 25 engineers to compensate. Those engineers read source code instead of docs and ask Slack questions that a functioning information system would already answer. That is the cost denominator that rarely appears in a documentation business case. | ||
| The inverse calculation is equally useful. A team where documentation problems consume 15 to 25% of engineering capacity, [an estimate DX documents](https://getdx.com/blog/developer-documentation/) for teams that run this kind of self-assessment, is effectively paying 15 to 25 engineers to compensate. Those engineers read source code instead of docs and ask Slack questions that a functioning information system would already answer. That is the cost denominator that rarely appears in a documentation business case. | ||
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. DX blog post: "This diagnostic typically reveals that documentation problems cost 15-25% of engineering capacity." Doc correctly frames as "an estimate DX documents ... for teams that run this kind of self-assessment," matching source's "this diagnostic typically reveals" framing. Same claim restated at line 79. |
||
|
|
||
| Support ticket deflection matters for customer-facing documentation. For internal and developer-facing docs, the engineering time lever is at minimum 10x larger. | ||
|
|
||
|
|
@@ -42,7 +42,7 @@ Few documentation teams make this argument, because the causal chain is harder t | |
|
|
||
| ### 1. Developer experience surveys | ||
|
|
||
| A single survey question asking developers to rate documentation quality on a scale of 1 to 10, tracked quarterly, is enough to build a trend line. Each point of improvement carries the 13-min/developer/week value above. For your specific team size and salary band, the dollar value is calculable and defensible. | ||
| A single survey question asking developers to rate documentation quality on a scale of 1 to 10, tracked quarterly, is enough to build a trend line. Documentation is one of the drivers feeding the DXI score, where each point of improvement carries the 13-min/developer/week value above. For your specific team size and salary band, the dollar value is calculable and defensible. | ||
|
|
||
| This is the most actionable metric for leadership conversations, because it is a number that moves and can be traced to investments. When you ship a documentation sprint, the DXI score should move. When it does not, that is also useful information. | ||
|
|
||
|
|
@@ -76,7 +76,7 @@ The documentation ROI argument is strong, but it is only sustained by accuracy m | |
|
|
||
| The engineering time calculation is what closes most leadership conversations. | ||
|
|
||
| Take your team size. Multiply by average salary. Multiply by 15%, which is the low end of engineering capacity lost to documentation problems. That is your annual cost denominator. Then estimate what a 5-point DXI improvement is worth at your team size, using the 13-min/week/developer figure. The gap between those two numbers is the available ROI and the investment case. | ||
| Take your team size. Multiply by average salary. Multiply by 15%, the low end of DX's estimated range for engineering capacity lost to documentation problems. That is your annual cost denominator. Then estimate what a 5-point DXI improvement is worth at your team size, using the 13-min/developer/week figure. The gap between those two numbers is the available ROI and the investment case. | ||
|
|
||
| Start with two tracking practices. First, run a quarterly developer survey with a single documentation quality question. Second, pull 50 escalated tickets and categorize them by root cause. Add a coverage metric against your product surface once those baselines exist. Together they convert "our docs need to be better" into a business case with a dollar value and a trend line. | ||
|
|
||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -47,7 +47,7 @@ The ownership gap explains the rest. Documentation coverage spans multiple teams | |
|
|
||
| The result is that most teams discover documentation coverage gaps reactively. A developer opens a support ticket asking how to use a feature that was never documented. The question surfaces in a Slack channel. A post appears on a forum. By then, the gap has already caused friction for at least one person and probably for many more who silently gave up. | ||
|
|
||
| Research from GetDX estimates that documentation problems consume 15 to 25 percent of engineering capacity. That cost includes time developers spend searching for information and interrupting colleagues for answers. A significant share of that cost comes from features that shipped without documentation. | ||
| DX, a developer-productivity vendor, estimates that documentation problems consume [15 to 25 percent of engineering capacity](https://getdx.com/blog/developer-documentation/). That cost includes time developers spend searching for information and interrupting colleagues for answers. A significant share of that cost comes from features that shipped without documentation. | ||
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. DX blog post: "This diagnostic typically reveals that documentation problems cost 15-25% of engineering capacity." Doc correctly frames this as "DX, a developer-productivity vendor, estimates" — no overclaiming. |
||
|
|
||
| <BlogNewsletterCTA /> | ||
|
|
||
|
|
@@ -69,8 +69,6 @@ A B2B technology company that analyzed its docs search logs found that 23 percen | |
|
|
||
| Zero-result searches do not show you everything that is undocumented. They only surface the gaps developers actively searched for. But they are actionable immediately and require no additional tooling if your docs platform has built-in search analytics. GitBook, Fern, Document360, and ReadMe all surface this data in their default dashboards. If you are not looking at it, you are leaving a direct signal unused. | ||
|
|
||
| For a deeper look at how to use search analytics alongside other documentation metrics, the [complete documentation metrics and analytics guide](/blog/technical/documentation-metrics-and-analytics-a-complete-guide-for-dev) covers the full measurement stack. | ||
|
|
||
| ### Map support tickets to documentation status | ||
|
|
||
| Support tickets are lagging indicators of coverage gaps. When a developer opens a ticket asking how to accomplish something your product supports, there is a reasonable chance it was never documented, or was documented incompletely. | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -18,7 +18,7 @@ That is documentation debt collecting interest. | |
|
|
||
| Documentation debt is the accumulated gap between what your docs say and what your product actually does, plus the sections where docs don't exist at all. Teams accumulate it passively by shipping without updating reference pages and releasing new features before the quickstart catches up. | ||
|
|
||
| The productivity cost inside your organization is real. Research consistently estimates that documentation problems consume 15 to 25% of total engineering capacity, as developers read source code instead of docs and ask Slack questions that accurate documentation would prevent. For a 100-person engineering team, that's the equivalent of 15 to 25 engineers whose time disappears into compensating for missing or inaccurate documentation. Annually, that translates to somewhere between $500,000 and $2 million in a mid-sized company. | ||
| The productivity cost inside your organization is real. DX, a developer-productivity vendor, [estimates that documentation problems consume 15 to 25% of engineering capacity](https://getdx.com/blog/developer-documentation/), as developers read source code instead of docs and ask Slack questions that accurate documentation would prevent. For a 100-person engineering team, that's the equivalent of 15 to 25 engineers whose time disappears into compensating for missing or inaccurate documentation. Annually, that translates to somewhere between $500,000 and $2 million in a mid-sized company. | ||
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. DX blog post (Taylor Bruneaux, Analyst; updated Dec 9 2025): "This diagnostic typically reveals that documentation problems cost 15-25% of engineering capacity." Uncited vendor estimate from DX's own self-assessment diagnostic, no third-party research backing. Framing in doc as "DX ... estimates" is accurate. |
||
|
|
||
| But those numbers capture only what happens inside your organization. The higher cost sits outside it. | ||
|
|
||
|
|
@@ -28,11 +28,11 @@ When code debt slows your engineers down, they show up to sprint planning and sa | |
|
|
||
| Documentation debt slows down the people trying to use your product. They don't show up in your standups. A developer who hits a broken quickstart doesn't file a support ticket and wait. Research consistently finds that [around 50% of developers abandon an API](https://userguiding.com/blog/user-onboarding-statistics) when documentation fails them, and the broken page stays up for the next developer who hits the same wall. | ||
|
|
||
| The scale of this is not subtle. [Postman's 2024 State of the API report](https://www.postman.com/state-of-api/2024) found that 68% of developers cite outdated documentation as their top frustration when working with APIs. 78% of development teams report challenges with outdated or insufficient documentation. 64% of developers spend four or more hours per week searching for project information that should already be accessible. | ||
| The scale of this is not subtle. [Postman's 2024 State of the API report](https://www.postman.com/state-of-api/2024) found that 39% of developers say inconsistent documentation is the biggest roadblock to API collaboration. 78% of development teams report challenges with outdated or insufficient documentation. 64% of developers spend four or more hours per week searching for project information that should already be accessible. | ||
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Postman 2024 State of the API Report, p.3: 39% figure re: inconsistent docs as "the biggest roadblock." The source attaches no "onboarding" qualifier, so the corrected prose reads "the biggest roadblock to API collaboration." Source: https://voyager.postman.com/doc/postman-state-of-the-api-report-2024.pdf |
||
|
|
||
| What makes documentation debt durable is the gap between how much pain it causes and how visible that pain is to the teams responsible for it. The people most affected are new developers and external integration partners, and they have no seat at the table when backlog priorities get set. | ||
|
|
||
| AI coding agents have made this more acute. A developer can often work around a stale code sample by recognizing a deprecated method and adapting. An AI agent follows your documentation literally and produces broken code. The developer then debugs the AI's output, traces the error back to your docs, and hits the same wall faster than they would have without the AI. Stale documentation is less forgiving when agents amplify whatever they read, [a pattern that shows up in onboarding data](https://promptless.ai/blog/technical/developer-onboarding-documentation-fails-after-launch). | ||
| AI coding agents have made this more acute. A developer can often work around a stale code sample by recognizing a deprecated method and adapting. An AI agent follows your documentation literally and produces broken code. The developer then debugs the AI's output, traces the error back to your docs, and hits the same wall faster than they would have without the AI. Stale documentation is less forgiving when agents amplify whatever they read. | ||
|
|
||
| <BlogNewsletterCTA /> | ||
|
|
||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -38,7 +38,7 @@ Documentation debt concentrates in three areas. | |
|
|
||
| **Support volume.** Outdated API references and incorrect setup guides generate support tickets at scale. Each ticket consumes the customer's time, the support engineer's time, and often the context-switch overhead of a developer pulled in for escalation. | ||
|
|
||
| **Engineering interruptions.** Every time a developer answers a question that should be in the docs, they lose 15-20 minutes recovering their working context. Research consistently estimates this compounds to [15-25% of total engineering capacity](https://promptless.ai/blog/technical/documentation-drift-detection-problem) in teams with significant documentation debt. That's 15-25 engineers per 100-person team spending their time compensating for missing documentation instead of building. | ||
| **Engineering interruptions.** Every time a developer answers a question that should be in the docs, they lose 15-20 minutes recovering their working context. DX, a developer-productivity vendor, [estimates this compounds to 15-25% of engineering capacity](https://getdx.com/blog/developer-documentation/) in teams with significant documentation debt. That's 15-25 engineers per 100-person team spending their time compensating for missing documentation instead of building. | ||
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. DX blog post: "This diagnostic typically reveals that documentation problems cost 15-25% of engineering capacity." Doc correctly frames this as DX's own vendor estimate. |
||
|
|
||
| <BlogNewsletterCTA /> | ||
|
|
||
|
|
@@ -64,7 +64,7 @@ Ownership matters just as much. Documentation without a named owner gets updated | |
|
|
||
| Some teams embed documentation review into the PR process itself. When a PR touches a public API endpoint, it triggers a review of the corresponding reference doc. When a config file changes, the setup guide gets flagged. Automated checks at the point of change are the same mechanic that makes code debt visible. | ||
|
|
||
| The [teams that keep docs current](https://promptless.ai/blog/technical/how-teams-keep-docs-up-to-date-with-promptless) build systems that surface what needs updating before it becomes a problem. Understanding [why onboarding docs break after launch](https://promptless.ai/blog/technical/developer-onboarding-documentation-fails-after-launch) gives you the specific failure modes to build prevention around. | ||
| The [teams that keep docs current](https://promptless.ai/blog/technical/how-teams-keep-docs-up-to-date-with-promptless) build systems that surface what needs updating before it becomes a problem. Understanding [why getting-started guides drift after launch](https://promptless.ai/blog/technical/sdk-documentation-best-practices-after-launch) gives you the specific failure modes to build prevention around. | ||
|
|
||
| ## The measurement gap is the starting point | ||
|
|
||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Postman 2024 State of the API Report, p.3: "58% of developers rely on internal documentation, but 39% say inconsistent docs are the biggest roadblock." Verified by direct PDF extraction. The source attaches no "onboarding" qualifier to this figure, so the corrected prose reads "the biggest roadblock to API collaboration" — matching the report's collaboration-challenges framing without narrowing the claim.
Source: https://voyager.postman.com/doc/postman-state-of-the-api-report-2024.pdf