Add embodied emissions to storage

docs/src/comparison.md

@@ -33,7 +33,7 @@ SPRUCE is part of a growing ecosystem of open source tools focused on measuring
 | **Data Privacy**       | Processes data locally, no external API calls for core functionality | Requires cloud provider credentials                   |
 | **Modularity**         | Highly modular with configurable enrichment pipelines                | Fixed calculation pipeline with configuration options |
 | **Output**             | Enriched Parquet/CSV files for custom analytics and visualization    | Pre-built dashboard and recommendations               |
-| **Embodied Carbon**    | Includes embodied emissions via Boavizta integration                 | Limited embodied carbon estimates                     |
+| **Embodied Carbon**    | Includes embodied emissions for compute via Boavizta, and for storage from drive LCAs | Limited embodied carbon estimates                     |
 | **Scalability**        | Designed for large-scale data processing with Apache Spark           | Suitable for smaller to medium deployments            |
 | **Carbon Intensity**   | Uses Ember average data                                              | Default factors outdated                              |
 | **Maintenance Status** | Actively maintained with regular updates                             | No longer actively maintained                         |

docs/src/methodology.md

@@ -1,6 +1,6 @@
 ---
 title: "How cloud carbon and water estimates are calculated"
-description: "The methodology behind SPRUCE: embodied emissions from Boavizta, energy models from Cloud Carbon Footprint, Ember grid carbon intensity, PUE and WUE."
+description: "The methodology behind SPRUCE: embodied emissions from Boavizta and drive LCAs, energy models from Cloud Carbon Footprint, Ember grid carbon intensity, PUE and WUE."
 ---
 
 # How SPRUCE calculates cloud carbon, energy and water estimates
@@ -30,7 +30,7 @@ The main columns added by SPRUCE are:
 : emissions of CO2 eq in grams from the energy usage.
 
 `embodied_emissions_co2eq_g`
-: emissions of CO2 eq in grams embodied in the hardware used by the service, i.e. how much did it take to produce it.
+: emissions of CO2 eq in grams embodied in the hardware used by the service, i.e. how much did it take to produce it. Estimated for compute instances, LLM inference, and the drives behind storage; other services leave it null rather than zero.
 
 The total emissions for a service are `operational_emissions_co2eq_g` + `embodied_emissions_co2eq_g`.
 

docs/src/modules.md

@@ -43,7 +43,7 @@ flowchart LR
 | Module | Providers | Writes | Based on |
 |---|---|---|---|
 | [RegionExtraction](#regionextraction) | AWS, Azure, FOCUS | `region` | — |
-| [Storage](#storage) | AWS, Azure | `operational_energy_kwh` | [Cloud Carbon Footprint](https://www.cloudcarbonfootprint.org/) |
+| [Storage](#storage) | AWS, Azure | `operational_energy_kwh`, `embodied_emissions_co2eq_g` | [Cloud Carbon Footprint](https://www.cloudcarbonfootprint.org/) for energy, [Boavizta](https://doc.api.boavizta.org/) and vendor LCAs for embodied |
 | [Networking](#networking) | AWS, Azure | `operational_energy_kwh` | [Boavizta](https://boavizta.org/) coefficients |
 | [Serverless](#serverless) | AWS | `operational_energy_kwh` | [Tailpipe](https://tailpipe.ai/methodology/serverless-explained/) |
 | [Accelerators](#accelerators) | AWS | `operational_energy_kwh` | [Cloud Carbon Footprint](https://www.cloudcarbonfootprint.org/) |
@@ -70,29 +70,66 @@ provider-neutral: it reads the standard `RegionId` column.
 
 ## Stage 2 — Energy and embodied estimates
 
-The modules in this stage estimate the energy used by a row of usage — and, for Boavizta
-and EcoLogits, the related embodied emissions. Each module handles a different kind of
+The modules in this stage estimate the energy used by a row of usage and, for Storage,
+Boavizta and EcoLogits, the related embodied emissions. Each module handles a different kind of
 usage (storage, networking, compute, …), so they complement each other rather than overlap.
 
 ### Storage
 
 Estimates the energy used for storage by applying a flat coefficient per GB, following the
-approach of the [Cloud Carbon Footprint](https://www.cloudcarbonfootprint.org/docs/methodology#storage) project.
-Service-specific replication factors are applied. On Azure, managed disks are estimated
-from their provisioned capacity.
+approach of the [Cloud Carbon Footprint](https://www.cloudcarbonfootprint.org/docs/methodology#storage) project,
+and the embodied emissions of the drives holding the data, amortised over their service life.
+Service-specific replication factors are applied to both, since the same bytes occupy that many
+times more physical drives. On Azure, managed disks are estimated from their provisioned capacity.
 
 | | |
 |---|---|
 | **Classes** | `com.digitalpebble.spruce.modules.ccf.aws.Storage`<br>`com.digitalpebble.spruce.modules.ccf.azure.Storage` |
-| **Writes** | `operational_energy_kwh` |
+| **Writes** | `operational_energy_kwh`<br>`embodied_emissions_co2eq_g` |
 
-**Configuration** (in Wh per TB-hour):
+**Energy configuration** (in Wh per TB-hour):
 
 | Key | Default | Description |
 |---|---|---|
 | `hdd_coefficient_tb_h` | 0.65 | Energy per TB-hour for HDD storage |
 | `ssd_coefficient_tb_h` | 1.2  | Energy per TB-hour for SSD storage |
 
+**Embodied emissions configuration**:
+
+| Key | Default | Description |
+|---|---|---|
+| `hdd_embodied_kg_per_drive` | 30.0 | Embodied emissions of one hard drive, in kg CO2eq |
+| `hdd_capacity_gb` | 15000.0 | Capacity assumed for one hard drive |
+| `ssd_embodied_kg_per_gb` | 0.055 | Embodied emissions of an SSD, per GB of capacity |
+| `storage_lifetime_hours` | 43800.0 | Service life the embodied emissions are amortised over (5 years) |
+
+The two media are modelled on different bases. A hard drive costs roughly the same to manufacture
+whatever its capacity, since the platters, motor, actuator, casing and PCB are near-fixed for a
+3.5" unit and areal density does the work, so its embodied emissions are a constant per drive
+divided by an assumed capacity. An SSD's die area scales with capacity, so its figure is a rate
+per GB. With the defaults that works out at 0.40 kg CO2eq per TB-year for HDD and 11 kg for SSD.
+
+Note that `hdd_capacity_gb` describes the physical drive rather than a provisioned volume, so it
+is not the same thing as an Azure Managed Disk size or an EBS volume size.
+
+**Data sources for the embodied figures**:
+
+| Figure | Value | Source |
+|---|---|---|
+| HDD, per drive | 31.11 kg CO2eq | [BoaviztAPI HDD component](https://doc.api.boavizta.org/Explanations/components/hdd/), from [Umweltbundesamt, *Green Cloud Computing* 2021](https://www.umweltbundesamt.de/sites/default/files/medien/5750/publikationen/2021-06-17_texte_94-2021_green-cloud-computing.pdf) |
+| HDD, Seagate Exos X22 LCA | 28.7 kg CO2eq for a 22 TB drive | [Tailpipe manufacture methodology](https://tailpipe.ai/methodology/embodied-emissions-methodology-manufacture/) |
+| HDD, per TB-year | 0.27 kg CO2eq | [Seagate, hard drives and data centre sustainability](https://www.seagate.com/blog/hard-drives-the-key-to-data-center-sustainability/) |
+| HDD, 24 vendor LCAs | 0.02 kg CO2eq/GB over a 512 GB to 6 TB sample | [Tannu &amp; Nair, *The Dirty Secret of SSDs: Embodied Carbon*](https://arxiv.org/pdf/2207.10793) |
+| SSD, die-area formula | 0.052 kg CO2eq/GB | [BoaviztAPI SSD component](https://doc.api.boavizta.org/Explanations/components/ssd/) |
+| SSD, 3D NAND study | 0.056 kg CO2eq/GB | [Tailpipe manufacture methodology](https://tailpipe.ai/methodology/embodied-emissions-methodology-manufacture/), from [*Embodied Carbon Footprint of 3D NAND Memories*](https://hal.science/hal-05015578v1/document) |
+| Drive capacity mix | 15 TB installed-fleet average | [Backblaze Drive Stats 2025](https://www.backblaze.com/blog/backblaze-drive-stats-for-2025/) |
+
+The first four rows converge on roughly 30 kg per drive across a 40x range of capacities, which
+is itself the evidence for treating HDD embodied carbon as capacity independent. The Tannu &amp;
+Nair rate cannot be applied per GB to current hardware: it encodes the drive sizes of a pre-2023
+corpus and overstates per-byte embodied carbon by about an order of magnitude. See
+[issue #102](https://github.com/DigitalPebble/spruce/issues/102) for the full derivation.
+
 ### Networking
 
 Estimates the energy used for networking in and out of data centres. The module

src/main/java/com/digitalpebble/spruce/Utils.java

@@ -107,6 +107,20 @@ public static Row withUpdatedValues(Row row, Map<String, Object> updates) {
         return new GenericRowWithSchema(values, row.schema());
     }
 
+    /**
+     * Reads a numeric module parameter, tolerating the whole-number literals JSON configs
+     * routinely carry as integers rather than doubles.
+     *
+     * @param params module configuration
+     * @param key    parameter name
+     * @param fallback value returned when the parameter is absent
+     * @return the configured value, or {@code fallback}
+     */
+    public static double doubleParam(Map<String, Object> params, String key, double fallback) {
+        Number value = (Number) params.get(key);
+        return value != null ? value.doubleValue() : fallback;
+    }
+
     /**
      * Utility conversions used throughout the codebase.
      *

src/main/java/com/digitalpebble/spruce/modules/ccf/aws/Storage.java

@@ -18,12 +18,38 @@
 import java.util.Map;
 
 import static com.digitalpebble.spruce.CURColumn.*;
+import static com.digitalpebble.spruce.SpruceColumn.EMBODIED_EMISSIONS;
 import static com.digitalpebble.spruce.SpruceColumn.ENERGY_USED;
 import static com.digitalpebble.spruce.Utils.loadJSONResources;
 
 /**
- * Provides an estimate of energy used for storage.
- * Applies a flat coefficient per Gb
+ * Provides an estimate of energy used for storage, and of the embodied emissions of the drives
+ * holding it. Applies a flat coefficient per Gb
+ *
+ * <p>Embodied emissions are amortised over a five year service life. The two media are modelled
+ * on different bases because they behave differently: a hard drive costs roughly the same to
+ * manufacture whatever its capacity, since the platters, motor, actuator, casing and PCB are
+ * near-fixed for a 3.5" unit and areal density does the work, whereas an SSD's die area scales
+ * with capacity. Hence a constant per drive for HDD and a rate per GB for SSD.
+ *
+ * <p>The 30 kg CO2eq per drive is the convergence point of four independent sources spanning a
+ * 40x range of drive capacities, which is itself the evidence for treating it as capacity
+ * independent: Boavizta / Umweltbundesamt <i>Green Cloud Computing</i> 2021 (31.11 kg per unit),
+ * a Seagate Exos X22 LCA (28.7 kg for a 22 TB drive), Seagate's published 0.27 kg per TB-year,
+ * and Tannu &amp; Nair's meta-analysis of 24 vendor LCAs (0.02 kg/GB over a 512 GB to 6 TB
+ * sample). Note that the last of those cannot be used as a per-GB rate on modern hardware: it
+ * encodes the drive sizes of a pre-2023 corpus and overstates current per-byte embodied carbon
+ * by an order of magnitude.
+ *
+ * <p>The 0.055 kg CO2eq per GB for SSD is where Boavizta's die-area formula (0.052) and the 2025
+ * <i>Embodied Carbon Footprint of 3D NAND Memories</i> study (0.056) agree. Tannu &amp; Nair's
+ * 0.16 kg/GB is roughly 3x higher because 3D NAND layer scaling has cut per-GB manufacturing
+ * carbon since their corpus closed.
+ *
+ * <p>The assumed 15 TB drive is the installed-fleet average implied by Backblaze's 2025 Drive
+ * Stats, which is the right basis for bytes sitting on hardware bought over several years;
+ * current nearline shipments average nearer 22 TB, which would give 0.27 kg per TB-year instead
+ * of 0.40. Both figures are configurable.
  *
  * <p>The values read (operations, usage types, units) are identical in CUR and FOCUS reports,
  * only the column labels differ: {@link #bindReportFormat(ReportFormat)} selects the bindings.
@@ -32,6 +58,13 @@
  *
  * @see <a href="https://www.cloudcarbonfootprint.org/docs/methodology#storage">CCF methodology</a>
  * @see <a href="https://github.com/cloud-carbon-footprint/cloud-carbon-footprint/blob/9f2cf436e5ad020830977e52c3b0a1719d20a8b9/packages/aws/src/lib/CostAndUsageTypes.ts#L25">resource file</a>
+ * @see <a href="https://doc.api.boavizta.org/Explanations/components/hdd/">Boavizta HDD embodied impacts</a>
+ * @see <a href="https://doc.api.boavizta.org/Explanations/components/ssd/">Boavizta SSD embodied impacts</a>
+ * @see <a href="https://tailpipe.ai/methodology/embodied-emissions-methodology-manufacture/">Tailpipe manufacture methodology, incl. the Exos X22 LCA</a>
+ * @see <a href="https://www.seagate.com/blog/hard-drives-the-key-to-data-center-sustainability/">Seagate, embodied carbon per TB-year</a>
+ * @see <a href="https://arxiv.org/pdf/2207.10793">Tannu &amp; Nair, The Dirty Secret of SSDs: Embodied Carbon</a>
+ * @see <a href="https://www.backblaze.com/blog/backblaze-drive-stats-for-2025/">Backblaze Drive Stats 2025, fleet capacity mix</a>
+ * @see <a href="https://github.com/DigitalPebble/spruce/issues/102">issue #102</a>
  **/
 public class Storage implements EnrichmentModule {
 
@@ -69,6 +102,20 @@ public void bindReportFormat(ReportFormat reportFormat) {
     //  1.2 Watt-Hours per Terabyte-Hour for SSD
     double ssd_gb_coefficient = 1.2 / 1024d;
 
+    /** Embodied emissions of one hard drive, in kg CO2eq; see the class javadoc for why this is
+     *  a constant per drive rather than a rate per byte. */
+    double hdd_embodied_kg_per_drive = 30d;
+    /** Capacity assumed for one hard drive, in GB. */
+    double hdd_capacity_gb = 15_000d;
+    /** Embodied emissions of an SSD, in kg CO2eq per GB of capacity. */
+    double ssd_embodied_kg_per_gb = 0.055d;
+    /** Service life over which embodied emissions are amortised, in hours (5 years). */
+    double storage_lifetime_hours = 43_800d;
+
+    /** Grams CO2eq per GB-hour of stored data, derived in {@link #init(Map)}. */
+    double hdd_embodied_g_per_gb_hour;
+    double ssd_embodied_g_per_gb_hour;
+
     List<String> ssd_usage_types;
     List<String> hdd_usage_types;
     List<String> ssd_services;
@@ -86,8 +133,19 @@ public void init(Map<String, Object> params) {
             ssd_gb_coefficient = coef / 1024d;
         }
 
+        hdd_embodied_kg_per_drive = Utils.doubleParam(params, "hdd_embodied_kg_per_drive", hdd_embodied_kg_per_drive);
+        hdd_capacity_gb = Utils.doubleParam(params, "hdd_capacity_gb", hdd_capacity_gb);
+        ssd_embodied_kg_per_gb = Utils.doubleParam(params, "ssd_embodied_kg_per_gb", ssd_embodied_kg_per_gb);
+        storage_lifetime_hours = Utils.doubleParam(params, "storage_lifetime_hours", storage_lifetime_hours);
+
+        hdd_embodied_g_per_gb_hour =
+                hdd_embodied_kg_per_drive * 1000d / (hdd_capacity_gb * storage_lifetime_hours);
+        ssd_embodied_g_per_gb_hour = ssd_embodied_kg_per_gb * 1000d / storage_lifetime_hours;
+
         log.info("hdd_gb_coefficient: {}", hdd_gb_coefficient);
         log.info("ssd_gb_coefficient: {}", ssd_gb_coefficient);
+        log.info("hdd_embodied_g_per_gb_hour: {}", hdd_embodied_g_per_gb_hour);
+        log.info("ssd_embodied_g_per_gb_hour: {}", ssd_embodied_g_per_gb_hour);
 
         try {
             Map<String, Object> map = loadJSONResources("ccf/storage.json");
@@ -109,7 +167,7 @@ public Column[] columnsNeeded() {
 
     @Override
     public Column[] columnsAdded() {
-        return new Column[]{ENERGY_USED};
+        return new Column[]{ENERGY_USED, EMBODIED_EMISSIONS};
     }
 
     @Override
@@ -180,6 +238,10 @@ private void computeEnergy(Row row, Map<Column, Object> enrichedValues, boolean
         //  to kwh
         double energy_kwh = amount /1000 * coefficient * replication;
         enrichedValues.put(ENERGY_USED, energy_kwh);
+        // the replication factor applies to the hardware as well as to the energy: the same bytes
+        // occupy that many times more physical drives, and so that much more embodied carbon
+        double embodied_coefficient = isHDD ? hdd_embodied_g_per_gb_hour : ssd_embodied_g_per_gb_hour;
+        enrichedValues.put(EMBODIED_EMISSIONS, amount * embodied_coefficient * replication);
     }
 
     /**