[Storage] Add support for storing some data in External EEPROM

[tonhuisman]

Resolves #1011
Resolves #2291
Resolves #5602
Forum requests: here here here here here here

Features:

• [Storage] Add settings for External EEPROM, supported models: AT24Cxx (xx: 32, 64, 128, 256, 512, 1024 and possibly AT24C2048), and also FRAM models: MB85RCxx (xx: 64, 128, 256, 512, 1M).
• Save numeric values in a 'slot' (float or double resolution, depending on build, always using double storage format) using WriteEE,<slot>,<value>.
• Save numeric values in external RTC (Realtime clock) SRAM for supported chips (DS1307 = 56 bytes/7 slots, DS3232 = 236 bytes/29 slots) using WriteRTC,<slot>,<value>
• Erase entire 'slots' range using command WriteEE,erase,erase or WriteRTC,erase,erase.
• Read numeric values from a 'slot' using [ReadEE#<slot>] generic Rules variables syntax
• Fetch values from external RTC SRAM using [ReadRTC#<slot>]. No checksum is calculated as there are not many slots available.
• Fetch max. number of available slots using [ReadEE#max]
• View current used 'slots' in the External EEPROM/RTC values page (/eepromvars endpoint), available from the Tools page. For 0-values no hex value display is included.
• Values in RTC SRAM can be viewed on the EEPROM/RTC page, from the Tools page.
• The RTC SRAM storage feature can be enabled independent from the EEPROM feature (in Custom builds, like for ESP8266)
  • FEATURE_EEPROM_EXTERNAL for enabling the External EEPROM storage.
  • FEATURE_RTC_SRAM_STORAGE for enabling the RTC SRAM storage.
• [CRC] Calculate checksum reading from a function
• Add check for write-ability of EEPROM, and set to ReadOnly mode if we can't write (WP (write-protect) can be enabled by a jumper on some boards)
• Add command WriteEE,Check,WP to re-evaluate the write-ability of the EEPROM
• Get Write-protected status using [ReadEE#WP], 1 = write-protected
• Show Write-protected state on Hardware page
• Show the available nr. of EEPROM slots on the Hardware page
• Separate EEPROM (AT24Cxxx) and FRAM (MB85RCxxx) models in the type selector (general term is still EEPROM)
• All EEPROM and RTC SRAM related code moved to new ESPEasy::eeprom namespace for better structure and maintainability.
• Reduce build size for BUILD_NO_DEBUG builds, so ESP8266 will fit. Still not including these features! See here

Note: I removed some of the original features, leaving only the saving and restoring of values in 'slots', making nearly the full storage space, minus 128 bytes in EEPROMs for future use, available. In RTC SRAM, all bytes are available for supported chips.

Original features, later removed:

• Save Task values that are stored in RTC also in EEPROM, when enabled per value in Device page
• Add settings on the Hardware page for restoring settings from EEPROM at Cold and/or Warm boot
• Add a checkbox per Task value to enable/disable storing the value in EEPROM, Disabled by default, only visible if the EEPROM (or FRAM) is enabled and available.
• Store and check some versioning and system parameters in EEPROM, avoiding incompatible data to be restored.
• Saving of UserVars to EEPROM is now handled in the 'background', to avoid a large time-penalty when a variable is saved in EEPROM. A save interval in seconds can be set to spread the load a bit more (see screenshot)

TODO:

• External EEPROM data layout to be determined
• Implement actual storing of data to EEPROM and RTC SRAM
  • Rules variables via command (WriteEE and WriteRTC commands)
• Implement loading/restoring from EEPROM and RTC SRAM
  • Rules variables via variable expansion ([ReadEE#<slot>] and [ReadRTC#<slot>])
• ? Export values to json ?
• Testing
  • EEPROM
  • FRAM
  • RTC SRAM
• Update documentation
• Add support for storage available in connected RTC chip. See Storing Generic Pulse Counter Values ​​in the RTC DS1307 SRAM #5602
• Fix ESP8266 build (size issues)

Hardware page configuration: (updated: 2026-04-06)

[chromoxdor]

@tonhuisman I got my EEPROMs today. Tell me when and what needs to be tested or how I can be of assistance.

[tonhuisman]

Ah, great 😃

You can of course test if the Task Values and Rules variables (writeEE,n,value / [readEE#n]) (NB: n = range 0..MAX-1 !) are restored correctly once you have enabled the corresponding settings on the Hardware page (see screenshot above). Restore should be successful after a power cycle 😄 to simulate the Cold boot situation.
And once you have some Rules values stored, you can view the /eepromvars endpoint, and even add ?tasks=1 to the url to also see the stored values per task.
For Cache values (C016) there's no 'restore' or 'view' yet.

NB: I'm thinking of limiting the Rules vars to f.e. 1024 (less on smaller EEPROM sizes!), so more space will be available for Cache values,

[chromoxdor]

Ok... i am already in testing mode (I didn't wait for your reply :P )

Everything seems to work fine. (I have everything covered you mentioned)
But two things for now.

1. Taskvalues are written to the EEPROM when they generate an Event. But often (at least in my case), I have multiple tasks that have an interval of one second. This would be a little bit too many write cycles for an EEPROM over time (not for an FRAM, of course). Maybe it would be better to choose which tasks will be written to the EEPROM. Just keep the layout for all tasks on the EEPROM but only write when enabled in the task? What do you think?
2. Why not showing zero values on /eepromvars? It shows, that this slot has been already used. (0 vs. NaN)

[chromoxdor]

NB: I'm thinking of limiting the Rules vars to f.e. 1024 (less on smaller EEPROM sizes!), so more space will be available for Cache values,

I see no reason not to do it that way unless somebody really wants to write a lot of vars :)

[tonhuisman]

1. Values are only written if the data is different from what's already stored, to avoid wearing out the EEPROM
2. Showing all values would generate a page of about 'a mile long', so when using value 1, 128 and 256 you would really be searching for your value..., especially on a small device (phone?) On FRAM, the values are initially 0, on EEPROM they are initially NaN, so both are ignored. Managing the Rules variables is the users' task, not ESPEasy, IMHO

[chromoxdor]

Would it be complicated to make the number of Rules var slots user selectable and put them at the end of the EEPROM and use the rest for C16? The ammount of user selectable slots could be also written to the EEPROM insted of storing it on the flash....

2. On FRAM, the values are initially 0, on EEPROM they are initially NaN, so both are ignored.

I get it. Would make a user-selectable amount of slots even more useful :)
But at least for the EEPROM, it would be OK to differentiate between 0 and NaN for convenience. (at least for the user.. not for the code size ;) )

[chromoxdor]

1. Values are only written if the data is different from what's already stored, to avoid wearing out the EEPROM

But for an analog reading of a LDR it still could mean 1 write per second

[tonhuisman]

I'll see where too put that extra checkbox per value 😅 I have already found a bit per value so I can store this setting 😎

Btw, I think it should be Enabled by default, and user can decide to turn it off

Ah, and there's a particular reason not to skip values, as I'm also storing the Checksum for this data, and validating that against the calculated checksum, so when skipping a value the checksum won't be correct anymore 😱

[chromoxdor]

I'll see where too put that extra checkbox per value 😅 I have already found a bit per value so I can store this setting 😎

Maybe it could also be stored on the EEPROM/FRAM?

[tonhuisman]

I'll see where too put that extra checkbox per value 😅 I have already found a bit per value so I can store this setting 😎

Maybe it could also be stored on the EEPROM/FRAM?

Hmm, not sure if that's such a good plan

[chromoxdor]

Hmm, not sure if that's such a good plan

Me neither :) Just thinking out loud...

[tonhuisman]

But for an analog reading of a LDR it still could mean 1 write per second

What to do for Taskvalues that are not stored, store 0 or NaN (once)? So the storage will be 'clean' for disabled values?

[chromoxdor]

What to do for Taskvalues that are not stored, store 0 or NaN (once)? So the storage will be 'clean' for disabled values?

Hmmm.. very good question. I would make sense to use NaN. But that can not be represented (only when choosing a specific other value for it i guess?)
The only thing I could think of would then be to store en/disabled in a bitmap on the first bytes to have a flag... but my knowledge is limited. 🤷‍♂️

Edit: ahhh since it is float 0xFFFFFFFF for NaN would be it right?

[chromoxdor]

The GPIO state could be saved in a bitmap and another bitmap for "en/disabled saving state".

[chromoxdor]

I am probably stating the obvious by the memory layout could be like:

Offset (bytes)	Size (bytes)	Purpose	Notes
0	8	GPIO enabled/disabled state	64 bits → 64 GPIOs max
8	8	GPIO state	64 bits → 64 GPIOs max
16	128	Task values	Fixed block
144	N × 4	Rules variables (slots)	User-selectable count of 4-byte values
...	Remaining	C016	Uses rest of EEPROM

[tonhuisman]

For GPIO pins we want to provide a way to store the states for MCP and PCF GPIOs, so 64 bits isn't going to cut it. And maybe we should store some more data, like PWM data and/or input/output, haven't coded anything for that yet. It's still slow-cooking in a background thread in my head 🤓

And the same is the case for String type variables; rewriting the entire data blob when a string length changes isn't very efficient, and neither is leaving 'growing space' between entries, as that'll often prove to be too small, causing another blob-rewrite, or leave much unused space 🫨

[chromoxdor]

To be able to compare data, the current EEPROM-data must be read to compare to the new data.

Sorry, I don't want to be a pain in the ass 😁 but it seems i don't get it.
Assume I don't need the ability to write task values and disable them all. (only maybe the var slots in rules) why would i need a comparison of anything for a task? Every active task running causes a read that is unnecessary then...

[tonhuisman]

First thing that's done is a validity-check of the current EEPROM: https://github.com/letscontrolit/ESPEasy/pull/5382/files#diff-87033636f7a2095187789ef49c13c2e2767c2e1d8eb19618aa17d932dd58ab53R175-R182

That involves a 'couple of bytes' being read, but those are not counted, and even if the params are not OK, the writes to update the params are not counted.
I could add a static or global bool to store the first params state, and never read/check it again (similar to the read-only test I do at startup), but you can in fact replace the EEPROM while the unit is running, and it could write (valuable) UserVar data in a clean EEPROM without correct params data, and then it won't restore that as that params-state is required to be OK before doing a restore on Cold or Warm boot 😨

NB: AFAIK, reading from an EEPROM is harmless to the device, as opposed to writing, that will eventually wear out the chip. (Most EEPROMs are specced at ~10⁶ writes per byte, though I haven't validated that.)

[tonhuisman]

Every active task running causes a read that is unnecessary then...

There currently is scheduled only a single write action per task update, if a new update comes while the data is still being processed, the new write action is ignored, as it will be handled on the next 'run'.

[chromoxdor]

NB: AFAIK, reading from an EEPROM is harmless to the device, as opposed to writing, that will eventually wear out the chip. (Most EEPROMs are specced at ~106 writes per byte, though I haven't validated that.)

I know. It’s not so much about the cycles but about the time that the read occupies the I2C bus. If you have multiple tasks with a 1-sec interval and let this be multiple I2C devices, then there is a lot of stuff going on on the bus. I even have some tasks with a 100-ms (or was it 50ms...) interval getting the analog data of a light sensor (LDR)... I would think that theoretically could lead to issues.
I would maybe, for swapping between EEPROMs, do an initialization option to prepare it for the new device....

[chromoxdor]

and even if the params are not OK, the writes to update the params are not counted.

to get a better understanding.. what params are we talking about?

[tonhuisman]

what params are we talking about?

See: https://github.com/letscontrolit/ESPEasy/pull/5382/files#diff-fa9213718305321296c98a66921379e1f7d2b6e78f02155044ff828739556a8cR77-R84

• Version (EEPROM content-version, currently 1)
• Max. tasks
• Vars per tasks
• RTC Cache address (in EEPROM, when included in the build)
• Pinstate address (in EEPROM)

[chromoxdor]

Would it be a problem to treat a task, where all "Eeprom" fields are disabled as an disabled task?
From the EEPRM point of view it would make no difference right....

[chromoxdor]

BTW: 0 is still not being displayed as a slot value in /eepromvars.

[tonhuisman]

Would it be a problem to treat a task, where all "Eeprom" fields are disabled as an disabled task? From the EEPRM point of view it would make no difference right....

Well, yes, that's the intention, but I now see that some improvements can be applied in the code. There's also the feature that I write a NaN value if the taskvalue is not enabled, but currently that's re-checked every time for an enabled task. Will fix that soon-ish.

[tonhuisman]

@chromoxdor and others that are building this PR locally:
⚠️
There has been a breaking change in the settings for enabling the storage of task values in EEPROM (there was overlap with another setting I added elsewhere), so please check your configurations! (this is of course the risk of working with PR code: Things may change or break 🫣)
⚠️

[chromoxdor]

Thanks for the heads up! What’s the other setting?

[tonhuisman]

What’s the other setting?

MQTT State Class per Value