Persistent Data APIs

Persistent Data

The PersistentData class stores one data structure (a settings struct, calibration data, a paired-device list) in a file on any filesystem reachable through the VFS, tracks the schema version of what is on disk in a small header (magic, version, a portable layout fingerprint, length, CRC32) and migrates older files forward through a chain of migrate() steps when the structure changes. Saves are atomic (temporary file plus rename), every operation reports a std::error_code (PersistentDataErrc), and encode() / decode() expose the same bytes for other storage backends.

Code examples for the Persistent Data API are provided in the persistent_data example folder.

API Reference

Header File

Classes

template<persistent_data::VersionedType... Versions>
class PersistentData : public espp::BaseComponent

File-backed storage for one data structure with schema versioning and forward migration.

Stores a serializable aggregate (see the serialization component) in a single file on any filesystem reachable through the VFS (LittleFS, SPIFFS, SD card). Every file carries a small header with a magic number, the schema version of the data it holds, a portable fingerprint of the type’s layout, the payload length and a CRC32, so a load distinguishes “no file yet”, “corrupt or truncated”, “written by newer firmware”, “version number reused for a different

layout” and “old version, migrated” instead of guessing by trying to parse. The format is identical on every target and on the host, so a PC tool can read and write a device’s file.

Versions are given as a type list, newest first; the first type is the current one. Each type declares `static constexpr uint32_t VERSION` (strictly decreasing along the list) and every type except the last provides `static T migrate(const Older &)` from the type after it. A file of an older version is decoded as that version and walked toward the front of the list one step at a time, so each version only knows how to come from the one before it. A new version is prepended; dropping support for the oldest version is removing the last entry: such files then load with PersistentDataErrc::unknown_version and the application decides what to do (typically load_or_create()).

Saves are atomic: the data is written to `<file>.tmp`, synced, then renamed over the file, so a reset mid-save keeps the previous contents.

All methods are thread-safe; get() returns a copy and update() runs the caller’s function and the save under the same lock. A save blocks other callers for the duration of the flash write. For convenience the object converts to a copy of the data (get()) and can be assigned from it (set()); saving stays an explicit call because it can fail.

Persistent Data Example

    // The type list is newest first; the first type is the current one. A file
    // written by any listed version loads and is migrated to Settings.
    using PersistentSettings = espp::PersistentData<Settings, SettingsV2, SettingsV1>;
    std::error_code ec;
    PersistentSettings settings({
        .file_path = settings_path,
        .auto_load = false, // we want to handle "no file yet" ourselves below
        .log_level = espp::Logger::Verbosity::INFO,
    });

    // load the file, or create it from defaults if there is none; any other
    // error (corrupt file, newer firmware's file) is reported, not papered over
    if (!settings.load_or_create(Settings{}, ec)) {
      logger.error("Could not load or create settings: {}", ec.message());
    }
    logger.info("Loaded (file version {}): {}", settings.loaded_version(), settings.get());

    // read-modify-write under one lock, saved atomically
    settings.update([](Settings &s) { s.boot_count++; }, ec);
    if (ec) {
      logger.error("Could not save settings: {}", ec.message());
    }

    // get() returns a copy; set() only changes memory, save() writes it
    auto copy = settings.get();
    copy.favorites = {"red", "green"};
    settings.set(copy);
    settings.save(ec);

    // the same through the conversion / assignment operators
    Settings s = settings; // == settings.get()
    s.display.brightness = 1.0f;
    settings = s; // == settings.set(s)
    settings.save(ec);

Migration Example

    // Write a version-1 file the way the first firmware would have, then load
    // it with the full version list: it comes back migrated v1 -> v2 -> v3 and
    // (save_after_migration, the default) the file is rewritten as version 3.
    {
      espp::PersistentData<SettingsV1> old_firmware(
          {.file_path = settings_path, .auto_load = false});
      old_firmware.save(SettingsV1{.brightness = 80, .device_name = "legacy"}, ec);
      check("v1 file written", !ec);

      PersistentSettings settings({.file_path = settings_path});
      check("v1 file loads", settings.is_loaded() && settings.loaded_version() == 1);
      check("v1 file was migrated", settings.was_migrated());
      auto s = settings.get();
      check("migration kept the name", s.device_name == "legacy");
      check("migration converted brightness 80 -> 0.80", s.display.brightness == 0.80f);
      check("migration filled v2 defaults", s.display.color_rgb == 0x00FF00 && s.favorites.empty());
      check("migration filled v3 defaults", s.boot_count == 0);

      PersistentSettings again({.file_path = settings_path});
      check("file was rewritten as the current version",
            again.loaded_version() == PersistentSettings::CURRENT_VERSION && !again.was_migrated());
    }
Template Parameters:

Versions – The schema versions, newest first; the first is the current type. Each must be an aggregate the serialization component can (de)serialize.

Public Types

using Current = typename List::Current

The current (newest) version type: what get(), set() and save() work with.

Public Functions

inline explicit PersistentData(const Config &config)

Construct the PersistentData and, if configured, load the file.

Parameters:

config – The configuration.

inline const std::filesystem::path &file_path() const

The file this object loads from and saves to.

inline bool exists() const

Whether a file exists at the configured path.

inline bool is_loaded() const

Whether the data in memory came from, or was saved to, the file.

False until a load or save succeeds; get() then returns a default-constructed Current.

inline uint32_t loaded_version() const

Schema version found in the file by the last successful load.

Returns:

The version, or 0 if nothing was loaded (a save does not change it).

inline bool was_migrated() const

Whether the last successful load migrated an older file.

inline Current get() const

Get a copy of the data in memory.

Note

Does not read the file; call load() for that.

inline void set(const Current &data)

Replace the data in memory.

Note

Does not write the file; call save() for that, or use save(data) / update().

inline PersistentData &operator=(const Current &data)

Assign new data, the same as set(): `settings = Settings{…};`.

Note

In memory only; save() still writes the file (and reports errors).

inline operator Current() const

Convert to a copy of the data in memory, the same as get(): `Settings s = settings;`.

inline bool load(std::error_code &ec)

Load the file, migrating it to the current version if needed.

On success the data in memory is replaced. On failure it is left as it was and ec says why: file_not_found, io_error, file_too_large, bad_header, bad_checksum, unknown_version or deserialize_failed.

Parameters:

ec – [out] The error, if any.

Returns:

True if the data was loaded.

inline bool load_or_create(const Current &defaults, std::error_code &ec)

Load the file, or create it from `defaults` if there is none.

Calls load(); if that fails with file_not_found, `defaults` is saved and becomes the data in memory. Any other load error is returned untouched (a corrupt file is not silently replaced; call save(defaults) or erase() for that).

Parameters:

  • defaults – The data to save when no file exists.

  • ec – [out] The error, if any.

Returns:

True if the data in memory is now valid (loaded or created).

inline bool save(std::error_code &ec)

Save the data in memory to the file (atomically).

Parameters:

ec – [out] The error, if any (serialize_failed or io_error).

Returns:

True if the file was written.

inline bool save(const Current &data, std::error_code &ec)

Replace the data in memory and save it.

Parameters:

  • data – The new data.

  • ec – [out] The error, if any.

Returns:

True if the file was written.

template<typename Fn>
inline bool update(Fn &&fn, std::error_code &ec)

Modify the data in memory and save it, under one lock.

Parameters:

  • fn – Callable taking `Current &`; it may return void, or a bool where false means “do not save” (the in-memory change is kept).

  • ec – [out] The error, if any.

Returns:

True if the file was written (false also when fn declined).

inline bool erase(std::error_code &ec)

Remove the file (and a leftover temporary file).

The data in memory is kept; is_loaded() becomes false.

Parameters:

ec – [out] The error, if any.

Returns:

True if no file exists afterwards (also when there was none).

inline const std::string &get_name() const

Get the name of the component

Note

This is the tag of the logger

Returns:

A const reference to the name of the component

inline void set_log_tag(const std::string_view &tag)

Set the tag for the logger

Parameters:

tag – The tag to use for the logger

inline espp::Logger::Verbosity get_log_level() const

Get the log level for the logger

See also

Logger::Verbosity

See also

Logger::set_verbosity

Returns:

The verbosity level of the logger

inline void set_log_level(espp::Logger::Verbosity level)

Set the log level for the logger

See also

Logger::Verbosity

See also

Logger::set_verbosity

Parameters:

level – The verbosity level to use for the logger

inline void set_log_verbosity(espp::Logger::Verbosity level)

Set the log verbosity for the logger

See also

set_log_level

See also

Logger::Verbosity

See also

Logger::set_verbosity

Note

This is a convenience method that calls set_log_level

Parameters:

level – The verbosity level to use for the logger

inline espp::Logger::Verbosity get_log_verbosity() const

Get the log verbosity for the logger

See also

get_log_level

See also

Logger::Verbosity

See also

Logger::get_verbosity

Note

This is a convenience method that calls get_log_level

Returns:

The verbosity level of the logger

inline void set_log_rate_limit(std::chrono::duration<float> rate_limit)

Set the rate limit for the logger

See also

Logger::set_rate_limit

Note

Only calls to the logger that have _rate_limit suffix will be rate limited

Parameters:

rate_limit – The rate limit to use for the logger

Public Static Functions

static inline bool encode(const Current &data, std::vector<uint8_t> &out, std::error_code &ec)

Encode data in the file format (header + payload) without writing it.

For storing the same bytes elsewhere (NVS blob, a transport); decode() reads them back, migrations included.

static inline bool decode(std::span<const uint8_t> bytes, Current &out, persistent_data::LoadInfo &info, std::error_code &ec)

Decode bytes produced by encode() (any supported version) into `out`.

Parameters:

  • bytes – [in] The encoded bytes.

  • out – [out] The decoded, migrated data.

  • info – [out] Version found and whether it was migrated.

  • ec – [out] The error, if any.

static inline uint32_t layout_fingerprint()

Portable fingerprint of the current type’s serialized layout (what save() writes into the header; see layout::fingerprint).

Public Static Attributes

static constexpr uint32_t CURRENT_VERSION = List::current_version

Schema version written by save().

static constexpr size_t HEADER_SIZE = persistent_data::HEADER_SIZE

Bytes of header in front of the payload in every file.

struct Config

Configuration for the PersistentData.

Public Members

std::filesystem::path file_path

File to load from / save to.

bool auto_load = {true}

Call load() in the constructor (its errors are logged, a missing file at info level)

bool save_after_migration = {true}

After loading an older version, save the migrated data so the next load is a plain read

size_t max_file_size = {64 * 1024}

Files larger than this are rejected before being read.

Logger::Verbosity log_level = {Logger::Verbosity::WARN}

Verbosity of the logger.

Header File