Non-volatile storage library

Introduction

The Non-volatile storage (NVS) library is designed to store key-value pairs in flash memory. This is a port of the Espressif Esp32 IDF nvs_flash library for Sming. It differs from the original in several ways:

  • Reworked to provide a consistent C++ interface

  • Methods return bool, and error code for last operation can be obtained via nvs_errno

  • IDF nvs::Storage has been renamed as Container; this handles data for a single Storage::Partition.

  • Containers may not be closed or erased if there are open handles - calls will fail. (IDF allows this, but invalidates all open handles in the process. It consumes RAM by keeping a list of all open handles for this purpose.)

  • The C API is retained as it’s required by esp_wifi. ‘C’ handles are created by casting the nvs::Handle pointer to a uint32_t. (IDF consumes RAM using a separate list for this.)

Underlying storage

The library performs all low-level reading, writing and erasing via the partition API: it does not access flash memory directly.

Partitions must have a type/subtype of data / nvs.

Storage block size is fixed at 4096 bytes, compatible with the ESP flash API.

Other types of storage device may be used. See the Storage Management library for details.

Note

If an NVS partition is truncated (by a change to the partition table), its contents should be erased.

Note

NVS works best for storing many small values, rather than a few large values of the type ‘string’ and ‘blob’. If you need to store large blobs or strings, consider using a regular file system.

The library also uses RAM for caching entries. This can be minimised by keeping the partition size small.

Keys and values

NVS operates on key-value pairs. Keys are ASCII strings; the maximum key length is currently 15 characters. Values can have one of the following types:

  • integer types: uint8_t, int8_t, uint16_t, int16_t, uint32_t, int32_t, uint64_t, int64_t

  • zero-terminated string

  • variable length binary data (blob)

Note

String values are currently limited to 4000 bytes. This includes the null terminator. Blob values are limited to 508000 bytes or (97.6% of the partition size - 4000) bytes, whichever is lower.

Keys are required to be unique. Assigning a new value to an existing key works as follows:

  • if the new value is of the same type as the old one, value is updated

  • if the new value has a different data type, an error is returned

Data type check is also performed when reading a value. Read operations will fail if the requested data type does not match that of the stored value.

Containers, Handles and Namespaces

A nvs::Container is used to manage storage within a single partition.

A container stores key-value pairs from different sources, each identified by a unique namespace.

Namespace names follow the same rules as key names, with a maximum length of 15 characters.

Containers are normally accessed using a nvs::Handle, which provides read-only or read/write access to a single namespace.

Handles are created using a call to nvs::Container::openHandle().

Security, tampering, and robustness

NVS is not directly compatible with the ESP32 flash encryption system. However, data can still be stored in encrypted form if NVS encryption is used together with ESP32 flash encryption. See Encryption for details.

If NVS encryption is not used, it is possible for anyone with physical access to the flash chip to alter, erase, or add key-value pairs. With NVS encryption enabled, it is not possible to alter or add a key-value pair and get recognized as a valid pair without knowing corresponding NVS encryption keys. However, there is no tamper-resistance against the erase operation.

The library does try to recover from conditions when flash memory is in an inconsistent state. In particular, one should be able to power off the device at any point and time and then power it back on. This should not result in loss of data, except for the new key-value pair if it was being written at the moment of powering off. The library should also be able to initialize properly with any random data present in flash memory.

Internals

See NVS Internals.

API

namespace nvs

Typedefs

typedef std::unique_ptr<Handle> HandlePtr
using TPageList = intrusive_list<Page>
using TPageListIterator = TPageList::iterator
using PartitionPtr = std::unique_ptr<Partition>

Enums

enum VerOffset

Used to recognize transient states of a blob. Once a blob is modified, new chunks with the new data are written with a new version. The version is saved in the highest bit of Item::chunkIndex as well as in Item::blobIndex::chunkStart. If a chunk is modified and hence re-written, the version swaps: 0x0 -> 0x80 or 0x80 -> 0x0.

Values:

enumerator VER_0_OFFSET
enumerator VER_1_OFFSET
enumerator VER_ANY
enum ItemType

The possible blob types

Values:

enumerator UNK
enumerator U8
enumerator I8
enumerator U16
enumerator I16
enumerator U32
enumerator I32
enumerator U64
enumerator I64
enumerator VARIABLE

Marker for start of variable-sized types.

enumerator STR
enumerator SZ
enumerator BLOB
enumerator BLOB_DATA
enumerator BLOB_IDX
enumerator ANY
enum OpenMode

Mode of opening the non-volatile storage.

Values:

enumerator ReadOnly
enumerator ReadWrite

Functions

inline constexpr size_t getAlignment(ItemType type)
inline bool isVariableLengthType(ItemType type)
template<typename T, typename std::enable_if<std::is_integral<T>::value, void*>::type = nullptr>
constexpr ItemType itemTypeOf()
template<typename T>
constexpr ItemType itemTypeOf(const T &var)

Obtain ItemType for given variable.

Template Parameters

T – Type of variable

Parameters

var – Variable to evaluate

inline HandlePtr openHandle(const String &partName, const String &nsName, OpenMode openMode)

Creating Handle object for a specified partition/namespace.

Parameters

  • partName – Name of partition

  • nsName – Namespace

  • openMode –

Returns

HandlePtr –

inline bool openContainer(const String &name)

Open and register storage container for given partition.

Partition manager tracks all open containers.

Parameters

name – Name of partition

Returns

bool – true on success, false if, for example, partition doesn’t exist

inline bool closeContainer(const String &name)

Close a container.

Parameters

name – Name of partition

Returns

bool – true on success, false if, for example, there are open handles on the container

Variables

PartitionManager partitionManager
class Container : public intrusive_list_node<Container>
#include <Container.hpp>

Manages data storage within a single NVS partition.

Read an entry

Method template to read an entry and deduce the data type

bool readItem(uint8_t nsIndex, ItemType datatype, const String &key, void *data, size_t dataSize)

Read an item into a user-supplied buffer, specify data type explicitly.

String readItem(uint8_t nsIndex, ItemType datatype, const String &key)

Read an item into a String object, specify data type explicitly.

Erase single value from container

Erase item matching data type and key

param datatype

May be ItemType::ANY to match any entry

param key

May be nullptr to match any key

inline bool eraseItem(uint8_t nsIndex, const String &key)

Erase item matching name only.

Parameters

key – May be nullptr to match any key

Public Functions

bool init()

Initialise container by reading and caching partition contents.

Fails if there are any open handles.

inline bool isValid() const

Determine if this container has been initialised successfully.

bool createOrOpenNamespace(const String &nsName, bool canCreate, uint8_t &nsIndex)

Open an existing namespace or create a new one.

Parameters

  • nsName – Namespace to open/create

  • canCreate – true if a new namespace entry may be created

  • nsIndex – On return, contains index of namespace

Returns

bool –

HandlePtr openHandle(const String &nsName, OpenMode openMode)

Open a new handle on this storage container.

Parameters

  • nsName – Namespace to qualify all entries

  • openMode – If read-only, all write operations will fail

Returns

HandlePtr – Created handle

bool getItemDataSize(uint8_t nsIndex, ItemType datatype, const String &key, size_t &dataSize)

Get size of stored value in bytes.

bool eraseNamespace(uint8_t nsIndex)

Erase all entries matching a single namespace.

inline Partition &partition() const

Get reference to underlying storage partition.

void debugDump()

Print contents of container for debugging.

void debugCheck()

Run extended check on container contents.

bool fillStats(nvs_stats_t &nvsStats)

Fetch statistics for this container.

bool calcEntriesInNamespace(uint8_t nsIndex, size_t &usedEntries)

Determine number of used entries for a given namespace.

inline size_t handleCount() const

Get number of open handles.

Certain container operations are prohibited whilst there are open handles.

bool checkNoHandlesInUse()

Check and set error if any handles are in use.

Called before performing certain operations to ensure last error is set appropriately. A debug message is also logged.

inline size_t pageCount() const

Determine number of pages in use.

inline const NamespaceList &namespaces() const

Get reference to list of registered namespaces.

class EncryptedPartition : public nvs::Partition
#include <EncryptedPartition.hpp>
class Handle
#include <Handle.hpp>

A handle allowing nvs-entry related operations on the NVS.

Note

The scope of this handle is the namespace of a particular partition. Outside that scope, nvs entries can’t be accessed/altered.

Set value for simple integral or enum type key

Method template to determine storage type based on provided value

param key

[in] Key name. Maximal length is (NVS_KEY_NAME_MAX_SIZE-1) characters. Shouldn’t be empty.

param value

[in] The value to set. Allowed types are the ones declared in ItemType as well as enums. Note that enums lose their type information when stored in NVS. Ensure that the correct enum type is used during retrieval with getItem!

return

  • ESP_OK if value was set successfully

  • ESP_ERR_NVS_READ_ONLY if storage handle was opened as read only

  • ESP_ERR_NVS_INVALID_NAME if key name doesn’t satisfy constraints

  • ESP_ERR_NVS_NOT_ENOUGH_SPACE if there is not enough space in the underlying storage to save the value

  • ESP_ERR_NVS_REMOVE_FAILED if the value wasn’t updated because flash write operation has failed. The value was written however, and update will be finished after re-initialization of nvs, provided that flash operation doesn’t fail again.

  • ESP_ERR_NVS_VALUE_TOO_LONG if the string value is too long

inline bool setItem(ItemType datatype, const String &key, const void *data, size_t dataSize)

Sets value given specific storage type.

Set value for a String key type

Fails if key exists and is of a different type. Maximum string length (including null character) is 4000 bytes.

brief get value for given key

Fails if key does not exist or the requested variable type doesn’t match the stored type. On error, value remains unchanged.

param key

[in] Key name. Maximal length is (NVS_KEY_NAME_MAX_SIZE-1) characters. Shouldn’t be empty.

param value

The output value. All integral types which are declared in ItemType as well as enums are allowed. Note however that enums lost their type information when stored in NVS. Ensure that the correct enum type is used during retrieval with getItem!

return

  • ESP_OK if the value was retrieved successfully

  • ESP_ERR_NVS_NOT_FOUND if the requested key doesn’t exist

  • ESP_ERR_NVS_INVALID_NAME if key name doesn’t satisfy constraints

  • ESP_ERR_NVS_INVALID_LENGTH if length is not sufficient to store data

Set variable-length binary values (Binary Large OBject)

set variable length binary value for given key

Note

compare to nvs_set_blob in nvs.h

param key

[in] Key name. Maximal length is (NVS_KEY_NAME_MAX_SIZE-1) characters. Shouldn’t be empty.

param blob

[in] The blob value to set.

param len

[in] length of binary value to set, in bytes; Maximum length is 508000 bytes or (97.6% of the partition size - 4000) bytes whichever is lower.

return

  • ESP_OK if value was set successfully

  • ESP_ERR_NVS_READ_ONLY if storage handle was opened as read only

  • ESP_ERR_NVS_INVALID_NAME if key name doesn’t satisfy constraints

  • ESP_ERR_NVS_NOT_ENOUGH_SPACE if there is not enough space in the underlying storage to save the value

  • ESP_ERR_NVS_REMOVE_FAILED if the value wasn’t updated because flash write operation has failed. The value was written however, and update will be finished after re-initialization of nvs, provided that flash operation doesn’t fail again.

  • ESP_ERR_NVS_VALUE_TOO_LONG if the value is too long

Read variable-length value

Read value into user-provided buffer

Fails if key does not exist or the requested variable type doesn’t match the stored type.

Use getString() for reading NUL-terminated strings, and getBlob for arbitrary data structures.

param key

Key name. Maximal length is (NVS_KEY_NAME_MAX_SIZE-1) characters. Shouldn’t be empty.

param outValue

Buffer to store value. Must have sufficient room for NUL terminator. On error, remains unchanged.

param length

Size of buffer.

return

  • ESP_OK if the value was retrieved successfully

  • ESP_ERR_NVS_NOT_FOUND if the requested key doesn’t exist

  • ESP_ERR_NVS_INVALID_NAME if key name doesn’t satisfy constraints

  • ESP_ERR_NVS_INVALID_LENGTH if length is not sufficient to store data

inline String getString(const String &key)

Read value into String object.

Parameters

key –

Returns

String – On error, will be invalid, i.e. bool(String) evaluates to false.

inline bool getBlob(const String &key, void *outValue, size_t len)

Read BLOB value into user-provided buffer.

Parameters

  • key –

  • outValue – Buffer to store value

  • len – Size of buffer

Returns

bool –

inline String getBlob(const String &key)

Read BLOB value into String object.

Parameters

key –

Returns

String –

Public Functions

inline bool getItemDataSize(ItemType datatype, const String &key, size_t &dataSize)

Look up the size of an entry’s data.

Parameters

  • datatype – Must match the type of stored data

  • key –

  • dataSize – For strings, includes the NUL terminator

Returns

bool –

inline bool eraseItem(const String &key)

Erases an entry.

inline bool eraseAll()

Erases all entries in the current namespace.

inline bool commit()

Commits all changes done through this handle so far.

inline bool getUsedEntryCount(size_t &usedEntries)

Determine number of used entries in the current namespace.

Parameters

usedEntries –

Returns

- ESP_OK if the changes have been written successfully. Return param used_entries will be filled valid value.

  • ESP_ERR_NVS_NOT_INITIALIZED if the storage driver is not initialized. Return param used_entries will be filled 0.

  • ESP_ERR_INVALID_ARG if nvs_stats equal to NULL.

  • Other error codes from the underlying storage driver. Return param used_entries will be filled 0.

inline Container &container()

Get reference to storage container.

Can be used to perform operations on the entire storage container, not just in the current namespace.

class HashList
#include <HashList.hpp>
class Item
#include <Item.hpp>
class ItemInfo
#include <ItemIterator.hpp>
class ItemIterator : public std::iterator<std::forward_iterator_tag, ItemInfo>
#include <ItemIterator.hpp>
class Page : public intrusive_list_node<Page>
#include <Page.hpp>
class PageManager
#include <PageManager.hpp>
class Partition
#include <Partition.hpp>

Implementation of Partition for NVS.

It is implemented as an intrusive_list_node to easily store instances of it. NVSStorage and NVSPage take pointer references of this class to abstract their partition operations.

Subclassed by nvs::EncryptedPartition

class PartitionManager
#include <PartitionManager.hpp>

Maintains list of all open storage containers.

Public Functions

Storage::Partition findPartition(const String &name)

Locate named partition and verify it’s a data/nvs type.

bool closeContainer(const String &name)

Close a storage container.

Parameters

name – Name of partition

Returns

bool – true on success, false if partition doesn’t exist or has open handles

Container *lookupContainer(const String &name)

Get container for given partition.

Parameters

name – Name of partition

Returns

Container* – Object owned by PartitionManager

HandlePtr openHandle(const String &partName, const String &nsName, OpenMode openMode)

Open a storage handle for a specified partition/namespace.

Parameters

  • partName – Name of partition

  • nsName – Namespace

  • openMode –

Returns

HandlePtr –

inline size_t handleCount() const

Get number of open handles.

Certain container operations are prohibited whilst there are open handles.

References

Used by

Environment Variables

  • ENABLE_NVS_DEBUGCHECK

  • ENABLE_NVS_ENCRYPTION