Storage Management

This Component provides support for using storage devices in a structured way by partitioning them into areas for specific uses. Partitions may can contain information such as:

Application (firmware) images
Filesystem(s)
Configuration/calibration/parameter data
Custom flash storage areas

A single partition table is located on the main flash device, Storage::spiFlash, and defines all partitions with a unique name and associated Storage::Partition::Type / Storage::Partition::SubType.

Hardware configuration

Each project has an associated Hardware configuration, specified by the HWCONFIG setting: this is a JSON file with a .hw extension.

For user convenience, the configuration file may contain comments however these are stripped before processing.

The build system locates the file by searching, in order:

{PROJECT_DIR} the root project directory
{SMING_HOME}/Arch/{SMING_ARCH}
{SMING_HOME}

Each architecture provides a standard configuration which defines such things as the partition table location and standard system partitions. Other configurations inherit from this by providing a base_config value.

You can list the available configs like this:

make hwconfig-list

This also shows the file path should you wish to view or edit it.

To select and view the resulting configuration, do this:

make hwconfig HWCONFIG=spiffs

or, to show the partition map:

make map HWCONFIG=spiffs

Note

You can set HWCONFIG in your project’s component.mk file, however as with other configuration variables it will be overridden by the cached value set on the command line.

For example, if you want to change from standard to standard-4m for your project, first add this line to your component.mk file:

HWCONFIG := standard-4m

Then either run make HWCONFIG=standard-4m or make config-clean.

Hardware configuration options

Commonly used settings can be stored in an option library for easier use. The library files are named options.json and located in the place as .hw files.

For example, we can do this:

make HWCONFIG=standard HWCONFIG_OPTS=4m,spiffs

This loads the ‘standard’ profile then merges the fragments found in the option library with the given names. This is how the standard-4m profile is constructed.

If using this approach, remember to updated your project’s component.mk with the desired settings, and verify the layout is correct using make map.

OTA updates

When planning OTA updates please check that the displayed partition map corresponds to your project. For example, the partition table requires a free sector so must not overlap other partitions.

Your OTA update process must include a step to write the partition table to the correct location. See Partition table migration.

It is not necessary to update the bootloader. See rBoot for further information.

Custom configurations

To customise the hardware configuration for a project, for example ‘my_project’:

Create a new configuration file in your project root, such as my_project.hw:
```
{
   "name": "My project config",
   "base_config": "spiffs",
   "options": ["vdd"]
}
```
You can use any available configuration as the base_config. Option fragments can be pulled in as shown. See Hardware configuration options.

If required, modify any inherited settings:

{
   "name": "My config",
   "base_config": "standard",
   "devices": {
      "spiFlash": {
         "speed": 80,
         "mode": "qio",
         "size": "2M"
      }
   },
   "partitions": {
      "rom0": {
         "address": "0x10000",
         "size": "0x80000"
      }
   }
}

This will adjust flash parameters (previously via SPI_SPEED, SPI_MODE and SPI_SIZE), and the location/size of the primary application partition.

Add any additional partitions:

{
   "name": "My config",
   "base_config": "standard-4m",
   "partitions": {
      "rom0": {
         "address": "0x10000",
         "size": "0x80000"
      },
      "spiffs1": {
            "address": "0x00280000",
            "size": "256K",
            "type": "data",
            "subtype": "spiffs",
            "filename": "$(FW_BASE)/spiffs1_rom.bin",
            "build": {
               "target": "spiffsgen",
               "files": "files/spiffs1"
            }
      }
   }
}

This adds a second SPIFFS partition, and instructs the build system to generate an image file for it using the files in the project’s files/spiffs1 directory.

Select the new configuration and re-build the project:
```
make HWCONFIG=my_project
```
You should also add this to your project’s component.mk file:
```
HWCONFIG := my_project
```
Program your device:
```
make flash
```
This will flash everything: bootloader, partition table and all defined partitions (those with a filename entry).

Note

The build system isn’t smart enough to track dependencies for partition build targets.

To rebuild these manually type:

make buildpart

These will be removed when make clean is run, but you can also clean them separately thus:

make part-clean

Partition maps

This is a concise view of your flash partitions. Display it like this:

make map

For the Basic Storage sample application, we get this:

Basic_Storage: Invoking 'map' for Esp8266 (debug) architecture
Partition map:
Device            Start       End         Size        Type      SubType   Name              Filename
----------------  ----------  ----------  ----------  --------  --------  ----------------  ------------
spiFlash          0x00000000  0x00001fff          8K                      Boot Sector
spiFlash          0x00002000  0x00002fff          4K                      Partition Table
spiFlash          0x00003000  0x00003fff          4K  data      phy       phy_init          $(FLASH_INIT_DATA)
spiFlash          0x00004000  0x00007fff         16K  data      sysparam  sys_param
spiFlash          0x00008000  0x000fffff        992K  app       factory   rom0              $(RBOOT_ROM_0_BIN)
spiFlash          0x00100000  0x001effff        960K                      (unused)
spiFlash          0x001f0000  0x001f3fff         16K  user      0         user0             user0.bin
spiFlash          0x001f4000  0x001f7fff         16K  user      1         user1
spiFlash          0x001f8000  0x001fffff         32K                      (unused)
spiFlash          0x00200000  0x0027ffff        512K  data      spiffs    spiffs0           $(SPIFF_BIN_OUT)
spiFlash          0x00280000  0x002bffff        256K  data      spiffs    spiffs1           $(FW_BASE)/spiffs1_rom.bin
spiFlash          0x002c0000  0x002fffff        256K  data      spiffs    spiffs2           $(FW_BASE)/spiffs2_rom.bin
spiFlash          0x00300000  0x003fffff          1M                      (unused)

For comparison, here’s the output for Esp32:

Basic_Storage: Invoking 'map' for Esp32 (debug) architecture
Partition map:
Device            Start       End         Size        Type      SubType   Name              Filename
----------------  ----------  ----------  ----------  --------  --------  ----------------  ------------
spiFlash          0x00000000  0x00007fff         32K                      Boot Sector
spiFlash          0x00008000  0x00008fff          4K                      Partition Table
spiFlash          0x00009000  0x0000efff         24K  data      nvs       nvs
spiFlash          0x0000f000  0x0000ffff          4K  data      phy       phy_init
spiFlash          0x00010000  0x001fffff       1984K  app       factory   factory           $(TARGET_BIN)
spiFlash          0x001f0000  0x001f3fff         16K  user      0         user0             user0.bin
spiFlash          0x001f4000  0x001f7fff         16K  user      1         user1
spiFlash          0x001f8000  0x001fffff         32K                      (unused)
spiFlash          0x00200000  0x0027ffff        512K  data      spiffs    spiffs0           $(SPIFF_BIN_OUT)
spiFlash          0x00280000  0x002bffff        256K  data      spiffs    spiffs1           $(FW_BASE)/spiffs1_rom.bin
spiFlash          0x002c0000  0x002fffff        256K  data      spiffs    spiffs2           $(FW_BASE)/spiffs2_rom.bin
spiFlash          0x00300000  0x003fffff          1M                      (unused)

To compare this with the partition map programmed into a device, do this:

make readmap map

JSON validation

When the binary partition table is built or updated, the configuration is first validated against a schema Sming/Components/Storage/schema.json.

This complements the checks performed by the hwconfig tool.

You can run the validation manually like this:

make hwconfig-validate

See JSON Schema for details about JSON schemas.

Configuration

HWCONFIG

default: standard

Set this to the hardware configuration to use for your project.

Default configurations:

standard

Base profile with 1MB flash size which should work on all device variants. Located in the Sming/Arch/{SMING_ARCH} directory.

standard-4m

Overrides standard to set 4Mbyte flash size

spiffs: Adds a single SPIFFS partition. See SPIFFS IFS Library.

Other configurations may be available, depending on architecture. You can see these by running make hwconfig-list.

For example, to select spiffs add the following line to your project:

HWCONFIG := spiffs

You will also need to run make HWCONFIG=spiffs to change the cached value (or make config-clean to reset everything).

HWCONFIG_OPTS: Set this to adjust the hardware profile using option fragments. See Hardware configuration options.

ENABLE_STORAGE_SIZE64

Build with ENABLE_STORAGE_SIZE64=1 to enable support for storage devices of more than 4GB capacity.

Device and partition addresses and sizes use the storage_size_t type, which by default is uint32_t. Setting this value changes it to uint64_t.

When enabling this setting, care must be taken in code especially with printf style format strings such as in debug statements. The safest way to handle both cases is like this:

debug_i("Partition size: %llu", uint64_t(part.size()));

Binary partition table

Sming uses the same binary partition table structure as ESP-IDF, located immediately after the boot sector. However, it is organised slightly differently to allow partitions to be registered for multiple storage devices.

Entries are fixed 32-byte structures, Storage::esp_partition_info_t, organised as follows:

The first entry is always a storage type defining the main spiFlash device.
This is followed by regular partition entries sorted in ascending address order. There may be gaps between the partitions.
The partition table md5sum entry is inserted as normal
If any external devices are defined: - A SMING_EXTENSION entry, which the esp32 bootloader interprets as the end of the partition table. - The next entry is a storage type for the external device. - This is followed by regular partition entries as before. - A second md5sum entry is inserted for the entire partition table thus far
The end of the partition table is identified by an empty sector (i.e. all bytes 0xFF).

Partition API

This is a C++ interface. Some examples:

Storage::Partition part = Storage::findPartition("spiffs0"); // Find by name
if(part) {
  Serial << part << endl;
} else {
  Serial << "spiffs0 partition NOT Found" << endl;
}

// Enumerate all partitions
for(auto part: Storage::findPartition()) {
  Serial << part << endl;
}

// Enumerate all SPIFFS partitions
for(auto part: Storage::findPartition(Storage::Partition::SubType::Data::spiffs)) {
  Serial << part << endl;
}

A Storage::Partition object is just a wrapper and can be freely copied around. It defines methods which should be used to read/write/erase the partition contents.

Each partition has an associated Storage::Device. This is usually Storage::spiFlash for the main flash device.

Other devices must be registered via Storage::PartitionTable::registerStorageDevice().

You can query partition entries from a Storage object directly, for example:

#include <Storage/SpiFlash.h>

for(auto part: Storage::spiFlash->partitions()) {
  Serial << part << endl;
}

External Storage

If your design has additional fixed storage devices, such as SPI RAM, flash or EEPROM, you can take advantage of the partition API to manage them as follows:

Implement a class to manage the storage, inheriting from Storage::Device.
Create a custom hardware configuration for your project and add a devices entry describing your storage device, plus partition entries: the device field identifies which device these entries relate to.
Create an instance of your custom device and make a call to Storage::registerDevice() in your init() function (or elsewhere if more appropriate).

See Disk Storage for how devices such as SD flash cards are managed.

API

namespace Storage

Get power of 2 for given value

See also

Use isLog2() to confirm value is power of 2

param value:: Must be an exact power of 2

retval uint8_t:: Result n such that value == 1 << n

template<typename T> constexpr std::enable_if<(sizeof(T) <= 4), uint8_t>::type getSizeBits(T value)

template<typename T> constexpr std::enable_if<(sizeof(T) > 4), uint8_t>::type getSizeBits(T value)

Enums

enum class Mode

Values:

enumerator ReadOnly

enumerator Write: Write but do not erase, region should be pre-erased.

enumerator BlockErase: Erase blocks as required before writing.

Functions

inline bool isSize64(uint64_t value): Determine if a value requires 64-bits to store.

inline bool isSize64(int64_t value): Determine if a value requires 64-bits to store.

template<typename T> constexpr bool isLog2(T value): Determine if a value is an exact power of 2.

void initialize(): Called early in the startup phase.

const Device::List getDevices(): Get read-only reference to device list.

bool registerDevice(Device *device)

Register a storage device.

Return values:: bool – true on success, false if another device already registered with same name

bool unRegisterDevice(Device *device)

Unregister a storage device.

Use extreme care: behaviour is unpredictable if partitions are in use

Device *findDevice(const String &name): Find a registered device.

Partition findPartition(const String &name): Find the first partition matching the given name.

inline Iterator findPartition(Partition::Type type = Partition::Type::any, uint8_t subType = Partition::SubType::any): Find partitions of the given type.

template<typename T> std::enable_if<std::is_enum<T>::value, Iterator>::type findPartition(T subType)

template<typename T> Storage::Partition findDefaultPartition(T subType)

Variables

constexpr uint16_t ESP_PARTITION_MAGIC = {0x50AA}: Identifies a valid partition.

constexpr uint16_t ESP_PARTITION_MAGIC_MD5 = {0xEBEB}: Identifies an MD5 hash block.

constexpr size_t ESP_PARTITION_TABLE_MAX_LEN = {0xC00}

ProgMem progMem

SpiFlash *spiFlash

SysMem sysMem

class FileDevice : public Storage::Device 

#include <FileDevice.h>

Create custom storage device using backing file.

Public Functions

inline FileDevice(const String &name, IFS::IFileSystem &fileSys, IFS::FileHandle file, storage_size_t size)

Construct a file device with custom size.

Parameters:

name – Name of device
fileSys – File system where file is located
file – Handle to open file
size – Size of device in bytes

inline FileDevice(const String &name, IFS::IFileSystem &fileSys, IFS::FileHandle file)

Construct a device using existing file.

Device will match size of existing file

Parameters:

name – Name of device
fileSys – File system where file is located
file – Handle to open file

inline virtual String getName() const override: Obtain unique device name.

inline virtual Type getType() const override: Obtain device type.

inline virtual storage_size_t getSize() const override

Obtain addressable size of this device.

Return values:: storage_size_t – Must be at least as large as the value declared in the hardware configuration

inline virtual size_t getBlockSize() const override: Obtain smallest allocation unit for erase operations.

virtual bool read(storage_size_t address, void *buffer, size_t len) override

Read data from the storage device.

Parameters:

address – Where to start reading
dst – Buffer to store data
size – Size of data to be read, in bytes.

Return values:

bool – true on success, false on error

virtual bool write(storage_size_t address, const void *data, size_t len) override

Write data to the storage device.

Parameters:

address – Where to start writing
src – Data to write
size – Size of data to be written, in bytes.

Return values:

bool – true on success, false on error

virtual bool erase_range(storage_size_t address, storage_size_t len) override

Erase a region of storage in preparation for writing.

Parameters:

address – Where to start erasing
size – Size of region to erase, in bytes

Return values:

bool – true on success, false on error

class Device : public LinkedObjectTemplate<Device>

#include <Device.h>

Represents a storage device (e.g. flash memory)

Subclassed by Storage::Disk::BlockDevice, Storage::FileDevice, Storage::ProgMem, Storage::SpiFlash, Storage::StreamDevice, Storage::SysMem

Public Types

enum class Type : uint8_t

Storage type.

Values:

enumerator partitionType

enumerator XX

Public Functions

inline const PartitionTable &partitions() const: Provide read-only access to partition table.

inline PartitionTable &editablePartitions(): Provide full access to partition table.

inline bool loadPartitions(uint32_t tableOffset)

Load partition table entries @tableOffset Location of partition table to read.

Return values:: bool – true on success, false on failure

bool loadPartitions(Device &source, uint32_t tableOffset)

Load partition table entries from another table.

Parameters:: source – Device to load entries from @tableOffset Location of partition table to read
Return values:: bool – true on success, false on failure

virtual String getName() const = 0: Obtain unique device name.

inline virtual uint32_t getId() const

Obtain device ID.

Return values:: uint32_t – typically flash chip ID

virtual size_t getBlockSize() const = 0: Obtain smallest allocation unit for erase operations.

virtual storage_size_t getSize() const = 0

Obtain addressable size of this device.

Return values:: storage_size_t – Must be at least as large as the value declared in the hardware configuration

virtual Type getType() const = 0: Obtain device type.

virtual bool read(storage_size_t address, void *dst, size_t size) = 0

Read data from the storage device.

Parameters:

address – Where to start reading
dst – Buffer to store data
size – Size of data to be read, in bytes.

Return values:

bool – true on success, false on error

virtual bool write(storage_size_t address, const void *src, size_t size) = 0

Write data to the storage device.

Parameters:

address – Where to start writing
src – Data to write
size – Size of data to be written, in bytes.

Return values:

bool – true on success, false on error

virtual bool erase_range(storage_size_t address, storage_size_t size) = 0

Erase a region of storage in preparation for writing.

Parameters:

address – Where to start erasing
size – Size of region to erase, in bytes

Return values:

bool – true on success, false on error

inline virtual uint16_t getSectorSize() const

Get sector size, the unit of allocation for block-access devices.

Override this method only if the device does not support standard 512-byte sector access. For example, ‘Advanced-Format’ drives use 4096-byte sectors.

inline virtual storage_size_t getSectorCount() const: Obtain total number of sectors on this device.

inline virtual bool sync()

Flush any pending writes to the physical media.

Devices with intermediate buffering should implement this method.

Return values:: bool – Return false if sync operation failed.

class Iterator: #include <Iterator.h>

class Partition

#include <Partition.h>

Represents a flash partition.

Confirm partition is of the expected type

bool verify(Type type, uint8_t subtype) const

Strong C++ type value.

Parameters:

type – Expected partition type
subtype – Expected partition sub-type

Return values:

bool – true if type is OK, false if not. Logs debug messages on failure.

inline bool verify(uint8_t type, uint8_t subtype) const: Weak ‘type’ value.

template<typename T> inline bool verify(T subType) const: Derive type from subtype, expressed as strong C++ enum.

Public Functions

bool read(storage_size_t offset, void *dst, size_t size)

Read data from the partition.

Parameters:

offset – Where to start reading, relative to start of partition
dst – Buffer to store data
size – Size of data to be read, in bytes.

Return values:

bool – true on success, false on error

bool write(storage_size_t offset, const void *src, size_t size)

Write data to the partition.

Note

Flash region must be erased first

Parameters:

offset – Where to start writing, relative to start of partition
src – Data to write
size – Size of data to be written, in bytes.

Return values:

bool – true on success, false on error

bool erase_range(storage_size_t offset, storage_size_t size)

Erase part of the partition.

Note

Both offset and size must be aligned to flash sector size (4Kbytes)

Parameters:

offset – Where to start erasing, relative to start of partition
size – Size of region to erase, in bytes

Return values:

bool – true on success, false on error

inline Partition::Type type() const: Obtain partition type.

inline uint8_t subType() const: Obtain partition sub-type.

inline FullType fullType() const: Obtain both type and subtype.

inline storage_size_t address() const

Obtain partition starting address.

Return values:: storage_size_t – Device address

inline storage_size_t lastAddress() const

Obtain address of last byte in this this partition.

Return values:: storage_size_t – Device address

inline storage_size_t size() const

Obtain partition size.

Return values:: storage_size_t – Size in bytes

inline String name() const: Get partition name.

inline Flags flags() const: Get partition flags.

inline bool isEncrypted() const: Check state of partition encrypted flag.

inline bool isReadOnly() const: Check state of partition readOnly flag.

bool getDeviceAddress(storage_size_t &address, storage_size_t size) const

Get corresponding storage device address for a given partition offset.

Parameters:

address – IN: Zero-based offset within partition, OUT: Device address
size – Size of data to be accessed

Return values:

bool – true on success, false on failure Fails if the given offset/size combination is out of range, or the partition is undefined.

String getDeviceName() const

Get name of storage device for this partition.

Return values:: String –

inline bool contains(storage_size_t addr) const: Determine if given address contained within this partition.

size_t getBlockSize() const: Obtain smallest allocation unit for erase operations.

uint16_t getSectorSize() const

Get sector size for block-addressable devices.

See also

See Storage::Device::getSectorSize

inline storage_size_t getSectorCount() const: Obtain total number of sectors in this partition.

bool sync()

Flush any pending writes to the physical media.

See also

See Storage::Device::sync

inline const Disk::DiskPart *diskpart() const: If this is a disk partition, return pointer to the additional information.

Public Static Functions

static inline SubType::App apptypeOta(uint8_t i): Convenience function to get SubType value for the i-th OTA partition.

struct FullType: #include <Partition.h>

Express both partition type and subtype together.

struct Info : public LinkedObjectTemplate<Info>, public Printable

#include <Partition.h>

Partition information.

Subclassed by Storage::Disk::PartInfo

struct SubType

#include <Partition.h>

Public Types

enum class App : uint8_t

Application partition type.

Values:

enumerator partitionType

enumerator XX

enumerator ota_min

enumerator ota_max

enumerator any

enum class Data : uint8_t

Data partition type.

Values:

enumerator partitionType

enumerator XX

enumerator any

struct esp_partition_info_t

#include <partition_info.h>

Internal structure describing the binary layout of a partition table entry.

Public Members

uint16_t magic: Fixed value to identify valid entry, appears as 0xFFFF at end of table.

Partition::Type type: Main type of partition.

uint8_t subtype: Sub-type for partition (interpretation dependent upon type)

uint32_t offset: Start offset.

uint32_t size: Size of partition in bytes.

Storage::Partition::Name name: Unique identifier for entry.

Storage::Partition::Flags flags: Various option flags.

class PartitionStream : public ReadWriteStream 

#include <PartitionStream.h>

Stream operating directory on a Storage partition.

To support write operations, the target region must be erased first.

Public Functions

inline PartitionStream(Partition partition, storage_size_t offset, size_t size, bool blockErase)

Access part of a partition using a stream.

If blockErase is false then region must be pre-erased before writing.

Deprecated:: Use mode parameter instead of blockErase

Parameters:

partition –
offset – Limit access to this starting offset
size – Limit access to this number of bytes from starting offset
blockErase – Set to true to erase blocks before writing

inline PartitionStream(Partition partition, bool blockErase)

Access entire partition using stream.

If blockErase is false then partition must be pre-erased before writing.

Deprecated:: Use mode parameter instead of blockErase

Parameters:

partition –
blockErase – Set to true to erase blocks before writing

inline PartitionStream(Partition partition, storage_size_t offset, size_t size, Mode mode = Mode::ReadOnly)

Access part of a partition using a stream.

Note

When writing in Mode::BlockErase, block erasure is only performed at the start of each block. Therefore if offset is not a block boundary then the corresponding block will not be erased first.

Parameters:

partition –
offset – Limit access to this starting offset
size – Limit access to this number of bytes from starting offset
mode –

inline PartitionStream(Partition partition, Mode mode = Mode::ReadOnly)

Access entire partition using stream.

Parameters:

partition –
mode – If blockErase is false then partition must be pre-erased before writing.

inline virtual int available() override

Return the total length of the stream.

Return values:: int – -1 is returned when the size cannot be determined

virtual uint16_t readMemoryBlock(char *data, int bufSize) override

Read a block of memory.

Todo:: Should IDataSourceStream::readMemoryBlock return same data type as its bufSize param?

Parameters:

data – Pointer to the data to be read
bufSize – Quantity of chars to read

Return values:

uint16_t – Quantity of chars read

virtual int seekFrom(int offset, SeekOrigin origin) override

Change position in stream.

Note

This method is implemented by streams which support random seeking, such as files and memory streams.

Parameters:

offset –
origin –

Return values:

New – position, < 0 on error

virtual size_t write(const uint8_t *buffer, size_t size) override

Write chars to stream.

Note

Although this is defined in the Print class, ReadWriteStream uses this as the core output method so descendants are required to implement it

Parameters:

buffer – Pointer to buffer to write to the stream
size – Quantity of chars to write

Return values:

size_t – Quantity of chars written to stream

inline virtual bool isFinished() override

Check if all data has been read.

Return values:: bool – True on success.

class PartitionTable

#include <PartitionTable.h>

Subclassed by Storage::ProgMem::ProgMemPartitionTable, Storage::SysMem::SysMemPartitionTable

Partition search

inline Iterator find(Partition::Type type = Partition::Type::any, uint8_t subType = Partition::SubType::any) const

Find partitions based on one or more parameters.

Parameters:

type – Partition type
subtype – Partition sub-type

Return values:

Iterator – Forward-iterator for matching partitions

Public Functions

inline Partition find(const String &name) const

Find partition by name.

Parameters:: Name – Name to search for, case-sensitive
Return values:: Partition – Names are unique so at most only one match

inline Partition find(uint32_t address) const

Find partition containing the given address.

Parameters:: address – Address to search for
Return values:: Partition –

inline Partition findOta(uint8_t index) const: Find the n’th OTA partition.

inline Partition add(const Partition::Info *info)

Add new partition using given Info.

Parameters:: info – Must be allocated using new: Device will take ownership
Return values:: Partition – Reference to the partition

class ProgMem : public Storage::Device 

#include <ProgMem.h>

Storage device to access PROGMEM using flash API.

Public Functions

inline virtual String getName() const override: Obtain unique device name.

inline virtual size_t getBlockSize() const override: Obtain smallest allocation unit for erase operations.

inline virtual storage_size_t getSize() const override

Obtain addressable size of this device.

Return values:: storage_size_t – Must be at least as large as the value declared in the hardware configuration

inline virtual Type getType() const override: Obtain device type.

virtual bool read(storage_size_t address, void *dst, size_t size) override

Read data from the storage device.

Parameters:

address – Where to start reading
dst – Buffer to store data
size – Size of data to be read, in bytes.

Return values:

bool – true on success, false on error

inline virtual bool write(storage_size_t address, const void *src, size_t size) override

Write data to the storage device.

Parameters:

address – Where to start writing
src – Data to write
size – Size of data to be written, in bytes.

Return values:

bool – true on success, false on error

inline virtual bool erase_range(storage_size_t address, storage_size_t size) override

Erase a region of storage in preparation for writing.

Parameters:

address – Where to start erasing
size – Size of region to erase, in bytes

Return values:

bool – true on success, false on error

class ProgMemPartitionTable : public Storage::PartitionTable 

#include <ProgMem.h>

Public Functions

Partition add(const String &name, const void *flashPtr, size_t size, Partition::FullType type)

Add partition entry for PROGMEM data access.

Parameters:

name – Name for partition
flashPtr – PROGMEM pointer
size – Size of PROGMEM data
type – Partition type and subtype

Return values:

Partition – Invalid if data is not progmem

inline Partition add(const String &name, const FSTR::ObjectBase &fstr, Partition::FullType type): Add partition entry for FlashString data access.

class SpiFlash : public Storage::Device 

#include <SpiFlash.h>

Main flash storage device.

Public Functions

virtual String getName() const override: Obtain unique device name.

virtual size_t getBlockSize() const override: Obtain smallest allocation unit for erase operations.

virtual storage_size_t getSize() const override

Obtain addressable size of this device.

Return values:: storage_size_t – Must be at least as large as the value declared in the hardware configuration

inline virtual Type getType() const override: Obtain device type.

virtual uint32_t getId() const override

Obtain device ID.

Return values:: uint32_t – typically flash chip ID

virtual bool read(storage_size_t address, void *dst, size_t size) override

Read data from the storage device.

Parameters:

address – Where to start reading
dst – Buffer to store data
size – Size of data to be read, in bytes.

Return values:

bool – true on success, false on error

virtual bool write(storage_size_t address, const void *src, size_t size) override

Write data to the storage device.

Parameters:

address – Where to start writing
src – Data to write
size – Size of data to be written, in bytes.

Return values:

bool – true on success, false on error

virtual bool erase_range(storage_size_t address, storage_size_t size) override

Erase a region of storage in preparation for writing.

Parameters:

address – Where to start erasing
size – Size of region to erase, in bytes

Return values:

bool – true on success, false on error

class StreamDevice : public Storage::Device 

#include <StreamDevice.h>

Read-only partition on a stream object.

Note

Writes not possible as streams always append data, cannot do random writes

Public Functions

inline virtual Type getType() const override: Obtain device type.

inline virtual bool read(storage_size_t address, void *buffer, size_t len) override

Read data from the storage device.

Parameters:

address – Where to start reading
dst – Buffer to store data
size – Size of data to be read, in bytes.

Return values:

bool – true on success, false on error

inline virtual bool write(storage_size_t address, const void *data, size_t len) override

Write data to the storage device.

Parameters:

address – Where to start writing
src – Data to write
size – Size of data to be written, in bytes.

Return values:

bool – true on success, false on error

inline virtual bool erase_range(storage_size_t address, storage_size_t len) override

Erase a region of storage in preparation for writing.

Parameters:

address – Where to start erasing
size – Size of region to erase, in bytes

Return values:

bool – true on success, false on error

class SysMem : public Storage::Device 

#include <SysMem.h>

Storage device to access system memory, e.g. RAM.

Public Functions

inline virtual String getName() const override: Obtain unique device name.

inline virtual size_t getBlockSize() const override: Obtain smallest allocation unit for erase operations.

inline virtual storage_size_t getSize() const override

Obtain addressable size of this device.

Return values:: storage_size_t – Must be at least as large as the value declared in the hardware configuration

inline virtual Type getType() const override: Obtain device type.

inline virtual bool read(storage_size_t address, void *buffer, size_t len) override

Read data from the storage device.

Parameters:

address – Where to start reading
dst – Buffer to store data
size – Size of data to be read, in bytes.

Return values:

bool – true on success, false on error

inline virtual bool write(storage_size_t address, const void *data, size_t len) override

Write data to the storage device.

Parameters:

address – Where to start writing
src – Data to write
size – Size of data to be written, in bytes.

Return values:

bool – true on success, false on error

inline virtual bool erase_range(storage_size_t address, storage_size_t len) override

Erase a region of storage in preparation for writing.

Parameters:

address – Where to start erasing
size – Size of region to erase, in bytes

Return values:

bool – true on success, false on error

class SysMemPartitionTable : public Storage::PartitionTable 

#include <SysMem.h>

Public Functions

inline Partition add(const String &name, const FSTR::ObjectBase &fstr, Partition::FullType type): Add partition entry for FlashString data access.

inline Partition add(const Partition::Info *info)

Add new partition using given Info.

Parameters:: info – Must be allocated using new: Device will take ownership
Return values:: Partition – Reference to the partition

namespace Debug

Functions

void listPartitions(Print &out)

void listPartitions(Print &out, const Device &device)

void listDevices(Print &out, bool fullPartitionInfo = true)

namespace Disk

Typedefs

using SysTypes = BitSet<uint8_t, SysType>

Enums

enum class Error

Values:

enumerator XX

enum class SysType : uint8_t

Identifies exact disk volume type.

Values:

enumerator unknown: Partition type not recognised.

enumerator fat12

enumerator fat16

enumerator fat32

enumerator exfat

enum SysIndicator

MBR partition system type indicator values.

Values:

enumerator SI_FAT12

enumerator SI_FAT16: FAT16 with fewer than 65536 sectors.

enumerator SI_FAT16B: FAT16B with 65536 or more sectors.

enumerator SI_IFS

enumerator SI_EXFAT

enumerator SI_FAT32X: FAT32 with LBA.

Functions

template<typename T> T align_up(T value, uint32_t align)

template<typename T> auto getBlockCount(T byteCount, uint32_t blockSize)

uint32_t crc32_byte(uint32_t crc, uint8_t d)

uint32_t crc32(uint32_t bcc, const void *data, size_t length)

inline uint32_t crc32(const void *data, size_t length)

inline bool operator!(Error err)

Error formatDisk(BlockDevice &device, GPT::PartitionTable &table, const Uuid &diskGuid = {})

Partition a device using the GPT scheme.

Parameters:

device –
table – Partitions to create

Return values:

Error –

Error formatDisk(BlockDevice &device, MBR::PartitionTable &table)

Partition a device using the MBR scheme.

Parameters:

device –
table – Partitions to create

Return values:

Error –

inline SysType getSysTypeFromIndicator(SysIndicator si)

Error validate(BasePartitionTable &table, storage_size_t firstAvailableBlock, storage_size_t totalAvailableBlocks, uint32_t blockSize)

Validate partition table entries.

If size <= 100 then the actual size is calculated as a percentage and updated
If offset = 0 then a suitable location is found and the offset updated

On success, partition entries are ordered by position.

Parameters:

firstAvailableBlock – First block number which may be allocated to a partition
totalAvailableBlocks – Number of blocks available for partition allocation
blockSize – Size of a block

Return values:

Error – For each partition:

bool scanPartitions(Device &device)

Variables

constexpr uint32_t PARTITION_ALIGN = {0x100000U}

static constexpr SysTypes fatTypes = SysType::fat12 | SysType::fat16 | SysType::fat32 | SysType::exfat 

class BlockDevice : public Storage::Device 

#include <BlockDevice.h>

Base class for sector-addressable (block) devices.

Inherited classes must set the sectorCount value, and implement the four raw_xxx methods.

This class supports byte-level access using internal buffering, which if required must be enabled by the application via the allocateBuffers method. Without buffering, read/writes must always be sector-aligned. Rrase must always be sector-aligned.

For power-loss resiliency it is important to call sync() at appropriate times. Filing system implementations should do this after closing a file, for example. Applications should consider this if leaving files open for extended periods, and explicitly call ‘flush’ on the filing system or ‘sync’ on the partition or device if necessary.

Subclassed by Storage::Disk::HostFileDevice, Storage::SD::Card, USB::MSC::LogicalUnit

Public Functions

virtual bool read(storage_size_t address, void *dst, size_t size) override

Read data from the storage device.

Parameters:

address – Where to start reading
dst – Buffer to store data
size – Size of data to be read, in bytes.

Return values:

bool – true on success, false on error

virtual bool write(storage_size_t address, const void *src, size_t size) override

Write data to the storage device.

Parameters:

address – Where to start writing
src – Data to write
size – Size of data to be written, in bytes.

Return values:

bool – true on success, false on error

virtual bool erase_range(storage_size_t address, storage_size_t size) override

Erase a region of storage in preparation for writing.

Parameters:

address – Where to start erasing
size – Size of region to erase, in bytes

Return values:

bool – true on success, false on error

inline virtual size_t getBlockSize() const override: Obtain smallest allocation unit for erase operations.

inline virtual storage_size_t getSize() const override

Obtain addressable size of this device.

Return values:: storage_size_t – Must be at least as large as the value declared in the hardware configuration

inline virtual storage_size_t getSectorCount() const override: Obtain total number of sectors on this device.

virtual bool sync() override

Flush any pending writes to the physical media.

Devices with intermediate buffering should implement this method.

Return values:: bool – Return false if sync operation failed.

bool allocateBuffers(unsigned numBuffers)

Set number of sector buffers to use.

Required to support byte-level read/write operations on block devices. Buffering can improve performance, with diminishing returns above around 4 sectors.

Parameters:: numBuffers – Number of buffers to allocate 1,2,4,8,etc. Pass 0 to deallocate/disable buffering.
Return values:: bool – false on memory allocation error, or if failed to flush existing buffers to disk

struct Stat

#include <BlockDevice.h>

Public Members

Func func[3]: Read, Write, Erase.

std::map<uint32_t, Func> sectors: By sector.

struct Func: #include <BlockDevice.h>

struct Buffer : public std::unique_ptr<uint8_t[]>: #include <Buffer.h>

class BufferList: #include <Buffer.h>

class HostFileDevice : public Storage::Disk::BlockDevice 

#include <HostFileDevice.h>

Create custom storage device using backing file.

Public Functions

HostFileDevice(const String &name, const String &filename, storage_size_t size)

Construct a file device with custom size.

Parameters:

name – Name of device
filename – Path to file
size – Size of device in bytes

HostFileDevice(const String &name, const String &filename)

Construct a device using existing file.

Device will match size of existing file

Parameters:

name – Name of device
filename – Path to file

inline virtual String getName() const override: Obtain unique device name.

inline virtual Type getType() const override: Obtain device type.

struct DiskPart

#include <PartInfo.h>

Adds information specific to MBR/GPT disk partitions.

Subclassed by Storage::Disk::PartInfo

Public Functions

size_t printTo(Print &p) const: Print full contents of this structure.

Public Members

Uuid typeGuid: GPT type GUID.

Uuid uniqueGuid: GPT partition unique GUID.

SysType systype = {}: Identifies volume filing system type.

SysIndicator sysind = {}: Partition sys value.

struct PartInfo : public Storage::Partition::Info, public Storage::Disk::DiskPart 

#include <PartInfo.h>

In-memory partition information.

A disk Storage::Partition refers to this instance.

Public Functions

inline virtual const Disk::DiskPart *diskpart() const override

Obtain additional disk information.

Accessed via Partition::diskpart() method

virtual size_t printTo(Print &p) const override: Print important fields only.

class BasePartitionTable : public OwnedLinkedObjectListTemplate<PartInfo>

#include <PartInfo.h>

Common type for MBR/GPT partition table.

Subclassed by Storage::Disk::GPT::PartitionTable, Storage::Disk::MBR::PartitionTable

class Scanner

#include <Scanner.h>

Class to iterate through disk partition tables.

Supports MBR and GPT partitioning schemes.

Public Functions

std::unique_ptr<PartInfo> next(): Obtains the next partition entry (if any)

class SectorBuffer : public std::unique_ptr<uint8_t[]>: #include <SectorBuffer.h>

Buffer for working with disk sectors.

namespace EXFAT

namespace FAT

namespace GPT

Functions

String getTypeName(const Uuid &typeGuid): Get string for known GPT type GUIDs.

struct SmingTypeGuid : public Uuid: #include <GPT.h>

class PartitionTable : public Storage::Disk::BasePartitionTable 

#include <GPT.h>

Public Functions

inline bool add(const String &name, SysType sysType, storage_size_t offset, storage_size_t size, const Uuid &uniqueGuid = {}, const Uuid &typeGuid = {}, Partition::Flags flags = 0)

Add a new standard GPT partition definition.

Parameters:

name – Partition name
sysType – Intended content for this partition
offset – Start offset, or 0 to have position calculated
size – Size of partition (in bytes), or percentage (0-100) of total partitionable disk space
uniqueGuid – Unique partition identifier (optional: will be generated if not provided)
typeGuid – Partition type GUID (default is BASIC_DATA)
flags –

Return values:

bool – true on success

inline bool add(const String &name, Partition::FullType type, storage_size_t offset, storage_size_t size, const Uuid &uniqueGuid = {}, Partition::Flags flags = 0)

Add a new GPT partition for a regular Sming filing system.

Note

These partitions use a custom type GUID and won’t be recognised by external software.

Parameters:

name – Partition name
type – Sming partition type/subtype
offset – Start offset, or 0 to have position calculated
size – Size of partition (in bytes), or percentage (0-100) of total partitionable disk space
uniqueGuid – Unique partition identifier (optional: will be generated if not provided)
flags –

Return values:

bool – true on success

namespace MBR

class PartitionTable : public Storage::Disk::BasePartitionTable 

#include <MBR.h>

Public Functions

inline bool add(SysType sysType, SysIndicator sysIndicator, storage_size_t offset, storage_size_t size, Partition::Flags flags = 0)

Add a new MBR partition definition.

Note

MBR does not have partition name field; this will appear as ‘mbr1’, ‘mbr2’, etc.

Parameters:

sysType – Intended content for this partition (or ‘unknown’)
SysIndicator – Appropriate system code SI_xxx
offset – Start offset, or 0 to have position calculated
size – Size of partition (in bytes), or percentage (0-100) of total partitionable disk space
flags –

Return values:

bool – true on success

namespace SD

class Card : public Storage::Disk::BlockDevice 

#include <Card.h>

Public Functions

bool begin(uint8_t chipSelect, uint32_t freq = 0)

Initialise the card.

Parameters:

chipSelect –
freq – SPI frequency in Hz, use 0 for maximum supported frequency

inline virtual String getName() const override: Obtain unique device name.

inline virtual uint32_t getId() const

Obtain device ID.

Return values:: uint32_t – typically flash chip ID

inline virtual Type getType() const: Obtain device type.

inline virtual size_t getBlockSize() const override: Obtain smallest allocation unit for erase operations.

struct CID

#include <CID.h>

Public Members

uint8_t mid: Manufacturer ID.

char oid[2]: OEM / Application ID.

char pnm[5]: Product name.

uint8_t prv: Product revision.

uint32_t psn: Product serial number.

uint16_t mdt: Manufacturing date.

uint8_t not_used: Always 1.

uint8_t crc: 7-bit checksum

struct CSD: #include <CSD.h>

Subclassed by Storage::SD::CSD1, Storage::SD::CSD2, Storage::SD::CSD3

struct CSD1 : public Storage::SD::CSD : #include <CSD.h>

struct CSD2 : public Storage::SD::CSD : #include <CSD.h>

struct CSD3 : public Storage::SD::CSD : #include <CSD.h>

References

Source Code

Used by

Sming (main) Component
Installable File System Component
Disk Storage Library

Environment Variables

ENABLE_STORAGE_SIZE64
HWCONFIG
HWCONFIG_OPTS
PARTITION_TABLE_OFFSET

SoC support

esp32
esp32c2
esp32c3
esp32s2
esp32s3
esp8266
host
rp2040