Storage Management
This Component provides support for using storage devices in a structured way by partitioning them into areas for specific uses. Partitions may can contain information such as:
Application (firmware) images
Filesystem(s)
Configuration/calibration/parameter data
Custom flash storage areas
A single partition table is located on the main flash device, Storage::spiFlash
,
and defines all partitions with a unique name and associated
Storage::Partition::Type
/ Storage::Partition::SubType
.
Hardware configuration
Each project has an associated Hardware configuration
, specified by the HWCONFIG
setting:
this is a JSON file with a .hw
extension.
For user convenience, the configuration file may contain comments however these are stripped before processing.
The build system locates the file by searching, in order:
{PROJECT_DIR}
the root project directory{SMING_HOME}/Arch/{SMING_ARCH}
{SMING_HOME}
Each architecture provides a standard
configuration which defines such things as the
partition table location and standard system partitions. Other configurations inherit
from this by providing a base_config
value.
You can list the available configs like this:
make hwconfig-list
This also shows the file path should you wish to view or edit it.
To select and view the resulting configuration, do this:
make hwconfig HWCONFIG=spiffs
or, to show the partition map:
make map HWCONFIG=spiffs
Note
You can set HWCONFIG
in your project’s component.mk
file, however as with other
configuration variables it will be overridden by the cached value set on the command line.
For example, if you want to change from standard
to standard-4m
for your project,
first add this line to your component.mk file:
HWCONFIG := standard-4m
Then either run make HWCONFIG=standard-4m
or make config-clean
.
Hardware configuration options
Commonly used settings can be stored in an option library for easier use.
The library files are named options.json
and located in the place as .hw files.
For example, we can do this:
make HWCONFIG=standard HWCONFIG_OPTS=4m,spiffs
This loads the ‘standard’ profile then merges the fragments found in the option library with the given names.
This is how the standard-4m
profile is constructed.
If using this approach, remember to updated your project’s component.mk
with the desired settings,
and verify the layout is correct using make map
.
OTA updates
When planning OTA updates please check that the displayed partition map corresponds to your project. For example, the partition table requires a free sector so must not overlap other partitions.
Your OTA update process must include a step to write the partition table to the correct location.
It is not necessary to update the bootloader. See rBoot for further information.
Custom configurations
To customise the hardware configuration for a project, for example ‘my_project’:
Create a new configuration file in your project root, such as
my_project.hw
:{ "name": "My project config", "base_config": "spiffs", "options": ["vdd"] }
You can use any available configuration as the base_config. Option fragments can be pulled in as shown. See Hardware configuration options.
If required, modify any inherited settings:
{ "name": "My config", "base_config": "standard", "devices": { "spiFlash": { "speed": 80, "mode": "qio", "size": "2M" } }, "partitions": { "rom0": { "address": "0x10000", "size": "0x80000" } } }
This will adjust flash parameters (previously via SPI_SPEED, SPI_MODE and SPI_SIZE), and the location/size of the primary application partition.
Add any additional partitions:
{ "name": "My config", "base_config": "standard-4m", "partitions": { "rom0": { "address": "0x10000", "size": "0x80000" }, "spiffs1": { "address": "0x00280000", "size": "256K", "type": "data", "subtype": "spiffs", "filename": "$(FW_BASE)/spiffs1_rom.bin", "build": { "target": "spiffsgen", "files": "files/spiffs1" } } } }
This adds a second SPIFFS partition, and instructs the build system to generate an image file for it using the files in the project’s
files/spiffs1
directory.Select the new configuration and re-build the project:
make HWCONFIG=my_project
You should also add this to your project’s
component.mk
file:HWCONFIG := my_project
Program your device:
make flash
This will flash everything: bootloader, partition table and all defined partitions (those with a
filename
entry).
Note
The build system isn’t smart enough to track dependencies for partition build targets.
To rebuild these manually type:
make partbuild
These will be removed when make clean
is run, but you can also clean them separately thus:
make part-clean
Partition maps
This is a concise view of your flash partitions. Display it like this:
make map
For the Basic Storage sample application, we get this:
Basic_Storage: Invoking 'map' for Esp8266 (debug) architecture
Partition map:
Device Start End Size Type SubType Name Filename
---------------- ---------- ---------- ---------- -------- -------- ---------------- ------------
spiFlash 0x00000000 0x00001fff 8K Boot Sector
spiFlash 0x00002000 0x00002fff 4K Partition Table
spiFlash 0x00003000 0x00003fff 4K data phy phy_init $(FLASH_INIT_DATA)
spiFlash 0x00004000 0x00007fff 16K data sysparam sys_param
spiFlash 0x00008000 0x000fffff 992K app factory rom0 $(RBOOT_ROM_0_BIN)
spiFlash 0x00100000 0x001effff 960K (unused)
spiFlash 0x001f0000 0x001f3fff 16K user 0 user0 user0.bin
spiFlash 0x001f4000 0x001f7fff 16K user 1 user1
spiFlash 0x001f8000 0x001fffff 32K (unused)
spiFlash 0x00200000 0x0027ffff 512K data spiffs spiffs0 $(SPIFF_BIN_OUT)
spiFlash 0x00280000 0x002bffff 256K data spiffs spiffs1 $(FW_BASE)/spiffs1_rom.bin
spiFlash 0x002c0000 0x002fffff 256K data spiffs spiffs2 $(FW_BASE)/spiffs2_rom.bin
spiFlash 0x00300000 0x003fffff 1M (unused)
For comparison, here’s the output for Esp32:
Basic_Storage: Invoking 'map' for Esp32 (debug) architecture
Partition map:
Device Start End Size Type SubType Name Filename
---------------- ---------- ---------- ---------- -------- -------- ---------------- ------------
spiFlash 0x00000000 0x00007fff 32K Boot Sector
spiFlash 0x00008000 0x00008fff 4K Partition Table
spiFlash 0x00009000 0x0000efff 24K data nvs nvs
spiFlash 0x0000f000 0x0000ffff 4K data phy phy_init
spiFlash 0x00010000 0x001fffff 1984K app factory factory $(TARGET_BIN)
spiFlash 0x001f0000 0x001f3fff 16K user 0 user0 user0.bin
spiFlash 0x001f4000 0x001f7fff 16K user 1 user1
spiFlash 0x001f8000 0x001fffff 32K (unused)
spiFlash 0x00200000 0x0027ffff 512K data spiffs spiffs0 $(SPIFF_BIN_OUT)
spiFlash 0x00280000 0x002bffff 256K data spiffs spiffs1 $(FW_BASE)/spiffs1_rom.bin
spiFlash 0x002c0000 0x002fffff 256K data spiffs spiffs2 $(FW_BASE)/spiffs2_rom.bin
spiFlash 0x00300000 0x003fffff 1M (unused)
To compare this with the partition map programmed into a device, do this:
make readmap map
JSON validation
When the binary partition table is built or updated, the configuration is first validated against a schema Sming/Components/Storage/schema.json.
This complements the checks performed by the hwconfig
tool.
You can run the validation manually like this:
make hwconfig-validate
See JSON Schema for details about JSON schemas.
Configuration
- HWCONFIG
default: standard
Set this to the hardware configuration to use for your project.
Default configurations:
- standard
Base profile with 1MB flash size which should work on all device variants. Located in the
Sming/Arch/{SMING_ARCH}
directory.- standard-4m
Overrides
standard
to set 4Mbyte flash size- spiffs
Adds a single SPIFFS partition. See SPIFFS IFS Library.
Other configurations may be available, depending on architecture. You can see these by running
make hwconfig-list
.For example, to select
spiffs
add the following line to your project:HWCONFIG := spiffs
You will also need to run
make HWCONFIG=spiffs
to change the cached value (ormake config-clean
to reset everything).
- HWCONFIG_OPTS
Set this to adjust the hardware profile using option fragments. See Hardware configuration options.
Binary partition table
Sming uses the same binary partition table structure as ESP-IDF, located immediately after the boot sector. However, it is organised slighly differently to allow partitions to be registered for multiple storage devices.
Entries are fixed 32-byte structures, Storage::esp_partition_info_t
, organised as follows:
The first entry is always a
storage
type defining the mainspiFlash
device.This is followed by regular partition entries sorted in ascending address order. There may be gaps between the partitions.
The partition table md5sum entry is inserted as normal
If any external devices are defined: - A SMING_EXTENSION entry, which the esp32 bootloader interprets as the end of the partition table. - The next entry is a
storage
type for theexternal
device. - This is followed by regular partition entries as before. - A second md5sum entry is inserted for the entire partition table thus farThe end of the partition table is identified by an empty sector (i.e. all bytes 0xFF).
Partition API
This is a C++ interface. Some examples:
Storage::Partition part = Storage::findPartition("spiffs0"); // Find by name
if(part) {
debugf("Partition '%s' found", part.name().c_str());
} else {
debugf("Partition NOT found");
}
// Enumerate all partitions
for(auto it = Storage::findPartition(); it; ++it) {
auto part = *it;
debugf("Found '%s' at 0x%08x, size 0x%08x", part.name().c_str(), part.address(), part.size());
}
// Enumerate all SPIFFS partitions
for(auto it = Storage::findPartition(Partition::SubType::Data::spiffs; it; it++) {
debugf("Found '%s' at 0x%08x, size 0x%08x", it->name().c_str(), it->address(), it->size());
}
A Storage::Partition
object is just a wrapper and can be freely copied around.
It defines methods which should be used to read/write/erase the partition contents.
Each partition has an associated Storage::Device
.
This is usually Storage::spiFlash
for the main flash device.
Other devices must be registed via Storage::PartitionTable::registerStorageDevice()
.
You can query partition entries from a Storage object directly, for example:
#include <Storage/SpiFlash.h>
for(auto part: Storage::spiFlash->partitions()) {
debugf("Found '%s' at 0x%08x, size 0x%08x", part.name().c_str(), part.address(), part.size());
}
External Storage
If your design has additional fixed storage devices, such as SPI RAM, flash or EEPROM, you can take advantage of the partition API to manage them as follows:
Implement a class to manage the storage, inheriting from
Storage::Device
.Create a custom hardware configuration for your project and add a
devices
entry describing your storage device, plus partition entries: thedevice
field identifies which device these entries relate to.Create an instance of your custom device and make a call to
Storage::registerDevice()
in yourinit()
function (or elsewhere if more appropriate).
API
-
namespace Storage
FileDevice.h
Copyright 2019 mikee47 mike@sillyhouse.net
This file is part of the IFS Library
This library is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3 or later.
This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this library. If not, see https://www.gnu.org/licenses/.
Functions
-
void initialize()
Called early in the startup phase.
-
bool registerDevice(Device *device)
Register a storage device.
- Returns
bool – true on success, false if another device already registered with same name
-
bool unRegisterDevice(Device *device)
Unregister a storage device.
Use extreme care: behaviour is unpredictable if partitions are in use
Variables
-
constexpr uint16_t ESP_PARTITION_MAGIC = {0x50AA}
Identifies a valid partition.
-
constexpr uint16_t ESP_PARTITION_MAGIC_MD5 = {0xEBEB}
Identifies an MD5 hash block.
-
constexpr size_t ESP_PARTITION_TABLE_MAX_LEN = {0xC00}
-
class CustomDevice : public Storage::Device
- #include <CustomDevice.h>
Class to support dynamic partitions.
Call
createPartition
to add partitions up to a maximum of 16 entries.Subclassed by Storage::FileDevice, Storage::ProgMem, Storage::StreamDevice, Storage::SysMem
-
class Device : public LinkedObjectTemplate<Device>
- #include <Device.h>
Represents a storage device (e.g. flash memory)
Subclassed by Storage::CustomDevice, Storage::SpiFlash
Public Functions
-
inline bool loadPartitions(uint32_t tableOffset)
Load partition table entries Location of partition table to read.
- Returns
bool – true on success, false on failure
-
bool loadPartitions(Device &source, uint32_t tableOffset)
Load partition table entries from another table.
- Parameters
source – Device to load entries from Location of partition table to read
- Returns
bool – true on success, false on failure
-
inline virtual uint32_t getId() const
Obtain device ID.
- Returns
uint32_t – typically flash chip ID
-
virtual size_t getBlockSize() const = 0
Obtain smallest allocation unit for erase operations.
-
virtual size_t getSize() const = 0
Obtain addressable size of this device.
- Returns
size_t – Must be at least as large as the value declared in the partition table
-
virtual bool read(uint32_t address, void *dst, size_t size) = 0
Read data from the storage device.
- Parameters
address – Where to start reading
dst – Buffer to store data
size – Size of data to be read, in bytes.
- Returns
bool – true on success, false on error
-
virtual bool write(uint32_t address, const void *src, size_t size) = 0
Write data to the storage device.
- Parameters
address – Where to start writing
src – Data to write
size – Size of data to be written, in bytes.
- Returns
bool – true on success, false on error
-
virtual bool erase_range(uint32_t address, size_t size) = 0
Erase a region of storage in preparation for writing.
- Parameters
address – Where to start erasing
size – Size of region to erase, in bytes
- Returns
bool – true on success, false on error
-
inline bool loadPartitions(uint32_t tableOffset)
-
struct esp_partition_info_t
- #include <partition_info.h>
Internal structure describing the binary layout of a partition table entry.
-
class FileDevice : public Storage::CustomDevice
- #include <FileDevice.h>
Read-only partition on a stream object.
Note
Writes not possible as streams always append data, cannot do random writes
Public Functions
-
inline virtual Type getType() const override
Obtain device type.
-
inline virtual size_t getSize() const override
Obtain addressable size of this device.
- Returns
size_t – Must be at least as large as the value declared in the partition table
-
inline virtual size_t getBlockSize() const override
Obtain smallest allocation unit for erase operations.
-
virtual bool read(uint32_t address, void *buffer, size_t len) override
Read data from the storage device.
- Parameters
address – Where to start reading
dst – Buffer to store data
size – Size of data to be read, in bytes.
- Returns
bool – true on success, false on error
-
virtual bool write(uint32_t address, const void *data, size_t len) override
Write data to the storage device.
- Parameters
address – Where to start writing
src – Data to write
size – Size of data to be written, in bytes.
- Returns
bool – true on success, false on error
-
virtual bool erase_range(uint32_t address, size_t len) override
Erase a region of storage in preparation for writing.
- Parameters
address – Where to start erasing
size – Size of region to erase, in bytes
- Returns
bool – true on success, false on error
-
inline virtual Type getType() const override
-
class Partition
- #include <Partition.h>
Represents a flash partition.
Confirm partition is of the expected type
- param type
Expected partition type
- param subtype
Expected partition sub-type
- returns bool
true if type is OK, false if not. Logs debug messages on failure.
Public Functions
-
bool read(size_t offset, void *dst, size_t size)
Read data from the partition.
- Parameters
offset – Where to start reading, relative to start of partition
dst – Buffer to store data
size – Size of data to be read, in bytes.
- Returns
bool – true on success, false on error
-
bool write(size_t offset, const void *src, size_t size)
Write data to the partition.
Note
Flash region must be erased first
- Parameters
offset – Where to start writing, relative to start of partition
src – Data to write
size – Size of data to be written, in bytes.
- Returns
bool – true on success, false on error
-
bool erase_range(size_t offset, size_t size)
Erase part of the partition.
Note
Both offset and size must be aligned to flash sector size (4Kbytes)
- Parameters
offset – Where to start erasing, relative to start of partition
size – Size of region to erase, in bytes
- Returns
bool – true on success, false on error
-
inline uint8_t subType() const
Obtain partition sub-type.
-
inline uint32_t address() const
Obtain partition starting address.
- Parameters
uint32_t – Device address
-
inline uint32_t lastAddress() const
Obtain address of last byte in this this partition.
- Parameters
uint32_t – Device address
-
inline uint32_t size() const
Obtain partition size.
- Returns
uint32_t – Size in bytes
-
inline Flags flags() const
Get partition flags.
-
inline bool isEncrypted() const
Check state of partition
encrypted
flag.
-
inline bool isReadOnly() const
Check state of partition
readOnly
flag.
-
bool getDeviceAddress(uint32_t &address, size_t size) const
Get corresponding storage device address for a given partition offset.
- Parameters
address – IN: Zero-based offset within partition, OUT: Device address
size – Size of data to be accessed
- Returns
bool – true on success, false on failure Fails if the given offset/size combination is out of range, or the partition is undefined.
-
inline Device *getDevice() const
Get storage device containing this partition.
- Returns
Device* – null if device isn’t registered
-
inline bool contains(uint32_t addr) const
Determine if given address contained within this partition.
-
size_t getBlockSize() const
Obtain smallest allocation unit for erase operations.
Public Static Functions
-
struct SubType
- #include <Partition.h>
-
class PartitionStream : public ReadWriteStream
- #include <PartitionStream.h>
Stream operating directory on a Storage partition.
To support write operations, the target region must be erased first.
Public Functions
-
inline virtual int available() override
Return the total length of the stream.
- Returns
int – -1 is returned when the size cannot be determined
-
virtual uint16_t readMemoryBlock(char *data, int bufSize) override
Read a block of memory.
- Todo:
Should IDataSourceStream::readMemoryBlock return same data type as its bufSize param?
- Parameters
data – Pointer to the data to be read
bufSize – Quantity of chars to read
- Returns
uint16_t – Quantity of chars read
-
virtual int seekFrom(int offset, SeekOrigin origin) override
Change position in stream.
Note
This method is implemented by streams which support random seeking, such as files and memory streams.
- Parameters
offset –
origin –
- Returns
New – position, < 0 on error
-
virtual size_t write(const uint8_t *buffer, size_t size) override
Write chars to stream.
Note
Although this is defined in the Print class, ReadWriteStream uses this as the core output method so descendants are required to implement it
- Parameters
buffer – Pointer to buffer to write to the stream
size – Quantity of chars to write
- Returns
size_t – Quantity of chars written to stream
-
inline virtual bool isFinished() override
Check if all data has been read.
- Returns
bool – True on success.
-
inline virtual int available() override
-
class PartitionTable
- #include <PartitionTable.h>
Partition search
Find partitions based on one or more parameters
- returns Iterator
Forward-iterator for matching partitions
Public Functions
-
inline Partition find(const String &name) const
Find partition by name.
- Parameters
Name – Name to search for, case-sensitive
- Returns
Partition – Names are unique so at most only one match
-
class ProgMem : public Storage::CustomDevice
- #include <ProgMem.h>
Storage device to access PROGMEM using flash API.
Public Functions
-
inline virtual size_t getBlockSize() const override
Obtain smallest allocation unit for erase operations.
-
inline virtual size_t getSize() const override
Obtain addressable size of this device.
- Returns
size_t – Must be at least as large as the value declared in the partition table
-
inline virtual Type getType() const override
Obtain device type.
-
virtual bool read(uint32_t address, void *dst, size_t size) override
Read data from the storage device.
- Parameters
address – Where to start reading
dst – Buffer to store data
size – Size of data to be read, in bytes.
- Returns
bool – true on success, false on error
-
inline virtual bool write(uint32_t address, const void *src, size_t size) override
Write data to the storage device.
- Parameters
address – Where to start writing
src – Data to write
size – Size of data to be written, in bytes.
- Returns
bool – true on success, false on error
-
inline virtual bool erase_range(uint32_t address, size_t size) override
Erase a region of storage in preparation for writing.
- Parameters
address – Where to start erasing
size – Size of region to erase, in bytes
- Returns
bool – true on success, false on error
-
inline virtual size_t getBlockSize() const override
-
class SpiFlash : public Storage::Device
- #include <SpiFlash.h>
Main flash storage device.
Public Functions
-
virtual size_t getBlockSize() const override
Obtain smallest allocation unit for erase operations.
-
virtual size_t getSize() const override
Obtain addressable size of this device.
- Returns
size_t – Must be at least as large as the value declared in the partition table
-
inline virtual Type getType() const override
Obtain device type.
-
virtual uint32_t getId() const override
Obtain device ID.
- Returns
uint32_t – typically flash chip ID
-
virtual bool read(uint32_t address, void *dst, size_t size) override
Read data from the storage device.
- Parameters
address – Where to start reading
dst – Buffer to store data
size – Size of data to be read, in bytes.
- Returns
bool – true on success, false on error
-
virtual bool write(uint32_t address, const void *src, size_t size) override
Write data to the storage device.
- Parameters
address – Where to start writing
src – Data to write
size – Size of data to be written, in bytes.
- Returns
bool – true on success, false on error
-
virtual bool erase_range(uint32_t address, size_t size) override
Erase a region of storage in preparation for writing.
- Parameters
address – Where to start erasing
size – Size of region to erase, in bytes
- Returns
bool – true on success, false on error
-
virtual size_t getBlockSize() const override
-
class StreamDevice : public Storage::CustomDevice
- #include <StreamDevice.h>
Read-only partition on a stream object.
Note
Writes not possible as streams always append data, cannot do random writes
Public Functions
-
inline virtual Type getType() const override
Obtain device type.
-
inline virtual bool read(uint32_t address, void *buffer, size_t len) override
Read data from the storage device.
- Parameters
address – Where to start reading
dst – Buffer to store data
size – Size of data to be read, in bytes.
- Returns
bool – true on success, false on error
-
inline virtual bool write(uint32_t address, const void *data, size_t len) override
Write data to the storage device.
- Parameters
address – Where to start writing
src – Data to write
size – Size of data to be written, in bytes.
- Returns
bool – true on success, false on error
-
inline virtual bool erase_range(uint32_t address, size_t len) override
Erase a region of storage in preparation for writing.
- Parameters
address – Where to start erasing
size – Size of region to erase, in bytes
- Returns
bool – true on success, false on error
-
inline virtual Type getType() const override
-
class SysMem : public Storage::CustomDevice
- #include <SysMem.h>
Storage device to access system memory, e.g. RAM.
Public Functions
-
inline virtual size_t getBlockSize() const override
Obtain smallest allocation unit for erase operations.
-
inline virtual size_t getSize() const override
Obtain addressable size of this device.
- Returns
size_t – Must be at least as large as the value declared in the partition table
-
inline virtual Type getType() const override
Obtain device type.
-
inline virtual bool read(uint32_t address, void *buffer, size_t len) override
Read data from the storage device.
- Parameters
address – Where to start reading
dst – Buffer to store data
size – Size of data to be read, in bytes.
- Returns
bool – true on success, false on error
-
inline virtual bool write(uint32_t address, const void *data, size_t len) override
Write data to the storage device.
- Parameters
address – Where to start writing
src – Data to write
size – Size of data to be written, in bytes.
- Returns
bool – true on success, false on error
-
inline virtual bool erase_range(uint32_t address, size_t len) override
Erase a region of storage in preparation for writing.
- Parameters
address – Where to start erasing
size – Size of region to erase, in bytes
- Returns
bool – true on success, false on error
-
inline virtual size_t getBlockSize() const override
-
void initialize()
References
Used by
Sming (main) ,Component