Disk Storage

Provides support for using disk storage devices.

Block Devices

Internal devices such as SPI flash may be read and written in bytes, however disk devices must be accessed in blocks. Historically these blocks are referred to as ‘sectors’, which are 512 bytes for SD cards.

Sector-addressable (block) devices should be based on Disk::BlockDevice. An example is the Storage::SD::Card provided by the SD Storage library. Note that devices larger than 4GB will require the ENABLE_STORAGE_SIZE64 project setting.

Partitioning

Partitions can be enumerated and created using this library. Both MBR and GPT partitioning schemes are supported.

Call Storage::Disk::scanPartitions() to populate the device partition table. For each partition, additional information can be obtained via Storage::Partition::diskpart() call.

Partition information is written using Storage::Disk::formatDisk().

Buffering

Block devices are typically used with a sector-based filing system such as FAT (see FatIFS). Note that these filing systems are generally unsuitable for use with SPI flash without some kind of intermediate wear-levelling management layer as provided by SD card controllers.

By default, block devices must be strictly accessed only by sector, i.e. for SD cards aligned chunks of 512 bytes. This is rather inflexible so BlockDevice supports byte-level access using internal buffering, which applications may enable using the allocateBuffers method.

This allows other filing systems to be used. LittleFS seems to work OK, although SPIFFS IFS Library does not. Partitions may also be used directly without any filing system.

Important

The Device::sync() method must be called at appropriate times to ensure data is actually written to disk. Filing systems should take care of this internally when files are closed, for example.

Testing

Linux provides excellent support for testing generated image files.

Windows users may find this tool useful: https://www.diskinternals.com/linux-reader/.

Configuration

DISK_MAX_SECTOR_SIZE

default: 512

Determines the minimum supported read/write block size for storage devices.

For example, AF disks use 4096-byte sectors so internal reads and writes must be a multiple of this value. This will increase the internal buffer sizes and so consume more RAM.

Acknowledgements

Code in this library is based on http://elm-chan.org/fsw/ff/00index_e.html. It has been heavily reworked using structures (from the Linux kernel) and separated from the actual FAT filing system implementation (FatIFS).

API

namespace Disk

Typedefs

using SysTypes = BitSet<uint8_t, SysType>

Enums

enum class Error

Values:

enumerator XX
enum class SysType : uint8_t

Identifies exact disk volume type.

Values:

enumerator unknown

Partition type not recognised.

enumerator fat12
enumerator fat16
enumerator fat32
enumerator exfat
enum SysIndicator

MBR partition system type indicator values.

See also

https://en.wikipedia.org/wiki/Partition_type#List_of_partition_IDs

Values:

enumerator SI_FAT12
enumerator SI_FAT16

FAT16 with fewer than 65536 sectors.

enumerator SI_FAT16B

FAT16B with 65536 or more sectors.

enumerator SI_IFS
enumerator SI_EXFAT
enumerator SI_FAT32X

FAT32 with LBA.

Functions

template<typename T>
T align_up(T value, uint32_t align)
template<typename T>
auto getBlockCount(T byteCount, uint32_t blockSize)
uint32_t crc32_byte(uint32_t crc, uint8_t d)
uint32_t crc32(uint32_t bcc, const void *data, size_t length)
inline uint32_t crc32(const void *data, size_t length)
inline bool operator!(Error err)
Error formatDisk(BlockDevice &device, GPT::PartitionTable &table, const Uuid &diskGuid = {})

Partition a device using the GPT scheme.

Parameters:

  • device –

  • table – Partitions to create

Return values:

Error –

Error formatDisk(BlockDevice &device, MBR::PartitionTable &table)

Partition a device using the MBR scheme.

Parameters:

  • device –

  • table – Partitions to create

Return values:

Error –

inline SysType getSysTypeFromIndicator(SysIndicator si)
Error validate(BasePartitionTable &table, storage_size_t firstAvailableBlock, storage_size_t totalAvailableBlocks, uint32_t blockSize)

Validate partition table entries.

  • If size <= 100 then the actual size is calculated as a percentage and updated

  • If offset = 0 then a suitable location is found and the offset updated

On success, partition entries are ordered by position.

Parameters:

  • firstAvailableBlock – First block number which may be allocated to a partition

  • totalAvailableBlocks – Number of blocks available for partition allocation

  • blockSize – Size of a block

Return values:

Error – For each partition:

bool scanPartitions(Device &device)

Variables

constexpr uint32_t PARTITION_ALIGN = {0x100000U}
static constexpr SysTypes fatTypes = SysType::fat12 | SysType::fat16 | SysType::fat32 | SysType::exfat
class BlockDevice : public Storage::Device
#include <BlockDevice.h>

Base class for sector-addressable (block) devices.

Inherited classes must set the sectorCount value, and implement the four raw_xxx methods.

This class supports byte-level access using internal buffering, which if required must be enabled by the application via the allocateBuffers method. Without buffering, read/writes must always be sector-aligned. Rrase must always be sector-aligned.

For power-loss resiliency it is important to call sync() at appropriate times. Filing system implementations should do this after closing a file, for example. Applications should consider this if leaving files open for extended periods, and explicitly call ‘flush’ on the filing system or ‘sync’ on the partition or device if necessary.

Subclassed by Storage::Disk::HostFileDevice, Storage::SD::Card, USB::MSC::LogicalUnit

Public Functions

virtual bool read(storage_size_t address, void *dst, size_t size) override

Read data from the storage device.

Parameters:

  • address – Where to start reading

  • dst – Buffer to store data

  • size – Size of data to be read, in bytes.

Return values:

bool – true on success, false on error

virtual bool write(storage_size_t address, const void *src, size_t size) override

Write data to the storage device.

Parameters:

  • address – Where to start writing

  • src – Data to write

  • size – Size of data to be written, in bytes.

Return values:

bool – true on success, false on error

virtual bool erase_range(storage_size_t address, storage_size_t size) override

Erase a region of storage in preparation for writing.

Parameters:

  • address – Where to start erasing

  • size – Size of region to erase, in bytes

Return values:

bool – true on success, false on error

inline virtual size_t getBlockSize() const override

Obtain smallest allocation unit for erase operations.

inline virtual storage_size_t getSize() const override

Obtain addressable size of this device.

Return values:

storage_size_t – Must be at least as large as the value declared in the hardware configuration

inline virtual storage_size_t getSectorCount() const override

Obtain total number of sectors on this device.

virtual bool sync() override

Flush any pending writes to the physical media.

Devices with intermediate buffering should implement this method.

Return values:

bool – Return false if sync operation failed.

bool allocateBuffers(unsigned numBuffers)

Set number of sector buffers to use.

Required to support byte-level read/write operations on block devices. Buffering can improve performance, with diminishing returns above around 4 sectors.

Parameters:

numBuffers – Number of buffers to allocate 1,2,4,8,etc. Pass 0 to deallocate/disable buffering.

Return values:

bool – false on memory allocation error, or if failed to flush existing buffers to disk

struct Stat
#include <BlockDevice.h>

Public Members

Func func[3]

Read, Write, Erase.

std::map<uint32_t, Func> sectors

By sector.

struct Func
#include <BlockDevice.h>
struct Buffer : public std::unique_ptr<uint8_t[]>
#include <Buffer.h>
class BufferList
#include <Buffer.h>
class HostFileDevice : public Storage::Disk::BlockDevice
#include <HostFileDevice.h>

Create custom storage device using backing file.

Public Functions

HostFileDevice(const String &name, const String &filename, storage_size_t size)

Construct a file device with custom size.

Parameters:

  • name – Name of device

  • filename – Path to file

  • size – Size of device in bytes

HostFileDevice(const String &name, const String &filename)

Construct a device using existing file.

Device will match size of existing file

Parameters:

  • name – Name of device

  • filename – Path to file

inline virtual String getName() const override

Obtain unique device name.

inline virtual Type getType() const override

Obtain device type.

struct DiskPart
#include <PartInfo.h>

Adds information specific to MBR/GPT disk partitions.

Subclassed by Storage::Disk::PartInfo

Public Functions

size_t printTo(Print &p) const

Print full contents of this structure.

Public Members

Uuid typeGuid

GPT type GUID.

Uuid uniqueGuid

GPT partition unique GUID.

SysType systype = {}

Identifies volume filing system type.

SysIndicator sysind = {}

Partition sys value.

struct PartInfo : public Storage::Partition::Info, public Storage::Disk::DiskPart
#include <PartInfo.h>

In-memory partition information.

A disk Storage::Partition refers to this instance.

Public Functions

inline virtual const Disk::DiskPart *diskpart() const override

Obtain additional disk information.

Accessed via Partition::diskpart() method

virtual size_t printTo(Print &p) const override

Print important fields only.

class BasePartitionTable : public OwnedLinkedObjectListTemplate<PartInfo>
#include <PartInfo.h>

Common type for MBR/GPT partition table.

Subclassed by Storage::Disk::GPT::PartitionTable, Storage::Disk::MBR::PartitionTable

class Scanner
#include <Scanner.h>

Class to iterate through disk partition tables.

Supports MBR and GPT partitioning schemes.

Public Functions

std::unique_ptr<PartInfo> next()

Obtains the next partition entry (if any)

class SectorBuffer : public std::unique_ptr<uint8_t[]>
#include <SectorBuffer.h>

Buffer for working with disk sectors.

namespace EXFAT
namespace FAT
namespace GPT

Functions

String getTypeName(const Uuid &typeGuid)

Get string for known GPT type GUIDs.

struct SmingTypeGuid : public Uuid
#include <GPT.h>
class PartitionTable : public Storage::Disk::BasePartitionTable
#include <GPT.h>

Public Functions

inline bool add(const String &name, SysType sysType, storage_size_t offset, storage_size_t size, const Uuid &uniqueGuid = {}, const Uuid &typeGuid = {}, Partition::Flags flags = 0)

Add a new standard GPT partition definition.

Parameters:

  • name – Partition name

  • sysType – Intended content for this partition

  • offset – Start offset, or 0 to have position calculated

  • size – Size of partition (in bytes), or percentage (0-100) of total partitionable disk space

  • uniqueGuid – Unique partition identifier (optional: will be generated if not provided)

  • typeGuid – Partition type GUID (default is BASIC_DATA)

  • flags –

Return values:

bool – true on success

inline bool add(const String &name, Partition::FullType type, storage_size_t offset, storage_size_t size, const Uuid &uniqueGuid = {}, Partition::Flags flags = 0)

Add a new GPT partition for a regular Sming filing system.

Note

These partitions use a custom type GUID and won’t be recognised by external software.

Parameters:

  • name – Partition name

  • type – Sming partition type/subtype

  • offset – Start offset, or 0 to have position calculated

  • size – Size of partition (in bytes), or percentage (0-100) of total partitionable disk space

  • uniqueGuid – Unique partition identifier (optional: will be generated if not provided)

  • flags –

Return values:

bool – true on success

namespace MBR
class PartitionTable : public Storage::Disk::BasePartitionTable
#include <MBR.h>

Public Functions

inline bool add(SysType sysType, SysIndicator sysIndicator, storage_size_t offset, storage_size_t size, Partition::Flags flags = 0)

Add a new MBR partition definition.

Note

MBR does not have partition name field; this will appear as ‘mbr1’, ‘mbr2’, etc.

Parameters:

  • sysType – Intended content for this partition (or ‘unknown’)

  • SysIndicator – Appropriate system code SI_xxx

  • offset – Start offset, or 0 to have position calculated

  • size – Size of partition (in bytes), or percentage (0-100) of total partitionable disk space

  • flags –

Return values:

bool – true on success

References

Used by

Environment Variables

SoC support

  • esp32

  • esp32c2

  • esp32c3

  • esp32s2

  • esp32s3

  • esp8266

  • host

  • rp2040