FlashFS for microcontrollers

In microcontroller development, it is good practice to have the device have number storage to remember numbers between power resets. All this is called in its own way: StoreFS, FlashFS, Non-volatile Key-Value Map (a), HashMap (ka), NVS and more.

FlashFS can be useful for storing configurations of smart fancy microcircuits such as wireless transceivers, stepper motor drivers, etc. The presence of FlashFS allows an order of magnitude decrease number assemblies in the repository, since devices can always before program already at Run-Time(s). Literally opening the UART Shell, you can directly write configs to FlashFS in TeraTerm / Putty and reboot the gadget so that the new settings are applied in initialization. Easy. And there is no need to cook separate firmware with some specific settings when FlashFS is available. FlashFs adds dynamics to the program.

Why exactly on-chip NorFlash?

1–It’s cheap. As a rule, after rolling the firmware, there is always a couple more empty pages (sectors) of Flash (a). For it “already paid” when buying a microcontroller. There is no need to install separate off-chip SPI-NorFlash chips and thereby increase the dimensions, cost, expand logistics, reduce reliability, and increase the complexity of the device. FlashFS is provided by microcontroller manufacturers themselves. For example, in STM32, the first 4 sectors according to 16kbytes the rest are larger at 128kByte. This is done on purpose, with the expectation that it will run on small sectors of 16kByte FlashFS(s) with a configurable size.

2–On-chip FS safer than off-chip FS, since there is no way to connect to SPI or SDIO wires on the PCB and thus read the transmitted data by the logic analyzer.

However, Nor Flash memory has one problem. It can only be erased (filled with 0xFF(kami)) pages several kilobytes each (usually 4kByte, 8kByte, 16kByte, 64kByte, 128kByte). At the same time, the record is that you can only reset bits 1 to 0. Sometimes even it is forbidden up reset already reset bits, for example in byte 0x55. It’s like writing with a ballpoint pen on paper. What you write cannot be erased with an eraser. It makes sense to write the whole sheet, and then tear it out and throw it away.

This is how the API for working with On-Chip Nor-Flash memory usually looks like.

#ifndef FLASH_DRV_H
#define FLASH_DRV_H

#include <stdbool.h>
#include <stdint.h>

#include "flash_const.h"

#ifndef HAS_FLASH
#error "Add HAS_FLASH"
#endif

#ifndef HAS_MCU
#error "Add HAS_MCU"
#endif

#ifdef HAS_FLASH_WRITE
bool flash_wr(uint32_t addr, uint8_t* array, uint32_t array_len);
bool flash_wr4(uint32_t flash_addr, uint32_t* wr_array, uint32_t byte_size);
bool flash_errase(uint32_t addr, uint32_t len);
bool flash_erase_pages(uint8_t page_start, uint8_t page_end);
#endif

bool Addr2SectorSize(uint32_t addr, uint32_t *sector, uint32_t *sec_size);
bool flash_read(uint32_t in_flash_addr, uint8_t* rx_array, uint32_t array_len);
bool flash_init(void);
bool flash_scan(uint8_t* base, uint32_t size, float* usage_pec, uint32_t* spare, uint32_t* busy);
bool is_errased(uint32_t addr, uint32_t size);
bool is_flash_spare(uint32_t flash_addr, uint32_t size);
bool is_flash_addr(uint32_t flash_addr);

#endif /* FLASH_DRV_H */

It is more convenient for us, people, to simply record data. on key. It’s like being in a theatre. You give me your number, you get your trench coat. Or like in a phone book. You give a name, you get a phone number.

Need a specific abstraction level, which would provide such an API and do all the rest of the work under the hood with arrays of raw Flash memory. You give number and an array, the Array is written down. In a week you give number, and you receive an array.

It’s like writing with a pencil. You can write and you can erase with an eraser and write something else. Conveniently? Highly.

I will try to list the most basic requirements for such embedded on-chip file systems FlashFS.

1-Rational Nor-Flash memory usage (endurance optimization)

2–Protected data from sudden disappearance nutrition (power off tolerance).

3–Simplicity implementation

4–The file must be located quickly name

5–File must quickly enroll

6–File must quickly be updated

7–File must quickly wash

8–compactness code that implements this FlashFS algorithm. The less code, the more files.

Define terminology. What File?

File it is a named binary array of bytes in memory. Memory can be RAM, ROM (Flash), FRAM, EEPROM, SD cards. What does named mean? This means that this data can be accessed by value. Let it be a natural 16-bit number. It’s easier. Since this is an array, next to the data you also need to store and length this array. This is what an example entry title in FlashFs might look like.

struct xFlashFsFileHeader_t {
    uint16_t id;
    uint16_t nid; /* bit inverted id*/
    uint16_t length;
    uint8_t crc8;   /*only for payload*/
} __attribute__((packed)); /*to save flash memory*/
typedef struct xFlashFsFileHeader_t FlashFsFileHeader_t;

Main idea of ​​Flash FS

If you write files with the same ID, they will be written sequentially. This achieves data protection against power failure. You can always take the previous file. Each file is equipped with an 8-bit checksum. This will only detect real files and not just random numbers in memory. In order not to calculate the checksum for each indent every time, there is a preamble from ID (shnikov). This is the FileID and its inverted copy. Why is an inverted ID needed? This is to speed up the search for the desired file. The fact is that the CRC calculation procedure is a very lengthy procedure. It would be wasteful to calculate the CRC for each indent to understand whether the file is there or not. Therefore, the algorithm calculates CRC only for those indents where a valid preamble is written.

And now the flash memory page is full. What to do? It is necessary to clean the second page and copy into it in pairs the excellent freshest files.

When copying pages, only latest file versions. The old ones remain and are waiting for their removal along with the entire page. The next time you switch, they will be cleared along with the entire NorFlash page.

In addition to the functions of reading and writing files, we also need an auxiliary function that will just monitor the file system itself. As soon as page “B” overflows, the background procedure should execute the procedure toggle flash page, that is, clear page “A”. Take the latest files from page “B” and copy them to page “A”.

This is what an API for a microcontroller file system might look like

#ifndef NOR_FLASH_H
#define NOR_FLASH_H

#include <stdbool.h>
#include <stdint.h>

#include "flash_drv.h"
#include "flash_fs_config.h"
#include "flash_fs_types.h"

#ifndef HAS_FLASH
#error "+ HAS_FLASH"
#endif

#ifndef HAS_NVS
#error "+ HAS_NVS"
#endif

#ifndef HAS_FLASH_FS
#error "+ HAS_FLASH_FS"
#endif

#ifndef HAS_CRC8
#error "+HAS_CRC8"
#endif

#ifdef HAS_FLASH_FS_WRITE
bool flash_fs_format(void);
bool flash_fs_erase(void);
bool flash_fs_invalidate(uint16_t data_id);
bool flash_fs_set(uint16_t data_id, uint8_t* new_file, uint16_t new_file_len);
bool flash_fs_maintain(void);
bool flash_fs_turn_page(void);
#endif

bool flash_fs_is_active(uint8_t page_num);
bool flash_fs_init(void);
bool flash_fs_proc(void);
bool flash_fs_get(uint16_t data_id, uint8_t* value, uint16_t max_value_len, uint16_t* value_len);
bool flash_fs_get_active_page(uint32_t* flash_fs_page_start, uint32_t* flash_fs_page_len);
bool flash_fs_get_address(uint16_t data_id, uint8_t** value_address, uint16_t* value_len);
bool is_flash_fs_addr(uint32_t addr);
uint32_t flash_fs_get_page_size(uint8_t page_num);
uint32_t flash_fs_get_page_base_addr(uint8_t page_num);
uint32_t flash_fs_cnt_files(uint32_t start_page_addr, uint32_t page_len, uint32_t* spare_cnt);
uint32_t flash_fs_get_remaining_space(void);
uint8_t addr2page_num(uint32_t flash_fs_page_start);

#endif /* MEMORY_MANAGER_NOR_FLASH_H */

The dependencies of software components for FlashFs can be shown like this

Or so

This is what a file system diagnostic might look like (Figure 2). This is a list of files in the file system, their addresses, size, data inside, checksum, ID (shnik)

Dealt with the files. But reading raw data in memory is also somehow not very convenient. For a person, this is just a sequence of nimbles (hex bits). Gotta somehow interpret these data into real physical quantities and data types. The software component will take care of this. Param.

On top of Flash Fs, another level of abstraction must work. I call it parameters (Param). It works like this. You give the file ID and data (hex array with a length), you get the data type and its value.

This is what function prototypes for a component might look like param

#ifndef PARAM_DRV_H
#define PARAM_DRV_H

#include <stdbool.h>
#include <stdint.h>

#include "param_types.h"

#ifndef HAS_PARAM
#error "+HAS_PARAM"
#endif /*HAS_PARAM*/

bool param_init(void);
bool param_proc(void);

#ifdef HAS_PARAM_SET
bool param_set(Id_t id, uint8_t* in_data);
#endif /*HAS_PARAM_SET*/

bool param_get(Id_t id, uint8_t* out_data);
ParamType_t param_get_type(Id_t id);
uint16_t param_get_real_len(Id_t id) ;
uint16_t param_get_len(Id_t param_id);
uint16_t param_get_type_len(ParamType_t type_id);
uint32_t param_get_cnt(void);

#endif /* PARAM_DRV_H  */

Now you can work with this. Variables are clear, values ​​are interpreted. Success.

Conclusion

As you can see, programs for microcontrollers are built from levels of abstraction that work one on top of the other.

Now you can imagine how to make a non-volatile file system for storing all sorts of things (settings, firmware). It is not necessary to make the file system on on-chip NorFlash. You can also use off-chip NorFlash or EEPROM.

Add file systems to your firmware. There is nothing difficult in this.

If you need source codes for the implementation of the file system in C with unit tests, then write in a personal.

If you have something to add, then write in the comments.