RFC: Flash Storage with Isolation

[atar13]

Hello!

@tyler-potyondy, @Samir-Rashid and I are porting OpenThread to Tock.

Goals

We are looking to utilize Tock's flash features so that OpenThread can save some state persistently.

We have the following two constraints:

1. We want full isolation between flash storage of user applications. Ex: another application should not be able to read/write to OpenThread's flash storage region.
2. We want to store up to around 4KB of flash data.

Issues

We have looked into Tock's flash offerings and have come across two potential options:

• app_flash_driver
• nonvolatile_storage_driver

According to our current understanding of these features, they are insufficient for our needs. Can anyone confirm if the following is an accurate understanding and that there is no way to do this flash isolation without keeping the flash region in memory?

app_flash_driver

• Here is how we understand the app_flash_driver to work:

  • The user application reserves region of flash at app-compile time. See this example in libtock-c
  • Tock maps flash region to user program memory (similar to mmap) See this libtock-c comment
  • Need to manually sync entire mapped memory region to flash device. See here

• Pros:

  • Isolation is given.

tock/capsules/extra/src/app_flash_driver.rs

Lines 101 to 111 in 01918fa

	// Check that this is a valid range in the app's flash.
	let flash_length = kernel_data
	.get_readonly_processbuffer(ro_allow::BUFFER)
	.map_or(0, |buffer| buffer.len());
	let (app_flash_start, app_flash_end) = processid.get_editable_flash_range();
	if flash_address < app_flash_start
	|| flash_address >= app_flash_end
	|| flash_address + flash_length >= app_flash_end
	{
	return Err(ErrorCode::INVAL);
	}

• Cons:
  • We must write entire flash region at once. This is problematic since we might need up to 4KB in flash and only need to make updates to a few bytes at a time.
  • Entire flash region must be mapped into memory. We are potentially memory constrained and storing 4KB in memory is not ideal.

nonvolatile_storage_driver

• How we think it works:
  • Can read/write anywhere on flash device. Only protection provided is between user and kernel flash.

tock/capsules/extra/src/nonvolatile_storage_driver.rs

Lines 218 to 229 in 01918fa

	// Do bounds check.
	match command {
	NonvolatileCommand::UserspaceRead | NonvolatileCommand::UserspaceWrite => {
	// Userspace sees memory that starts at address 0 even if it
	// is offset in the physical memory.
	if offset >= self.userspace_length
	|| length > self.userspace_length
	|| offset + length > self.userspace_length
	{
	return Err(ErrorCode::INVAL);
	}
	}

• Pros:
  • Allows us to write a few bytes at a time (unlike our entire flash region in app_flash_driver)
  • Don't need to keep our flash region mapped to memory (unlike app_flash_driver).
• Cons:
  • Provides no isolation of flash storage between user applications.

Question

With these two existing offerings, it seems that it is not possible for us to have flash isolation while avoiding mapping our entire persistent state to memory. One potential solution is to add application-level isolation to the existing non-volatile driver. This can re-use the mechanisms that app_state uses for reserving space at app-compile time. Otherwise, we are open to suggestions on how to proceed.

[bradjc]

I agree with your assessment.

In brief: don't use app_state. This breaks app signing and we don't want to prevent signing thread apps.

In nonvolatile_storage_driver, I did write:

tock/capsules/extra/src/nonvolatile_storage_driver.rs

Lines 9 to 10 in 01918fa

	//! the entire memory space that has been provided to userland. Future revisions
	//! should update this to limit applications to only their allocated regions.

At the time we didn't have a mechanism for restricting regions to specific apps. Now we do with ShortID. I would edit the nonvolatile_storage_driver with something like what is in screen_shared, i.e.

tock/capsules/extra/src/screen_shared.rs

Lines 78 to 81 in 01918fa

	pub struct AppScreenRegion {
	app_id: kernel::process::ShortID,
	frame: Frame,
	}

[bradjc]

I should add, I don't think changing nonvolatile_storage_driver.rs should require changing the syscall interface. So it should be possible to work on open thread with full flash before any isolation is supported, or in parallel with updating the capsule.

[atar13]

Sounds good! We'll move forward with making the appropriate changes to nonvolatile_storage_driver.

[atar13]

I should add, I don't think changing nonvolatile_storage_driver.rs should require changing the syscall interface. So it should be possible to work on open thread with full flash before any isolation is supported, or in parallel with updating the capsule.

I agree. We already have OpenThread ported to use nonvolatile_storage_driver, so nothing should have to change on the user-space side.

[atar13]

@bradjc , we were looking into implementing isolation between apps for the non volatile storage driver and came up with two potential approaches. We wanted to get some feedback from you on which approach makes more sense to have in Tock.

Option 1: hardcode each app's flash regions in board's main.rs

In the screen_shared capsule that you referenced here we found that the screen regions were allocated at compile time in the board's main.rs where the capsule is initialize. In the following snippet you can see that 3 screen regions for 3 apps are hardcoded:

tock/boards/makepython-nrf52840/src/main.rs

Lines 518 to 545 in 25d9f42

	let apps_regions = static_init!(
	[capsules_extra::screen_shared::AppScreenRegion; 3],
	[
	capsules_extra::screen_shared::AppScreenRegion::new(
	kernel::process::ShortId::Fixed(core::num::NonZeroU32::new(crc("circle")).unwrap()),
	0, // x
	0, // y
	8 * 8, // width
	8 * 8 // height
	),
	capsules_extra::screen_shared::AppScreenRegion::new(
	kernel::process::ShortId::Fixed(core::num::NonZeroU32::new(crc("count")).unwrap()),
	8 * 8, // x
	0, // y
	8 * 8, // width
	4 * 8 // height
	),
	capsules_extra::screen_shared::AppScreenRegion::new(
	kernel::process::ShortId::Fixed(
	core::num::NonZeroU32::new(crc("tock-scroll")).unwrap()
	),
	8 * 8, // x
	4 * 8, // y
	8 * 8, // width
	4 * 8 // height
	)
	]
	);

We could mimic this design for nonvolatile storage. Before the user flashes the kernel, they have to manually specify which range of flash storage correspond to each app. Instead of specifying address ranges, the user could specify how many bytes of flash they want for each app and the kernel will figure out where to put them (like app_state?).

Pros

• Simple to implement
• We know exactly which regions of flash belong to each process when the board boots up

Cons

• Requires modifying main.rs before flashing kernel
• Can't dynamically use more flash than given at compile time

Option 2: dynamic flash allocation

The other approach is to dynamically allocate Pages of non-volatile storage to apps. The flash would be split up into many fixed size pages. Then processes can ask for more flash blocks when they need space.

Implementing this will require a master table inside the capsule to keep track of which AppId owns which page.

Freeing pages is complicated. Processes would have to explicitly free pages because processes may still want to read the data on next power cycle. The OS should not delete user data without permission. Also, the virtual addressing makes it complicated to delete any page that is not at the end of the allocation.

This table gets lost if the board loses power, so allocating new pages has to synchronously commit the table to flash.

Having a dynamic amount of space available also makes the abstraction confusing to the application developer. This is a worse version of a filesystem.

Pros:

• Do not need to specify regions at compile time, which can be error prone
• User doesn't have to recompile the kernel
• Can grow storage usage to the whole flash

Cons:

• Complex logic for address translation
• Need to store tables in memory
• Need tables to be persistent in flash so that we know which pages belong to which process on board reboot

[alistair23]

Option 3: There is a third option which gets a bit of the best parts from both.

main() calls a function like this to setup the flash storage

/// Reserve a region of flash for application usage.
/// This flash must not be used by the kernel
///
/// start_address is the flash address to start the reservation and size
/// is the number of bytes to reserve.
/// app_size is the maximum size a single application can reserve. It must
/// be smaller or equal to `size`
fn reserve_app_flash(start_address: usize, size: usize, app_size: usize) -> Result<>

Which reserves a region of flash for applications and sets a maximum application size.

Then when an application starts it run a syscall to get the app_size

// Return the maximum allocation size for a single application
size_t app_flash_get_max_alloc_size();

An application can then allocate up to that size

// Allocate a region of flash of size `size` bytes in flash
// The access permissions are determined by read and write IDs in
// the `perms` struct
int app_flash_allocate_size(size_t size, access_permissions perms);

The Tock capsule just needs to append a header at the start of the region with the reserved size and read/write permission information. That way on power up the capsule can iterate through the reserved region to find all of the existing regions.

An application can also free a region, in which case the capsule will just erase the entire region (if it can, otherwise it will just write 0's to clear the data). There is a possibility of fragmentation, but it hopefully won't be too bad.

The other issue is alignment as you are adding a header to the region the app requested. That could end up wasting an entire flash region to just store a few bytes. But if the size isn't page aligned that shouldn't matter.

Pros

• Do not need to specify regions at compile time, which can be error prone
• User doesn't have to recompile the kernel
• Can grow storage usage to the whole flash
• Applications can request the storage they need, while also avoiding a single application using all of the flash
• We can determine exactly which regions of flash belong to each process when the board boots up
• Moves address complexity to the kernel, each application just operates on a region of flash from 0 to size

Cons

• Need to store headers for each region
• Has address translation in the kernel capsule
• Doesn't allow full dynamic memory growth/shrinkage
• Applications need to estimate how much flash they need at startup
• Possible fragmentation if apps free the region

[bradjc]

I think all of these are good designs, and the benefit of capsules is we don't have to choose, we could support all of them, and let board authors decide what meets their needs.

I'm not too worried about freeing allocated storage as that doesn't seem like a common use case for nonvolatile storage.

I think for options 2/3, the main question is: use a linked list-esque structure (like TBFs), or a single table for tracking allocations and metadata. The table approach seems more straightforward.

[alistair23]

I think for options 2/3, the main question is: use a linked list-esque structure (like TBFs), or a single table for tracking allocations and metadata. The table approach seems more straightforward.

A table approach is difficult in flash. You basically need to store the table in memory and erase the section at some point, which is risky in terms of data corruption.

linked list-esque is probably the way to go