-
Notifications
You must be signed in to change notification settings - Fork 764
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feature Request: Extra programming buffer for metadata or waiting to commit the new file until closed? #942
Comments
Even a |
Well, if you already have some file content planned I see the idea but the api for FS in Zephyr for example does not allow for that. It does however allow for a sync() and as stated, filesystem is in a valid state if filename is not yet committed to fs or if it has been, but users intent is not reflected by a 0 length file when in fact he may even have written several blocks of data but never got to close the file. Powerloss would be nicer if the initial file were either what the user wanted, or not at all? |
I think if the state upon power-loss would be no file this could be closer to what the user wanted (assuming the close to be a sync point). But on the other hand IIRC open+create on POSIX would be atomic too (and thus create an empty file), thus both views are valid. |
This issue comes at a really interesting time as I recently tried to implemented this (delaying file creation until first sync/close). Though my motivation was more to avoid 0-sized files, as they a real pain point for users, rather than performance/disk util. This commit may be worth a read, though I'll summarize: ba505c2 ba505c2 Implemented scratch file basics
Initially, my thinking was the same as @kyrreaa's, wait until the first sync/close to write out anything about the file. But as I was trying to implement this, I kept running into problems:
I think all of these problems ended up with at least theoretical solutions. But it also feels like the design is sort of fighting itself. And if we already would need to scan for empty mdirs during mount... Just setting an So that's my thoughts: an (or more likely a new file type because of encoding reasons, but that's details) But I hadn't really considered the performance savings of not writing the filename. Still, if the tradeoff is one
Yeah, POSIX expects open to create a file, so I think you're right. But given the number of issues that have been opened over this, I think avoiding 0-sized files is the best option for allowing users to easily write power-safe code. They can always call sync if they really want a 0-sized file. We already deviate from POSIX a bit when it comes to power-loss quality-of-life things. Truncate doesn't take effect until sync/close for example. ... Actually, reading the POSIX O_CREAT docs (here), I think they don't actually specify what size the new file should be? We could make new files 12-bytes for fun :) |
I do want to add these eventually. I was thinking just They're a win-win for everyone as they're a simpler API if you don't really need to manipulate file content, and if you only use get/set you can likely trim off a decent amount of code. All non-disk APIs are strictly low-priority for me atm, given other things.
Zephyr could always map sync to get/set behind the scenes if that accomplishes better behavior. Though while I don't want to make Zephyr dev's jobs harder, littlefs really shouldn't be limited by what APIs do or don't exist elsewhere. |
I think the issues people have really stems from the same that I did, the loss of power or SW reset due to bug in code etc causing the file to appear as 0-length even though data may actually have been written to many blocks. Since no reference is ever written regarding file length change or blocks allocated and filled the average user can not recover any data. |
I think this discussion is starting to converge with #523 (comment).
Journals are logs and littlefs's mdirs are logs, so I think this is roughly equivalent to calling Note that There's a batch of work that aims to address a number of the performance problems in littlefs. I think we need to at least wait for that to land before considering new APIs that are mostly performance motivated. |
I've been looking at how lfs currently creates a new file and it appears it could use more caching if RAM is not at a premium.
Committing a file id, and then a record of filename accompanied by a inline tag with length 0 to be able to remember the name and the fact that it is 0 size will in most cases not be very useful.
If one could wait with committing the file until one has a write that determines if an inline data attribute is needed, or if a CTZ may be used could be useful? One could potentially even wait until file is closed as there really is not much of a difference between a set of metadata saying a file exist and is 0 length or nothing at all when a whole CTZ linked array of blocks now may exist on the disk without any references to it. Regardless of what was committed the filesystem would still be in a valid state.
If a full write_size buffer of memory is provided for the metadata it could potentially be written by calling sync() or when filled up. Allowing the pcache to work with file-data only? This could also reduce the need for garbage collection / compacting as well.
I could see it would be nice to allow user control of this metadata flush to file/dir close, it being filled up or upon manual sync depending on situation and user needs.
The text was updated successfully, but these errors were encountered: