So this is something I'd consider posting to StackOverflow, but it feels like it may not be 100% appropriate for SO and as we know, if you ask something not appropriate, you get knocked down pretty quickly. While this is a technical question, it is also an opinion one as well, so I'm not sure. I figure if I'm not sure, I might as well just ask here and not worry about the SO Overlords approving of my question. I approve, and that's all that matters. ;)
Ok, so let me start with some background. Almost two years ago I created a simple Cordova app that demonstrated writing to a file: Cordova Example: Writing to a file. I won't repeat all of that code here, but this is the simple utility function I wrote to support a basic logging system.
function writeLog(str) {
if(!logOb) return;
var log = str + " [" + (new Date()) + "]\n";
console.log("going to log "+log);
logOb.createWriter(function(fileWriter) {
fileWriter.seek(fileWriter.length);
var blob = new Blob([log], {type:'text/plain'});
fileWriter.write(blob);
console.log("ok, in theory i worked");
}, fail);
}
logOb
is a File object made on startup. This method simply takes input, like "Button clicked", prefixes it with a timestamp and then writes it to the end of a file. Notice, and this is crucial, it has no result. It just does its work and ends. To me, that's appropriate for a logging system, you don't really care when it finished.
Until you do...
Back on that earlier post, someone noticed an issue when they did:
writeLog("actionOne fired 00000");
writeLog("actionOne fired AAAAA");
writeLog("actionOne fired BBBBB");
Basically only one or two items were being appended. Looking at the code now, it is pretty obvious why. Since the logic is async, the multiple calls to it firing at the same time mean the file object is being manipulated by multiple calls at the same time, leading to random, unreliable results.
So the fix is easy, right? Add a callback to writeLog
or perhaps a Promise (shiny). Then it is the user's responsibility to ensure they never fire call #2 till call #1 is done. Maybe something like this:
writeLog('msg').then(function() {
writeLog('msg2');
}).then(function() {
writeLog('msg3');
});
That's not too bad, but here's where my question comes in. What if we made writeLog
(and pretend for a moment that I have it in a module) more intelligent so it simple queued up writes? So in other words, if I did N calls, it would simply handle storing the 2-N calls in an array and processing them one by until done. I'd essentially hide the async nature from the caller.
Now everything about me says that smells wrong, but on the other hand, I'm curious. Is there ever a time where you would think this is appropriate? Certainly for a logging system it seems ok. At worse I send 100K calls to it and the app dies before it can finish, but in theory, we could live with that.
Thoughts?
Archived Comments
Since you mention at the end you can live with missed writes, I do think this would be reasonable. A couple of thoughts which might make it more robust if you were going to rework it a bit. First, you could have the actual write process handle all pending writes when it runs. Second, and while this wouldn't catch everything, you could process.on('exit',writeOutstanding) or do the same with window.unload or the current preferred browser/cordova closing event.
Good thoughts - thanks!
I would definitely wrap this in something that looks/feels/behaves synchronous but hides its inner "asynchronisityishness." You just want to log; you don't want to deal with your logging tool handing you promises and stuff to deal with.
When building brackets-git project I run into a similar problem. The application is coded in a way that it considers Git to be async but that doesn't work well because Git works with a file system and fails when two executions are trying to lock the same files. So I had to hide this behavior implementing a promise queue (because we also care about the result, unlike the samples above). Looks something like this:
For a problem like this check out Subjects and Observables in rxjs, instead of using promises. There is a great presentation on this by Jafar Husain at Netflix. If you haven't seen it I can Post the link.
What you want for something like this is a system with back pressure support. That way, you can pump messages in slow, fast, or in bursts. Then the observer can processes them as fast as it can. With Observables you can take messages one at a time, in batches, filter out messages, etc. There are a lot of options for working with the data coming through.
There are definitely situations where sending async calls to the background is a huge advantage. Such as sending an email after closing an http connection. This is a major benefit with aysnc I/O runtimes such as nodejs. Accomplishing such things in many languages can be painful.
My guess is this code is crashing because you are initializing and reading the length of the file on every single log call so this will consume cpu and memory exponentially.
If you open only one file descriptor you should be able to treat `fileWriter.write(blob);` as a queue in that the messages to the file will happen in the order it is called.
FWIW - I think this would make an excellent interview question as its very common "real world" scenario.
It isn't crashing, just writing data incorrectly. But I can see how it could crash too which is just another problem. :)
I'm fighting learning that. ;) My first encounter was with Angular2 and it felt overly complex. But so did Promises at first. At FluentConf, the most packed session was on rxjs.
It takes some getting used to, but now that I have I'm a big fan. I use them for my Actions and properties in my Stores in my reactjs app. Someday I'll write up a blog on it.
I can't help but share this video, I saw a version of this at a conference and it was the best session I've seen in a long time. https://youtu.be/5uxSu-F5Kj0