Twitter: raymondcamden


Address: Lafayette, LA, USA

Proof of Concept - Build a download feature for IndexedDB

08-23-2012 9,435 views jQuery, JavaScript, HTML5 10 Comments

Before I begin, a quick editorial note. I almost didn't write this blog entry. After working on the code and getting everything working right, things quickly went to crap when I switched from Mac to Windows. I had odd results in Firefox as well. Overall, I feel that the solution I've come up with here is solid, but the current browser implementations are... less than ideal. So, please keep that in mind. Perhaps you are reading this a year from now while cruising around on your jetpack and the browsers have settled down in terms of their IndexedDB support. Perhaps. Until then, please consider that what follows is going to be less than perfect in your browser.

IndexedDB is a nice way to store massive amounts of data on the user's machine. This allows for personal storage of - well - just about anything. Browsers are still working on their implementation, and the feature can be a bit... tricky (see my earlier posts), but overall it can be an incredibly powerful feature.

I was thinking that it might be interesting to build a way to export and save data from an IndexedDB datasource. Why bother when the data is local? I don't know. Maybe as a way to save a 'version' to a USB stick. Maybe as a way to upload later to another machine. To be honest, I just wanted to build it and see what it took.

Thinking about the process, I broke it down to a few steps.

First - we need to get all the data from our datasource. IndexedDB has a simple way to iterate over an objectstore (think table). What isn't so easy though is handling the fact that this is an async operation. If you have more than one objectstore you have to wait until all are done.

Second - once you have all the data, you need to serialize it. Luckily we can rely on the browser's native JSON support to quickly convert it.

The third and final step is to stream it to the user. Fellow Adobian Christian Cantrell has a good blog entry on saving JavaScript data. But I used a modified version that made use of HTML5's new "download" attribute for anchors.

Simple enough? I decided to begin with an earlier application I wrote that allowed for simple Note creation. If you've got a recent Firefox installed, you can play with it right now:

http://www.raymondcamden.com/demos/2012/aug/23/test.html

This will not work in Chrome... sometimes... due to the issue I reported here. Oddly - Google Canary, on Mac only, seems to work now - perfectly. That's the main browser I used for testing. But the exact same Canary on Windows did not work. Confusing - I know.

Even if you aren't using a browser that will handle the demo, I encourage you to hit the page and view source. I'm going to be sharing lots of snippets as we go on.

To make the application a bit more complex I added a new objectstore called log. This is defined here:

This objectstore will simply contain log messages. I modified my code so that when I created and deleted notes it would simply log the actions. To simplify this I wrapped the logic into a nice utility function:

The end result is that my application is using two objectstores. One main "note" objectstore for the actual content and one called "log" which isn't actually shown to the user.

I began the process of adding export support by adding a nice button to the top right of the application. Clicking this button begins the process I defined above. As I mentioned, the first step was to actually get all the data. Because this is involves N asynchronous processes, I decided to make use of jQuery Deferreds. jQuery Deferreds are black magic to me still. I have the hardest time wrapping my head around them but I was able to get something working. I'm betting there are nicer ways of doing this and I hope my readers can share some tips. Basically though I loop through each objectstore in the database (and note this code is entirely abstract - it should work for any IndexedDB instance) and create a new Deferred to handle the data collection for an individual objectstore. When done looping over the data, I resolve the deferred by returning an array of objects. Finally, $.when is used to collect all of this.

Let's talk about the last few lines above. You can see where I stringify the entire data set in one line. That's damn convenient. Any browser that supports IndexedDB will support the JSON object so it's a no brainer to use.

Sending the data to the user was also pretty simple. You can see where I - initially - made use of the "Cantrell Solution" (yes, I'm using that term because it sounds cool ;). While this worked, it didn't allow for a file name.

To get around this, I added an empty link to my DOM. That may not be necessary, I could have just made one in JavaScript, but it was quick and worked. If you view source you will see this in the layout: <a id="exportLink" download="data.json"></a>

Again - I feel kinda bad just dropping this into the page like that. I'd probably do it entirely virtually in the future. But note the download attribute. That's all it takes to 'suggest' a filename for downloading. That's it! So given that I had a jQuery hook to the link already, I simply set the HREF equal to my serialized data.

I initially tried to trigger a click, but for some reason, this didn't work correctly. Luckily I found a solution on StackOverflow - the fakeClick function. You can see it yourself if you view source.

Unfortunately, Firefox does not quite work right with the download attribute. It should be coming soon, but for me, it never worked right. That means to truly test this demo, the only browser I know of that can do it all is Google Canary on OS X. Hopefully that will change soon.

So - despite all the buts and warnings above - I hope this is an interesting demonstration for my readers. As always, I'd love to hear your feedback on how this could be improved.

10 Comments

  • Commented on 01-22-2013 at 2:32 PM
    I think this is a great idea. I'm very actively pursuing the idea of IndexedDB databases as individual files, at least in the sense of one thing per database. IndexedDB makes a great basis for structure.

    Backups of your own data are paramount, and this is a key reason to support moving data into local storage, I think.

    However, I want to make one caveat with the solution mentioned here, because any time you take IndexedDB data and push it through JSON, you can loose information or cause errors. IndexedDB can store anything valid by the HTML Structured Clone Algorithm, which is anything in JSON + Dates, Files, Blobs, RegExps, and undefined values. Any of these types will choke JSON.stringify(), sadly.

    This is still a good technique, but you either need 1) to write your own serializer and parser or 2) use it on a database you only allow a subset of possible types into.
  • Commented on 01-22-2013 at 2:37 PM
    Great point. I haven't played much with binary data in IndexedDB yet.
  • Rob #
    Commented on 02-07-2013 at 7:37 PM
    Just wondering if you could use a webWorker per objectStore to retrieve the data?
  • Commented on 02-07-2013 at 8:33 PM
    It isn't yet supported, at least not in Firefox:

    http://stackoverflow.com/questions/14479431/use-in...
  • Mann #
    Commented on 07-10-2013 at 4:30 AM
    Hi Raymond,

    Really this is a very very useful thing, but when i included the downloadable feature to my application, it is not downloading the JSON file, even if it is not responding to the download button.

    Can you please help me for that problem.
  • Commented on 07-10-2013 at 6:08 AM
    I'd need more details. What browser are you using? Can you add a console.log to the event handler and see if it even runs?
  • Mann #
    Commented on 07-10-2013 at 11:39 PM
    Hi,

    Like exporting IndexedDB data as a JSON file is possible as described by you above, is it possible to import data from a JSON file to the IndexedDB is possible, if possible, then can you give some code snipset.
  • Commented on 07-11-2013 at 6:42 AM
    It is possible. I don't have a code snippet, but it would be something like so:

    1) Add a file field.
    2) When the user hits a button, check the value of the file field and if they picked something, read it. (You can do that now in modern browsers.)
    3) Once you confirm it was a JSON packet, you can then iterate and insert.
  • Mike Jones #
    Commented on 06-08-2014 at 9:49 AM
    Hello Raymond

    Thank you very much fro your example. I have been trying for two and half days to get a project working right where I needed Indexeddb to return some data before something else should happen.

    I adapted your code to my situation and it worked perfectly.

    Thanks and best wishes from York, UK
  • Commented on 06-08-2014 at 10:07 AM
    Glad to know!

Post Reply

Please refrain from posting large blocks of code as a comment. Use Pastebin or Gists instead. Text wrapped in asterisks (*) will be bold and text wrapped in underscores (_) will be italicized.

Leave this field empty