Proof of Concept - Build a download feature for IndexedDB

This post is more than 2 years old.

Before I begin, a quick editorial note. I almost didn't write this blog entry. After working on the code and getting everything working right, things quickly went to crap when I switched from Mac to Windows. I had odd results in Firefox as well. Overall, I feel that the solution I've come up with here is solid, but the current browser implementations are... less than ideal. So, please keep that in mind. Perhaps you are reading this a year from now while cruising around on your jetpack and the browsers have settled down in terms of their IndexedDB support. Perhaps. Until then, please consider that what follows is going to be less than perfect in your browser.

IndexedDB is a nice way to store massive amounts of data on the user's machine. This allows for personal storage of - well - just about anything. Browsers are still working on their implementation, and the feature can be a bit... tricky (see my earlier posts), but overall it can be an incredibly powerful feature.

I was thinking that it might be interesting to build a way to export and save data from an IndexedDB datasource. Why bother when the data is local? I don't know. Maybe as a way to save a 'version' to a USB stick. Maybe as a way to upload later to another machine. To be honest, I just wanted to build it and see what it took.

Thinking about the process, I broke it down to a few steps.

First - we need to get all the data from our datasource. IndexedDB has a simple way to iterate over an objectstore (think table). What isn't so easy though is handling the fact that this is an async operation. If you have more than one objectstore you have to wait until all are done.

Second - once you have all the data, you need to serialize it. Luckily we can rely on the browser's native JSON support to quickly convert it.

The third and final step is to stream it to the user. Fellow Adobian Christian Cantrell has a good blog entry on saving JavaScript data. But I used a modified version that made use of HTML5's new "download" attribute for anchors.

Simple enough? I decided to begin with an earlier application I wrote that allowed for simple Note creation. If you've got a recent Firefox installed, you can play with it right now:

https://static.raymondcamden.com/demos/2012/aug/23/test.html

This will not work in Chrome... sometimes... due to the issue I reported here. Oddly - Google Canary, on Mac only, seems to work now - perfectly. That's the main browser I used for testing. But the exact same Canary on Windows did not work. Confusing - I know.

Even if you aren't using a browser that will handle the demo, I encourage you to hit the page and view source. I'm going to be sharing lots of snippets as we go on.

To make the application a bit more complex I added a new objectstore called log. This is defined here:

This objectstore will simply contain log messages. I modified my code so that when I created and deleted notes it would simply log the actions. To simplify this I wrapped the logic into a nice utility function:

The end result is that my application is using two objectstores. One main "note" objectstore for the actual content and one called "log" which isn't actually shown to the user.

I began the process of adding export support by adding a nice button to the top right of the application. Clicking this button begins the process I defined above. As I mentioned, the first step was to actually get all the data. Because this is involves N asynchronous processes, I decided to make use of jQuery Deferreds. jQuery Deferreds are black magic to me still. I have the hardest time wrapping my head around them but I was able to get something working. I'm betting there are nicer ways of doing this and I hope my readers can share some tips. Basically though I loop through each objectstore in the database (and note this code is entirely abstract - it should work for any IndexedDB instance) and create a new Deferred to handle the data collection for an individual objectstore. When done looping over the data, I resolve the deferred by returning an array of objects. Finally, $.when is used to collect all of this.

Let's talk about the last few lines above. You can see where I stringify the entire data set in one line. That's damn convenient. Any browser that supports IndexedDB will support the JSON object so it's a no brainer to use.

Sending the data to the user was also pretty simple. You can see where I - initially - made use of the "Cantrell Solution" (yes, I'm using that term because it sounds cool ;). While this worked, it didn't allow for a file name.

To get around this, I added an empty link to my DOM. That may not be necessary, I could have just made one in JavaScript, but it was quick and worked. If you view source you will see this in the layout: <a id="exportLink" download="data.json"></a>

Again - I feel kinda bad just dropping this into the page like that. I'd probably do it entirely virtually in the future. But note the download attribute. That's all it takes to 'suggest' a filename for downloading. That's it! So given that I had a jQuery hook to the link already, I simply set the HREF equal to my serialized data.

I initially tried to trigger a click, but for some reason, this didn't work correctly. Luckily I found a solution on StackOverflow - the fakeClick function. You can see it yourself if you view source.

Unfortunately, Firefox does not quite work right with the download attribute. It should be coming soon, but for me, it never worked right. That means to truly test this demo, the only browser I know of that can do it all is Google Canary on OS X. Hopefully that will change soon.

So - despite all the buts and warnings above - I hope this is an interesting demonstration for my readers. As always, I'd love to hear your feedback on how this could be improved.

Raymond Camden's Picture

About Raymond Camden

Raymond is a senior developer evangelist for Adobe. He focuses on document services, JavaScript, and enterprise cat demos. If you like this article, please consider visiting my Amazon Wishlist or donating via PayPal to show your support. You can even buy me a coffee!

Lafayette, LA https://www.raymondcamden.com

Archived Comments

Comment 1 by Calvin Spealman posted on 1/23/2013 at 1:32 AM

I think this is a great idea. I'm very actively pursuing the idea of IndexedDB databases as individual files, at least in the sense of one thing per database. IndexedDB makes a great basis for structure.

Backups of your own data are paramount, and this is a key reason to support moving data into local storage, I think.

However, I want to make one caveat with the solution mentioned here, because any time you take IndexedDB data and push it through JSON, you can loose information or cause errors. IndexedDB can store anything valid by the HTML Structured Clone Algorithm, which is anything in JSON + Dates, Files, Blobs, RegExps, and undefined values. Any of these types will choke JSON.stringify(), sadly.

This is still a good technique, but you either need 1) to write your own serializer and parser or 2) use it on a database you only allow a subset of possible types into.

Comment 2 by Raymond Camden posted on 1/23/2013 at 1:37 AM

Great point. I haven't played much with binary data in IndexedDB yet.

Comment 3 by Rob posted on 2/8/2013 at 6:37 AM

Just wondering if you could use a webWorker per objectStore to retrieve the data?

Comment 4 by Raymond Camden posted on 2/8/2013 at 7:33 AM

It isn't yet supported, at least not in Firefox:

http://stackoverflow.com/qu...

Comment 5 by Mann posted on 7/10/2013 at 1:30 PM

Hi Raymond,

Really this is a very very useful thing, but when i included the downloadable feature to my application, it is not downloading the JSON file, even if it is not responding to the download button.

Can you please help me for that problem.

Comment 6 by Raymond Camden posted on 7/10/2013 at 3:08 PM

I'd need more details. What browser are you using? Can you add a console.log to the event handler and see if it even runs?

Comment 7 by Mann posted on 7/11/2013 at 8:39 AM

Hi,

Like exporting IndexedDB data as a JSON file is possible as described by you above, is it possible to import data from a JSON file to the IndexedDB is possible, if possible, then can you give some code snipset.

Comment 8 by Raymond Camden posted on 7/11/2013 at 3:42 PM

It is possible. I don't have a code snippet, but it would be something like so:

1) Add a file field.
2) When the user hits a button, check the value of the file field and if they picked something, read it. (You can do that now in modern browsers.)
3) Once you confirm it was a JSON packet, you can then iterate and insert.

Comment 9 by Mike Jones posted on 6/8/2014 at 6:49 PM

Hello Raymond

Thank you very much fro your example. I have been trying for two and half days to get a project working right where I needed Indexeddb to return some data before something else should happen.

I adapted your code to my situation and it worked perfectly.

Thanks and best wishes from York, UK

Comment 10 by Raymond Camden posted on 6/8/2014 at 7:07 PM

Glad to know!

Comment 11 by cregox posted on 11/30/2015 at 9:49 AM

> "I was thinking that it might be interesting to build a way to export and save data from an IndexedDB datasource. Why bother when the data is local? I don’t know. Maybe as a way to save a ‘version’ to a USB stick. Maybe as a way to upload later to another machine. To be honest, I just wanted to build it and see what it took."
Awesome! ^_^

But do you happen to know of any way without using stringify?

https://codenewbie.slack.co...

I'd love to find such mechanism for enabling sharing saved states on my Mariox ported Ai ANN neat algo, you can see on basiux.org or livecoding ( https://www.livecoding.tv/c... ).

Comment 12 (In reply to #11) by Raymond Camden posted on 11/30/2015 at 11:53 AM

I assume you also mean w/o building the string by hand, since you could do that too.

If so - then no - I don't. Sorry. :\

Obviously you could take the string and zip it and save a zip file instead, but you still need to get the data in a usable form first.

Comment 13 by cregox posted on 11/30/2015 at 5:13 PM

This is what I've tried using, but it's not quite fully working for me (yet): https://gist.github.com/rob...

Comment 14 (In reply to #13) by Raymond Camden posted on 11/30/2015 at 9:31 PM

How is it not working?

Comment 15 (In reply to #14) by cregox posted on 11/30/2015 at 11:21 PM

Same reason from previous comment. `Stringify` crashes on my object. Before it was giving a message that lead to believe it was due to object size, now it just completely kill the process. It's always coming back to serializing...

Comment 16 (In reply to #15) by Raymond Camden posted on 12/1/2015 at 1:59 AM

What does your Object look like? Could it be an infinite loop? Ie, User has Roles and Roles have Users and Users have Roles and etc etc.

Comment 17 (In reply to #16) by cregox posted on 12/6/2015 at 5:13 AM

It's `pool.state` under basiux.org and if it has any circular reference it's within `.species` and it does make a lot of sense! I should have thought of that, but... Surprisingly, I couldn't find any such thing there.

Comment 18 (In reply to #17) by Raymond Camden posted on 12/6/2015 at 3:58 PM

Cool - glad you found it!