Earlier this week I posted about hitting the limits of LocalStorage (Blowing up LocalStorage) and today I thought I'd do a bit of testing around IndexedDB. Unfortunately, I don't really have a simple "if you do this, X happens" type story to tell, but I did find out some interesting things about storage limits. I want to thank the following people for help in writing this post: Ben Kelly of Mozilla, Joshua Bell of Google, Addy Osmani of Google, and Paul Irish of Google.
So before we begin, let's talk limits. This is what MDN has to say:
There isn't any limit on a single database item's size, however there is in some cases a limit on each IndexedDB database's total size. This limit (and the way the user interface will assert it) varies from one browser to another:
- Firefox has no limit on the IndexedDB database's size. The user interface will just ask permission for storing blobs bigger than 50 MB. This size quota can be customized through the
dom.indexedDB.warningQuota
preference (which is defined in http://mxr.mozilla.org/mozilla-central/source/modules/libpref/src/init/all.js).- Google Chrome: see https://developers.google.com/chrome...rage#temporary.
This is - unfortunately - not quite correct. (But frankly, MDN being as awesome as it is gets a pass for not being perfect.) I mentioned to Ben Kelly that it was a bit weird that Firefox would only prompt for one big insert but be ok with a bunch of small ones. In fact, my test script (more on that below), inserted a 3 meg-ish Base64 image. I ran it about 100+ times or so and never got a prompt. Ben pointed out in this GitHub issue (Rethink about the storage model) that the prompting for "one big blob" had been removed.
One thing I'll point out before going further - storage as a general concept for browsers is in a huge state of flux right now. There is chaos. That's a bit frustrating, but it is really good that these conversations are happening now. In my mind this should have happened before Web Audio and Animation crap, but I'm a nerd who likes databases and IDB doesn't demo as well as Unreal in the browser. ;) If you want to see some of the current thinking in regards to storage, see: WhatWG Storage
Ok, so going back to that MDN quote - the link for Chrome is also incorrect. If you follow it, you will see that IndexedDB is described as temporary. Under persistent storage, it even says this:
Persistent storage is storage that stays in the browser unless the user expunges it. It is available only to apps that use the Files System API, but will eventually be available to other offline APIs like IndexedDB and Application Cache.
But I got confirmation that this is no longer true. But... it is possible that Chrome may delete your IDB. If space on the host machine is low, then Chrome will clear out an IDB data based on a LRU policy. It will delete the entire local database - it will not trim. And to be clear, we are talking about one IDB database instance, not all of them.
Firefox will also follow a similar procedure. If disk space becomes an issue, it will clear out IDB.
So given that there isn't a real good way to test quota, I was kinda curious to see what would happen if I abused IDB a bit. I wrote the following script which, on button click, would insert the base64 version of a 3 meg ish image. By the way, this code is pretty bad. I just noticed I convert the image on every click. I should cache the string in RAM while I test. But you get the idea - click a button - insert a bunch of crap.
<!doctype html>
<html>
<head>
<meta name="viewport" content="user-scalable=no, initial-scale=1, maximum-scale=1, minimum-scale=1, width=device-width, height=device-height" />
</head>
<body>
<script>
var db;
imgurl = "baby.jpg";
function urlTo64(u, cb) {
var xhr = new XMLHttpRequest();
xhr.open('GET', imgurl, true);
xhr.responseType = 'blob';
xhr.onload = function(e) {
if (this.status == 200) {
// get binary data as a response
var blob = this.response;
var reader = new FileReader();
reader.readAsDataURL(blob);
reader.onloadend = function() {
base64data = reader.result;
cb(base64data);
}
}
};
xhr.send();
}
function indexedDBOk() {
return "indexedDB" in window;
}
document.addEventListener("DOMContentLoaded", function() {
//No support? Go in the corner and pout.
if(!indexedDBOk()) return;
var openRequest = indexedDB.open("bighonkingtest",1);
openRequest.onupgradeneeded = function(e) {
var thisDB = e.target.result;
console.log("running onupgradeneeded");
if(!thisDB.objectStoreNames.contains("crap")) {
thisDB.createObjectStore("crap", {keyPath:"id",autoIncrement:true});
}
}
openRequest.onsuccess = function(e) {
console.log("running onsuccess");
db = e.target.result;
console.log("Current Object Stores");
console.dir(db.objectStoreNames);
//Listen for add clicks
document.querySelector("#addButton").addEventListener("click", addData, false);
}
openRequest.onerror = function(e) {
//Do something for the error
}
},false);
function addData(e) {
console.log("About to add data");
urlTo64(imgurl, function(s) {
console.log("s size",s.length);
//Get a transaction
//default for OS list is all, default for type is read
var transaction = db.transaction(["crap"],"readwrite");
//Ask for the objectStore
var store = transaction.objectStore("crap");
//Define data
var data = {
img:s
}
//Perform the add
var request = store.add(data);
request.onerror = function(e) {
console.log("Error",e.target.error.name);
//some type of error handler
}
request.onsuccess = function(e) {
console.log("Woot! Did it");
}
});
}
</script>
<button id="addButton">Add Data</button>
</body>
</html>
I did a lot of testing, and by testing, I mean I just clicked like crazy. It took a while, but I finally got an error in Firefox:
Ben Kelly and I spoke more on Twitter (like, a few seconds) ago, and he added some more information about Firefox:
- Yes, it will evict (i.e. kill) an IDB by a LRU (Least Recently Used) policy.
- The max is dynamic and based on your hard drive.
He had these details to add: "Heurestic is roughly: all origin combined can take up to 50% available disk space, no one origin more than 20% available." "Err... no one origin more than 20% of the total allowed for all origins. So thats actually 20%*50%=10% of available disk."
Jonathan Smith wrote an interesting little JS snippet you can paste into your console to check the size of an IDB table: getIndexedDbSize.
If I read his results right, I got Firefox up to about 2.8 gigs of storage before it threw that error. My drive maxes out at 500 gigs. So if Firefox can take 10% of that and one origin can take 20%, then 2.8 feels certainly within the ballpark.
For Chrome, I couldn't get it to throw a QuotaErr, and eventually Smith's test script ended up crashing the tab. It is also possible I just gave up before I hit the upper limit.
I didn't test in Safari because of how horrible they have screwed up IDB. I don't even want to think of it. Opera worked pretty much the same as Chrome.
So - take aways? IDB is still good "persistent" storage in terms of how it has never been 100% perfect persistent storage. A user has always been able to go into dev tools and screw crap up. So knowing the browser itself may nuke it based on storage issues isn't a deal breaker for me. And as I mentioned above - this whole area is in motion and needs to improve. And it will - I have faith.
Archived Comments
Great work here, Raymond. As always, thanks for sharing. I've been doing my own research into this. I need to reliably store up to 50MB of JSON data, which will include Base64 encrypted photos, on mobile devices. I found a couple of FANTASTIC resources I'd like to share with you.
http://www.html5rocks.com/e...
http://demo.agektmr.com/sto... (<-- this would have saved you a ton of work)
One more, here's an IndexDB shim...
https://github.com/axemclio...
Which I learned about from this smashingmag article...
http://www.smashingmagazine...
I didn't mind writing my own "Abuser" - it was kind of fun. ;)
Oops, my reply is to your other comment - not this one. :)
I just read this post after discovering for myself that Safari IDB is still horribly broken. I get a dialog asking if I want to increase the allocated space above what a site originally required, and even though I allow it, Safari always throws an error. This was one of the many bugs that I encountered the last time I was trying to get IDB working on Safari via IndexedDBShim, and it's still broken after all this time. </rant>
Great article, thanks! But what about IE11?
I didn't test yet. I've got a VM though...
I tested. Ouch. Going to post a followup.
Just posted.
We posted an update about how this all works in Firefox, here:
https://developer.mozilla.o...
I will reach out to other browser vendors soon and ask them to start adding more info to cover their products too. Hope this is useful.
Chris Mills, Mozilla
Interesting. I'm still reading but the first thing I noticed was that if you do *not* specify a storage type, it will default to temporary storage for web apps. I didn't know that.
Looks like a great doc, thank you for adding this.
Any toughts on www.forerunnerdb.com ??
Never seen it. On a quick glance, it looks like a nice library.
Small correction about chrome's behavior when space is low. The origins are sorted by use and the temporary storage associated with them are deleted, starting with least recently used, until space is not low. We delete origins mostly wholesale. Cookies and localstorage are excepted, for odd reasons. Wholesale means that ALL of an origins' IDB instances are deleted together (along with appcache, cache storage, websql, etc), not one at a time as is kind of implied in your post.
Who told you that IDB instances are deleted one at a time? (respond privately if you'd prefer) I will follow up with them.
Thanks for the update. To be honest, I don't remember who it was. It may have been Addy or Paul, but it was a twitter conversation, and I'd say it was much more likely that I misunderstood.
Hey Raymond, Great work,
I have few queries regarding indexeddb.
I am developing a hybrid application for android and I need to know the size limit of indexeddb. How can I get it. And other thing I need to ask, is there any way to get total size of your indexeddb?
Afaik, no, outside of getting all the data and checking the length. I
believe there are some Chrome extensions that will do this.
hey Raymond, Very nice article, I am working on IDB, My Question is, I wanted to keep at max 50 recent records into IDB, is there way to do that, Currently what I am thinking is i will check the number of rows and it exceeds 50 I will delete older records. Is there any better way to handle this scenario.
As far as I know that's the only way to do it - count and then insert if < 50. Don't forget that the objectStore API includes a count() method.