IndexedDB and Limits

This post is more than 2 years old.

Earlier this week I posted about hitting the limits of LocalStorage (Blowing up LocalStorage) and today I thought I'd do a bit of testing around IndexedDB. Unfortunately, I don't really have a simple "if you do this, X happens" type story to tell, but I did find out some interesting things about storage limits. I want to thank the following people for help in writing this post: Ben Kelly of Mozilla, Joshua Bell of Google, Addy Osmani of Google, and Paul Irish of Google.

So before we begin, let's talk limits. This is what MDN has to say:

There isn't any limit on a single database item's size, however there is in some cases a limit on each IndexedDB database's total size. This limit (and the way the user interface will assert it) varies from one browser to another:

This is - unfortunately - not quite correct. (But frankly, MDN being as awesome as it is gets a pass for not being perfect.) I mentioned to Ben Kelly that it was a bit weird that Firefox would only prompt for one big insert but be ok with a bunch of small ones. In fact, my test script (more on that below), inserted a 3 meg-ish Base64 image. I ran it about 100+ times or so and never got a prompt. Ben pointed out in this GitHub issue (Rethink about the storage model) that the prompting for "one big blob" had been removed.

One thing I'll point out before going further - storage as a general concept for browsers is in a huge state of flux right now. There is chaos. That's a bit frustrating, but it is really good that these conversations are happening now. In my mind this should have happened before Web Audio and Animation crap, but I'm a nerd who likes databases and IDB doesn't demo as well as Unreal in the browser. ;) If you want to see some of the current thinking in regards to storage, see: WhatWG Storage

Ok, so going back to that MDN quote - the link for Chrome is also incorrect. If you follow it, you will see that IndexedDB is described as temporary. Under persistent storage, it even says this:

Persistent storage is storage that stays in the browser unless the user expunges it. It is available only to apps that use the Files System API, but will eventually be available to other offline APIs like IndexedDB and Application Cache.

But I got confirmation that this is no longer true. But... it is possible that Chrome may delete your IDB. If space on the host machine is low, then Chrome will clear out an IDB data based on a LRU policy. It will delete the entire local database - it will not trim. And to be clear, we are talking about one IDB database instance, not all of them.

Firefox will also follow a similar procedure. If disk space becomes an issue, it will clear out IDB.

So given that there isn't a real good way to test quota, I was kinda curious to see what would happen if I abused IDB a bit. I wrote the following script which, on button click, would insert the base64 version of a 3 meg ish image. By the way, this code is pretty bad. I just noticed I convert the image on every click. I should cache the string in RAM while I test. But you get the idea - click a button - insert a bunch of crap.


<!doctype html>
<html>
<head>
<meta name="viewport" content="user-scalable=no, initial-scale=1, maximum-scale=1, minimum-scale=1, width=device-width, height=device-height" />
</head>

<body>

<script>
var db;
imgurl = "baby.jpg";

function urlTo64(u, cb) {
  var xhr = new XMLHttpRequest();
  xhr.open('GET', imgurl, true);
  xhr.responseType = 'blob';

  xhr.onload = function(e) {
    if (this.status == 200) {
      // get binary data as a response
      var blob = this.response;
      var reader = new FileReader();
      reader.readAsDataURL(blob);
      reader.onloadend = function() {
        base64data = reader.result;
        cb(base64data);
      }
    }
  };
  xhr.send();

}

function indexedDBOk() {
	return "indexedDB" in window;
}

document.addEventListener("DOMContentLoaded", function() {

	//No support? Go in the corner and pout.
	if(!indexedDBOk()) return;

	var openRequest = indexedDB.open("bighonkingtest",1);

	openRequest.onupgradeneeded = function(e) {
		var thisDB = e.target.result;

		console.log("running onupgradeneeded");

		if(!thisDB.objectStoreNames.contains("crap")) {
			thisDB.createObjectStore("crap", {keyPath:"id",autoIncrement:true});
		}

	}

	openRequest.onsuccess = function(e) {
		console.log("running onsuccess");

		db = e.target.result;

		console.log("Current Object Stores");
		console.dir(db.objectStoreNames);

		//Listen for add clicks
		document.querySelector("#addButton").addEventListener("click", addData, false);
	}

	openRequest.onerror = function(e) {
		//Do something for the error
	}


},false);


function addData(e) {
	console.log("About to add data");

  urlTo64(imgurl, function(s) {
    console.log("s size",s.length);
  	//Get a transaction
  	//default for OS list is all, default for type is read
  	var transaction = db.transaction(["crap"],"readwrite");
  	//Ask for the objectStore
  	var store = transaction.objectStore("crap");

  	//Define data
  	var data = {
  		img:s
  	}

  	//Perform the add
  	var request = store.add(data);

  	request.onerror = function(e) {
  		console.log("Error",e.target.error.name);
  		//some type of error handler
  	}

  	request.onsuccess = function(e) {
  		console.log("Woot! Did it");
  	}
  });

}
</script>

<button id="addButton">Add Data</button>

</body>
</html>

I did a lot of testing, and by testing, I mean I just clicked like crazy. It took a while, but I finally got an error in Firefox:

ff1

Ben Kelly and I spoke more on Twitter (like, a few seconds) ago, and he added some more information about Firefox:

  1. Yes, it will evict (i.e. kill) an IDB by a LRU (Least Recently Used) policy.
  2. The max is dynamic and based on your hard drive.

He had these details to add: "Heurestic is roughly: all origin combined can take up to 50% available disk space, no one origin more than 20% available." "Err... no one origin more than 20% of the total allowed for all origins. So thats actually 20%*50%=10% of available disk."

Jonathan Smith wrote an interesting little JS snippet you can paste into your console to check the size of an IDB table: getIndexedDbSize.

If I read his results right, I got Firefox up to about 2.8 gigs of storage before it threw that error. My drive maxes out at 500 gigs. So if Firefox can take 10% of that and one origin can take 20%, then 2.8 feels certainly within the ballpark.

For Chrome, I couldn't get it to throw a QuotaErr, and eventually Smith's test script ended up crashing the tab. It is also possible I just gave up before I hit the upper limit.

I didn't test in Safari because of how horrible they have screwed up IDB. I don't even want to think of it. Opera worked pretty much the same as Chrome.

So - take aways? IDB is still good "persistent" storage in terms of how it has never been 100% perfect persistent storage. A user has always been able to go into dev tools and screw crap up. So knowing the browser itself may nuke it based on storage issues isn't a deal breaker for me. And as I mentioned above - this whole area is in motion and needs to improve. And it will - I have faith.

Raymond Camden's Picture

About Raymond Camden

Raymond is a senior developer evangelist for Adobe. He focuses on document services, JavaScript, and enterprise cat demos. If you like this article, please consider visiting my Amazon Wishlist or donating via PayPal to show your support. You can even buy me a coffee!

Lafayette, LA https://www.raymondcamden.com

Archived Comments

Comment 1 by Chris Geirman posted on 4/19/2015 at 12:44 AM

Great work here, Raymond. As always, thanks for sharing. I've been doing my own research into this. I need to reliably store up to 50MB of JSON data, which will include Base64 encrypted photos, on mobile devices. I found a couple of FANTASTIC resources I'd like to share with you.
http://www.html5rocks.com/e...
http://demo.agektmr.com/sto... (<-- this would have saved you a ton of work)

Comment 2 by Chris Geirman posted on 4/19/2015 at 12:46 AM

One more, here's an IndexDB shim...
https://github.com/axemclio...

Which I learned about from this smashingmag article...
http://www.smashingmagazine...

Comment 3 (In reply to #2) by Raymond Camden posted on 4/19/2015 at 11:22 AM

I didn't mind writing my own "Abuser" - it was kind of fun. ;)

Comment 4 (In reply to #3) by Raymond Camden posted on 4/19/2015 at 11:22 AM

Oops, my reply is to your other comment - not this one. :)

Comment 5 by matt posted on 4/20/2015 at 2:33 PM

I just read this post after discovering for myself that Safari IDB is still horribly broken. I get a dialog asking if I want to increase the allocated space above what a site originally required, and even though I allow it, Safari always throws an error. This was one of the many bugs that I encountered the last time I was trying to get IDB working on Safari via IndexedDBShim, and it's still broken after all this time. </rant>

Comment 6 by Vladimir Havenchyk posted on 4/24/2015 at 12:42 PM

Great article, thanks! But what about IE11?

Comment 7 (In reply to #6) by Raymond Camden posted on 4/24/2015 at 12:46 PM

I didn't test yet. I've got a VM though...

Comment 8 (In reply to #7) by Raymond Camden posted on 4/24/2015 at 1:00 PM

I tested. Ouch. Going to post a followup.

Comment 9 (In reply to #7) by Raymond Camden posted on 4/24/2015 at 1:51 PM

Just posted.

Comment 10 by Chris Mills posted on 6/11/2015 at 3:05 PM

We posted an update about how this all works in Firefox, here:

https://developer.mozilla.o...

I will reach out to other browser vendors soon and ask them to start adding more info to cover their products too. Hope this is useful.

Chris Mills, Mozilla

Comment 11 (In reply to #10) by Raymond Camden posted on 6/22/2015 at 3:24 PM

Interesting. I'm still reading but the first thing I noticed was that if you do *not* specify a storage type, it will default to temporary storage for web apps. I didn't know that.

Comment 12 (In reply to #10) by Raymond Camden posted on 6/22/2015 at 3:26 PM

Looks like a great doc, thank you for adding this.

Comment 13 by Luis Rodriguez posted on 7/11/2015 at 6:23 PM

Any toughts on www.forerunnerdb.com ??

Comment 14 (In reply to #13) by Raymond Camden posted on 7/12/2015 at 11:54 AM

Never seen it. On a quick glance, it looks like a nice library.

Comment 15 by dgrogan posted on 7/23/2015 at 5:39 PM

Small correction about chrome's behavior when space is low. The origins are sorted by use and the temporary storage associated with them are deleted, starting with least recently used, until space is not low. We delete origins mostly wholesale. Cookies and localstorage are excepted, for odd reasons. Wholesale means that ALL of an origins' IDB instances are deleted together (along with appcache, cache storage, websql, etc), not one at a time as is kind of implied in your post.

Who told you that IDB instances are deleted one at a time? (respond privately if you'd prefer) I will follow up with them.

Comment 16 (In reply to #15) by Raymond Camden posted on 7/23/2015 at 6:03 PM

Thanks for the update. To be honest, I don't remember who it was. It may have been Addy or Paul, but it was a twitter conversation, and I'd say it was much more likely that I misunderstood.

Comment 17 by Bilal Jawwad posted on 9/9/2015 at 5:59 AM

Hey Raymond, Great work,
I have few queries regarding indexeddb.
I am developing a hybrid application for android and I need to know the size limit of indexeddb. How can I get it. And other thing I need to ask, is there any way to get total size of your indexeddb?

Comment 18 (In reply to #17) by Raymond Camden posted on 9/12/2015 at 1:41 AM

Afaik, no, outside of getting all the data and checking the length. I
believe there are some Chrome extensions that will do this.

Comment 19 by Gururaj posted on 11/28/2016 at 5:29 AM

hey Raymond, Very nice article, I am working on IDB, My Question is, I wanted to keep at max 50 recent records into IDB, is there way to do that, Currently what I am thinking is i will check the number of rows and it exceeds 50 I will delete older records. Is there any better way to handle this scenario.

Comment 20 (In reply to #19) by Raymond Camden posted on 11/28/2016 at 4:00 PM

As far as I know that's the only way to do it - count and then insert if < 50. Don't forget that the objectStore API includes a count() method.