Based on some discussion earlier today on Twitter, I wanted to take a quick look at what happens when you exceed the quota limit in a browser's LocalStorage system. I knew an error would be thrown, but I was curious about the type, message, etc. I built a quick test and threw some browsers at it. This probably isn't the most scientific test, but here's what I found.
First, for my test I wanted a quick way to hit the "typical" limit of 5 megs per domain. To do that, I found an image of around one meg in size and then wrote code that would grab the binary bits, convert to base64, and then store it. Here's my test script.
/*
credit:
http://stackoverflow.com/a/18650249/52160
*/
function urlTo64(u, cb) {
var xhr = new XMLHttpRequest();
xhr.open('GET', imgurl, true);
xhr.responseType = 'blob';
xhr.onload = function(e) {
if (this.status == 200) {
// get binary data as a response
var blob = this.response;
var reader = new FileReader();
reader.readAsDataURL(blob);
reader.onloadend = function() {
base64data = reader.result;
cb(base64data);
}
}
};
xhr.send();
}
$(document).ready(function() {
console.log("Lets add some stuff");
imgurl = "xmax2010.jpg";
urlTo64(imgurl, function(s) {
console.log("urlTo64 result length",s.length);
for(var i=0;i<500;i++) {
if(i > 0 && i % 100 === 0) console.log("A set done");
try {
localStorage["bigimage"+i] = s;
} catch(e) {
console.log(e.toString());
console.dir(e);
break;
}
}
console.log("Done setting big images");
});
});
I'm using jQuery here, but I don't really need to. I loaded it up when I began but I didn't end up needing it. This isn't really important but I'm just trying to defend my somewhat shoddy test script. ;) The idea is simple - use XHR to fetch the bits of an image (hard coded to one on my test server), convert it to a DataURL and read in the base64 data. You can see that in the urlTo64 function. I then put this in a loop and tried to store the result. You'll notice my loop goes from 1 to 500. Originally it just went to 5. I'll explain the 500 when we get to Internet Explorer.
So - that's it. Read the binary, convert, store, and on error, print it out and stop working. Ok, so let's look at how different browsers handle this.
Chrome 41 (OSX)
Throws an exception with the name QuotaExceedError. The code is 22. The message is nice as it tells you what key it was trying to set:
Firefox 37 (OSX)
Throws an exception, but with a completely different name/code. The name is NS_ERROR_DOM_QUOTA_REACHED and the code is 1014.
Safari 8.0.5
Throws the same exception as Chrome (name and code anyway):
Internet Explorer 11 (Windows 10)
So, IE really threw me for a loop. When I ran my code (again, I had started with a loop of 5), it didn't throw an error. So I thought, ok, it has a bigger cache. So I added a 0. And then another 0. And another. I got it up to 5K calls and it still worked. That seemed... wrong. I did some Googling and turns out that IE supports a non-standard remainingSpace property. (Non-standard but a good idea imo. Client-storage does not help developers at all in terms of managing space.) When I inspected that value, it never seemed to change no matter how many additional 0s I added. I could inspect localStorage in the console and see all the values just fine.Then on a whim I tried something. I killed IE, re-opened it, and discovered that localStorage only had 2-3 of my images stored. From what I can tell, IE11 stopped storing items, but never threw an error! Which is really, really bad. Even worse, if you try to read the value, it reads it just fine, but on restart, it is gone. I'm not exactly sure what to think about that, outside of the fact that "silent fail" is the worst thing to happen to development since starting arrays at 0.
I tried an interesting little test. I checked remainingSpace before and after a set, and when the set fails, you can clearly see the space does not change. In theory, you could use this (on IE11 anyway) to confirm a proper save.
As an aside, Jonathan Sampson tried the latest Spartan build with my code and saw the same.
iOS Safari and PhoneGap/Cordova
It throws the exact same error as desktop Safari.
Android Chrome and PhoneGap/Cordova
It throws the exact same error as desktop Chrome.
Takeaway
So, yeah, what's the practical takeaway from this?
- Pretty much all client-side storage options suck really bad at quota. I love client-side storage (and have presented on it many times) but this is the biggest area of concern.
- Try to keep track of how you use client-side storage. So for example, if you saving the last 10 searches so you can display them to the user, know that and note it somewhere so that when you make use of client-side storage in the future ("Hey, can we cache some fonts?") you can check and see what you've stored already and see if you'll be hitting the limit possibly.
- Wrap everything in a try/catch? Um - maybe. ;) The "Super Strict Lets Do Everything Perfect" side of me says yes, but the "Practical I Live in the Real World" says that would probably be overkill. Again, going to my last point, as long as you keep track of what you're doing then I think you will be ok. Obviously I think folks will disagree with me here. Let me have it in the comments.
Archived Comments
Link to IE bug since it wasn’t linked in the post :) https://connect.microsoft.c...
Thanks - and thanks for including me in the daily.
I'd say it would be best to estimate your space usage first as a determiner for what storage type you use. I've been bitten by exceeding localStorage capacity in the past, so now use a mixture of creating/modifying local files (through cordova) and localStorage.
I have to disagree with this sentence: "I’m not exactly sure what to think about that, outside of the fact that “silent fail” is the worst thing to happen to development since starting arrays at 0." Silent failure is the worst, but starting arrays at 0 was a good idea. Nice post though.
No one starts counting at 0. I get that it was - originally - a performance thing - but that shouldn't matter now.
localStorage should always be in a try/catch because Safari in private mode will always throw #22. Breaks many sites.
I didn't test Opera until this morning, but as expected, it has the same result as Chrome, DOMException code 22.
It does take a little while to get used to, and it's not without its downsides (especially for beginners). To me though, it makes more sense once you get used to it because it leads to 0 <= index < length indexing, which makes things like representing empty ranges or accessing the (index % length)th array element more natural. Of course, part of it could be that starting with 0 for so long has shaped the way I think about indexing.
I'm pretty sure the issue with starting arrays at 0 is the off by one errors e.g The number of items in a list is different from the index of the last item in the list. This has caused a whole variety of problems and bugs that aren't always easy to spot.
Ahh so you can't even store anything in private mode? Damn.
https://github.com/chrisben...
Yes? I mean, can you clarify why you are linking to that and how it relates to the discussion?
Array indices starting at 0 is fine, the months starting there on the other hand...
That, I agree with 100%.
not sure I got the "overkill" part of using setItem inside a try/catch ... I mean, localStorage is already overkill since it's synchronous and heavy/blocking on slow SD cards, using the only possible safety belt as try/catch is in that case cannot overkill much. Altohugh, the problem is that once you have a failure, usually nobody has any idea what to do. There could be keys you don't own within the database, so clear is not an option unless you want be the most obtrusive code out there. I've used in many real-world projects this project which use at least a self-contained access to the database https://github.com/WebRefle... including localStorage, so that if you clear it, you remove only things that you own and not every entry of the db.
"I mean, localStorage is already overkill since it's synchronous and heavy/blocking on slow SD cards, using the only possible safety belt as try/catch is in that case cannot overkill much."
I guess it depends then. For me, I've never encountered a site slowed down by LS. If it was a performance issue for you, try/catch wouldn't really help there.
"Altohugh, the problem is that once you have a failure, usually nobody has any idea what to do."
Why? Unless your app *requires* LS, then it should be fine to handle cases where it isn't supported, or a read fails. If you are using LS to _enhance_ your app, then having it not be available should be something you can handle well.
"There could be keys you don't own within the database, so clear is not an option unless you want be the most obtrusive code out there."
I'm not really sure what you mean here. You only have access to keys w/n your own domain. You can easily clear your own values w/o impacting other sites. What you describe is only an issue if you are working on a large site w/ multiple teams and not coordinating your development efforts. But outside of that, you (foo.com) can't remove keys someone else made (goo.com).
> You only have access to keys w/n your own domain
no, every script in the same domain has access to it ... now tell me how well you ensure your keys do not conflicts with other keys, and how often you loop over all keys to grab only yours and eventually clean them if you have a size failure.
LS is more problematic when full and on "not-so-powerful" mobile device, on desktop evergreen browsers is not a big issue but also it makes little sense to use it since there a re better alternatives.
Um, I don't think you are explaining yourself right. If I set a localStorage value called name on raymondcamden.com, and you try to read it on foo.com, it will not work. Period. That's how LocalStorage works. Maybe you are trying to say something else and I'm not understanding you. Can you clarify? If not, please provide proof of how a script on another domain can read the values on my domain.
let's try again with a spoiler: hi, I am WebReflection, I've been doing web and mobile web development for the last 16 years and used and talked about JS/client-side databases for the last 6 or more: how are you doing?
- - - - - -
back to the problem:
**every script in the same domain has access to it** so, unless you are the only script and your script is not a library that other scripts could use, feel fre to use LS.clear();
In every other case you should loop over all keys an dbe sure what you are either setting or removing or overwriting is your own key ... YES, PER DOMAIN, unless you are the owner of such domain and you are the only script in it.
Developers don't usually do this, nobody is prefixing LS entries, not as common practice. So once you have the try/ catch failing, what do you do exactly? Show me one single block that deal with the problem and solve it, 'cause if you do, good for you, but I've never seen other libraries be that kind. Most common practice they just clear after keeping temporary in memory most needed own keys (without checking if anyn other key is own)
TL;DR ... the localStorage is a very poor API when it comes to cooperative scripts in the same domain: it does not scale in quote, it does not offer a way to be non obtrusive, it does not offer a way to easily avoid overwrites or clear of own property keys.
try/catch in all this mess is the last problem you have when it comes to "overkill", assuming we have the same meaning for the word overkill.
Err, are we measuring sexual organs here? I've been doing web dev for 20 years. It doesn't make me any more smarter than you or more right. Let's keep to the topic, shall we?
"**every script in the same domain has access to it** so, unless you are the only script and your script is not a library that other scripts could use, feel fre to use LS.clear();"
So basically this goes back to what I said earlier. If you are working on a team at Foo.com, then obviously you need to coordinate your use of LS. Just like you would need to coordinate Cookies too. And IndexedDB. Pretty much - hopefully - everything since you are working on the same web site. Your comments did not make it clear that you were speaking about that situation. I don't consider that an issue honestly. If you aren't coordinating with your coworkers to build your site than you are going to have multiple issues anyway.
In the *typical* use of LS, given that "you" represent the team for *your* site, then you are absolutely free to clear out whatever and use whatever w/o worry of messing stuff up.
In terms of library code - any library code that makes use of LS should clearly document it (ditto for cookies, IDB). I'm sure some libraries *do* make use of LS and don't document it, but you should blame the library, not LS. That's poor judgement/management on the author of the library and poor judgement on you (the generic you) on not reviewing the library before making use it.
it wasn't about competition, it was about "please stop teaching me and rather try to understand what I am saying" , thank you.
Although, it ended up clearly with a different vision/PoV of the matter.
Put in this way: I add a listener to the document, who can remove it? **only** only me, and only if I've kept a reference to such listener or the listener itself has a mechanism to be removed. **no other library/script/console-with-god-powers** should be able to remove that script. This is what I call a good cooperative API, and talking about DOM events, we have an excellent example of an API that protect your own business, and if you attach your business to some component and the page, other scripts, or your very same script, destroy such component in a dirty way, all your listeners attached to any removed node will be freed from RAM accordingly. You are protected, and so are others.
Now, you have local/sessionStorage, and a .clear() method that is handy when you have a situation like the one you described in this post: no more space.
What do you do? Some third-part library/script might have set keys that if you clean them will make the entire siite stop working or ... who knows ... and if it's not your own code that caused the saturation of the quota, what else can you do if not clearing the entire LS?
In few words, a try/catch around a setItem is not the real issue or the overkill, ,because everything else could go wrong.
A try catch will not slow down anything with an already synchronous and slow API as LS **is** but at least will give you a chance to fix something in case you are the only owner of the running code.
And by you, I mean **your team**, which could be either yourself alone, or 1000 engineers.
I hope I've explained better why I've commented that try/catch, compared with all other problems we have in the LS world, couldn't really be consider as the overkill practice, rather a good one that won't cause problems but **might** help solving them if we know how to and we don't want to be obtrusive.
Did I say I don't use localStorage since every other option is better in terms of quota, plus I have a library that works with named databases that fallbacks with named LS too so that .clear() wont' remove anything but your own data? The link was in the first answer.
Have a nice day
I've been doing some programming in languages where arrays *did* start indexing at 1. I loved it initially, but when doing more complex things with arrays (storing a two-dimensional strycture in a one-dimensional array and calculating the index from x and y coordinates) I quickly found that when the index starts at one you end up having to write -1 very often.
So, even though it initially seems easier, when you get more experienced it actually becomes an annoying hassle to deal with.
Yes, so true!! It's really annoying. Here is what I am doing to normalize Safari's Private Browsing mode now:
var storage;
try {
var x = Date.now().toString();
localStorage.setItem(x, x);
var y = localStorage.getItem(x);
localStorage.removeItem(x);
if (y !== x) throw new Error();
storage = localStorage;
}
catch(e) {
storage = {
storage: {},
getItem: function(key) {return this.storage[key];},
setItem: function(key, val) {this.storage[key] = val;},
removeItem: function(key) {delete this.storage[key];}
}
}
I would argue that perhaps making it easier for 99% of usage makes it ok if it makes it less easy for 1% of usage. ;)
Unfortunately it is more the other way around :)
Once you get used to array indexes starting at zero, it's much easier to work with. Besides, even if some smart language designer would now introduce a new language where indexes do start at 1, it would be totally annoying because everyone programs in more than one language these days so you would have to keep switching.
But now that we are talking about breaking conventions... What about QWERTY? How annoying is that? And who ever came up with the week having 7 days? A prime number? Really? How convenient :p
Or the months having 30 and 31 days except for july and august where the pattern suddenly breaks... And don't get me started on leap days!
These things are what makes programming so difficult. I heartily recommend Jon Skeet's fantastic article 'OMG! Ponies! (AKA Humanity? EPIC Fail!)' on this subject. It's a great read!
http://codeblog.jonskeet.uk...
You are ridiculous and alone that you encounter those problems just show that you are a wannabe scripter. Period.
Did you ever got an error when using localForage API? Me never :/
http://www.badpenguin.org/s...
localStorage just uses IDB/WebSQL/LocalStorage in the back end, so it would do whatever they do when storage goes over. You *can* max out WebSQL and IDB.