A new javascript minifier: JSMin+

By crisp on Friday 10 April 2009 01:37 - Comments (30)
Categories: JSMin+, Javascript, PHP, Tweakers.net, Views: 32.721

For some time we have been looking for ways to minify the javascript and CSS files for Tweakers.net but were unable to find the right tool for this. If finding the right tool takes too much time there is only one other option: create your own tool, which is exactly what we did. Even better: we are releasing this tool to the public so you can use it too!

Why use minification? you might ask, don't you guys already use gzip compression? Sure we do, but a gzipped minified file is still marginally smaller than the gzipped version of the original. Besides, it gives us a higher score on the YSlow scale which should be a reason by itself :+

So why couldn't we find the right tool? There are plenty tools available that provide javascript minification such as JSMin and derivatives, and even obfuscation such as YUI Compressor and DoJo Shrinksafe, but these just didn't meet our criteria. And besides, building your own tool is much more fun :P

So what criteria did we have? Here's the shortlist:
  • It should be able to handle our javascript files.
    That seems a no-brainer, but most minifiers actually force you to cohere to a very strict syntax. That means: no omission of semicolons that are otherwise optional, always using curly braces for if-else statements and always explicitly use parentheses when combining pre- or postfix increments in expressions. We felt this requirement was way too restrictive, and this criterium actually eliminated all tools that were based on Douglas Crockford's JSMin.
  • Have support for JScript's conditional compilation which we use in some of our files instead of the more brittle browser-sniffing techniques (and actually as a browser-sniffing technique :P).
    Most, if not almost all, minifiers however do not have support for conditional compilation which takes the form of a regular comment-statement and is therefore stripped by most tools.
  • Should not have a negative impact on clientside performance.
    The major goal of minification is to improve clientside performance. A tool like Dean Edwards' Packer is very effective in drastically reducing javascript filesize, but it comes with the added cost of the client having to 'unpack' the javascript files on every pageview. The benefit of the smaller size doesn't really outweigh that cost.
  • It should be easy to setup and implement in our current semi-automated deployment process.
The first two criteria actually prove that minifying javascript isn't easy; using regular expression based tools or tools that use a very simple tokenizer will easily break your scripts. Basically the only way to make sure that you interpret and minify the syntax of the original sourcecode correctly is to use a real javascript parser.

Tools like DoJo Shrinksafe and YUI Compressor do just that; they use Mozilla's Java-based SpiderMonkey implementation, called Rhino, to tokenize the actual input. The Yahoo! team even went a step further and hacked the Rhino sourcecode to add support for JScript conditional comments.

So out of all minification tools we looked at only YUI Compressor met our first three criteria, but being a Java-based tool it didn't seem easy to setup and integrate in our script-based deployment tool (actually, me being not so familiar with Java is most of the reason ;)).

So we decided to built our own tool based on a javascript parser. I myself did some effort some time ago to create such a parser from scratch but building a good parser is hard, especially if you have to figure out the exact syntax requirements and all of its quirks from a 188 page technical specification...

Since I'm not totally about reinventing the wheel (just a little :P) I started looking for already existing javascript parsers and the most obvious place to start was Mozilla's SpiderMonkey and it's Java-derivate Rhino. It was when browsing through Mozilla's repository that I came across this little gem: Narcissus, a javascript engine written in javascript by Brendan Eich (the inventor of javascript) himself :)

This engine contained a very straight-forward javascript parser that proved easy enough to port to PHP. After that all that was needed was a routine that could reserialize the output tokenstream of the parser to a minimized version of the original scriptfile. As this is also exactly what for instance YUI Compressor does I was also able to 'steal' some of their optimizations in order to reduce filesize a bit more :P

Since our goal was not to do obfuscation but merely fast and effective minification we didn't go to lengths to replace variable names with shorter versions and the likes. Not only is that a very complex problem (due to the expressive nature of javascript), but it adds new constraints to development with javascript as well.

Another advantage of having a real parser for the minification process is the fact that it can actually notify you when a javascript file contains syntax errors. This is part of the output from our deployment tool when it tries to minify a new version of a javascript file that contains a syntax error:

Updating: tweakers.net
Minifying javascripts and css in tweakers/x/
Minifying general.js
Parse error: Missing ) in parenthetical in file 'general.js' on line 191

But when all is right, we see this:
Done! - saved 11508 bytes (23%)

And this is what it looks like (original). As you can see we put function declarations on it's own line, that will at least give us a clue which function is involved in case a javascript error occurs. Of course we have easy ways to switch to the same page with non-minimized versions of the javascript files in order to be able to debug any errors.

Back to the promise we made in the first paragraph of this article: we are giving back our effort to the community. Open Source is a wonderful thing; we use it every day and in this case it would be a waste if we were the only one to benefit from the hard work of others. We dubbed our little project "JSMin+" because essentially it acts as Douglas Crockford's JSMin but is far less restrictive, and released it under the same MPL/GPL/LGPL tri-license as the original Narcissus code.

You can download the PHP sourcecode here: JSMin+.

Note that this script is provided "as is": there is no support or guarantee. It works for us but may not work for you. Can you contribute? Yes you can! We'll take patches when we find them useful. Especially performance can be improved; we already changed the tokenizer a bit to be more performant in PHP and to add support for JScript conditional comments but there's much more room for improvements. Note that we don't recommend using this tool for on-the-fly minification!

Update 12-04: version 1.1 has been released which fixes some issues and improves performance.

Volgende: JSMin+ version 1.1 04-'09 JSMin+ version 1.1
Volgende: IE6: exit 03-'09 IE6: exit

Comments


By Jaap, Friday 10 April 2009 09:30

I like it, and most of all I like your reasoning in the first few pragraphs!

By Tweakers user Vincenz0, Friday 10 April 2009 09:35

Hey,

This is great!
Do you have some performance improvement values for tweakers.net?
It's quite interesting what your minimizer does but will it make a significant impact on performance?

By Tweakers user Tjoekbezoer, Friday 10 April 2009 09:54

Does this minifier automatically add curly brackets to conditional statements that do not have them? eg, does
if( cond )
   do();
become
if( cond ) {
   do();
}
I ask this because my code breaks after using JSMinPlus, and I have a lot of conditional statements without curly brackets. Using my own very simple minification where new line feeds remain intact, the code runs.

By Tweakers user crisp, Friday 10 April 2009 10:14

Tjoekbezoer: well, actually it tries to remove those curly braces whenever possible, in particular when the body of a conditional or loop stament is also just one statement. The if-statement is an exception when it contains a nested if and is followed by an else-statement because that would result in an ambiguity when we would strip the curly braces, so that should be taken into account.

I'd like to see a full example of your case (you can email or DM me if you like), it's not entirely impossible that you found a bug in the minifier itself ;)

Edit: luckily this wasn't a bug in our minifier. You shouldn't just echo the minfied version to a browserwindow and copy/paste, that doesn't keep newlines and special characters intact :)
Use something like
PHP:

1
<?php
echo '<pre>' . htmlspecialchars($output) . '</pre>';
?>
instead.

[Comment edited on Friday 10 April 2009 11:43]


By Tweakers user RobIII, Friday 10 April 2009 10:42

Nice work crisp! _O_

By Tweakers user Mr.Zop, Friday 10 April 2009 10:48

I always use this fancy script:
http://code.google.com/p/minify/
It let's you combine, minify and include css and js on the fly so you can update whenever you like without minifying it again manually.

By Tweakers user Mr.Zop, Friday 10 April 2009 10:49

Mm, can't edit my comment but I'd like to add the description from the project page: "Minify is a PHP5 app that can combine multiple CSS or Javascript files, compress their contents (i.e. removal of unnecessary whitespace/comments), and serve the results with HTTP encoding (gzip/deflate) and headers that allow optimal client-side caching."

By Tweakers user crisp, Friday 10 April 2009 11:22

Mr. Zop: that's indeed a better version than the original PHP port of JSMin, but the minification level is still a bit less than our own minifier :P - it overcomes the optional semicolon problem by just keeping a lot of newlines in place.

Besides that I would not recommend doing on-the-fly minification (without some form of serverside cacheing), especially not on heavy-traffic sites such as Tweakers.net and/or with large scripts. For instance, minifying our general.js takes almost 2 seconds (using either that specific version of JSMin or our own JSMin+) on average webserver hardware; that really hurts the first-page loadtime when a visitor comes in with an empty cache, and that's a very bad first impression for that visitor.

Even with serverside cacheing (it looks like 'minify' does support that, which is good) I prefer a static version of the minified files to link to because webservers can deal with that much more efficiently. As you may know we use a seperate domain for our static files which are being served by lighttpd.

[Comment edited on Friday 10 April 2009 11:37]


By Tweakers user Bosmonster, Friday 10 April 2009 12:34

Even een testje op de jquery 1.3.2 library:

YUI (nomunge): 73.536 bytes
JSMin: 73.690 bytes
JSMinPlus: 72.830 bytes

We have a winner :+

By Tweakers user Tjoekbezoer, Friday 10 April 2009 14:59

My own project's compression rate: 33%. Notteh bad! :)

By Tweakers user BtM909, Friday 10 April 2009 21:50

Simple stuff like
As you can see we put function declarations on it's own line, that will at least give us a clue which function is involved in case a javascript error occurs.
shows how clever, but simple, the design is :)

Nice work! Will give it a try in my projects!

By timd, Saturday 11 April 2009 01:29

Why all the efforts to reduce a single file, which eventually will be cached site-wide. I still do not consider reducing javascript and css files, while serving a > 15 million pageview/day website...

By Tweakers user iceheart, Saturday 11 April 2009 02:10

well indeed timd, you seem to be quite good at taking up an unneccessary amount of space regardless what you're typing :+

By Tweakers user crisp, Sunday 12 April 2009 01:31

timd: you'd be surprised how many of your daily visitors do not have a primed cache.

Not considering is a poor excuse for not wanting to weigh cost versus benefit. I cannot judge the cost of implementing JS and CSS minification in your case, but in our case the actual implementation took less than an hour.

By Steve Clay, Monday 13 April 2009 22:35

Since taking over the Minify project I've wanted to do this (port a real JS parser to PHP) for awhile, good that you found Narcissus! Porting Rhino to PHP was just out of the question :)

Do you have a test script for JSMin+? Here are the simple tests I run for Minify's extended JSMin port.

If JSMin+ passes all the tests (and all the popular JS libraries), we'll undoubtedly use it in Minify.

By Tweakers user crisp, Monday 13 April 2009 23:53

Hi Steve,

Our testcases currently consist of our own production files like our general JS toolkit (original - minified). Sofar the minification did not introduce any issues for us, and Tweakers.net is a large-traffic site in the Netherlands ;)

I also tested against the most recent versions of prototype and jQuery which it was able to parse without errors and convert into a minified version that parsed without errors in recent browsers. I also crosschecked wether the minified version could be parsed again by JSMin+ and if that produced the same result.

I just checked your testcases and it passes your issue74.js in sofar that our output does not have the semicolon before the closing curly brace, but that doesn't seem to impact any current browser. I believe this test is meant to see if the expression is recognized as a regexp literal, which it is in Narcissus and thus also in our port :)

We do fail the testcase in before.js because our support for IE's conditional compilation syntax obviously isn't complete... I'l need to work on that.

By Mathias Bynens, Tuesday 14 April 2009 13:37

"Should not have a negative impact on clientside performance.
The major goal of minification is to improve clientside performance. A tool like Dean Edwards' Packer is very effective in drastically reducing javascript filesize, but it comes with the added cost of the client having to 'unpack' the javascript files on every pageview. The benefit of the smaller size doesn't really outweigh that cost."

Packer's so-called "base62-encode" feature is optional. If you don't want to use it, just don't check the box, and client-side performance won't be affected at all.

By Tweakers user crisp, Tuesday 14 April 2009 13:51

Packer's so-called "base62-encode" feature is optional. If you don't want to use it, just don't check the box, and client-side performance won't be affected at all.
But then still Packer doesn't meet our first requirement ;)

By Tweakers user --MeAngry--, Tuesday 14 April 2009 14:13

It appears 400KB of uncompressed JavaScript is a bit too much for JSMin+. :P Apache 2.2 crashes every time I try to run it. :)

Further specs:
Core 2 Duo E6550 @ 2.33 GHz
2GB PC5400 DDR2
Windows Vista SP1
Apache 2.2.11, PHP 5.2.9 (through SAPI)

I'm aware that 400KB is quite a lot of code. But I thought minifying the code might help speed up our old CMS which uses multiple libraries of all sorts.

[Comment edited on Tuesday 14 April 2009 14:17]


By Tweakers user crisp, Tuesday 14 April 2009 14:20

--MeAngry--: define 'crashes'? Don't you just hit the memory limit or max execution time?

By Tweakers user --MeAngry--, Tuesday 14 April 2009 14:56

Wouldn't that result in an error from either PHP or Apache? Because normally it does. Now I just get an entry in my Application Error log:
Faulting application httpd.exe, version 2.2.11.0, time stamp 0x493f5d44, faulting module php5ts.dll, version 5.2.9.9, time stamp 0x49a56925, exception code 0xc00000fd, fault offset 0x00151eff, process id 0xb78, application start time 0x01c9bcf99937d460.
Nothing is to be found in my Apache error log, neither does PHP give a message.
edit:
Excuse my English. I don't know what my problem is today.

[Comment edited on Tuesday 14 April 2009 15:01]


By Tweakers user crisp, Tuesday 14 April 2009 15:03

--MeAngry--: aj, that looks like a PHP bug :P Is it possible you can sent me that javascript file so I can test if I can reproduce this?

By Tweakers user CptChaos, Saturday 25 April 2009 12:40

To bad there isn't a list with the (software) requirements in order to make this fancy script work.
I think I'm missing something, because it doesn't work here.

[Comment edited on Saturday 25 April 2009 12:40]


By Tweakers user crisp, Saturday 25 April 2009 14:14

Here's the list of software requirements:

- PHP5 or higher

that's it afaik

By Florent V., Sunday 26 April 2009 18:19

Hello,

I have yet to test JSMin+, but i really like the announced features and the reasoning behind it.

Are there plans to move the project to some place with source browsing, issue tracking, and “static” documentation (i.e. a documentation that's not in an array of blog posts)? Like Google Code, or something else?

By sunnybear, Wednesday 09 September 2009 02:10

Guys, I lead development of open source solution to automate all clientside optimization actions (like Minify, also on PHP, but with UI, dozens options, simple install, CSS Sprites + data:URI included, etc) - Web Optimizer ( http://code.google.com/p/web-optimizator/ ).

I'm interested in applying this tool as an alternative for JSMin (patched to avoid conditional compilation), Packer (which is slow on client side) and YUI Compressor (which requires java). But what I see (initial file -- mootools-uncompressed.js from Joomla! 1.5):

Filesize before: 183588
Total time for JSMin: 1.3537750244141
Total memory usage for JSMin: 189376
Filesize after JSMin: 74276
Total time for JSMinPlus: 3.3452179431915
Total memory usage for JSMinPlus: 4367656
Filesize after JSMinPlus: 73182

JSMinPlus wins about 0,6% in size (and about 0,1% after gzipping), but consumes 3 times more time and 23 (!!!) times more memory. I think it's cool. But it can't be applied to real time compression on big real projects.

I think 400Kb of JavaScript code leads to about 20Mb of memory consumption, so some PHP environments will fall out. And this is PHP5 'fast by default'...

Please consider memory / performance review, so we can include it as alternative for Web Optimizer.

By Tweakers user crisp, Wednesday 09 September 2009 09:58

Hi sunnybear
Filesize before: 183588
You are aware that that is pretty large for a JS file? ;)
But it can't be applied to real time compression on big real projects.
I generally don't recommend doing real time compression, even the 1.35 seconds from JSMin would be unacceptable in that case. We apply compression during deployment.
Please consider memory / performance review, so we can include it as alternative for Web Optimizer.
I mentioned in my article that performance indeed is something that can be improved, but JSMin+ being built upon a full-fledged javascript parser means that we will probably never match the performance of the original JSMin, which has a mutch simpler tokenizer and doesn't actually do a full syntax-scan based upon the parsed tree. That last thing also explains why the memory consumption is much higher (plus the fact that PHP just isn't really efficient at that).

I have been considering doing the minification in the tree-building stage. That way we could just throw away scoped parts of the tree once they are minified. That might give as some better performance as well. The only question is if that does not interfere with our plans to support variable-name minification.

Unfortunately my time is very limited and I don't know when I can pick-up this project again. However, being open-source you could give it a try yourself. We accept patches as well ;)

By Isaac, Sunday 29 November 2009 00:01

I haven't tried this yet, but can't wait to get my hands on it!
Looks Awesome (and I've spent some time looking at the alternatives), many thanks for releasing it- you rock ;)

ps. totally with you on Not using any minification tool for real-time deployment,
that almost defeats the purpose of why me minify in the First (emphasis added) place

By Kirk Olson, Tuesday 02 February 2010 10:23

Am I right in understanding JSMin+ only minifies pure js files and not html?

By Tweakers user crisp, Tuesday 02 February 2010 10:26

Kirk: no need to post your question twice ;)

Anyway, JSMin+ only minifies pure javascript, not HTML. If you want to minify the javascript contained in an HTML file you would have to write a script that takes the content of each scripttag, minify it and replace it in the HTML file.

Comments are closed