Weblogs: Javascript

Another November, and the third Full Frontal one-day conference took place at the Duke of York cinema in Brighton. Last year's conference was fairly interesting, if a little demo-heavy. This year's seems geared toward development tools and web applications. The headline speaker was ex-Yahoo Nicholas Zakas, but the supporting speakers were more impressive and fresh.

Most interesting talk was Jeremy Ashekenas' CoffeeScript Design Decisions - which surprised me somewhat considering my opinion on the impracticability of CoffeeScript in a real-world web development team. I was also pleasantly surprised by Rik Arends' talk about the Cloud9 IDE. Phil Hawkesworth's talk was eminently listenable, but unfortunately impossible to live blog. Eloquent JavaScript author Marjin Haverbeke was incredibly interesting, but perhaps too technical for a conference talk. The final two talks, from Brendan Dawes and Marcin Wichary were both entertaining and (more importantly) inspiring. Glenn Jones provided us a very useful look at almost-ready features of browsers as a platform for sharing data. And Zakas rolled out a two-year old talk.

CoffeeScript Design Decisions by Jeremy Ashkenas

I covered Ashkenas on CoffeeScript Design Decisions in a separate blog post. It was that interesting and bloggable.

Respectable Code-Editing in the Browser by Marijn Haverbeke

Marijn Haverbeke talks about code editors in a browser; basically glorified text areas. Text areas themselves are too primitive for building a code editor on; indentation is unworkable, and good luck finding which line of your code you are currently on.

The code editor in a browser received a massive boost from Bespin, a code editor written entirely in canvas, and basically recreated every pixel of a UI. As a demo Bespin was interesting, but using canvas is inappropriate solution. Thankfully this abysmal mess of inaccessibility is now mostly abandoned, and the code editors that have arisen from this work are well on track to being more conducive to the web environment, and more perceptive of the DOM. One significant limitation of canvas is its portability.

So far there are only 3 serious implementations of in-browser code editors:

• ACE, which took the good parts of Bespin, and with new techniques it is much faster and more portable than Bespin.
• Orion, which is an Eclipse project, uses content editable / design mode as a basis.
• CodeMirror, which is Marjin's project. Currently in its second iteration.

All three are open source projects, and to date there isn't a single commercial implementation of a browser-based code editor.

Marijn is the sole developer behind CodeMirror. The first major version of CodeMirror used content-editable (or design mode); this is a browser supported mechanism of inline editing. But Marijn found the feature both underspecified and the browser support buggy or unsuitable for building a code editor. Issues ranged from Internet Explorer inserting paragraphs whenever it could, to the general unavailability of useful events.

The original version of CodeMirror grew out of Marjin's previous project, an online JavaScript book called Eloquent JavaScript. The book contains exercises and demonstrations of JavaScript, code that could be written and run inline on the page. For the book Marjin's solution of a Firebug-like textarea console and output window worked. He later added colour syntax highlighting by overlaying the text area with coloured text DOM nodes, but this turned out to be too slow with the pure JavaScript DOM changes. This gave rise to the first version of CodeMirror.

After content-editable proved insufficient (code folding was impossible, for example), Marjin started version 2.0 of CodeMirror which was based pure DOM. He limited the size of the DOM by creating a viewport of the document, so only the visible part of the document needed to be rendered on page, thus speeding up the overall rendering of the document. Marjin calls this a fake editor, just lots of changing DOM nodes. The cursor is a lie, the text looks editable, with copy and paste and moving text working. Marjin crafted his own text selection algorithms, and that worked pretty well since CodeMirror has full control over the document model.

CodeMirror with its API approach allows undo/redo, code-folding, and an extensible mechanism for supporting multiple languages with both syntax highlighting and code completion. Building a language extension is about implementing one method, token(), which is in a SAX-like way called for each token within the edited document. There are hooks for managing state through the document. The typical features of a good editor such as search-in-place and text replacement end up being scripts on top of the CodeMirror API.

CodeMirror supports the identification of local variables, helping the developer identify potential bugs by it's ability to distinguish the scope of variables as the code is being edited. This support isn't limited to JavaScript. XML-based languages has mismatched tag detection, for example.

Also CodeMirror can handle syntax highlighting of multiple languages in the same document. (composed modes) Thus an HTML page containing CSS and JavaScript - and even PHP - each piece can be independently supported. I guess this is done on a line-by-line basis, so perhaps an inline piece of JavaScript surrounded on the same line as non-JavaScript might not by syntax highlighted in the same way as a block of JavaScript.

Internally, the document being edited is stored as a doubly-indexed B-tree as this allows the two way mapping of the actual line of code with the line currently being edited on screen. These two lines can be different because of code-folding, long lines being wrapped, and the region being displayed is offset from the top of the document.

The editor listens for scroll events, and avoids startup freezes as it renders the DOM nodes onces there's sufficient information available to render the current visible region. So scrolling and loading large documents doesn't slow down the interface.

Copy and pasting is a clever hack, CodeMirror detects a right-click and inserts a hidden textarea underneath the cursor. Then the context menu at this point offers copy and paste options, because they are native to text fields.

With demos of the above features and talking through how they were implemented Marjin delivered an interesting tech-heavy talk.

How We Architected Cloud9 IDE by Rik Arends

Rik Arends introduces Cloud9 IDE as the easiest way to work with node.js. Cloud9 IDE is an online IDE and runs on node.js.

Though, why build and IDE with JavaScript? JavaScript developers lack tooling. Each main language has an IDE geared for their language; Java has Eclipse, C++ has Visual Studio, JavaScript is added to IDEs and editors as an afterthought, always treated as a second hand language. By using JavaScript developers will already know how they can extend the IDE they use.

We evangelise the Web, but we don't develop on it. Rik accepts the challenge that if many applications can be hosted on the web. why not a developer IDE. Surely web developers should be able to work using a cloud-based IDE. If the web can self-host its entire toolchain isn't this, he asks, the best way of pushing the web forward?

IDEs don't need to be ugly or clunky, so Cloud9 takes a designed approach to IDEs. One year ago it looked a little like Eclipse. Today it will look very familiar to TextMate users (and indeed it takes some useful cues from the Sublime Text Editor). They are even currently importing TextMate themes.

Cloud9 is betting the company on building a cloud-based IDE. It is currently funded by Accell and Atlassian. And, Cloud9 IDE is developed using the Cloud9 IDE. The IDE is open source, and downloadable from github to run on your own sever. (I caught up with Rik later on in the day and he confirmed it can be up and running on a VPS as long as node.js has at least 128Mb of RAM available. So a cleanly configured 256Mb VPS should be a useful starting point).

Rik demos the Cloud9 IDE; projects are brought in via github (after oAuthing with github). The main editing interface uses the ACE code-editing component. This renders a viewport of the current code as DOM Fragments (similar to CodeMirror2, I gather). This means rendering is lightning fast, and syntax highlighting is easy to extend for different languages and markup schemas. One recently added feature to the editor is a mini-map of the code, which is a good visual overview of the code; this is based on the findings that developers recognise code block by how the look, rather than the actual code.

Cloud9 IDE has a Console for running git commands. It's actually an emulation of a server shell, but very limited range of unix commands. One feature coming soon is a link to a virtual machine, and that will allow a real unix console right in the IDE.

Projects can have node.js servers configured with them (essentially on the fly and managed through the CloudIDE interface). This allows inline node.js debugging, so the developer has the whole gamut of breakpoints and means of stepping through code, peeking at stack traces and in-scope variables (much like the Firebug debugger for browser-based JavaScript debugging).

The IDE has support for Heroku based deploys (as git commit hooks, I think). The console has code-complete (for example git commands and options).

CloudIDE takes advantage of a number of publicly available libraries and toolkits, including:

• APF - a high performance UI library
• ACE as the code editing component.
• Socket.io which offers realtime socket networking. Rik demoed inline chat and collaborative editing within the CloudIDE interface.
• jsDAV a pure JavaScript implementation of the WebDAV protocol
• Connect as the HTTP middleware (which Rik compares to a build-your-own customised Apache server).

Lessons learned using node.js

Rik lists some useful pointers in building applications on node.js:

• structured exceptions do not fit in node.js., it doesn't make sense in an evented system, since there's no clean way of surfacing errors back up to where they can be seen. And because of the asynchronicity of node, you don't want an exception to break the whole call chain.
• Beware of globals. Definitely "use strict" to catch these. Beware of modifying the prototype of objects since these changes persist; prototype changes are global, so this can cause unintended side-effects elsewhere in your code.
• minimise your callback nesting by grouping functions together into a more linear-like order. (I guess this also hints at considering using promises).
• for dense business logic and tests, CPS transforms makes sense, use a code analyser to refactor into a top-down looking code.

From a systems point of view:

• keep an eye on state, preferably have none because it persists across requests
• Monitor nodeJS: use a piece of software called forever which runs node apps in the background and keeps monitoring that it's still alive, restarting it when it fails.
• Log everything so you can reconstruct bugs. Use logging over UDP, so it's a fire-and-forget, for example
• Thick client + stateless server = win
• Put node behind a proxy— makes it very manageable. load balancing, server upgrades, start processes.
• Do not blindly trust a library or module — nodejs is pretty new, so there are many examples of "my first node js module" that then never get updated with fixes and improvements. Don't be afraid to get your hands dirty and building your own libraries and modules, when necessary.

Rik's excellent talk covers useful ground in building and architecting node.js applications. It's also a great example of what is considered an impossible to build application implemented in a highly usable, flexible and powerful way. I guess this is the first serious node.js app I've seen under the covers of, and a great demonstration of what's possible with node.js

Scalable JavaScript Application Architecture by Nicholas Zakas

We live in a globally connected world. Conferences happen regularly, they are recorded in video and posted on-line for the world to watch. The JavaScript community is small and closely-knit, so we keep up with what's going on outside of the UK borders. With that in mind I find the idea of paid conference speakers who get top billing (both local speakers, and flown in from outside countries) giving the same talk multiple times at different events a deplorable and unacceptable behaviour.

Nicholas Zakas' talk "Scalable JavaScript Application Architecture" is unchanged from the original talk he gave at the Bayjax conference hosted at Yahoo, Sunnyvale in September 2009. Two years ago. I've already watched the video of this talk (several times), so I was unimpressed to see him wheel it out again verbatim.

This repetition may make sense if this was a last-minute stand-in replacement for another speaker, or one to make up the numbers speaker-wise. But not for a top-billed speaker flown in from the USA, that we as attendees were paying for.

Beyond the Page by Glenn Jones

One page web applications are appearing all over the place, and as Glenn accurately notes they are siloes, but they don't need to be. Glenn shows a couple of demos he has been working on that allow data from one web application to be communicated to another.

His central application is a people store, storing vcards (or the microformat equivalent hcards). He shows how drag and drop between two separate browsers can be used to copy across the vcard information purely in the browser, meaning that this transfer can be done on any web-based application hosted anywhere.

Glenn shows further capabilities, of dragging and dropping vcards from the file system to the browser, and saving vcards from the browser back to the file system.

This shows that the drag-and-drop API and the filesystem based APIs (including uncompressing zip files in pure JavaScript) are getting to to point of being useful to the developer. Glenn demos the applications first, and then goes back over each application showing how each demo is done, along with the APIs used, and the current set of browsers these APIs are working in.

What impressed me was that the data wasn't just text, or an image, but a set of structured data plus an image. The drag and drop looks very useful, but Glenn notes that the HTML5 specification for dragdrop is a disaster - concurring with PPK's viewpoint. Currently advances in drag-drop seem to be driven by GMail.

Another clever demo was copying in a chunk of HTML (using the Clipboard). Glenn shows that under the covers he is using content-editable / design mode to receive the HTML, and then run a standard parser over that to extract out the structured information. Certainly a sensible use of microformats parser.

As an additional extra Glenn is using the Google Social API which, given an email, returns a list of social network or XFN type information about the person represented by that email address.

The last part of Glenn's talk covered Web Intents. These are used to map common actions with services, such as posting a status, bookmarking a link, editing an image, picking a profile. Glenn shows the two main ways Web Intents improves the flow of information:

Many blogposts have social bookmarking links, instead Glenn shows that replacing these with a "Bookmark this link" and identifying it as such to the browser, that browser can then assign that link to the service that the user uses. So clicking on that link opens the Delicious bookmarking dialogue for one user, and a Pinboard dialogue for another user.

The other direction is for filling in forms - like street addresses. He demos a webintent that maps a "Fill in this address using a profile" link and maps it to his own people store. That way he can select the right profile in his contacts system appropriate for the form such as his personal profile address, or his business profile for his office address.

A very informative talk about the features that are on the way to being ready for use in web sites, features that improve the user experience, and thus help reduce the risk of customers disengaging with sites.

Beyond the Planet of the Geeks by Brendan Dawes

Brendan Dawes is a designer and a geek, and he delights in design of items that escape us - such as Japanese and Brazillian paperclips (and pencils). And he lets his creativity loose. When a high-profile architecture company hires Brendan's company to design a new experience for their portfolio of buildings, they see in Brendan the very same boundless creativity, amazing things happen.

Brendan's narrative draws you in, he shows that sometimes it's good to build things that are not user-friendly because in that way you get experiences that are memorable, enriching and deeply rewarding, as well as expressing a brand identity. One side-effect of having a rich exploratory interface is that it opens the visitor up to serendipitous discovery, one of the most wonderful experiences for curious people.

Brendan's talk is about visualisation of data into an engaging experience. He talks about the ideas that worked, how the idea evolved, how the customer asked to make the logo smaller, about what it's like creating with no limits in place.

Yet he also shows that the idea is still grounded in technology, it's built in HTML5 because it's something Brendan doesn't know (Brendan is a firm believer in moving on to different technologies when you have mastered the current one).

An utterly enjoyable, and inspiring talk, and a nice counter-balance with the other technical-focused talks.

Google Doodles by Marcin Wichary

Marcin works for Google, and his 20% project is helping the Google Doodles team build their next Doodle. He's been involved in several of the new interactive doodles that have appeared on the Google homepage over the last year. From the faithful implementation of PacMan, to the ??? machine, to the magnifying glass doodle, and back to the Jules Verne underwater exploration doodle, and the ??? animation.

After a quick summary of the history of the Google Doodle (from the first Burning Man doodle, right to when Marcin starts to get involved), Marcin dives into the interesting stories behind the interactive ones, covering the usability tests and technical challenges of the doodle. We are shown several versions of the Jules Verne doodle and see the changes made and how they considered IE support.

Marcin talks about how each of the doodles evolved to their end-result, the user-experience factors, the technical limitations and reinventing new animation techniques. Marcin has a healthy interest in retrogame programming techniques, and it's eye-opening some of those techniques have been adapted in these famous doodles. The Martha Graham dancing woman doodle is a magnificent example of his Crushinator technique, which reminded me of the bit-blitting techniques of the Amiga.

Animation tricks like a thick lens holder to hide the coarseness of the rectangular clip-regions, thus saving performance and still producing a jaw-dropping result. Similar to the Dark Sceptre approach to masking large animated characters.

Marcin shows some of the easter eggs present in the Doodles, like the Jules Verne one controllable by the accelerometer built into the Macbook Pros. Many people first learnt about that particular hardware feature because of the Jules Verne doodle. Marcin notes some of the bugs they uncovered, for example the same generation of Macbook pro having different accelerometer chips which resulted in the movements being completely reversed.

A fascinating exploration of animation, interactivity and rich user-experience on the world's most trafficked page. And shows how the old animation techniques are repurposed in the ever-evolving web.

Conclusion

Full Frontal was an enjoyable day spent watching masters of JavaScript showing their efforts and taking us behind the scenes. I like Remy & Julie's choice of speakers - the people who actually ply their trade and craftsmanship, the ones who don't seek out the limelight and attention, and generally just do a wonderful job and are passionate about what they do.

Full Frontal delivers in a way no other conferences do, it's a small audience in a homey venue. Free from marketing and hype, and delivers great value, inspiration, and a focus on the web developer. It is my favourite event.

---

[ Weblog | Categories and feeds | 2011 | 2010 | 2009 | 2008 | 2007 | 2006 | 2005 | 2004 | 2003 | 2002 ]