The RADAR Architecture: RESTful Application, Dumb-Ass Recipient

Yesterday, I posted a brief note about changes in the URL conventions in Rails 2.

Sam Aaron responded with a comment:

Maybe I just haven't yet grokked the whole REST thing yet (most probably the situation), but the seeming re-marriage of verb and noun (or at least the loss of ability to easily distinguish them) disturbs me slightly. So (just out of interest) my question is: why was a great Huzzah! heard throughout the lands?

Let's look at how we got to where we are. And then let's look at a different way of thinking about the problem that might help simplify things.

It Starts With Verbs and Nouns

Roy Fielding's REST work showed us that the design of the interaction between a client and a server could make a major difference to the reliability and scalability of that application. If you follow a set of conventions embodied in REST when designing your application, is means that (for example) the network will be able to cache responses to certain requests, offloading work from both upstream network components and from the application.

But REST was about more than performance. It offered a way of separating the actions performed by your applications (the verbs) from the things acted upon (the nouns, or in REST terms, the resources). What's more, REST said that the verbs that you use should apply consistently across all your resources.

When using HTTP, the HTTP verbs (such as GET, PUT, POST, and DELETE) are also the verbs that should be used with REST applications. This makes a lot of sense, because things like routers and proxies also understand these verbs.

One of the successes built using these principles is the Atom Publishing Protocol. Atom allows clients to fetch, edit, and publish web resources. Atom is probably most commonly applied to allow people to use local editors to create and maintain blog articles—client software on the user's computer uses Atom to fetch existing articles and to save back edited or new articles.

Let's look at how that happens with Atom.

I fire up my Atom-based local client. It needs to fetch the updated list of all the articles currently on the server, so it issues a GET request to a URL that represents the collection of articles: the verb is GET, and the resource is the collection.

   GET   /articles

What comes back is a list of resources. Included with each resource are one or two URIs. One of these URIs can be used to access that resource for editing. Let's say I want to edit a particular article. First, my client software will fetch it, using the URI given in the collection:

   GET   /article/123

As a user of the editing software, I then see a nicely formatted version of the article. If I click the [EDIT] button, that view changes: I now see text boxes, drop down lists, text areas, and so on. I can now change the article's contents. When I press [SAVE], the local application then packages up the changed article and sends it back to the server using a PUT request. The use of PUT says “update an existing resource with this new stuff.”

Browsers are Dumb

If you use a browser to interact with a server, you're using a half-duplex, forms-based device. The host sends down a form, you fill in the form, and send it back when you're ready. This turns out to be really convenient for people running networks and servers. Because HTTP is stateless, and because applications talk to browsers using a fire-and-forget model, your servers and networks can handle very large numbers of devices. This is exactly the architecture that IBM championed back in the 70s. The half-duplex, poll-select nature of 3270-based applications allowed mainframes with less processing power than a modern toaster to handle tens of thousands of simultaneous users. The browser is really not much better than a 3270 (except the resolution is better when displaying porn). (Recently, folks have been trying to circumvent this simplicity by making browser-based applications more interactive using technologies such as Ajax. To my mind, this is just a stop-gap until we throw the browser away altogether—Ajax is just lipstick on a pig.)

How well do browsers play in a RESTful world?

Well, for a start, they really only use two of the HTTP verbs: GET and POST. Is this a problem? Yes and no.

We can always hack around the lack of verbs by embedding the verb that you wanted to use somewhere in the request that you send to the server. It can be embedded in the URL that you use to access a resource or it can be embedded in the body of the request (for example as an element of the POST data). Both work, assuming the server understands the conventions. But both techniques also mean that you're not using REST and the HTTP verb set; you're simply using HTTP as a transport with your own protocol layered on top. And that means that the network is less able to do clever things to optimize your traffic.

But a browser talking HTML over HTTP is also lacking in another important area. It isn't running any application functionality locally. Remember our Atom-based blogging client? It had smarts, and so it knew how to display an article for editing. Browsers don't have that (without a whole lot of support in terms of bolted on Javascript or Flash). So, as a result, people using RESTful ideas to talk to browsers have to put the smarts back on the server. They invent new URLs which (for example) return a resource, but return it all wrapped up in the HTML needed to display it as a form for browser-based editing. This is how Rails does it. Using the Rails conventions, if I want to fetch an article for viewing on a browser, I can use

  GET   /article/1

If, instead, I want to allow the user to edit the resource, I issue

  GET  /article/1;edit

The application then sends down a form pre-populated with the data from that article. When I hit submit, the form contents are sent back to the server, along with a hidden field that tells the server to pretend that the HTTP POST request is actually a RESTful PUT request—the result is that the resource is updated on the server.

The ;xxx notation looks like it was tacked on. I think that in this case, looks aren't deceiving—these modifiers really were something of an afterthought, added when it became apparent that a pure Atom-like protocol actually wouldn't do everything needed in a real-world browser-based web application. That's because the client of an Atom server is assumed to be more that just a display device. An Atom client would not say "fetch me this resource and (oh, by the way) send it down as a form so that I can edit it." The client would just say GET, do what it had to do, then issue a PUT.

So, in pure REST, the client is a peer (or maybe even in charge) of the interchange. But when talking to a dumb browser, the client is no better than an intermediary between the human and the application. That's why it was necessary to add things such as ;edit.

Client-independent Applications

One of the other benefits of designing your applications against the limited verbs offered by HTTP REST is that you can (in theory) support many different clients with the same application code. You could write blogging server that (for example) looked at the HTTP Accepts header when deciding what to send back to a client. The response to a GET request to /articles could be a nicely formatted HTML page if the client wants an HTML response, an XML payload if the client asks for XML, or an Atom collection if the client asks for application/atomcoll+xml. In Rails terms, this is handled by the respond_to stanza that is duplicated in each controller action of a RESTful server.

But the benefits run deeper than simply being able to serve up different styles of response.

The HTTP verb set maps (with a little URL tweaking) pretty nicely onto the database verb set. POST, GET, PUT, and DELETE get mapped to the database CRUD verbs: Create, Read, Update, and Delete.

So, in theory at least, you should be able to write your application code as a fairly thin veneer over a set of resources. The application exports the verbs to manipulate the resources and the clients have fun accessing them. This is similar to the vision of Naked Objects.

However, I personally think it was optimistic to try to treat the two styles of interaction (smart and dumb clients) using the same protocol. The ugliness of the ;xxx appendages was a hint.

I think there's a lot of merit in following a CRUD-based model for interacting with your application's resources. I'm not convinced all the hassle of bending dumb browser interactions into a REST-based set of verbs, and then extending those verbs to make it work, is worth the extra hassle. To put it another way, I believe the discipline and conventions imposed by thinking of your application interface as a RESTful one are worthwhile. They lead to cleaner and easier-to-understand designs. The hoops you have to jump through to implement it with a browser as a client seem to suggest that a slavish adherence to the protocol might be trying to push it too far.

An Alternative Model

Does that mean I'm down on the RESTful, CRUD based approach to application development? Not at all. For some categories of application, I think it's a great way of structuring your code. But REST isn't designed for talking to people. So let's accept that fact when creating applications. And, while we're at it, let's take advantage of the fact that HTTP is such a flexible transport. Rather than trying to design one monolithic application than has both the CRUD functionality and the smarts to be able to talk HTML to end users, why not split it into two smaller and simpler applications?

Put the main application logic into a RESTful server. This is where all the CRUD-style access to resources takes place.

Then, write a second proxy server. This is an HTTP filter, sitting between dumb browsers and your core resources. When browser-based users need to interact with your resources, they actually connect to this proxy. It then talks REST to your main server, and interprets the RESTful responses back into a form that's useful on the browser. And this filter doesn't have to be a dumb presentation layer—there's nothing to say that it can't handle additional functionality as well. Things like menus, user preferences, and so on could all sit here.

Where does this filter sit? That depends. Sometimes it makes sense to have it running locally on the user's own computer. In this case you're back in a situation like the one we looked at with the blogging software. Sometimes it makes sense to have the proxy running on the network. In this case you gain some interesting architectural flexibility—there's no rule that says you must colocate your REST and presentation servers. Potentially this can lead to better response times, lower network loads, and increased security.

With this approach, we can lose all the hacks we need to make to our URLs and POST data to make them act RESTfully. And we can lose all the ugly respond_to stanzas in our Rails server code. We might even be able to get better reuse, sharing both resource servers and presentation servers between applications.

Let's call it RADAR. We have a RESTful Application talking to Dumb-Ass Recipients.

Does it work? I don't know. You tell me.

Comments

[Piers Cawley]

The advantage of having REST and presentation controllers in the same process is that the REST part can serve Ruby objects to the presenter (actually, I think there might be case for having the RESTful kernel itself just serve up objects and have two (or more presenters), one very thin one which would simply go from object to XML/JSON/CSV/Whatever and a fatter presenter for talking to the browser).

I do like the idea though; working with RESTful controllers right now can feel like trying to multiplex two applications through the same pipe.

[Dave Thomas]

True, but we can proxy Ruby objects using XML or other forms of marshaling. And, much though I hate XML, it is also a pipe protocol: there are plenty of filters that can work with it (such as XSLT) to help reduce the amount of coding we have to do.

Separating the presentation from the core like this also helps us overcome the temptation to put code that helps the view into model classes. Passing pure data across means that this code will now reside in decorators in the presentation server.

Dave

[Piers Cawley]

Yeah, but marshalling through XML means you go from an object with behaviour to a data structure with fields, and that gives me the heebie jeebies.

[Piers Cawley]

Plus: Dave, this is Ruby, not Java. If you have behaviour which depends solely on a single object, then shove it on that object. If this were Smalltalk, I'd say to stick it in a 'printing' protocol on the object, but, as it's Ruby, you can declare your intent by shoving it in a Model::Printing module and mixing it in.

[Dave Thomas]

Yeah, but marshalling through XML means you go from an object with behaviour to a data structure with fields

I'm thinking that might be an advantage, but I can be convinced either way. There's some behavior that needs to be shared (such as validation), but most of the behavior in an AR object is just baggage in the view. I'd like to experiment with using POROs as the transport, and shared mixins for any behavior that needs to be shared.

Dave

[nicholas a. evans]

This is just some hypothetical conjecturing, but the HTML Presentation layer would be doing more than simply displaying forms (or other alternative views) for dumb browsers. It would also have to layout a reasonable workflow for the dumb browsers. Might this be an appropriate location for some continuations based server logic? Proponents of REST and continuations-based webservers often seem to be diametrically opposed, but it seems to me that there is a natural fit here. (I'm speaking out of my ignorance, because I've never actually learned how Seaside or UnCommon Web (et al.) work.)

Also, I've heard you (and others) say before that AJAX is a hack. And I'm inclined to agree. But HTML/CSS client/server web-based apps have certainly solved some of the pains that developers were feeling in previous environments, or else they wouldn't be so common today. Where do you think the technology stack should move? In an incrementally better direction (such as future versions of Firefox and Safari being less dumb clients and more application platforms), or in a drastically different direction?

[Piers Cawley]

Dave: I guess this is where the ActiveRecord pattern falls down, you end up with awfully 'heavy' objects. I've done an object mapper (turned out to be one to throw away) that worked with plain old perl objects, but I'm not sure how to get started on doing one in Ruby.

[Bob Hutchison]

As best I can tell, this is the architecture my company used starting in late 2002 to develop a trading partner application and four loan management applications. We had bought into the REST ideas but in these applications couldn't see how we were going to provide a useful UI in a browser using html alone (e.g. validation was very tricky, involved, subtle, and really a *huge* part of the UI; Editing of something that on paper was in excess of 20 pages) -- we went so far as to use Laszlo 1.x, then 2.x, to develop parts of the application. The trading partner application even had a CSV over kermit 'UI' (who remembers kermit :-) Anyway we ended up with five running applications.

The architecture worked very well. In fact we were able to hide three of the loan management applications behind one Laszlo/html UI. We were able to script batch and off-hour work in Javascript (Rhino), Python, or Ruby. There is also a wonderful place in the architecture against which to run functional tests.

We actually used a similar architecture within the bigger adaptors -- we aren't called 'Recursive' for nothing.

You guys were talking about using XML between the proxy (we called them adaptors) and the REST application. This is precisely what we did using a fancy XML binding tool that I've been developing since SAX first appeared. We could store the XML that went back and forth, and better yet we could *play it back* -- think about that for a second...

These proxies/adaptors are also very handy when integrating existing systems into the application. We wrapped money transfer systems, credit checking, insurance purchasing, government reporting, and lender reporting with these adaptors. I hate to go in this direction, but SOA begins to make a little sense.

As far as AJAX goes, based on our experience using Laszlo and since then, AJAX does fit into this proxy thing quite well. And it doesn't pester the application with tons of noisy little queries.

[Joe Gregorio]

Dave,
You've linked to a rather old version of the APP. If you change
the link to:

http://bitworking.org/projects/atom/

everyone will see latest version, and all
the old versions too. Thanks, -joe

(Done. Thanks, Dave)

[Steve]

RE turning smart AR objects into dumb POROs: could you marshal code back and forth also? Obviously there'd be some limits, but Ruby code is just plain text. You'd want a different caching strategy for the code than the data, but we do this already with HTML (data) and Javascript (code) when sending stuff to the browser.

What does Fielding's thesis say about code on demand? (I must admit, I haven't read it.)

[Sam Aaron]

Thanks very much for the answer Dave. I think I'm going to have to let all of this swish about in my head for a bit :-)

From an initial inspection, I think that what you propose holds a lot of merit. However, I wonder what the implications of having two implementations of the application workflow would be (presentation server, and smart client). Another thing that concerns me is the potential move back to having to manage updates on a remote machine (to my mind, this is one of the major benefits of web-applications).

Definitely food for thought...

[Dave Thomas]

Sam: You only have the smart client if you want it--it'sw just there to show wheree such a client would fit it.

[Asbjørn Ulsberg]

A very interesting take on the issue and a great solution, in my opinion. Separating the application into one REST and one for presentation also gives you the option to switch the presentation application while still using REST in the backend.

I do see a huge deficiency in the browsers and perhaps even in HTML itself here, which I would like to address and fix. I have some ideas on how we can do it, but I'd like to ask: How would you picture a request and response would look like between a web browser acting on a resource to get it represented as an HTML form so it can edit the information which it then can PUT back to the same URI to update the data?

This can be done with MIME types; should we create a new MIME type for HTML forms that the client can put in its Accept-header when it wants to edit a resource? How does the user indicate to the browser that it wants to trigger this behaviour? How can we allow the web application to trigger this behaviour? I'd like to explore this a bit and discuss it further on the WHAT WG and HTML WG mailing lists!

[Sam Aaron]

Ah ok! That makes complete sense. From what I understand of it in many ways your REST server is like a smart and friendly database: for putting stuff in, and getting stuff out (or am I on the wrong track here).

[Dave Thomas]

Sam:

It's more than a databaseit's really the heart of the application. It is where all the business logic lives.

[Sam Aaron]

I'm liking this idea the more I think of it :-)

[roberthahn]

There is a blogging system called Blosxom; one interesting characteristic of the system is that if you ask for a file with an html extension, it would fetch the data, wrap it in an HTML template, then return that representation. However, if you ask for the same data with an rss extension, it would wrap it in an RSS template, and send that back.

The basic premise is that the file extension should dictate the representation, since the dumb browser doesn't provide the user with a way of submitting requested Mime types. The end user is also free to request the data in any format they see fit.

This notion makes a lot of sense to me.

However, it seems weird to ask for different *kinds* of HTML representations by adding a filename extension.

Still, the problem you're trying to solve is: how can I fetch an alternate HTML representation? Well, the alternative representations are still nouns - in this example, they're called forms. Wouldn't it therefore make sense to make a request to /forms/articles/1 if you wanted such a representation?

(this isn't to say your proposal is bad - I like it a lot, actually, but it seems to be better matched to high traffic sites -- it doesn't scale down very well)

[John]

roberthahn: That's how Rails currently works. If you add a .rss extension, for example, to the URL the rss type is invoked in the respond_to block. You can even add .html as the extension, though html is usually the default format, thus the extension is redundant.

[roberthahn]

John: true, but the problem that needs solving is how to request two different representations of the same data that require the same mime type. The current best practice appears to be using urls that conform to the /articles/1/edit or /articles/1;edit style. They're both retrieved with GET, and they both carry identical data (conventionally speaking), but the representation is not identical.

All I'm suggesting is to rethink the nouns for alternative representations of the same thing. In my previous comment, I suggested collecting all forms under the /forms/ namespace, which is convenient for routes design (just check for an optional forms/ in the path).

In my opinion, this type of approach would eliminate the proxy and keep the infrastructure size down, and still give you clean, RESTful URIs.

[Dave Thomas]

Robert:

I think that's a problem, but there are other, bigger ones. In particular, I think that the idea of separating the business logic from the presentation is a powerful one. We've tried in the past to do this, but with some exceptions, it's always broken down a bit. I was hopeful with Rails, but I find the way that controllers get muddied with respond_to is a step in the wrong direction.

What I'm suggesting is moving that entire decision away from the business logic component, instead wrapping it in presentation servers that each specialize in one particular rendering.

Although the URL change initiated this discussion, I think I'm now looking at a broader picture.

Dave

[Steve]

Just to play devil's advocate: I appreciate the value in logically separating business and presentation logic. But what benefit does physical separation add?

This was one of the touted benefits of EJB sessions beans. By separating your business and presentation servers you can put them on different machines, leading perhaps to better perfomance in some cases, teams can specialize in presentation or business tiers, and you get a clean separation of concerns.

However in practice the extra remote procedure call and mashalling usually made things slower in practice. Putting everything in one app and clustering it proved simpler. As far as separating business and presentations concerns, well you can still create a business server running in a separate process that mixes in presentation logic and is tightly coupled to one presentation. Separation of concerns is a matter of programmer discipline, not an architecture. Then there's the issue of marshalling; it can be expensive and time consuming. Although marshalling Ruby objects should be much more transparent. And with agile teams tended not to specialize by tier, so all the marshalling and gunk was just more YAGNI violations.

Finally there's the issue of where to put the business logic if you don't put it in your AR objects and domain model? EJB was partially responsible for the anemic domain model pattern. If you don't put your business logic in your domain objects (or objects your domain objects delegate to), where do you put it? In your controllers??!??!

So nowadays the move in Java circles is to put everything in one process with a lightweight 'container' like Spring (which runs in the Servlet process).

[roberthahn]

Steve:

I don't know anything about EJB's, particularly how heavy they are, but I don't think they're nearly as lightweight as HTTP is, with all the things you can do with it.

Assume a presentation server with 3 features: Deep Etags baked in, HTTP Status code savvy, and caching of fully-rendered pages to the file system.

With those 3 features, most of the traffic between the presentation server and the app server would only be a few tens of bytes large.

I don't think 'heavy AR objects' would be required, as any data being sent to the presentation server should only really be enough to satisfy the rendering requirements. I'd probably would prefer to use JSON instead of Atom, because a JSON marshaller would translate the payload directly into simple script-native data structures. For 90% (maybe more -- I only build simple systems, so this would come close to 100%) of the cases, this is really all you need.

The only problem left that *I* see is that we now need to construct a model that shows how to design URL's for easy, discoverable use between the browser and the presentation server. I'd be interested in seeing what people would come up with there. I think it's a tricky problem because it looks to me like we're removing REST as a constraint on URL design.

[CRUD is naked without transactions]

Dave talks about database CRUD - Create, Read, Update, Delete - and how it is a RESTful model for apps distributed across the web.

It is... but it is missing one thing. Databases usually have transactions, so the CRUD is wrapped up in transactions. But if you do not have transactions in your web based application, you run the risk of losing information.

E.g. two of us could be editing the same page using Atom. E.g. two of us could be editing the same wiki page.

Some flavours of Update get around this because the transaction is inherent in the Update: i++, rather than read i->tmp;tmp++;update tmp->i.

Many applications have a locking protocol.

But it would be best if web based apps extended CRUD using a compare-and-swap operation: an Update of the form "Here is the new value, and here is the old value. If the old value is the same as the current value, do the update, else warn."

You can s/the same as/compatible with/.

With such a CAS operation, web based apps would lose data less often.

[Dave Thomas]

Dear CRUD:

Actually, you wouldn't use database transactions in these scenarios anyway--locking a row while someone edits it is considered bad form (if for no other reason than they could go off to lunch). Most long running transactions are implemented via variants of optimistic locking, as you describe.

So, in these scenarios, the RADAR architecture is no different to the more conventional one.

However, transactions do have a role to play. But that role should be entirely on the REST server--they should never leak out into the interface server. You need to design your REST protocol accordingly.

Dave

[Steve]

Robert,

Thinking about it, I usually stick my AR object in an instance variable in the controller and consume it in the view. Going the other way I usually just call update_attributes. Pace Dave, I do often have business logic in my AR objects, but not much that a presentation server would need to invoke directly.

I didn't know we were now moving away from RESTful URLs. In any case you could make them discoverable via a central page that listed all the root resources that the server provides, and so on.

One key to making this work would be to make things as transparent as possible. Another concern is speed -- adding an extra network hop doesn't usually help. Also I think it's harder to create generic resource/service interfaces than one thinks. The ATOM guys are really smart and were working in a fairly constrained domain, but it still took them several tries to get the protocol right. For apps that only have one kind of client you really wouldn't know if you had created a generic interface or not. (Not that you should care if you have no plans for another client - RADAR might be the wrong architecture for you in that case.)

I wonder if it would be possible to package this as a plugin that you could co-locate if you wanted to? If it was running in the same process as the client (both would have to be Ruby in this case) it would by-pass XML marshalling and network hop, similar to local EJB beans.

Another issue would be lazy-loading of related resources. That should be pretty easy to do; ActiveResource might already do it. SDO I guess is similar.

This approach is somewhat similar to the 'Service Layer' pattern by David Rice (I think) in Martin Fowler's 'Patterns of Enterprise Application Architecture' book, except that it's a 'Resource Layer' instead.

Steve