Is REST === ROA?

I’ve had this feeling of unease around REST for awhile, and I’m coming to the conclusion that there’s a central disconnect in the abstraction we’re using for the RESTful model.

REST is “Representational State Transfer”, so at least nominally it centers on the ideas of “state” and “representation”. But the state in a RESTful communication is tied to the representation, and only indirectly to the underlying resource.

That is, if I request a data resource using a GET, I get data back that represents the state of something in the system. But the data, whether it’s XML or JSON or tab-delimited lines, is just a convenient way to represent the data, rather than a pure serialization of the underlying database rows or whatnot.

For example, I might throw some things into the representation, like calculated values, or maybe hyperlinks, that don’t exist in the underlying object. Or I might leave some stuff out that the client has no business seeing.

What makes it worse is that the “pure” data objects in my system might have several representations. Or else I might be able to GET resources that combine several underlying data objects. Or a lot of underlying data objects. Worse, the GET might return a representation of a calculated result, for which there’s no persistent underlying data object at all.

There is this idea, that I am fond of, that RESTful services are the same as Resource-oriented Architecture. That if you expose a service, follow all the REST rules, and diligently follow the GET PUT POST DELETE model, then you have an ROA system.

But trying to claim that REST is ROA puts us in a hard place when we look at questions like, “What about searches?” Search is one of those hard problems in the REST world, because it clearly belongs there, and yet there isn’t a persistent underlying resource that maps to a search result.

Ultimately, I think it comes down to a shared misconception that RESTful communications are Resource-oriented. But I don’t think that’s right — they are Representation-oriented.

I’m still kicking the idea around, but in the end I think we’re going to have to get rid of the idea that REST is ROA. They are very compatible, but still not the same.

There are already standards around for communicating data structure as well as data, but I think we’re going to have to rely on those to provide our ROA. REST is a useful model for shaping communications to remote services, but there’s still a big disconnect with what we’d really expect from a true ROA.

EDIT: I just did a little more reading, and it sounds like it boils down to: “resource” doesn’t have a single definition. Lots of specs mean different things when they say “resource”. So that is an area of emerging clarity. In essence, saying you’re doing “ROA design” is like saying you’re doing “?OA design”. I guess I’m too practical, so it’s easy for me to discount the “resource” part and focus on the practical “representation” side. 😉

Will REST give us an Internet OS?

We were in a week of training, and it was pretty exhausting. The last day was the most interesting, because we got into the “advanced” stuff. The guy training us was a really smart guy, and had some good ideas. At one point he offered his vision of a sort of file system distributed across the web, where he could have, say, pictures scattered all over the place and just pull them in.

I perked up at that and observed that’s more or less the vision of REST… that by making access to resources uniform, you could just go out and grab them from whatever service was holding them at the time.

I didn’t mention that I hold a patent — for what that’s worth hehe — or at least my employer does 😉 — on a system for managing arbitrary resources by relating URIs to each other, with a state machine for managing the lifecycles of those relationships. Which is kind of part of what he was talking about.

Then, in another conversation, one of the guys on my team — a really, really sharp guy — was creating a RESTful interface for launching Map/Reduce jobs in Hadoop. As we were chatting I recommended he actually expose three addressible resources for that purpose: a mapper resource, a reducer resource — and a control resource that ties the other two together through URIs.

Anyway, the upshot of all this is that as I pondered it some more, it occurred to me that I’ve always been talking about REST in the context of *services*. That is, how cool would it be if my service were just like yours and I didn’t have to spend 2 days of coding to write a client to your service.

And that’s a noble thought, but I think the broader, more powerful model that’s going to emerge is one of combined data and processing services across the internet. What would it take to turn the Internet into a giant OS?

An OS needs to store data, and it also needs to provide execution units. And we’re getting there with cloud computing. But the units of execution are still tightly bound to the idea of “a box”. We call our boxes “virtual”, but they are still boxes.

So what if “the box” an application ran on was The Internets?

This blog is a place for me to throw out (up?) any half-baked, often whiny ideas that pop into my head, so I don’t know if that’s just me being artistic, or it’s me missing the boat by 5 years again, or if that’s actually a good idea. But it seems like there’s something there.

And maybe if that happens, I can put that stupid patent of mine to work, finally.

The Central Problem in REST

So after much deep thought, I have uncovered the central problem in the modern interpretation of REST. I’ve had this nagging feeling that something doesn’t quite line up between “resource-oriented” design, and the restrictions on design declared for the sake of a “uniform interface”.

It’s not so much a problem with the original REST architecture, but a conflict between that original vision, and the notion of what we call RESTful these days.

Specifically, the “uniform” part of the uniform representation is there so a server can serve up data that a consumer can just pick up and render in a uniform way. So JPG images, HTML pages, and PDFs are all perfect examples. A browser just gets the data, then renders them. It doesn’t know or care what’s inside the representation, because it conforms to a uniform standard.

But the moment you have your REST service send back an application-specific representation, it’s not uniform anymore. Suddenly the client needs to know what the fields *mean* in order to use that information.

And it doesn’t matter if you’re using the HTTP verbs or not. As soon as you return any representation that only a specific set of clients are specially coded to understand, then it’s not uniform anymore.

The classic example where this shows up is the problem REST has with search. All the search engines have different URLs for their searches, and all of them return different representations. So they are, on their face, not uniform.

Another place where the central problem shows up is schema versioning. If the client can only understand a certain version of the representation, that conforms to a specific schema, then REST doesn’t have a good answer for that. The community is trying to come up with an acceptable standard: use content negotiation, put it in the URL, put it in a query string, put it in a special header? But there isn’t a clear answer, or a clear winner, because REST isn’t about how to manage non-uniform resources.

Designing your service interfaces closely to the HTTP spec is a good idea, and leads to nice clean design. But it still isn’t consistent with REST, even if you disavow RPC and use all the HTTP verbs right. But then we should stop talking services and how REST-ful they are, and start talking about HTTP-ish they are. Citing REST just doesn’t work when you’re serving up specialized, non-uniform application data that requires specialized logic on the client side.

In the end, it may turn out that REST just isn’t an architecture well-suited to the purpose of sending around application data.

Representation … Resource … and …?

OK I’m trying to get my brain wrapped around the terminology in REST. I think the important thing to bear in mind that REST is a client-server, client-pull-message (or client-message-push) architecture. So it’s about how to identify and move information initiated by a client.

The main things that REST boils down to are:

• Identifier: some arbitrary string which points at an instance of data from an abstract class of data
• Representation: just some way to bundle up information for transmission
• Resource: a conceptual idea of some information you want to either get, or put in place
• Static resource: a resource that is backed by some data that’s going to stick around for a while, so you expect to get the same information back for multiple retrievals over time, more or less
• Dynamic resource: a resource that is more ephemeral, like a calculation done on the fly, or an aggregation of other data that might be changing rapidly
• …? : behind the resource, there will be some data storage, or processing, that results in a bundle of data we can call a resource and roll up into a representation and point to with an identifier

I see the last bit called an “entity” frequently, but really the REST architecture definitions, including Roy Fielding’s, mostly stop at the resource  level and leave the rest to our imagination.

Personally, my main stumbling block is at the use of the term “resource”. I think of that word as something static, or even worse, a specific instance that I can put my hands on. So when I say, “a resource”, I usually think “the blue mouse in the top drawer of my desk.” In common usage, that blue mouse is a resource. It’s something I can use. But in the REST world, the Resource is “computer mouse”, and the Identifier adds “/desk/drawer?which=top”. So what I normally think of “a resource” in common usage is really a Resource + Identifier in REST parlance.

In the OO world, an abstract class is a Resource, and an instance is Resource + Identifier. More or less.

I think one of the main reasons for all the blurriness is that what we call REST is really a collision of several worlds:

1) The REST architecture itself, as laid out by Roy Fielding. Which is really more about ways to string together client-server systems in a uniform way, granting interoperability and scalability at the expense of efficiency.

2) The HTTP protocol, which is the primary protocol people use to implement REST. It’s not at all correct to say “REST is HTTP”, but if you are doing the REST thing, and putting it on top of HTTP, then REST commands you, “Use HTTP strictly! Don’t use your own personalized variation of HTTP!”

A personalized variation of HTTP is essentially what 99% of the industry uses today.

After all, despite it’s pretensions, the software industry enjoys freedom from the tyranny of rational thought.

3) The ROA, or resource-oriented architecture crowd, which tends to take the basic terminology of REST and wrap it around a design based heavily on “static resources” or “nouns”. I say noun-based and not object-based, because the strict ROA guys require hyperlinks, but don’t allow intermixed data and methods like you’d see in OO.

4) All the personal preferences, biases, superstitions, agendas and personality disorders of everyone who’s involved with developing software for the Internets. Which, taken together, somehow fails to oblitherate points 1-3. Most of the time.

Based on the conversations I’ve seen where I work, the primary religious conflict is between the static resource and dynamic resource crowd. Let’s face it, the Internets were built not just on a series of tubes, but on the GET/POST verbs. Chopping RPC out of REST flies in the face of what Fielding was trying to do in his paper — to capture what made the Internets work.

For example, some guys in my shop who were brave enough to dive into the Flex realm have been horrified to discover that Flex doesn’t support even the basic set of functionality required to implement a static-resource-over-strict-HTTP design. It basically just supports the GET/POST model, and no more. For some reason, the Flash guys thought that was all they needed.

All that said, I fall heavily on the “noun” side of that religious war, if only because it’s more challenging and fun.

So the upshot is, from my readings, I think that REST is a very open-ended idea that almost every web developer has been using for the past 10 years anyway, just because that’s the way browsers and web servers work. Now we’re getting strict about the HTTP protocol — which is a really good thing — and raising the awareness of the power and simplicity of noun-based design. Which is also a good thing.

But I’m going to try to keep the terminology straight, because I think the core ideas of REST are worth keeping in mind, and it’s important to keep the terminology clear in discussion.

Not that I’m good at that, especially where it doesn’t help my argument.

REST is… representation?

[Note: since writing this, my understanding of the “real” REST terminology is getting better, so I’m not sure a lot of this discussion is on mark. I would delete the post altogether and start over, except that the comments are actually better than the post. So I appreciate the comments, and I’ll let the post stand, to my shame to honor those comments …just heads-up.]

We’re in the process of creating a big data aggregator, which is going to be a massively distributed and parallelized data dump, where all our business units can just dump their data, and then that data will be available for them to run whatever map-reduce jobs they want. The problem I’m thinking about now is how to make that work, when the form of that data will vary from source to source, and even for a given source, it will vary over time.

Also, as our shop starts to put together all these disparate REST systems, every team is putting their own spin on the problems of URI definition and resource representation. It would be cool to pretend that we could just draft a standard, and it would be clear and perfect, and everyone would code to it strictly. But whoops — I work in the software industry, not in Nirvana. So that’s not going to happen.

Or to use another metaphor which is appropriately vaguely insulting, in this zoo, I work in the monkey house. At least, not anywhere that reason dominates.

So I’m starting to think about representation. We have transformation technology lying around that will help us get there, at the expense of CPU cycles, but it means laying out all the resources that span our business space in a sort of directed graph, and filling in the links with transformations. Do-able, but people internally are going to whine a lot. Which will be a side benefit.

So I started thinking about REST — representational state transfer. In kicking ideas around, I’m wondering whether it’s correct to call REST resource-based, when really it’s representation-based.

If I come to a REST service, it’s because I want to GET or PUT some data (ignoring the bad-boy POST and the troublesome DELETE for now). And I’m going to GET or PUT the data I’m interested in a particular representation.

That data will probably map to some underlying resource — but it might not. The data might map to something ephemeral, like a search result. A search result isn’t really a resource in the system, it’s the *representation* of some *work* that I just did. I *might* create a key for that search, shove it in a data store, and treat it as persistent, but odds are I won’t.

A resource might be mapped with multiple URLs, and it might have multiple representations. A resource might be included in the representation of another related resource.

Even if I’m grabbing data from another source — let’s say a table of user accounts — the service is going to decide how deep to drill to add related records to the representation. My representation might be a very deep rendering of the data, or it might be shallow, with links that tell me where to get the rest of it.

It might come back as XML with a strict XSD schema, or it might just come back as a JSON bundle and you take-what-you-get. It might be a raw binary stream.

There’s a many-to-many relationship between representations and resources — so why are we equating them?

The point is, I think that calling a REST service resource-based is misleading. In nice, simple systems serving up static resources like JPG images or HTML files, sure the representation maps directly and easily to the actual data in the data store. But if you step away from that and spend any CPU cycles at all “preparing” the data — even rendering columns in a table into JSON — then your interface is not resource-based anymore, it’s representation-based.

I haven’t thought this all the way through — obviously — but I’m having the nagging feeling that this is related to my problem with the big data dump, and all these disparate REST systems we’re building. I’m having the feeling that asking or even wishing for uniform representations is wrong-headed. That we should be thinking of these systems as a federation of *representation* services, not a federation of resource services.

Right now, I have the feeling that making that fine distinction will be liberating and make things easier, but I have to noodle on it some more.