Work has begun on the Servlet Expert Group to come up with a standard set of atoms for WebSocket applications in a Java EE environment.  I'm not 100% convinced that the servlet spec is the best place to do this but I suppose it saves us the hassle of submitting another JSR and all the work that would entail.  However, that's where the work is being done for now, so whatever.  This is something I've naturally thought a lot about over the last couple of years as I've worked to implement and promote WebSockets so here is roughly how I would do it. There's a lot of "we have to have something in the servlet spec" comments floating around.  I don't necessarily agree with that sentiment but there is one thing I would: support for the upgrade request header.  This would have the servlet container automatically detect a request with this header.  The container would then query the system for all implementations of upgrade request handlers registered.  Then, based on which protocol each handler has said it deals with, hand off processing to that handler.  At this point, the request is not necessarily in the servlet stack at all but a completely separate API that might not even be back by a JSR.  This has multiple advantages in my mind.

1. WebSockets uses this header and so this feature  is already going to be needed for WebSockets to work.
2. It provides a general hook and customization point at which developers can choose to extend the container without resorting to modifying the source of the container itself to handle the new protocol.  More importantly, it does so in a standardized fashion allowing for portability at least among the higher levels of the API.  Certain container specific code might be needed for such things as suspending/parking requests and the like.  But these could easily be abstracted away such that the protocol logic itself could be fairly portable.
3. This also decouples new APIs such as WebSockets from the constraints of the legacy(?) servlet API.
4. Using this hook, someone could homebrew their own protocols using the upgrade mechanism and be able to integrate in to their container with little fuss.

Barring some tweaks here and there, that's all the servlet spec really needs to offer.  So where does that leave WebSockets?  Why, in its own JSR, of course!  I really think it needs its own separate JSR with its own EG populated by folks who know and understand the technology.  More importantly, this separate EG would feel less bound to make this new API consistent/compatible with the servlet API.  While there is certainly a need for integration with the servlet spec for access to things like the session data, binding the two APIs would really hamstring the interesting things we could do with WebSockets.  So what would all this look like?  Let's look at some mostly java mockups.

First, the servlet side upgrade handler. It would look something like this. If you've used @WebServlet, this should look familiar to you.

@Target({ElementType.TYPE})
@Retention(RetentionPolicy.RUNTIME)
@Documented
public @interface HttpUpgradeHandler {    
    /**
     * The name of the target protocol
     */
    String value() default {};

    /**
     * The description of the handler
     */
    String description() default "";

    /**
     * The display name of the handler
     */
    String displayName() default "";
}

And the interface exposed for implementations:

public interface UpgradeHandler {
    public void doUpgrade(ServletRequest request, ServletResponse response) throws UpgradeException;
}

This provides a general mechanism for any application developer to register a new handler thus extending the functionality of the container without need for hacking the container itself. So what would one of these handlers look like? Maybe something like this:

@HttpUpgradeHandler(protocol ="websocket", urlPatterns={"/sample"})
public class SampleWebSocketContext extends WebSocketContext {
    @Override
    public void customizeHandshake(ServletRequest request, ServletResponse response) {
        super.customizeHandshake(request, response);
        // insert cookies here
    }

    @Override
    public WebSocketContext createContext(ServletRequest request, ServletResponse response) {
        return new SampleWebSocketContext(request, response);
    }

    private static class SampleWebSocket extends DefaultWebSocket {
        private String userName;

        public SampleWebSocketContext(ServletRequest request, ServletResponse response) {
            super(request, response);
            userName = request.getParameter("user");
        }

        public String getUserName() {
            return userName;
        }
    }
}

By the time control gets handed off to WebSocketContext, the appropriate headers requesting the upgrade have already been found and validated against name provided in the annotation. All that's left to do here is perform the actual upgrade. In this case, I added a method to customize the handshake for cases where, perhaps, the developer wants to add a header field or validate an existing header value. So what comes from the servlet specification is that annotation, the interface, and an exception. There might be more as details are hashed out, but that's about all I would say that the servlet specification should deliver. That paves the way for a WebSocket JSR export group to define its own atoms.

Not entirely surprisingly, I imagine a WebSocket JSR would end up with something that approximates what I built in Grizzly. That said, there are certain ... rough points that need to be smoothed over but that can most easily be smoothed over by 1) an expert group and 2) user feedback. Still, here's how I see one approach at cracking this nut.

In Grizzly, we have a WebSocketApplication which is the center of the developer's world when building WebSocket applications. I've recently come to prefer WebSocketContext instead. It's largely semantic but whatever. This is where all the application logic, if there is any, would likely want to live. Between the WebSocketContext and any potential custom implementation of WebSocket, there's a lot of room of custom logic. But none of this is tied to the servlet spec beyond the initial use of the ServletRequest and ServletResponse which I think is key.

This separation allows the WebSocket JSR evolve without needing to worry about servlet evolution. More importantly, it allows innovation to happen without needing a JSR at all. Because there's a generic upgrade hand off, developers are free to extend their container in whatever crazy directions they choose. Imagine implementing a custom JMS transport over HTTP and tunneling through your web server. Or FTP. Or SNMP. Or some completely custom, proprietary protocol. This extension point takes the EE platform one huge step down the road of enabling developers to build applications on and extending the EE platform without the need for container specific hacks. It aslo, of course, paves the way for future JSRs to provide new functionality without extraneous requirements and constraints from external JSR lifecycles.

At the risk of repeating myself, I'd like to say again that this proposal is not perfect. It almost certainly has some corner cases I haven't thought of or addressed here. But it's at least along the lines of how I've been imagining something like this would work for the last couple of years. It's a start. I have the code I showed above and more in a git hub repository if you'd like to take a look in more detail. I'd love to hear any comments you might have.