« Social Networking Webinar | Main | Quick Summary Of Client Meetings At Catalyst »

July 11, 2008

TrackBack

TrackBack URL for this entry:
http://www.typepad.com/services/trackback/6a00d834515a5969e200e553956dd98833

Listed below are links to weblogs that reference Twitter & XMPP Scale:

Comments

Feed You can follow this conversation by subscribing to the comment feed for this post.

Matt Tucker

I think you have it right -- there's really no information about XMPP not scaling (and there really shouldn't be since it's about the perfect protocol for this type of use case). The issue is that Twitter sending out every single update on its network is an *enormous* amount of data that would stretch the boundaries no matter what technology was being used.

The typical solution to this problem is to provide feeds for just the data that's relevant to each party. In the case of the Zappos integration mentioned -- it would be listening for updates from just Zappos employees and not the entire network. In other words, a pubsub architecture. Building out a robust pubsub stack isn't trivial, but it's probably the best option for Twitter and will get them out of the horrible polling problem.

Mike Gotta

Thanks Matt ... that's how I felt - I don't see XMPP per se as the problem with Twitter vs. the design pattern and implementation of XMPP specific to Twitter's need.

kael

I read that Twitter uses Openfire and I wonder if Ejabberd wouldn't be better for it.

Matt Tucker

Actually, what we've heard is that ejabberd wasn't coming close to scaling for them and that they are now experimenting with Openfire.

Eric Marcoullier

Heya -- Sorry for the confusion. I wasn't pointing the finger at XMPP at all. I was simply pointing out that in the Twitter post I linked to, Biz said this:

Despite delivery over a faster and cheaper technology, this entire public feed of Twitter updates is resource intensive—we had to be very careful about giving it out.

Mickael Rémond

Actually, I am quite familiar with the discussion, so let me provide a few points:

- XMPP is not at the core of the architecture. It means that XMPP does not have any influence on Twitter scalability. XMPP is used on the side of the Twitter architecture, very much like the SMS gateway for example. Twitter is based around a website and a database. It is not event based but statefull.

- Twitter is a Java / Scala shop. They are experimenting with a Java XMPP server because they need to reuse their internal workforce knowledge.

Peter Saint-Andre

I think Mickael hits the nail on the head -- XMPP is not at the core of how Twitter built things out. So even though it may seem to us now that Twitter is a messaging / pubsub system, I don't think it was designed that way from the beginning. Thus some of the scaling challenges.

Mike Gotta

Thanks everyone for providing additional info and clarification!

The comments to this entry are closed.

Become a Fan

Twitter Updates

    follow me on Twitter