Hmm … I’ve read the original Twitter blog post but I cannot find a strong direct statement that points the finger at XMPP not scaling … perhaps the Jabber folks or someone can clarify this entry (or perhaps the point is simply being stretched a bit to make the case for Gnip). Is it the Twitter design or a wall that XMPP hits for this type of social application. My inclination so far was to think that XMPP itself was not the problem, that it had to do with the design, topology and infrastructure related to Twitter. But the folks at Gnip seem to imply that XMPP might be an issue when used in this manner at least – thoughts?
Let’s address the first issue: How we would benefit Twitter and anyone that wants to integrate with Twitter data.
Twitter has found that XMPP doesn’t scale for them and as a result, people are forced to poll their API *a lot* to get updates for their users. MyBlogLog has over 25,000 Twitter users that they throw against the Twitter API every 15 minutes. This results in nearly 2.5 million queries against the API every day, for maybe 250K updates. Now add millions of pings from Plaxo and SocialThing and Lijit and heaven forbid Yahoo starts beating up their API…
If Twitter starts pushing updates to us, via our dead simple API or Atom or their XMPP server, we can immediately reduce by an order of magnitude the number of requests that some very large sites are making against their API. At the same time, we reduce the latency between when someone Tweets and when it shows up on consuming sites like Plaxo. From 15 minutes or more to 60 seconds or less.