Stream processing systems (and associated analytical components) will become a critical underpinning for much of what is talked about in terms of workstreaming, lifestreaming, attention streams, collective intelligence and so on. Discovering patterns across people, interactions, information, activities and social networks and assessing those relationships is difficult enough. It becomes even more challenging when you also want the results to be communicated in a manner that is contextual, relevant and sensitive to attention (and confidentiality) needs.
Of note are aggregation and correlation hubs that can also add value on top of the analytics. A good example is Feedburner which allows augmentation of information with additional insertions that can alter attention, influence participation and encourage action on the information being communicated. The result is a type of network effect. Augmentation could include insertion of tagging options, references to relevant communities or workspaces, real-time counts on how other people are treating that information, pointers to workspaces, as well as widgets. I've alluded to these trends in a recent presentation on collective intelligence (via SlideShare here) and the relevant slide below.
A near-perfect world would:
Take what's going on in my life...
Directory of Lifestreaming
I probably should lump all these into the Directory of Attention, but I’m not going to.
Don’t look for a definition of lifestreams on Wikipedia, because it will take you to a Final Fantasy VII page. The term actually goes back to at least 1997, when Eric Freeman and David Gelernter saw it “as a network-centric replacement for the desktop metaphor. As their project page (last updated in 2000) at Yale put it:
A lifestream is a time-ordered stream of documents that functions as a diary of your electronic life; every document you create and every document other people send you is stored in your lifestream.
Source: loose wire blog
Add more context about what I'm doing...
My Data Stream
After a year and a half of using social applications heavily, I recently had to revisit the plan to aggregate all my activity into one data stream. As the calendar rolled to 2007, I kept wishing I could look at all my social activity from 2006 in context: time, date, type of activity, location, memory, information interest, and so on. What was I bookmarking, blogging about, listening to, going to, and thinking about? I still had the urge to have an information and online activity mash-up that would allow me to discover my own patterns and to share my activity across the web in one chronological stream of data (to start with anyway).
In the context of my interaction patterns
/Message: Traffic And Flow
Social applications are -- at their basis -- a means for us to communicate. Not just point-to-point communication, as in email or in IM, but increasingly a more general communication from me to the network of others that believe that I matter. This is what blogging affords us, and Flickr streams, and even Twitter.
We are sending all sorts of traffic -- different sorts of messages -- flowing through the various implicit and explicit social networks that we define ourselves through, and through which we discover meaning, belonging, and insight.
This traffic flow -- made more liquid by RSS and instant messaging style real-time messaging -- is the primary dynamic that I believe we will see in all future social apps. Yes, we will want to have our traffic cached -- for search and analysis purposes -- but we will increasingly move toward a flow model: where the various bits that we craft and throw into the ether -- blog posts, calendar entries, photos, presence updates, whatever -- will be picked up by other apps, either to display them to us, or to make sense of them. We want to consolidate all into one flow -- a single time-stamped thread -- that all apps can dip into.
Source: /Message: Traffic And Flow
Correlate everything in an intelligent manner...
Complex Event Processing for EDA
Organizations are increasingly adopting EDA as a platform to effectively manage the increasing number of events generated from IT systems, business processes and physical sensors such as RFID. With this in mind, Oracle is developing new Complex Event Processing (CEP) capabilities that support high-volume, programmatic analysis of events to identify patterns and correlations across multiple heterogeneous event sources. These capabilities will complement Oracle's existing EDA offerings, such as Oracle Business Activity Monitoring, that provides real-time operational dashboards for tracking business key performance indicators, multi-channel alerting, and invoking automated or manual response actions.
Continue to analyze continuously - past and present...
Streaming Analytics vs. Perpetual Analytics (Advantages of Windowless Thinking)
The terms "streaming" and "perpetual" probably sound like the same thing to most people. However, in the context of intelligent systems, I think there is a big difference.
[Note: when I use the term "observation" below, feel free to think about this as a synonym for "transaction" or "record."]
Streaming analytics involves applying transaction-level logic to real-time observations. The rules applied to these observations take into account previous observations as long as they occurred in the prescribed window – these windows have some arbitrary size (e.g., last five seconds, last 10,000 observations, etc.).
Perpetual Analytics, on the other hand, evaluates every incoming observation against ALL prior observations. There is no window size. Recognizing how the new observation relates to all prior observations enables the publishing of real-time insight (i.e., The Data Finds the Data and the Relevance Finds the User). And another unique property is Sequence Neutrality (i.e., future observations can affect earlier outcomes).
And discover what is important to me (even if I may not know it)...
Creators of "The Content Router for the Web"
The Real Time Matrix provides technology to find what is relevant to "Me" from the continuous flow of live, Web content and send it to me on my personal channel. And, the content I want finds me in real time on my cell phone, IM, browser, reader, PDA, or any other Internet-connected device.
It is available today through the free iJ.am ™ web site as well as a mashup service that you may incorporate into your own products.
Source: The Real Time Matrix#
Augment that information before you communicate it to me...
Give your subscribers easy ways to email, tag, share, and act on the content you publish by including as many or few of the services listed below. FeedFlare places a simple footer at the bottom of each content item, helping you to distribute, inform and create a community around your content.
The FeedFlare API (FlareAPI) allows anyone to extend our existing FeedFlare service. Provide new actions and incorporate outside services to make your content more interesting and engaging — both in your FeedBurner feed and on your website.
Signal such information or messages relevant to my work context and focus...
The New Presence Model
New Presence is a user-centric view of presence. Instead of merely reflecting the crude, device specific “availability awareness” of today, New Presence systems understand our context, relationships, wants and desires. The New Presence model reflects the integrated conversation web we live in today.
The New Presence model has three building blocks: relationships, context, profile. Each of these is a core component in a model which is fundamentally richer, and more user-driven than any presence model previously.
and, of course, in a manner that is aware of my attention priorities...