It’s been a few months since I wanted to make a data visualization project based on Angular and d3. The one thing that was missing was some data to work with …
I tried to find something that fit me on open-data sites but I was unsuccessful. I finally had the idea to use the Twitter Stream API, which would get me lots of data to process and moreover, in realtime …
To handle the tweets post-processing part, I built a node module : twitter-stream-channels that I plugged to socket.io and express to send processed data from the Twitter Stream API to the frontend via the node server.
Finally on my Angular app, I made a service that handles the transport/persistance layer to share the realtime data collected from the websockets to the d3 directive. All of them are home made and responsive.
I was looking for a project where I could train myself mixing Angular and d3 … I ended up doing a little more backend thant I thought 😉 …
Tophe
PS : See the README on the github repository of the project for further informations about the implementation.
posted this at HN, but seeing as i’m not sure you submitted there:
“this is great, well done, I’ve really enjoyed playing around with this.
Does twitter-stream-channels support the follow [1] part of the of the twitter streaming api? Didn’t see any mention of it in the docs.
What I’d really like to do would be to give it a big list of users to follow and then filter for keywords on that list. I did try changing “track” in channelsDescription.json to “follow” but to no avail.
[1]: https://dev.twitter.com/streaming/overview/request-parameters#follow “
Hello bvm,
I must say that I never tested this specific use case, though, I took a rapid look at my code, I had it in mind while coding it :
https://github.com/topheman/twitter-stream-channels/blob/master/lib/StreamChannels.js#L92
https://github.com/topheman/twitter-stream-channels/blob/master/lib/StreamChannels.js#L31
Try this kind of snippet (just add a “follow” attribute aside of the “track” parameter). Please let me know if that works (it could help other people looking for the same feature and I’ll update the doc). Otherwise, open an issue on the github bugtracker and I’ll get on it : https://github.com/topheman/twitter-stream-channels/issues
var client = new TwitterStreamChannels(credentials); var channels = { "languages" : ['javascript','php','java','python','perl'], "js-frameworks" : ['angularjs','jquery','backbone','emberjs'], "web" : ['javascript','nodejs','html5','css','angularjs'] }; var stream = client.streamChannels({ track:channels, follow:"fooAccount,barAccount" });Apple is not a fruit, at least on Tweeter in most of cases 😉
Next implementation task: manage multi-sense words in data filters to get pertinent responses by analysing the context or the presence of some related words …
I suspect that your choice is oriented by the will to get some fast data responses, isn’t it ?
However, great job, very interesting way to take the best of modern tools.
@alexk Of course, my choice was oriented that way : get fast data, to make the point that it’s real time, but not too fast though, so that the user could catch up some how.
And about “apple” choice, I did that on purpose so you could have this kind of reaction, that the twitter-stream-channels module underneath is not about semantics but raw post-processing (the keyword “apple” could be anything : a fruit or a brand, like you implied).
Thanks for the comment alexk 😉
@bvm Now that I’m at home I checked out your problem, it doesn’t seem to be possible (at least with the current version of the module I use for the network layer to communicate with Twitter : twit). I opened an issue here : https://github.com/ttezel/twit/issues/135
@bvm and anyone who would like to use the “follow” attribute to narrow down the tweets to only a couple of accounts, this isn’t the way the Twitter stream API works.
Explanation from the twitter doc : The track, follow, and locations fields should be considered to be combined with an OR operator.
https://github.com/ttezel/twit/issues/135#issuecomment-56269867
Thanks a lot Christophe, I’ll look at implementing the filtering client side.
And thanks so much for looking so thoroughly into my question! It’s really appreciated.
When i launch Datavisual, i could not see the main container whereas i can see the header with count tweet per channel. Could you help me please ?
Hi, I’m having the same issue, no D3 visuals, just the top header. Any help you could give would be great as I’m keen to understand this a bit more. thanks.
Error: [$injector:unpr] Unknown provider: $templateRequestProvider <- $templateRequest <- $route <- ngViewDirective
http://errors.angularjs.org/1.2.28/$injector/unpr?p0=%24templateRequestProvider%20%3C-%20%24templateRequest%20%3C-%20%24route%20%3C-%20ngViewDirective
at angular.js:78
at angular.js:3801
at Object.getService [as get] (angular.js:3929)
at angular.js:3806
at getService (angular.js:3929)
at Object.invoke (angular.js:3956)
at angular.js:3807
at getService (angular.js:3929)
at Object.invoke (angular.js:3956)
at angular.js:5716
persistance.service.js:157 getState Object connected null
persistance.service.js:62 connected Object
persistance.service.js:93 twitter:connect Object
persistance.service.js:157 getState Object connecting connected
stateNotifications.directive.js:18 state.twitter connecting connecting
stateNotifications.directive.js:30 state.socket connected connected
stateNotifications.directive.js:49 state.socket 0 0
persistance.service.js:98 twitter:connected Object
persistance.service.js:56 updateStateConnexionTime – connect Object
stateNotifications.directive.js:18 state.twitter connected connecting
@Big Predict @NeOnBlade , I’ve opened an issue here : https://github.com/topheman/topheman-datavisual/issues/1 to track the bug, I don’t have this kind of problem but I added some questions that might help me to find the origin of your bug.
Please continue the conversation on github issues.