Went to the django-nyc meetup last night, it was in Manhattan and much easier to get to then when it’s at huge in bk. Wrote down a few things to read up on:
Celery – Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well.
Django ideas – Voting on things with limited numbers of votes to spread around, ala UserVoice
Every once in a while you need some old fashion SQL style queries in django. This is a common one for reporting and aggregation. Its fairly easy to replicate in a queryset. Say I wanted to get the authors and the number of articles they have written going back to the beginning of 2009 to the present:
from django.db.models import Count
Article.objects
.filter(created_date__gte=datetime.datetime(2009,1,1))
.values('author')
.annotate(Count('author'))
I’ve had this idea in my head for a while and just got it to work this morning.
Basically you send a tweet from Twitter and this script runs, picks up your current twitter status, and if need be, updates your gChat status. I’m working on a service version of this where you could use gpowered.net to do all this for you. Stay tuned =) If I change the version of the script that I have running in the service, you will be able to see the source code that I have checked in here
For those of you that don’t know what Twitter is:
“Twitter is a free social networking and micro-blogging service that allows users to send “updates” (or “tweets”; text-based posts, up to 140 characters long) to the Twitter website, via short message service (e.g. on a cell phone), instant messaging, or a third-party application such as Twitterrific or Facebook.
Updates are displayed on the user’s profile page and instantly delivered to other users who have signed up to receive them. The sender can restrict delivery to those in his or her circle of friends (delivery to everyone is the default). Users can receive updates via the Twitter website, instant messaging, SMS, RSS, email or through an application. For SMS, four gateway numbers are currently available: short codes for the USA, Canada, and India, as well as a UK number for international use. Several third parties offer posting and receiving updates via email.” (Wikipidia)
import sys, xmpp, os, twitter
class Twitter2gChat:
twitter_login = os.environ['TWITTER_LOGIN']
twitter_pass = os.environ['TWITTER_PASS']
google_login = os.environ['GOOGLE_LOGIN']
google_pass = os.environ['GOOGLE_PASS']
twitter_status = None
updated = False
catches = 0
#keep looping and wait for xmpp response
def GoOn(self,conn):
while self.StepOn(conn):
pass
#keep listening for responses
def StepOn(self,conn):
try:
conn.Process(1)
except KeyboardInterrupt:
return 0
return 1
#handle responses
def iqHandler(self, conn,iq_node):
print 'in iqHandler'
self.catches = self.catches + 1
#we have looped enough, die
if self.catches == 4:
print 'i think we did it'
sys.exit(0)
#print response, don't need to send anything back
if self.updated == True:
print iq_node
#havn't updated yet, sent status update
else:
#we can build of response
node = iq_node.getChildren()[0]
#remove what we don't ned
node.delAttr('status-list-max')
node.delAttr('status-max')
node.delAttr('status-list-contents-max')
iq_node.delAttr('from')
iq_node.delAttr('type')
iq_node.delAttr('to')
#update the current status
curr_status = node.getChildren()[0]
#no need to update
if curr_status.getData() == self.twitter_status:
print 'status is already tweet'
sys.exit(0)
curr_status.setData(self.twitter_status)
#set response
iq_node.setType('set')
print 'sending'
print iq_node
self.updated = True
conn.send(iq_node)
print 'end of iqHandler\n\n'
#start talking to the server and update status
def updateGtalkStatus(self):
#connect
jid=xmpp.protocol.JID(self.google_login)
cl=xmpp.Client(jid.getDomain(),debug=[])
if not cl.connect(('talk.google.com',5222)):
print 'Can not connect to server.'
sys.exit(1)
if not cl.auth(jid.getNode(),self.google_pass):
print 'Can not auth with server'
sys.exit(1)
#build query to get current status
iq = xmpp.Iq()
iq.setType('get')
iq.setTo('timothy.broder@gmail.com')
node = xmpp.Node()
node.setName('query')
node.setAttr('xmlns', 'google:shared-status')
iq.addChild(node=node)
print iq
#register with server and send subscribe to status updates
cl.RegisterHandler('iq',self.iqHandler)
cl.send(iq)
self.GoOn(cl)
cl.disconnect()
#get current twitter status
def getTwitterStatus(self):
api = twitter.Api(username=self.twitter_login, password=self.twitter_pass)
self.twitter_status = api.GetUserTimeline(self.twitter_login, 1)[0].text
#don't want to use replies
if self.twitter_status.find('@') >= 0:
sys.exit(0)
t = Twitter2gChat()
t.getTwitterStatus()
t.updateGtalkStatus()
I’ve been wanting to share what I subscribe to in Google Reader and using the functions I wrote I was able to do just that. Check out the article for the full run down on the unofficial Google Reader API. This is written in python but should be easily portable to php. If i get around to it, I want to make a WordPress plugin so bloggers can share what they read with their readers. This will be followed (or in parallel depending on my mood) with a Javascript version so Blogspot users can do the same in the sidebar. On to the code!
To start off we’ll just copy the functions we need from last time. Generally this is the login and SID token functions, as well as the feed list function.
from django.shortcuts import render_to_response
from django.template import Library
from elementtree import ElementTree
import urllib
import urllib2
import re
login = 'timothy.broder@gmail.com'
password = '***'
source = 'gPowered'
google_url = 'http://www.google.com'
reader_url = google_url + '/reader'
login_url = 'https://www.google.com/accounts/ClientLogin'
token_url = reader_url + '/api/0/token'
subscription_list_url = reader_url + '/api/0/subscription/list'
#login / get SED
def get_SID():
header = {'User-agent' : source}
post_data = urllib.urlencode({ 'Email': login, 'Passwd': password, 'service': 'reader', 'source': source, 'continue': google_url, })
request = urllib2.Request(login_url, post_data, header)
try :
f = urllib2.urlopen( request )
result = f.read()
except:
print 'Error logging in'
return re.search('SID=(\S*)', result).group(1)
#get results from url
def get_results(SID, url):
header = {'User-agent' : source}
header['Cookie']='Name=SID;SID=%s;Domain=.google.com;Path=/;Expires=160000000000' % SID
print url
request = urllib2.Request(url, None, header)
try :
f = urllib2.urlopen( request )
result = f.read()
except:
print 'Error getting data from %s' % url
return result
#get a specific feed. It works for any feed, subscribed or not
def get_feed(SID, url):
return get_results(SID, get_feed_url + url.encode('utf-8'))
#get a token, this is needed for modifying to reader
def get_token(SID):
return get_results(SID, token_url)
#get a list of the users subscribed feeds
def get_subscription_list(SID):
return get_results(SID, subscription_list_url)
Then we’ll want to get rid off all the information in the feed that we don’t want and load what we do into a data dictionary. After its in the dictionary, feed names and links (and the folders they are in) are ready to be displayed. As usual, I use Django to display my pages, but everything is the same up to the final return in the Feeds method. Below is an example of what each subscription looks like in the Google Reader Feed, and below that is how to process it
class myFeed:
def __init__(self, name, link):
self.name = name
self.link = link
def Feeds(request):
SID = get_SID()
feeds = get_subscription_list(SID)
tree = ElementTree.fromstring(feeds)
d = dict()
#loop through each feed
for object in tree.findall('list')[0].findall('object'):
strings = object.findall('string')
key = object.findall('list')[0].findall('object')[0].findall('string')[1].text
#tag already exists, add to the list
try:
d[key].append(myFeed(strings[1].text, strings[0].text.replace('feed/', '')))
#tag doesn't exist, create list
except KeyError:
d[key] = [myFeed(strings[1].text, strings[0].text.replace('feed/', ''))]
return render_to_response('pages/feeds.html', {
'feeds': d,
})
For those of you that use django or are just curious how I end up displaying the feeds, this is what i have in my view:
After 2 of my posts were on the Digg front page this morning (Thank you all very much to those that dugg them), I took my first look into the Digg API. I wanted a way to take a quick look to see how many Diggs certain stories were getting. In some ways it is similar to GData: make a call to a URL, get some XML back, parse it, etc. It does, however, feel lighter, probably due to its streamlined nature. It has one purpose, get information off of Digg. Using this, I’ve added a section in the Post List section of gPowered.net that shows the Diggs of a few of the articles that I have submitted on Digg.
The API is broken into 5 main sections or endpoints. Each of these will return related types of data:
- Stories
- Events
- Users
- Topics
- Errors
In this quick HOWTO I’m going to take a quick look into the Stories endpoint so I can display the number of Diggs specific stories have. We’ll start off by making a small class to hold our returned data (useful to send to a template or just for working with later on. We don’t want to keep having to hit the ElementTree to get data out). All of the calls will be send to ‘http://services.digg.com/’. In this example I will only be querying ‘http://services.digg.com/story/{story clean title}’.
import httplib2
from elementtree import ElementTree
#for storing
class MyDigg:
def __init__(self, title, link, digg, diggs):
self.title = title
self.link = link
self.digg = digg
self.diggs = diggs
def __str__(self):
return self.title + ' ' + self.diggs
#stories to get diggs of
posts = [
'Google_NOT_releasing_it_s_Goobuntu_Desktop_OS_STOP_DIGGING_IT',
'New_Digg_Home_Page_breaks_the_Linux_section_on_IE',
'Google_Reader_API_Functions'
]
#hold returned info
my_diggs = []
#all calls go through this
digg_service = 'http://services.digg.com/'
#just looking at stories
service_endpoint = digg_service + 'story/%s'
#only need 1 result back
trailer = '?count=1&appkey=http%3A%2F%2Fgpowered.blogspot.com'
#keep track of total diggs
total_diggs = 0
After we are set up, we will want to loop through each story we want to get Digg data for. Add the well formed title into the query string, and send it to the Digg service. Then, parse the response, and get the information we need.
for story in posts:
curr_story = service_endpoint % story
url = curr_story + trailer
h = httplib2.Http()
resp, content = h.request(url, "GET", body="nt", headers={'content-type':'text/plain'} )
story = ElementTree.fromstring(content).findall('story')[0]
d = MyDigg(story.findall('title')[0].text, story.get('link'), story.get('href'), story.get('diggs'))
total_diggs = total_diggs + int(d.diggs)
my_diggs.append(d)
print d
print 'Total: ' + str(total_diggs)
And that’s that. my_diggs now has all the information we need!