Being a good Twitter Spammer

First of all let’s get the big announcement out of the way…

Days till oauthpocalypse

August 16th 2010 is oauthpocalypse. Please note: the powers that be discovered you can’t have an apocalypse during the world cup because apparently, no one notices (the old date is 30th of June). Also, for those panicking, unlike the original plan, the guys at Twitter have decided we can’t go cold turkey so they will start to reduce the rate limit over a few days. All non Oauth/Xauth apps will die after August 31, unless something else catches our attention.

World Cup Killing Twitter?

Who would have thought kicking a cows bladder around could cause so much trouble? It would seem the World Cup is mostly to blame for Twitter’s current fail whale issues. In response, today’s second important public service announcement is: For the time being, applications will be subjected to variable rate limits!

What Does This Matter?
Well, normally an Oauth’d user can make 350 calls/hour, but during the high load time (so I’m guessing between 3pm – 7pm BST) this limit maybe dropped, indeed as this post launches the current rate limit is 175 for all.

Now, this is an emergency measure, and for it to have any real effect, Twitter clients have to start adapting and dynamically reducing or increasing their calls.

Three ways to tell if your hitting the rate limit:

• 400 HTTP Status code – means you hit your limit
• X-ratelimit- Headers sent in return to all rate limited API calls
• account/rate_limit_status API Call

Now, your average PHP developer making CURL requests will start using number 3. I know. It was my first thought because you would save having to parse through headers. However, there is an obvious flaw with this of course of action.

The whole point of this exercise is to reduce the number of calls against Twitter. One developer I was speaking to yesterday used a rate_limit API call after every action. In effect, he was definitely not reducing the number of calls. He wa doubling them!

Developing Sensible Twitter Spamming Patterns

If you’re like me and run intensive Twitter clients that send out hundreds of tweets an hour and monitor lots of accounts etc, you are going to get caught, if you’re not careful. Repeatedly sending tweets to 400 HTTP will result in blacklisting, so this clearly is not a solution. Now, I’m assuming you are using php/CURL for the rest of this article as (no offense guys) you are most likely the people causing problems.

First Of All Stop Dead the chance of blacklisting
Really, you want to completely stop your twitter client’s activities the moment you receive a 400. So, after the

curl_exec($ch);


and before we look at the response message, we need to know if everything is ok. This is particularly important if your application is threaded or on a 1min cron job etc.

$status_code = curl_getinfo($ch, CURLINFO_HTTP_CODE);

if($status_code != ’200′){
//some thing has gone utterly wrong
die();
}

Now, obviously the above just kills the current process. You will want to switch off the tweets. I use a simple on/off switch for my systems that allow me to kill twitter apps from posting dead. Then, I use other techniques on top. This is normally a tiny int stored in a db or memcached. It is fine except for an obvious problem…

When Do You Restart Your Spamming?

At this point, you would normally interogate the header for the rate limit information, except doh! The 400 doesn’t have any! However, the good news is the account/rate_limit_status calls is still available to you. Make sure you re pass your Oauth credentials when calling, otherwise you will get the IP rate limit back. This call, you will be pleased to know, won’t count against your rate limit.

The two reset values, reset_time and reset_time_in_seconds, will give you the time to resume your spamming. Add at least 30 seconds to that before you start.

I would suggest using reset_time and storing it against your dead switch, or indeed as your dead switch. So far, it’s fairly simple, but this is very much reactive management, not proactive. If you are sending out large amounts of data, you will want to have a high delivery rate, and the only way to do that is traffic shaping.

Managing the Twitter API Yourself

Assuming you are on the edge of the API limit, you are going to have a clue as to how much data you will need to pass in a given hour because an application (IP address) is limited to 20,000 calls. While it hasn’t been ruled out, this will be reduced. The current rate limit adjustments seem to be aimed at clients, who are limited to a maximum of 350/hour.

This means an IP can have 57 clients using the maximum limit per hour (unless it’s white listed).

Now, if your clients are receiving and not posting data, there is a very simple way around the rate limit: increase the number of clients. As one client gets close to its limit, it hands over to the next client, and so on. I think I can hear someone screaming at me now, and of course this is not a nice option for Twitter, but an option none the less.

If this is not really an option, you will end up having to limit your posting schedule. Most of my applications target a UK audience, so it “posts” most often between 7am and 6pm GMT.

I’m also taking several additional steps with high posting accounts. Because I want people to see information, I am currently avoiding the dreaded Whale Zone from around 4-7pm BST, I also try to get most of the posts out prior to it. This means the account is putting a higher load on Twitter during early afternoon, but is vastly reducing that load during Twitter’s peak and during the night.

You will notice a peak around 10pm. This is due to pushing off last high priority tweets.

Now, ahead of traffic shaping, I have been working on equalizing tweets, basically this means more constant tweeting, with “low” value tweets being fired off at night ,and high value tweets during the day.

Dealing With World Cup Matches

Normally, the above is fine, but what about world cup matches? Initially, I was more worried about the run up and post games (both show high spikes), but a match itself causes a fair amount of traffic. Therefore, I reduce my tweets during this time. Gareth Poole has provided World Cup data on fixtures and matches as sql and json. If you are taking matches into account, I suggest you take 15 minutes before kick off and 30 after as reduced times.

What if you do all this

It can still go utterly wrong as demonstrated above. However, you will just have to live with it, (That page scrolled through a lot of whales oops )

While it is a lot of work, the above will save you in the long run. With Twitter having high loads, its error reporting will get flaky. One issue I have been having recently is lots of duplicate content error messages caused by posting at times when the API is on edge of breaking. The best way to avoid such things is to simply not post at these times, which is why I posted this post at oh 16.31 BST sorry Twitter.

Are you using Twitter on the edge of the API limits? What is your experience with doing your own API shaping?