When this happens it is because we are getting HTML back from Twitter due to them blocking us. They had told us they whitelisted our IP addresses so we'll follow up with them and see if there is an issue.
the problem is still there, any progress on the subject ?
There has been a lot of progress but its not terribly encouraging:
We asked Twitter about the issue, that they are returning a fairly weird looking response now and again:
HTTP/1.1 200 OK Date: Wed, 11 Feb 2009 19:02:25 GMT Server: --- Content-Type: text/html; charset=UTF-8 Cache-Control: max-age=300 Expires: Wed, 11 Feb 2009 19:07:25 GMT Content-Length: 122 Vary: Accept-Encoding X-Varnish: 1848104217 Age: 0 X-Cache-Svr: searchweb001.twitter.com X-Cache: MISS Set-Cookie: _search_twitter_sess=BAh7...; path=/
Status: 500 Internal Server Error Content-Type: text/html
You can see they appear to be having an internal server error to our XML request, which they are returning inside an HTML response as a 200 OK status. Not pretty.
The good news was that they told us they would get the issue fixed.... and things seemed to work again - yay. .... And then it happened again. We contacted them again, and they said they'd get it fixed.... yay!... and then it happened again...
We have since contacted them again and they have said they will get it fixed... but you can see the pattern.
In the meantime, depending on your needs, gnip provides a nice way of monitoring twitter, e.g.:
select * from gnip.activity where publisher='twitter'