README.md: *custom-streaming-callback* not working?

[bzg]

While trying to reproduce the async streaming example from the README, I get an error:

#error {
 :cause JSON error (end-of-file)
 :via
 [{:type java.io.EOFException
   :message JSON error (end-of-file)
   :at [clojure.data.json$_read invokeStatic json.clj 182]}]
 :trace
 [[clojure.data.json$_read invokeStatic json.clj 182]
  [clojure.data.json$_read invoke json.clj 177]
  [clojure.data.json$read invokeStatic json.clj 272]
  [clojure.data.json$read doInvoke json.clj 228]
  ...

Replacing custom-streaming-callback with bodypart-print (from twitter.callbacks.handlers) only yields blank lines.

Is there something to update in the README.md?

Thanks for maintaining this library, it promises to be useful!

[bzg]

A quick follow-up on this: using cheshire instead of clojure.data.json (from Clojure 1.9) to parse the response seems to get around some "end of line" bug. Not sure where this should be fixed, but I'd be interested in knowing if I'm not alone.

[chbrown]

In my experience (or to my taste), http.async.client doesn't have a good story for streaming responses. Before I forked twitter-api as twttr in order to replace http.async.client with aleph, I was using a hacky jumble of lazy-seq and java.io.ByteArrayOutputStream, which was hard to work with and tedious to debug. My primary use of the Twitter API is the streaming endpoints, which aleph elegantly handles together with its manifold / byte-streams siblings, but migrating to aleph required some pretty large architectural changes — thus the hard fork (and name change).

At the moment, I haven't been able to devote as much time to twttr as I'd like, and the API is a bit of a work-in-progress (more volatile than twitter-api, certainly), but you might want to check it out: https://github.com/chbrown/twttr

If you cobble together an update to the obsolete *custom-streaming-callback* documentation and send along a PR, I'll merge it in, but for now my focus is primarily on twttr. Closing for now.

P.S. I'm guessing cheshire by default parses empty strings as nil, whereas clojure.data.json/read(-str) defaults :eof-error? to true. Set that option to false and I'd bet you get the same results.

[bzg]

P.S. I'm guessing cheshire by default parses empty strings as nil, whereas clojure.data.json/read(-str) defaults :eof-error? to true. Set that option to false and I'd bet you get the same results.

Indeed! I wasn't aware of :eof-error?.

Thanks for https://github.com/chbrown/twttr - I didn't know it, I will probably check it at some point, but for now I got something working with this small bot.

I'll see if I can suggest a useful PR, it's fine to close this issue.

[bzg]

Okay, http.async.client seems really to brittle, and I'm having a hard time debugging this.

At the moment, I haven't been able to devote as much time to twttr as I'd like, and the API is a bit of a work-in-progress (more volatile than twitter-api, certainly), but you might want to check it out: https://github.com/chbrown/twttr

So I checked twttr and it looks quite complete! Thanks a lot for it.

I just miss a few lines of guidance on how to get started with (api/statuses-filter creds ...): how can I setup a bot filtering through a list of followed users and retweeting tweets matching against a string?

Shall I use core.async to get the output of the stream?

Thanks in advance for your help!

[chbrown]

Here's an example. Doesn't require core.async, but you could incorporate it if you need a pub/sub setup for some reason.

• Be sure you're using the latest [twttr "3.0.0-beta3"]
• The doseq is important.
• The "Finished!" println will never run (barring Twitter closing up shop).

(require '[clojure.string :as str]
         '[twttr.api :as api]
         '[twttr.auth :as auth])

(defn statuses-filter-friends
  "Get the (first 5000) friends of @`screen_name`, then return a lazy
  (& infinite!) sequence of the tweets authored by those friends."
  [credentials screen_name]
  (println "Getting friends for @" screen_name)
  (let [{:keys [ids]} (api/friends-ids user :params {:screen_name screen_name})
        ids-set (set ids)]
    (println "Following" (count ids) "users:" ids)
    (->> (api/statuses-filter credentials :params {:follow (str/join "," ids)})
         ; /statuses/filter.json returns a lot more than just the 'follow' set's tweets
         (filter (fn [status] (->> status :user :id (contains? ids-set)))))))

(defn tweet-numbers
  "(Re-)tweet each status in `statuses` that contains a number,
  after anonymizing the user names."
  [credentials statuses]
  (doseq [status statuses]
    (let [text (get-in status [:extended_tweet :full_text] (:text status))]
      (if-let [number (re-find #"\b\d+\b" text)]
        (let [anonymized-text (str/replace text #"@\w+" "<user>")
              new-status (format "%s!\n%s" number anonymized-text)
              new-status-280 (subs new-status 0 (min 280 (count new-status)))]
          (println "Tweeting new status!" new-status-280)
          (api/statuses-update credentials :params {:status new-status-280}))
        (println "*sigh* No numbers in tweet:" text)))))

(defn tweet-friends-numbers
  [credentials screen_name]
  (->> (statuses-filter-friends credentials screen_name)
       (tweet-numbers credentials))
  (println "Finished! (LOL yeah right)"))

(def user (auth/env->UserCredentials)) ; or whatever
; kick it all off for some account with a reasonable number of friends
(tweet-friends-numbers user "ACLU")

[bzg]

Hi Christopher,

thank you very much! That helped a lot. I've now switched to using twttr and it works nicely so far - this is a toy bot for now, no problem if it breaks from time to time.

Thanks again for the help, I do really appreciate it.