Siscia Tech

Programming, tech related blog

0 notes &

Tumblr really sucks to show source code…

I want to move this blog as soon as possible, if you have any suggestion, I would be very glad to hear it….

In the mean time, please be patience…

0 notes &

Parallel Colt first basic benchmark in clojure.

Inspired by the article of Mike Anderson I decide to benchmark also my implementation of clojure.matrix.core: parallel-colt-matrix that is using Parallel Colt under the hood.

I believe to have nice result, especially for very big matrix, and I really mean big.

Before to show the result I want to remember you that Parallel Colt (PC) run  in parallel for every computation, and it is not always good, especially for little data set:

user=> (time (pmap inc [1 2 3]))
"Elapsed time: 10.830703 msecs"
(2 3 4)
user=> (time (map inc [1 2 3]))
"Elapsed time: 0.036178 msecs"
(2 3 4)
Plus to achieve good performance, of course, I needed to type hit everything.
Lets so take a very quick look at the code:
(matrix-add ^DenseDoubleMatrix2D [m ^DenseDoubleMatrix2D a]
   (let [sum (. DoubleFunctions plus)
          other (.copy ^DenseDoubleMatrix2D m)]
      (.assign ^DenseDoubleMatrix2D other ^DenseDoubleMatrix2D a sum)))
matrix-add need to return a new matrix so I needed to copy the old one, of course it is slowing down everything a lot.
(matrix-add! [m a]
    (let [sum (. DoubleFunctions plus)]
      (.assign ^DenseDoubleMatrix2D m ^DenseDoubleMatrix2D a sum)))
matrix-add! on the other side just modify the old matrix, so everything is going faster, about twice as fast.
Let’s now take a look at the result, add together two matrix with 1 million of element each for one thousand time take about:
user> (let [a (get-matrix (vec (partition 10 (take 1e6 (repeatedly #(Math/random)))))) 
 b (get-matrix (vec (partition 10 (take 1e6 (repeatedly #(Math/random))))))]
 (quick-bench (dotimes [i 1e3] (matrix-add a b))))
Execution time mean : 6.269815 sec
Same computation, but modifing the original matrix instead takes less than half of the time:
user> (let [a (get-matrix (vec (partition 10 (take 1e6 (repeatedly #(Math/random))))))
                 b (get-matrix (vec (partition 10 (take 1e6 (repeatedly #(Math/random))))))]
                       (quick-bench (dotimes [i 1e3] (matrix-add! a b))))

Execution time mean : 2.237656 sec
These are really big computation, and I don’t know how usefull they can be, something more usefull could be a little smaller matrix, like a 10000 element matrix that runs in:
user> (let [a (get-matrix (vec (partition 10 (take 1e4 (repeatedly #(Math/random))))))
                 b (get-matrix (vec (partition 10 (take 1e4 (repeatedly #(Math/random))))))]
                       (quick-bench (dotimes [i 1e3] (matrix-add a b))))

Execution time mean : 48.680351 ms
or again in about the half of the time if we want to use mutability:

user> (let [a (get-matrix (vec (partition 10 (take 1e4 (repeatedly #(Math/random))))))
                 b (get-matrix (vec (partition 10 (take 1e4 (repeatedly #(Math/random))))))]
                       (quick-bench (dotimes [i 1e3] (matrix-add! a b))))

Execution time mean : 26.798289 ms
How good are those result ? Well, I believe they are pretty good, even because I don’t know how to go any further, in comparision a simil computation but using map and normal sequence takes a lot of more time, about a million times more…

(quick-bench (dotimes [_ 1e3] (doall (map + (take 1e4 (repeatedly #(Math/random)))
                                                                             (take 1e4 (repeatedly #(Math/random)))))))

Execution time mean : 4.955435 sec
But to be fair i should use parallel computation, so:
(quick-bench (dotimes [_ 1e3] (doall (pmap + (take 1e4 (repeatedly #(Math/random)))
                                                                        (take 1e4 (repeatedly #(Math/random)))))))
Execution time mean : 39.231347 sec
(That is even worse… and very badly, I should find out why…)

Everything is on a i7 with fedora18.

Anyway, this was just a little article about the state of parallel-colt-matrix.

Here the benchmark: of PC without type hitwith type hit and of the sequence

Write this implemetation is very instructive, and if you are looking for some project to follow it may be a very good idea implement some other java libraries into the clojure.core.matrix ecosystem.

A very last thing, I am actually for hire.

Filed under clojure parallel colt java matrix hire for hire type hit performance

0 notes &

Login with Twitter clojure/java, this is a little tricky…

Last time I implemented the login with facebook and I was guessing that login with twitter was about the same, just change a couple of url, some secret key and some appid and I thought I would have been set.

Well I was wrong.

The basic idea is about the same, but it is implemented in a different way, facebook is way more straight and I would say easy or at least less complex, but it is just my idea.

Before to show the code I need to do a full disclaimer, I am not sure I am doing it right, it works but there might be better way to do the same, if so just let me know.

Just going straight to the code (how can I show gits on tumblr ???)

(ns morgan.models.social
  (:require [clj-http.client :as client])
  (:import [org.scribe.oauth OAuthService]
           [org.scribe.builder ServiceBuilder]
           [org.scribe.model Token Verifier OAuthRequest Verb])
  (:import [org.scribe.builder.api FacebookApi TwitterApi$Authenticate]))


(def twitter-service
  (-> (doto
          (ServiceBuilder.)
        (.provider (TwitterApi$Authenticate.))
        (.apiKey "Pqc8RQMeJwVKpv0vPqPg")
        (.apiSecret "4UvfkrZbmT4wmwhKrddsqZqIwPQRQ7X4akQW2zCLM")
        (.callback "https://morgan-siscia.rhcloud.com/tw"))
      (.build)))

(def tokens (atom {}))

(defn get-token []
  (.getRequestToken twitter-service))

(defn get-twitter-url [token]
  (swap! tokens assoc (.getToken token) token)
  (.getAuthorizationUrl twitter-service token))

(defn get-access-token-twitter [oauth-token oauth-veri]
  (let [token (get @tokens oauth-token)
        verifier (Verifier. oauth-veri)
        access-token (.getAccessToken twitter-service token verifier)]
    (do
      (swap! tokens dissoc oauth-token))
    access-token))

(defn get-info-twitter [access-token user-id]
  (let [request (doto
                    (OAuthRequest. (Verb/GET) "https://api.twitter.com/1.1/users/show.json")
                  (.addQuerystringParameter "user_id" user-id))]
    (do
      (.signRequest twitter-service access-token request))
    (.getBody (.send request))))

and here the views:

(ns morgan.views.social
  (:use [noir.core :only [defpage]]
        [noir.response :only [redirect]]
        [morgan.models.social :only [get-access-token-twitter
                                     get-info-twitter
                                     get-twitter-url
                                     get-token]]))

(defpage "/tw" {:keys [oauth_token oauth_verifier]}
  (let [access-token (get-access-token-twitter oauth_token oauth_verifier)
        user-id (:user_id ((fn [req] ;;homemade implementation of
                             ;;URLDecode by amaloy
                             ;;http://stackoverflow.com/a/6591708/869271
                             (into {}
                                   (for [[_ k v]
                                         (re-seq #"([^&=]+)=([^&]+)" req)]
                                     [(keyword k) v]))
                             ) (.getRawResponse access-token)))]
    (get-info-twitter access-token user-id)))

(defpage "/twitter" []
  (redirect (get-twitter-url (get-token))))

The interface it’s the same of last time, a page “/twitter” (that tomorrow can be behind a button, a link, or whatever) that redirect to a page that twitter give to me, twitter then redirect the user back on my site “/tw” where I show what info I get when you give me the access.

Let’s analyze the model.

The first thing to notice would be on the line

(.provider (TwitterApi$Authenticate.))

TwitterApi$Authenticate is a inner class defined inside the TwitterApi class, you can use and import it like so.

In few line we can find the definition of a atom, 

(def tokens (atom {}))

I hate use mutable in clojure but I didn’t find a better solution.

Twitter need the token to both generate the url where redirect the user so the user can authorize your app and to get the access-token (the token that give you the access to the user information). The token is defined by two string, a “public” one that you pass by GET more times and a secret one.

If a define a single immutable token then I can use it only with one user all the other request will fail, if I re-define the token every single time that I finish to use one the are some issue, take the example of a user who is already signed to my service and a very new user, the new user will take way more time of the old one to login, so while the new user is looking at the access I am getting the new user will generate a new token (destroying the old user one) and use it, as result the new user will not be able to sign in and he will need to try again. (I hope it is clear, if it  is not just let me know)

Every time that I generate a new token I store it in the atom, I am using a map, so I use the string representation of the token as key and the object token like value.

(defn get-twitter-url [token]
  (swap! tokens assoc (.getToken token) token) ;;HERE
  (.getAuthorizationUrl twitter-service token))

And when I finish to work with the token I remove it from the map

    (do
      (swap! tokens dissoc oauth-token)) ;;HERE
    access-token))

With the user authentication I can get some more info about the user itself with the function 

get-info-twitter

That is pretty easy to understand.

To note is the page “/tw” in the view, it is where twitter redirect the user after the authentication, the page get two parameter, the verifier and the token, the same token that I am using like key to get the token object.

In the same response where I get the verifier and the token twitter send me also very little info about the user such user_id and screen_name, since I need at least one of this to get some info about the user I parse the response and the I ask twitter the basic info that I show you in the page.

Please note that the twitter app has permission “Read and Write”, asking also the direct message permission will make it a little more difficult.

You can try it out here:

https://morgan-siscia.rhcloud.com/twitter

Please note that if you visit the page twice after that you authenticate the first time it SHOULD NOT ask the permission again.

The code I am running behind the page is the same you are looking at, be aware that it might change in the future.

I hope that my english is fine, if you don’t get something just let me know in a comment.

Here the gits: https://gist.github.com/4218318

@siscia_

Filed under clojure scribe login with twitter twitter oauth auth login with twitter clojure

1 note &

Login with Facebook clojure/java

I spend a couple of days trying to implement OAuth2 (aka Login with -{facebook, twitter, google, & more}) in clojure when I finally gave up.

There are several libraries out there but honestly I don’t like any of those, or they don’t work.

Even worse I didn’t find any patter to follow, even if I didn’t look very careful and this is why I am writing this micro-post.

First of all I am gonna use a java library, scribe it’s looks like it is widely used in the java community but I didn’t know its existence before, its look like an amazing library for OAuth and I didn’t spend to much time experimenting at the REPL thanks to the amazing documentation and examples.

Ok, let’s go into something more interesting, how “Login with Facebook” works ?

First of all you need to make an app on facebook, it’s not difficult, remember to put the real domain when they ask for, the App ID and the App Secret are very important but they won’t go away so…

Then it’s pretty simple, your user click the button and you redirect him to an URL that FB gave to you which lead to the FB page where the user can login in facebook and authorized the app, when everything is done, so the user has authorized your app, facebook re-direct the user to your domain in a specific URL passing via GET a huge string, now you exchange that string with a token and finally the token with the info that you need, stuff like name, email, where the user live and more — a lot more even if I asked only the basic information.

Ok, I am boring… Let’s go straight to the code:

The model

(ns morgan.models.social

  (:require [clj-http.client :as client])

  (:import [org.scribe.oauth OAuthService]

           [org.scribe.builder ServiceBuilder]

           [org.scribe.model Token Verifier OAuthRequest Verb])

  (:import [org.scribe.builder.api FacebookApi TwitterApi]))

(def facebook-service

  (-> (doto

          (ServiceBuilder.)

        (.provider (FacebookApi.))

        (.apiKey “your api key”)

        (.apiSecret “your api secret”)

        (.callback “URL where FB is gonna re-direct your user”))

      (.build))) ;;It’s necessary to call .build outside the doto,

;;otherwise we would get a ServiceBuilder and not a OAuthService…

(def facebook-url

  (.getAuthorizationUrl facebook-service nil))

(defn get-info-facebook [code]

  (let [verifier (Verifier. code)

        token (.getAccessToken facebook-service nil verifier)

        request (OAuthRequest. (Verb/GET) “https://graph.facebook.com/me”)]

    (do

      (.signRequest facebook-service token request))

    (.getBody (.send request))))

and the view:
(ns morgan.views.social
  (:use [noir.core :only [defpage]]
        [noir.response :only [redirect]]
        [morgan.models.social]))
(defpage “/facebook” []
   (redirect facebook-url))
(defpage “/fb” {:keys [code]}
   (get-info-facebook code))
That’s it, nothing hard, isn’t it ?
In the model: 
The var called facebook-service is the core, it is an object that take care of everything, you just need to pass it what service are you gonna use, your App ID, your App Secret, and the url when the user is going to be redirect.
facebook-url is the url that the user need to follow in order to login into facebook and to authorize the app, should be behind a “LogIn with Facebook” button.
The last function let you get the info from the user’s facebook account, the function return a string of the json that facebook send, you may want to parse it with chesmire or something similar and then store the information.
In the view:
The page /facebook is pretty pointless, I was simply to lazy to add a button, it, anyway, redirect the user to the facebook-url page where the user can authenticate and authorize your app.
After that everything went smoothly the user is redirect by facebook in the /fb page with the code via GET (where the user is redirect depends on what you said in the .callback when you are creating the facebook-service object); the page get the code, call the proper function and show back what I got from facebook about the user, pretty straight.
You can check everything here https://morgan-siscia.rhcloud.com/facebook 
Please be aware: 
right now I am NOT saving any of your information.
Hope you enjoy the reading
Simone Mosciatti
PS: Since you are please check this out https://morgan-siscia.rhcloud.com and let me know what you think.

Filed under clojure facebook sign in with login login with facebook

2 notes &

Webnoir on Openshift

After one whole day spent trying to figure out how make a webnoir application work in openshift  I decide that I should share what I have learned.

The process is not very hard. I am assuming that you already have a openshift account— if not is not really hard to register, Although I won’t use it in the tutorial, it could be useful have rhc (the openshift client) installed; you will definitely need git.

The easiest way is to start creating the application on openshift— we need to create a Do-It-Yourself (DIY) application.

I call my application examplenoir.

Now we download the code that we will need, using git.

After the creation of our application openshift will give us the URL of the git repository. If you forget to save it, don’t worry, you will get the same URL in the application console.

Now we will just clone the repository in our computer.

It will ask you to add the rsa key to the know keys, just type yes and keep going.

It is now time to build our application, so let’s move in the diy directory and just launch lein, please note that I used the noir script for lein.

Now that your application is ready, we can push it on the cloud (or you can wait the next step)

Visiting now your webpage, you will probably see something like that:

This seems to be meaningless, but it gives us important clues, it say to modify the .openshift/action_hooks/{start, stop} 

Before we do that we need to understand what those file are; they are simply bash script that openshift will execute every time that you start or stop your script, we will use them to run the webserver through lein or to kill every clojure process.

Since we will need lein, before of everything we can “install” it, just make another directory called bin and download inside the lein script and make it executable.

Now we can think about the start script, my looks like that:

#!/bin/bash
# The logic to start up your application should be put in this
# script. The application will work only if it binds to
# $OPENSHIFT_INTERNAL_IP:8080
# save as .openshift/action_hooks/start

export HTTP_CLIENT="wget --no-check-certificate -O"
export PORT=$OPENSHIFT_INTERNAL_PORT
export HOST=$OPENSHIFT_INTERNAL_IP
export HOME=$OPENSHIFT_DATA_DIR/home
export LEIN_JVM_OPTS=-Duser.home=$HOME

cd $OPENSHIFT_REPO_DIR/diy/examplenoir

$OPENSHIFT_REPO_DIR/bin/lein deps

$OPENSHIFT_REPO_DIR/bin/lein run >${OPENSHIFT_LOG_DIR}/lein.log 2>&1 &
disown

And this other is the stop script:

#!/bin/bash

# The logic to stop your application should be put in this script.

# save as .openshift/action_hooks/stop

kill `ps -ef | grep clojure | grep -v “grep clojure” | awk ‘{ print $2 }’` >${OPENSHIFT_LOG_DIR}/stop.log 2>&1

exit 0

(Thanks to Philipp Meier)

Now it is time to push all back to Openshift.

It should print out a lot of stuff, (it is lein that is downloading the deps)

Now everything should work just fine, it is time to visit our URL, and…

It is not working.

Ok keep calm, and let’s try to figure out what went wrong.

You can log in your application via ssh. On the Openshift page there is a link “WANT TO LOG IN TO YOUR APPLICATION?” it should expand to show an ssh command; copy it and paste it to your terminal and log in.

Now that we are in we could look for the log of lein. So just move in the right directory >examplenoir(YOUR_NAME_APP)>logs and take a look at the lein.log (cat lein.log)

The problem is that since Openshift is a PaaS you cannot use whatever ip address or port you want— you need to use your allocated address and port. We can just adjust noir/jetty and solve the problem.

It is prettty simple, just modified your server.clj in something like that:

(ns YOUR_NAME_APP.server

  (:require [noir.server :as server]))

(server/load-views “src/YOUR_NAME_APP/views/”)

(defn -main [& m]

  (let [mode (keyword (or (first m) :dev))

        port (Integer. (get (System/getenv) “OPENSHIFT_INTERNAL_PORT” “8080”))

host (get (System/getenv) “OPENSHIFT_INTERNAL_IP”)]

    (server/start port {:mode mode

                        :ns ‘YOUR_NAME_APP

                        :jetty-options {:host host}})))

Now push it again, and wait— it will take a little while…

In the meantime you can log in in the ssh and keep looking at the logs, to see if it is finished.

when you see that means that it is running.

http://examplenoir-siscia.rhcloud.com/

Now just keep coding and make your app ;-)

Here the repository on github

https://github.com/siscia/examplonoir

Filed under DIY PaaS clojure diy clojure openshift noir openshift redhat web web developing webnoir