I was wondering how hard would it be to setup Asterisk with Google Voice.

It seems that it is not hard at all :) 

Using chan_gtalk and res_jabber you will be able to utilize your Google Voice account from inside your Asterisk installation, as shown in this wiki article