Like many people in the NLP community, I use large files that I often need to move around computers (at home, at the office, etc.) I normally rely on cloud-based storage services such as Dropbox and SkyDrive to keep my files accessible across all computers, which is OK even if a bit messy. However, I recently learnt about SMEStorage, a service that integrates all your cloud-based storage services and presents them to you as one single cloud-based virtual drive. One of their killer apps allows you to mount your SMEStorage virtual drive in Ubuntu! This is great because it makes SkyDrive (which is normally only integrated in Windows and Mac OS) available in Ubuntu just as if it was a USB hard drive that you plug in.
The real power of SMEStorage comes when SkyDrive, Dropbox and other services that you use can all be listed as sub-folders in your mounted virtual drive! And of course, you can do this in all your comptuers!!! Note: in the free version of SMEStorage you can add up to three cloud services, in addition to a 5 GB storage service that SMEStorage gives you for free. Paid services get more stuff. But as it is, I think the free service is good for me for now.
These two blog posts should get you started in mounting SMEStorage in Ubuntu: part 1 and part 2.
Now, I really hit a problem when I tried to mount my cloud-based SMEStorage drive from my desktop computer at the office in the University, as we have a proxy server. Unfortunately, the SMEStorage Linux client tool is NOT designed to be run behind a proxy server. It doesn’t have options to specify a proxy server and it doesn’t read your http_proxy envorionment variable. I searched online but couldn’t find anything to make it work with the proxy server. Soon, however, I realised that the smestorage command line tool (the program used to mount the cloud-based SMEStorage drive as a local drive) was just a Perl program. I inspected the code and discovered that with just a tweak I could make it read my http_proxy server. After I implemented the tweak, I managed to successfully mount the cloud-based SMEStorage drive across the network, using my Univeristy’s proxy server correctly.
I now describe this tweak.
As a prerequisite, you have to get an account with SMEStorage and install the Linux Cloud Desktop Tools (Linux Cloud Drive) from them: http://eu.smestorage.com/?p=static&page=LinuxDrive
Now, edit the program smestorage that was installed. You can find its location in your computer this way (in a terminal):
$ which smestorage /usr/local/bin/smestorage
In my computer’s case it was installed in that path. To edit the file:
$ sudo gedit /usr/local/bin/smestorage
Within gedit (or your favourite text editor) look for sub get_page. This the subroutine that makes all Internet connections. It should look like this:
# load HTML page
sub get_page {
my $path=shift;
my $method=shift;
$method="GET" unless defined $method;
my $browser = LWP::UserAgent->new();
$browser->agent('FuseSMEStorage '.$VERSION);
my $response;
if( $response = $browser->get($path) ){
$response = $response->content;
}
return $response;
}
If you know programming, you’ll see that this subroutine is opening a web session but it doesn’t specify a proxy. In order to make the code use the proxy server specified in your http_proxy environment variable (which should be set in the /etc/environment file) add “$browser->env_proxy();” below the line “my $browser = LWP::UserAgent->new();”
Your subroutine should look like this (added line shown in bold):
# load HTML page
sub get_page {
my $path=shift;
my $method=shift;
$method="GET" unless defined $method;
my $browser = LWP::UserAgent->new();
$browser->env_proxy();
$browser->agent('FuseSMEStorage '.$VERSION);
my $response;
if( $response = $browser->get($path) ){
$response = $response->content;
}
return $response;
}
<BEGIN EDIT>
I posted too quickly. After posting this quick tweak I noticed that it would allow you to mount the cloud drive and read files from it alright, but it wouldn’t let you write files to the cloud! So, please also perform the following edits in order to be able to write files from Ubuntu back to the cloud drive:
a) There is another line that says my $browser = LWP::UserAgent->new(); inside the subroutine smestorage_save_file_to_local. Make sure you add this line after that line:
$browser->env_proxy();
b) The program makes a few Internet requests via IO::Socket::INET, which doesn’t seem to respect Proxy settings. I manage to fix the program so it can make the connection using this module via the proxy server by using the HTTP CONNECT method. Unfortunately, not all proxy servers allow this method, but it should work well in most situations. Also, I haven’t implemented any authentication, so if this is a requirement you’ll have to implement it (shouldn’t be too difficult).
You need to add a new subroutine to the program. You can add it pretty much anywhere in the program. I added it before the # run fuse ### section. Here’s the subroutine you need to add:
sub EstablishConnection
{
my $dest_server = shift;
my $dest_port = shift;
my $socket = "";
my $proxy_url = $ENV{'http_proxy'};
if (!$proxy_url)
{
$proxy_url = $ENV{'https_proxy'};
}
if (!$proxy_url) # if no proxy was set in environment variables, open direct connection (just like in original code)
{
$socket = new IO::Socket::INET( #open socket to the server
PeerAddr => $dest_server, #this is development server
PeerPort => $dest_port, #standart http port
Proto => 'tcp');
}
else # establish connection via proxy using HTTP CONNECT method
{
$proxy_url =~ s/^https?\:\/\///;
$proxy_url =~ s/\/$//;
my ($proxy_server, $proxy_port) = split(/:/, $proxy_url);
$socket = new IO::Socket::INET(
PeerAddr => $proxy_server,
PeerPort => $proxy_port,
Proto => 'tcp');
print $socket "CONNECT $dest_server:$dest_port HTTP/1.1\r\n";
print $socket "Host: $dest_server:$dest_port\r\n";
print $socket "User-agent: SMEStorage as fixed by Alfredo\r\n";
print $socket "Proxy-Connection: Keep-Alive\r\n";
print $socket "Pragma: no-cache\r\n";
print $socket "\r\n";
my $rrr= ""; my $acum="";
do
{
$rrr = <$socket>;
chomp($rrr);
$rrr =~ s/\r//g;
$acum .= $rrr;
} until (!$rrr);
if(index($acum, ' 200 ')==-1)
{
if( $DEBUG){ print "Cannot establish connection via proxy. Message returned: $acum\n"; }
return "";
}
}
return $socket;
}
Now, you need to change how connections are opened in three places:
b.1) Under subroutine smestorage_save_file_to_local, change:
$socket = new IO::Socket::INET( #open socket to the server PeerAddr => $server_sme, PeerPort => $port, Proto => 'tcp');
to:
$socket = EstablishConnection($server_sme, $port);
b.2) Under subroutine quickUpload, change:
my $socket = new IO::Socket::INET( #open socket to the server PeerAddr => $server_sme, #this is development server PeerPort => 80, #standart http port Proto => 'tcp');
to:
my $socket = EstablishConnection($server_sme, 80);
b.3) Under subroutine Post1, change:
my $socket = new IO::Socket::INET( #open socket to the server PeerAddr => $server_sme, #this is development server PeerPort => 80, #standart http port Proto => 'tcp');
to:
my $socket = EstablishConnection($server_sme, 80);
This should do the trick. I realise these changes are a bit too much and it would probably be much easier just to post a fixed version of the file. However, I’m not sure if this particular program is open source. I’ll check, and if it is, I’ll post the fixed file.
</END EDIT>
And that’s it! This should do the trick.
Now, in order to actually mount the cloud drive in your computer, follow these steps (adapted from this blog post):
First: Enable your own user account to be able to mount the drive:
$ sudo modprobe fuse $ sudo usermod -a -G fuse <username>
Where <username> is your Ubuntu login username, e.g. jbloggs
$ sudo gedit /etc/fuse.conf
Uncomment the last line: #user_allow_other
(i.e. remove the #)
I also found a good idea to take ownership of the /etc/fuse.conf file:
$ sudo chown <username> /etc/fuse.conf
Now, create the folder where you want your SMEStorage drive to be mounted. In my case, I chose /media/sme, as that way it appears mounted alognside my DVD drive, USB drives and other samba drives that I connect to:
$ sudo mkdir /media/sme $ sudo chown <username> /media/sme
And now, we can perform the actual mount operation:
$ smestorage /media/sme '<smeusername>:<smepassword>' --server=eu.smestorage.com --ao --DEBUG
Where <smeusername> and <smepassword> is the username and password for your SMEStorage account. I use the European SMEStorage server. If you use the US-based SMEStorage server, specify –server=smestorage.com instead.
I find the –DEBUG flag useful as if there are any errors, it will tell you what the error was, instead of just failing silently.
If everything worked OK, you should be able to see your SMEStorage cloud folders in /media/sme
And that’s all! You can also add this command call to your Startup Applications so that the drive gets automatically mounted every time you log in Ubuntu (see this blog post for details).
Enjoy your life in the cloud!
