Accessing Google Cloud Storage with Go and AppEngine

I spend the last night figuring out how to access Google Cloud Storage (GCS) with Go. This short howto is explains how to create a GCS bucket (for free, no credit cards required) and how to access this GCS bucket from a Go program, an AppEngine instance and the AppEng dev server.

Creating a bucket

To store data, you need a bucket. It is possible to create one for free, using https://console.cloud.google.com/storage/ but it will ask to turn on billing.

Another option is to create a free AppEngine project which comes with a 5GB bucket for free and does not require any billing to be turned on. Whether you want to use AppEngine or not does not matter. Simply go to https://console.cloud.google.com/appengine and create a new project (the drop down button on the top right corner). When asked for a name type "myproject". AppEngine will come back to you with an app-id somewhat like "myproject-1234". Now you should be able to select "Settings" on the left side of the screen. There is a blue button that will finally create a new GCS bucket. The name of the bucket is "myproject-1234.appspot.com".

Now you can go to https://console.cloud.google.com/storage/ and browse the bucket. So far it is empty, but you can already upload and download files manually this way.

Installing the Client Libraries

You need to install the Go client libraries to work with GCS and appengine.

go get -u golang.org/x/oauth2
go get -u google.golang.org/cloud/storage
go get -u google.golang.org/appengine/...

Installing the SDK

Install the SDK from here: https://cloud.google.com/sdk/ and follow the installation instructions on the web page. The SDK is required if you want to work with GCS from your computer. It is not required to use GCS from AppEngine in a production environment. However, it is required to use GCS from the AppEngine dev server. Now initialize the SDK:

gcloud init

This will prompt you to grant access to your google account. You should select "myproject-1234" (or whatever the name of your app is) as the active project.

Accessing GCS on AppEngine

Since the bucket is linked to your AppEngine project, AppEngine knows the name of the bucket. You can programmatically retrieve it like this

import (
        "golang.org/x/net/context"
"google.golang.org/cloud/storage"
        "google.golang.org/appengine"
        "google.golang.org/appengine/file"
)

...
ctx := appengine.NewContext(r)
bucketname, err := file.DefaultBucketName(ctx)
client, err := storage.NewClient(ctx)

Now you can use the client to work on the bucket and its objects. See https://godoc.org/google.golang.org/cloud/storage for more information.

However, this will only work if you deploy the app to AppEngine. It will not work with the dev server. The reasons seems to be that the dev server has neither a GCS locally nor the right credentials to access the GCS

Accessing GCS on AppEngine Dev Server

AppEngine's dev server assigns your app a different default bucket. So we have to convince it to choose our already existing bucket. Start your application like this

dev_appserver.py --default_gcs_bucket_name "myproject-1234.appspot.com" myapp/

The important part is not to use goapp. Instead launch the dev server as shown above as this allows us to pass along the right bucket name.

Accessing GCS this way is slow, but it works for dev stuff somehow. Please note that this solution uses the online GCS. The dev server does not come with a GCS emulator (not yet).

Accessing GCS from a Go Application

You need to grant your program access to your GCS data. See https://developers.google.com/identity/protocols/application-default-credentials for more details. There are several options. The most convenient one for development is to use the following command:

gcloud beta auth application-default login

Now each process running within your account on your local machine will have the credentials to access your GCS data. Accessing the bucket is now as easy as:

client, err := s.NewClient(context.Background())
bucket := client.Bucket("myproject-1234.appspot.com")