By Ugorji Nwoke   16 Mar 2012   /blog   appengine geek golang technology

Streamlining Go App Engine Runtime

With App Engine, the Go Runtime is a mashup of Python Runtime, Go SDK and glue code (Go and Python). This poses some challenges during development, which this proposal addresses with solutions.

Current setup of Go Runtime

The current Go Runtime uses:

  • Python Runtime:
    1. RPC/Api server, for all RPC code which should not be handled directly by Go Code (e.g. datastore interaction, tasks, etc)
    2. Frontend Proxy: Requests come into Python Runtime, and are proxied to Go App if it is not a match for an RPC/Api request or a static file.
    3. Tools: appcfg is used for all interaction with production environment
  • Go SDK:
    1. Go Runtime is a restricted runtime (limited access to some packages like syscall, unsafe, etc, and API’s like io.write, etc). A Restricted Go SDK is bundled with the Go Runtime.
  • Glue Code:
    1. Go Runtime provides go-app-builder (command-line tool written in Go) which can build Go App on demand
    2. Python Glue Code is run on each request, and checks if any file in the app tree has changed. If so, it rebuilds and/or restarts Go App.

Potential Issues with Current Setup

The potential issues with these are:

  1. Too much bundled into Go App
    1. I currently include a symbolic link to my GOPATH in my app
    2. go-app-builder looks for any file in there which has an init(…) method, and includes that in the synthetic main.
    3. Consequently, I have all my GOPATH code in my Go App.
    4. The other option is to either copy packages I use, or selectively add symbolic links to said packages in my app directory. Both of these have issues:
      1. For copying, I always have to ensure multiple directories are in sync, which defeats version-control and ease-of-development
      2. For symlinks, it gets hard when I only want some sub-packages and not others. For example, I have gae, gae/app, gae/db, gae/counter packages. In a simple app, I only need gae and gae/app. But I can’t make gae and gae/app symbolic links (because gae has to be a directory for gae/app to be a symlink). So my only option is to include the whole gae directory with all its subpackages.
  2. Testing is a challenge:
    1. I have to use an installed Go SDK (not the bundled SDK in Go Runtime), since the installed Go SDK is very restricted and incomplete.
    2. I have to make symbolic links to appengine and appengine_internal from the bundled SDK goroot directory in my GOPATH, since I have to use an installed Go SDK.
    3. There isn’t a nice exported API in appengine_internal which allows us interact with RPC/API server via simple appengine.NewContext(…) just as Go App does.
  3. Documentation is a challenge
    1. godoc from bundled SDK in Go Runtime does not honor GOPATH appropriately, to show all my code
  4. Performance Testing
    1. This is hard because Python Runtime only allows one request at a time, causing false expectation of performance E.g. A page request which should load App Code and 10 images in parallel and take 10ms, will take like 100ms run through Python Runtime.
    2. Concurrent Testing is hard, since requests via Python runtime only go one at a time.
  5. Promote Go development
    1. App Engine is the current poster child for Go Language. It will be nice if the usage experience clearly shows Go at its best:
      1. Encourage installing a Go SDK
      2. Encourage seamlessly using shared Go Code (not ones written specifically for the app)
      3. Encourage using a single godoc instance to see all code:
        In my case, I put all my go code (even app code) as packages in my GOPATH.
      4. Clearly showcase, even in development, Go’s benefits (concurrency and performance)
      5. Show how easy it is to build generic Go Apps
  6. No Easy way to build for dev or production
    1. I have some handlers which I want to bind during development (for dev testing), but not during production
    2. I have some code which I want to include and which are used during development, but should not be in production
    3. the // +build mechanism can help here

Proposed Solution

The summary of the solution is to:

  1. Developers interact with a gaetool executable (written in go) which solves all problems above elegantly.
    1. This is hard because Python Runtime only allows one request at a time, causing false expectation of performance E.g. A page request which should load App Code and 10 images in parallel and take 10ms, will take like 100ms run through Python Runtime.
    2. Concurrent Testing is hard, since requests via Python runtime only go one at a time.
  2. Promote Go development
    1. App Engine is the current poster child for Go Language. It will be nice if the usage experience clearly shows Go at its best:
      1. Encourage installing a Go SDK
      2. Encourage seamlessly using shared Go Code (not ones written specifically for the app)
      3. Encourage using a single godoc instance to see all code:
        In my case, I put all my go code (even app code) as packages in my GOPATH.
      4. Clearly showcase, even in development, Go’s benefits (concurrency and performance)
      5. Show how easy it is to build generic Go Apps
  3. No Easy way to build for dev or production
    1. I have some handlers which I want to bind during development (for dev testing), but not during production
    2. I have some code which I want to include and which are used during development, but should not be in production
    3. the // +build mechanism can help here

Proposed Solution

The summary of the solution is to:

  1. Developers interact with a gaetool executable (written in go) which solves all problems above elegantly.
    1. This is hard because Python Runtime only allows one request at a time, causing false expectation of performance E.g. A page request which should load App Code and 10 images in parallel and take 10ms, will take like 100ms run through Python Runtime.
    2. Concurrent Testing is hard, since requests via Python runtime only go one at a time.
  2. Promote Go development
    1. App Engine is the current poster child for Go Language. It will be nice if the usage experience clearly shows Go at its best:
      1. Encourage installing a Go SDK
      2. Encourage seamlessly using shared Go Code (not ones written specifically for the app)
      3. Encourage using a single godoc instance to see all code:
        In my case, I put all my go code (even app code) as packages in my GOPATH.
      4. Clearly showcase, even in development, Go’s benefits (concurrency and performance)
      5. Show how easy it is to build generic Go Apps
  3. No Easy way to build for dev or production
    1. I have some handlers which I want to bind during development (for dev testing), but not during production
    2. I have some code which I want to include and which are used during development, but should not be in production
    3. the // +build mechanism can help here

Proposed Solution

The summary of the solution is to:

  1. Developers interact with a gaetool executable (written in go) which solves all problems above elegantly.
    1. Hide the Python api_server.py behind the scenes.
    2. Proxies requests to Python api_server, or Go App, or serves static files appropriately (with fine concurrency)
    3. Handles rebuilding/restarting Go App appropriately, vetting app for “restrictions”
    4. Handles assembling full app for uploading to the cloud.
  2. Use an installed Go SDK (as opposed to bundling a restricted Go SDK)
  3. Do not bundle any compiled artifacts in AppEngine SDK (everything in source)
    1. I know this will be controversial, but please hear me out.
    2. This also means that there will be only one Go Runtime SDK (not one per supported platform/arch combination).
  4. Use a single downloadable AppEngine SDK bundled as a zip, which includes:
    1. Generic Python Runtime (without Python demos, docs, etc)
    2. Source of Go SDK (gaetool, appengine packages, as gosrc//.go files)
    3. Demos, documentation, other complementary files
  5. Export appengine_internal.initAPI(netw, addr string). This way, testing just involves:
    1. Start appengine dev server
    2. call appengine_internal.initAPI(netw, addr string) in init() method of _test.go file.
    3. Use appengine.NewContext(…) as usual in test code.
  6. Support a build context flag for dev-time building i.e. // +build appengine appenginedev
    1. Allows us easily have separate top-level files for development and production respectively.

Tool written in Go (gaetool): developer interaction

The gaetool execution depends on the following:

  1. Build context tags:
    1. // +build appengine (controls inclusion within appengine build)
    2. // +build appenginedev (controls inclusion within appengine dev build)
  2. app.json (instead of app.yaml):
    1. Go has builtin support for json and templates, so can easily read and comprehend a app.json, and spit out a app.yaml (for use by api_server.py and appcfg.py)
    2. This is necessary because gaetool now takes over serving static files and all front-end operations (e.g. ensuring secure-only urls), but api_server.py and appcfg.py need app.yaml
    3. Limits app configuration to what is supported in Go Runtime:
      • app identity (name, version, api-runtime)
      • static files/directories (with url, mime type, cache expiration) and app resources
      • secure-only urls
      • enabled services (xmpp, email, etc)
      • admin console custom pages
      • custom out-of-app error responses (for over_quota, dos_api_denial, timeout)
  3. Go App Source in the Go App Directory/src.
    1. Go App Directory will be added to GOPATH during a gaetool build
  4. top level package main (in src/main/main.go)
    1. This allows the developer determine the initialization sequence, and what should be included in his Go App (and not all the code which is in GOPATH)
    2. AppEngine synthetic main will just call appengine_internal.Main() or whatever is applicable.

Wrappers for the gaetool exist as gaetool.sh and gaetool.bat files, and basically do the pseudocode:

if __directory_sdk_extracted_to__/gosrc/gaetool/gaetool[.exe] does not exist
-> cd __directory_sdk_extracted_to__/gosrc/gaetool
-> go build
->  cd -
__directory_sdk_extracted_to__/gosrc/gaetool/gaetool[.exe] [...]

To run dev server or update app, developer will run gaetool as below:

cd __my_app_dir__
__directory_sdk_extracted_to__/gaetool[.sh|.bat] [...]

To do other interaction with the production app engine system (like downloading logs, updating indexes, etc), developer will use appcfg.py as before:

cd __my_app_dir__
__directory_sdk_extracted_to__/appcfg.py [...]

To see full sources, and leverage app engine sources within test code, or godoc, developer will:

Add __directory_sdk_extracted_to__/gosrc to GOPATH

That’s the full interaction that the developer has with the Go Runtime SDK. All the magic is hidden behind the gaetool.

Tool written in Go (gaetool): Logic: How it works

The gaetool running as dev server does the following:

  1. Exec api_server.py, so that go app can interact with it
  2. Given the “main” package, it will do like “go list” to find out all the dependencies, and note the corresponding directories for polled tracking (in lieu of a cross-platform fsnotify functionality)
    1. This allows app sources to live outside the app directory, but within GOPATH
  3. It also walks the app directory, and keeps track of the full directory tree
  4. Every 2+ seconds, it looks to see if any directory has changed (last mod modified or deleted).
    1. If any src directory, it will:
      1. reset list of src directories
      2. delete and rebuild packages for modified directories
      3. kill running Go App
      4. rebuild Go App
      5. Restart Go App
    2. If any other app directory, it will
      1. reset list of app directories
      2. kill running Go App
      3. Restart Go App
  5. Setup http listener on the Go App
  6. Setup http listener on the gaetool, and http proxies to Go App and Api Server, and file server handler for the static files
    1. All http interaction goes through the http listen port on the gaetool, which proxies to Go App or Api Server, or passes request to its file server for static files

An app build now has some intelligence to respect the “restricted” nature of the production app engine runtime. At start of each build, app will inspect the result of “go list” and ensure that no “restricted” APIs (syscall, unsafe, io write, etc) are used. If this inspection passes, then the app is built. Else, the app build fails.

During an update, the gaetool will copy the whole app and all its dependencies into a tmp directory, and call appcfg.py update on it.

gaetool, when shutdown via ctrl-c or a process kill, will also shutdown all the processes it is managing (i.e. api_server python process, Go App process).

Tags: appengine geek golang technology

Subscribe: Technology
© Ugorji Nwoke