Dev Tool for GO AppEngine

Development Tool for go app engine development, that presents an easier to use wrapper for App Engine development with GO Runtime, bypassing some pitfalls caused by integration with the Python SDK.

The source is available online, and the motivation for building this is described below. What irks one person may not irk the other, so your utility of this tool may differ from mine. For me, the utility is really high:

• Requests are much faster
• I get a better appreciation of how fast my GO App is
• I can see how many concurrent requests my app can handle
• I can now do load tests which really stress out my GO App, giving me finer results and seeing how it handles resources

Get Source

Source code for the tool is available at: http://code.google.com/p/go-gae-dev/

To clone, build and test it out:

mkdir -p /tmp/go-gae-dev
cd /tmp/go-gae-dev
git clone https://code.google.com/p/go-gae-dev/ .
export GOPATH=`pwd`
cd src/gogaedev
goinstall -nuke .
./gogaedev

Read on to see how to test it out on your own setup.

Background

Currently, the GO App Engine SDK bundles the Python App Engine SDK, and uses it as a http proxy and an API server. This means that all request come into the Python SDK first.

The Python SDK does the following on behalf of the GO SDK:

• If the Python SDK determines that it is a static file request, it serves it directly from the filesystem
• If the Python SDK determines that it is an internal appengine request, it serves it directly
• Else it proxies it over to the GoApp
  • If the GoApp is not running, it starts it
  • If any file has changed since the last request was initiated, it will rebuild and/or restart the GoApp
  • It then proxies the request over to the GoApp, and returns the response back to the client

The problems with these are:

• Python SDK is single-threaded. It handles one request at a time. Consequently:
  • GO App development cannot support concurrent requests
  • For each request to GO App, all files in the directory are checked to see if the app should be reloaded
  • Web Requests are slower since each static file must be loaded one at a time.
• Python SDK checks all files for each request, making things slower.
• GoApp Log files are interspersed with the Python SDK log, and it’s hard to separate log files for each run.
• App.yaml is used for both development and production:
  • In itself, this is not a problem
  • However, in practice, it can be. For example, you may want to use a different initialization file in development than in production. You may want to skip some files (tests, dev setup) in production, but use them in development. (Python SDK even has the allow_skipped_files flag for this).
• Performance Profiling
  • The Python SDK frontend prevents me from really seeing how well my GO App performs.
  • I cannot do load tests, or stress the server, or see how concurrent requests may cause things to fail because I’m sharing memory where I should be using some of the concurrency primitives effectively.

The beauty of GO is that all these features are supported by GO libraries in very elegant ways, and we do not have to put up with all the Python SDK limitations if all we need Python SDK for is to serve as an RPC server for AppEngine services (ie datastore, memcache, etc), while leaving Go App to do the same thing it would be doing in a production environment.

There’s a saying that:

"If there's an itch I really want to scratch and I don't, ... I eventually will, ... just so I can move on".

It’s paraphrased, but I think Brad Fitzpatrick eloquently talks to it here: http://bradfitz.com/talks/2011-09-Djangocon . Anyhow, I finally gave it and wrote a tool that does everything, relegating python to just serve as an RPC Server and serve admin console, and other /_ah/* requests.

What the tool does

This tool will do the following:

1. Launch Api Server (Python SDK)
   if nothing is listening on the Python API Unix Socket
2. Launch go app (just like python sdk does) and writes log files to a certain directory
   if nothing is listening on the Go App Unix Socket
3. Create a proxy to do the following (by default):
   • requests matching / or /_ah/warmup, etc go to GO
   • requests matching /_ah/, /form go to Python
   • requests matching static files are served by this “helper” process
   • all others go to GO
4. Watches for changes to directories for my app
   • If any .go file there changes, it will rebuild and restart the app
   • If any other file changes, it will just restart the app
5. Watches for some files and keeps them in sync.
   • If any source file changes, it will copy it over to the corresponding dest

Setup of Python SDK:

With this setup, only requirement from Python SDK is that it should initialize its API server socket at startup, not at first GoApp CGI request.

To setup the API server accordingly, Two files have to be edited:

1. google/appengine/tools/dev_appserver_main.py
   to create and listen on the Go API Server Socket at startup
2. google/appengine/tools/dev_appserver.py
   to remove (comment out) call to execute_go_cgi

Edit google/appengine/tools/dev_appserver_main.py:
to create and listen on the Go API Server Socket at startup (i.e. before call to http_server.serve_forever).

 #ugorji: add call to setup api hook port
 if appinfo.runtime == 'go':
     import threading, getpass, atexit, asyncore
     import google.appengine.ext.go as go
     from google.appengine.ext.remote_api import handler
     user_port = '%s_%s' % (getpass.getuser(), port)
     go.SOCKET_API = go.SOCKET_API % user_port
     go.SOCKET_HTTP = go.SOCKET_HTTP % user_port
     go.GAB_WORK_DIR = go.gab_work_dir() % user_port
     go.cleanup()
     atexit.register(go.cleanup)
     go.RAPI_HANDLER = handler.ApiCallHandler()
     ds = go.DelegateServer()
     def asynCoreLoop():
       while ds.connected or ds.accepting:
         asyncore.loop(map=ds._map, count=1)
     th = threading.Thread(target=asynCoreLoop)
     th.setDaemon(True)
     th.start()

Edit google/appengine/tools/dev_appserver.py
to remove call to execute_go_cgi, by commenting out the whole code block.
This way, Python SDK does not try to proxy requests over to the Go App.

 # Ugorji: remove hook for _go_app
 # if handler_path == '_go_app':
 #   from google.appengine.ext.go import execute_go_cgi
 #   return execute_go_cgi(root_path, handler_path, cgi_path,
 #       env, infile, outfile)

How To Use This Tool:

User creates a file go-gae-dev-cfg.json, and puts in their app directory. A complete one looks something like this (below, we show what the defaults are):

   {
     "Verbose": false,
     "Succinct": true,
     "UseFSWatch": false,
     "IncludeChildProcLogs": false,
     "GaeSdkDir": "",
     "AppId": "app",
     "AppVersion": "1",
     "GoFilesToIgnore": "abc",
     "WatchPathsToIgnore": "(.* /)?(_go_\\.[0-9]|_obj|.*[#~].*)",
     "WatchDirNamesToSkip": "^(cmds|_obj|\\..+|_.*)$",
     "StaticPaths": "/web(/.*)?|(.*\\.(gif|png|jpg|jpeg|ico|css|js|json))",
     "InitialCheckGoPaths": "/$|(/_ah/(warmup)(/.*)?)",
     "ApiPaths": "/_ah/.*|/form.*",
     "ApiServerHttpURL": "http://localhost:8080",
     "GoServerHttpURL": "http://localhost:8088",
     "ManageGoApp": true,
     "ManageApiServer": true,
     "ProxyAddr": ":8888",
     "ApiParams": [ "--allow_skipped_files", "--skip_sdk_update_check" ],
     "LogDir": "/tmp/gogaedev_USERNAME_logs",
     "StaticFilesDir": ".", 
     "ManageGoAppIntervalSecs": 5,
     "FilesToSync": { },
     "DirsToWatch": [
       "."
     ]  
   }

A lot of these entries are reasonable/adequate defaults and can be omitted. At a minimum, configure these:

   {
     "GaeSdkDir": "/opt/go-appengine-1.6.0/google_appengine",
     "GoFilesToIgnore": ".* /((main|yaml)/.*|app_(dev|prod)\\.go)"
   }

To sync some files, e.g. if app_dev.go changes, copy it over to app_for_env.go, use:

   {
     "FilesToSync": {
       "app/server/app_dev.go": "app/server/app_for_env.go"
     }
   }

User must add a http tcp listen address to their app. A quick way to do this is, within an init() method in your app, add:

   if appengine.IsDevAppServer() { go http.ListenAndServe(":8888", nil) }

Within your app directory, run gogaedev:

   gogaedev

Access application as usual, but using the proxy (instead of going through Python SDK).

   E.g.
   http://localhost:8888/_ah/admin
   http://localhost:8888/