Objective Gripes with New Google App Engine Pricing

This attempts to objectively address areas where the new App Engine pricing may not not fair, and what Google may do to alleviate these concerns.

Disclaimer
I am an unabashed fan of Google App Engine, and have been for over three years. I don’t think there’s anything else on the market that comes close. I think it is a fine platform for any applications, from simpler small ones to large complex ones.

My only gripes are about the new pricing costs, how it is being rolled out, and some lack of transparency. These gripes are founded because they could cause the platform to lose its momentum, leading to an untimely death which is what they seem to be trying to avoid in the first place.

Hypothesis on making App Engine a “sustainable” business
Google said that the pricing model change was necessary to make App Engine a sustainable business. Many people took that to mean that App Engine may have been running at a loss. I interpret that differently.

My guess is not that they were making a loss, but that they were not generating enough revenue/profit. For a company Google’s size, and with it’s new focus on high-value products, I believe each product had to have a material impact on the bottom line to survive. With the current user base and trajectory, the profits was not enough to make a material impact to the bottom line. Consequently, Google decided to hike up the prices while giving us short notice before implementing them, even though their own ducks were not in order (buggy, non tested scheduler, no concurrency in Python, etc).

Pricing Gripes
Google App Engine provides a Platform-As-A-Service system, with the following:

• Data Hosting:
  We leverage IP Google has built for high performance, highly available and scalable storage. For these, we are charged seperate premium costs for datastore storage and per operation.
• Application Hosting:
  We leverage Google’s hosting and routing infrastructure at the frontend and optionally backend. This comes with some value-added services, mainly:
  1. Disaster recovery (multiple data centers)
  2. Smart Routing (horizontal scaling, activity-based routing)
  3. Distributed Caching (with non-deterministic eviction policy)
  4. Scheduled Tasks
  5. API’s to use their services
• Other Services:
  A few other premium-priced services which are optional to use (email, XMPP, channels, etc)

The key thing to note is that these are all charged separately. Discussing the cost of one of these should not be mixed up with value provided by the other services.

There are two big pricing changes which will affect users of the service in a way that is not deterministic:

1. Cost of hosting application instances (front end and backend).
2. Cost of datastore operations

Cost of hosting application instances (front end and backend)
Beyond the free quote, the application hosting is priced at about $60 ($0.08 X 24 X 30) if you “average” 1 instances of 600Mhz CPU, 128MB RAM, and increases by $60 for each new instance per month.

Contrast this with other providers e.g. http://order.1and1.com/xml/order/ServerPremium where you get a 4.4 GHz CPU with 2G RAM for the same price. That’s about 7X CPU and 15X RAM for the same price.

Given that the other services provided by App Engine are charged separately at a premium and there are tens of thousands of billing-enabled applications, this margin is a pretty hefty multiplier for each application (even taking the value-added features into consideration).

Beyond the fairness of the hosting cost, there is also the concern that some things are not deterministic and could cause our charges to balloon. These include:

• Scheduler algorithms (which may cause unnecessary instances to be created)

Cost of datastore operations
We can’t really comment on the fairness of the datastore prices until we’ve had a chance to see it for a while. These prices may be fair regardless, and developers will just have to go back to the drawing board to update their applications and re-optimize.

However, there are also things which are not deterministic and could cause our charges to balloon. These include:

• Memcache eviction policy (which could result in further datastore access)

What Google may do to help alleviate concerns:
Google may do the following to alleviate our immediate concerns:

1. Make the hosting prices fairer (ie reduce the hosting prices to a more palatable multiplier) for both frontend and backend instances.
2. Implement some fair and transparent scheduler algorithm: This allows users direct the scheduler to use as few instances as technically possible and only sparingly start up extra instances. The following should be inputs to the scheduler mechanism:
   1. Request Latency
   2. Server Resource Consumption (CPU, RAM).
      This is more critical on a runtime like GO which (in theory) is not as limited by the number of available threads, and so can handle a lot more concurrent requests.
   3. Min Idle Instances:
      Allow and respect a min-idle-instances to be settable to 0. Also prefer that setting over the max-idle-instances (which could be 2).
   4. Maximum Number of concurrent requests per instance
3. Support Throttling of requests.
   If I hit my reach my threshold of max-budget and/or max-active-instances, start throttling requests instead of the options of shutting down for the rest of the day or having an extremely high bill. This will prevent me from getting slashdotted to broke, or having my site completely down.
4. Implement a fairer and transparent memcache eviction policy:
   For example, evict from Memcache for larger users first. If there is still memory pressure, then for all applications using over a threshold, evict a certain percentage of their memcache, and continue doing that until memory usage is below the machine-level threshold.
5. Make Python 2.7 Generally Available at least one month in advance of leaving Preview:
   This allows Python users have concurrency support. The current temporary half-price suggests that an instance can only handle 2 concurrent requests which is not enough.

In addition, these may help moving forward:

1. Give us a better roadmap.
   With premium pricing should more premium treatment. Many of the features on the roadmap have been there for over a year. Users have no insight into when these features may come. A roadmap should allow users plan accordingly. For example, I might wait on building a feature if I know that Google plans to release it within the next 3-6 months.
2. Let us know what experimental means.
   Currently, GO runtime is defined as experimental. What does this mean? Does this mean that the plug could be pulled on the GO runtime at any time, or will we be given a heads up and how long (e.g. 1 year, 3 years, etc)?
3. Create a better solution for downloading your application’s data.
   For example, allow users request their data and charge them a fee comprising of a flat admin fee, and a variable cost based on the size of the data. Then ship the encrypted data to the user in a set of portable media (DVD’s, portable hard drive, etc).

Responses to critical responses to our outcry
Users outcry has got some critical responses from people more sympathetic to Google with regards to their new app engine pricing. Let’s respond to them.

Google is not a charity:
No one expects Google to give this out free. Google is also free to decide to start charging us more for the significant IP they’ve built over the years even before App Engine was conceived. Our gripe is that the charge of hosting in isolation is very high for the server resources we are being given.

App Engine was in preview before:
Forgive me, but after 3 years with no idea that the price may change, there’s no way we could have seen this coming. Google even introduced pricing of some things that were in-line. For example, Always On was released in December, and gave us 3 Always-On instances for $9 less than a year ago and over 2 years into the preview. Contrast that now with 1 instance for $56. It’s not an apples-to-apples comparison, but my point is still made.

An argument can even be made that App Engine exited Preview in February 2009 when billing was enabled. Look at the first blog post from http://googleappengine.blogspot.com/2008/04/introducing-google-app-engine-our-new.html, and I quote:

"During this preview period, applications are limited to 500MB of
storage, 200M megacycles of CPU per day, and 10GB bandwidth per
day. We expect most applications will be able to serve around 5
million pageviews per month. In the future, these limited quotas
will remain free, and developers will be able to purchase additional
resources as needed."

In summary, Google had over 8 months to monitor usage of early adopters, with over 100,000 applications, before releasing a billing plan which at the time was considered fair. It also introduced new pricing within the last year which was in line with its typical low pricing. We were not even aware that App Engine was still in preview.

References:
http://googleappengine.blogspot.com/2010/12/happy-holidays-from-app-engine-team-140.html
http://code.google.com/appengine/docs/roadmap.html
http://googleappengine.blogspot.com/2008/04/introducing-google-app-engine-our-new.html
http://googleappengine.blogspot.com/2009/02/new-grow-your-app-beyond-free-quotas.html
http://code.google.com/appengine/docs/roadmap.html
http://code.google.com/appengine/kb/postpreviewpricing.html
http://www.google.com/enterprise/cloud/appengine/pricing.html
http://code.google.com/appengine/docs/quotas.html#Requests
http://code.google.com/appengine/docs/adminconsole/performancesettings.html
http://www.google.com/enterprise/cloud/appengine/pricing.html
http://order.1and1.com/xml/order/ServerPremium