Background

PostgresSQL is great. But administering it can suck up a lot of time and for small teams using SaaS service is great value. We use and love Amazon RDS.  Until recently it was the only reasonable choice in the market. But in 2017 new options ore on the verge of becoming available.  Both Google and Azure clouds announced support and Amazon is also launching their Aurora service with PostgreSQL compatibility.

We did a quick comparison of those options.

TLDR:  Google and Aurora are a bit faster for the same money but it’s no free lunch. Azure tests are not yet done.

The Test

Use case we care about is “mid size” database and queries that process gigabytes of data in seconds-minutes. More specifically we test 45GB table containing JSON documents (PostgreSQL is great for working with JSON).   

We won’t test CRUD/web app backend performance. pg_bench is fine for that. In reality for small/medium apps any service will be fine and cost negligible.

We won’t test “big data” scenarios – Google BigQuery, AWS Redshift and Athena are great for that.

To run this test we provisioned High Availability setups with 16GB RAM and minimum number of possible CPUs.  We also ran tests on a desktop machine with a modern SSD.

Instance sizes and costs

All amounts are US dollars per month, using on demand pricing.

	AWS RDS	AWS Aurora	Google
Notes		PostgreSQL pricing is unavailable at this time. So we are using MySQL pricing:	High availability for PostgreSQL is not yet available.  Assuming 2x costs
Cost items	db.r3.large multi AZ (2 cpu, 15GB ram) $376 1TB General Purpose (SSD) Storage $230	db.r3.large $0.290 $215 1TB storage   $.100/GB $100	CPU Master 2 vCPU $94 CPU Slave 2 vCPU $94 RAM Master 13GB $103 RAM Slave 13GB $103 SSD Storage Maser 1TB $187 SSD Storage Slave 1TB $187
Total	$606	$315	$758

Results

First times to restore and parse 45GB table containing JSON.

	AWS RDS	AWS Aurora	Google	Desktop
Restore (seq write)	50:28	41:44	21:15	24:00
Scan (seq read)	32:30	5:02	15:58	2:57
Parse JSON (CPU)	1:46:50	46:54	51:55	31:51

 

Dividing time by size we get processing speed in MB/sec:

	AWS RDS	AWS Aurora	Google PG	Desktop
Restore (seq write)	14.53	17.57	34.50	30.55
Scan (seq read)	22.56	145.69	45.92	248.58
Parse JSON (CPU)	6.86	15.64	14.12	23.02

 

Observations

JSON parsing is CPU bound and only runs in one thread so this is a tight bottleneck on all platforms. Still seems that Google gives faster virtual cpus than RDS.

Writing to Aurora is slow and reading is fast.  Could be because of their 6 way replication.

Google claims to give many more IOPS than Amazon RDS (15000 vs 3000).  Our test shows only about 2x speedup.

As of June 2017 Google is the best value, but no HA support yet.  Aurora is not yet ready but might mature to be the best option.  RDS is still a winner for production work.

SQL queries used

Sequential scan:

select count(*) row_count, sum(length(content)) text_length from js

JSON parse:

with jp as (select json_array_elements(content::json) x from js) select distinct x->>'ID' id, x->>'Name' name from  jp

Intro

AWS RDS is very convenient and takes care of almost all DBA tasks. It just works as long as you stay inside AWS. But if you want to have a local copy of your database or need to move data to another host it can be tricky.

TL;DR – For our solution skip to part last section

What doesn’t work

AWS daily backups

AWS RDS by default creates daily backups of your data. First thought would be get such backup and restore it locally. But those are not regular Postgres backups. They probably are VM image copies, but no way to know, as you cannot copy or see them.  The only option is to restore them to another RDS instance.

Postgresql replication

AWS uses replication to maintain hot standby, but do not give access to that.

Cold backups

No access to RDS filesystem, so can’t do that either

Amazon Database Migration Service (DMS)

AWS DMS sounds too good to be true.   Thea features that would be killer if they worked:

• Any to any DB connections (RDS PG to local MySQL for example is possible)
• Change data capture (CDC) and continuous replication without modifications to database schema
• DDL capture

In reality we couldn’t get it to do even basic data copying.  It corrupts/truncates data, throws errors and there’s very little talk about it in Stackoverflow or AWS forums.  Not ready for prime time but worth keeping an eye on it.

Foreign data wrappers (FDW) – pushing from RDS to remote

It works and can be used to push incremental changes like so:

ON RDS: 
insert into remote_bigtable 
select * from bigtable where id>(select max(id) from remote_bigtable)

But speed is only few rows per second.  Can’t use it for anything

Foreign data wrappers (FDW) – pulling data

It’s faster than pushing but more difficult.    If you run query on local db like this:

ON Local:

select * from rds_bigtable where id>(select max(id) from bigtable)

It will fetch entire bigtable and do filtering locally (9.6 fixes this).

So for full data pulls this method is still too slow and for incremental requires extra work like so:

ON local: select max(id) from bigtable
ON RDS: create tmp_bigtable as select * from bigtable where id>maxid_on_local
ON local: insert into local_bigtable select * from rds_tmp_bigtable
ON RDS: drop table tmp_bigtable

We use this approach for incremental data pulls.   It’s pain.  Hopping better alternatives will come up.

pg_dump running on local server

It works, but much slower than what’s possible

pg_dump running on tempororay EC2 instance

Finally a solution that works and is fast!

Steps to create it

1. Create EC2 instance with enough space to store full db dump in the same datacenter/zone as RDS
2. Install postgres tools on it: sudo yum install postgresql95
3. Install aws command line tools (optional, need for file copying)
4. Open firewall in RDS to allow connections from temporary instance
5. Run pg_dump –host=xxx.rds.amazonaws.com –username=xxx –file=x.dmp –format=c
6. Copy file to s3 (aws s3 cp)
7. Delete instance
8. On local host get and restore file:  pg_restore –host=localhost –username=x  x.dmp

It your db is large enough pg_dump dies in the middle of export.  It’s an obscure bug with ssl.  AWS will force ssl socket renegotiation after preset amount has been transfered (seems to be about 6GB).  PostgreSQL doesn’t renegotiate and kills connection instead.

Solution is to temporarily disable SSL on RDS (all security implications and requires a db restart). In RDS Parameter group set flag ssl=0

Conclusion

Copying Postgresql RDS data to local server is harder than it should to be. Hopefully my long adventures, misadventures and discoveries will save you some time.