requirements.rdoc

Jetpants Requirements and Assumptions¶ ↑

The base classes of Jetpants currently make a number of assumptions about your environment and database topology.

Plugins may freely override these assumptions, and upstream patches are very welcome to incorporate support for alternative configurations. We’re especially interested in plugins or pull requests that add support for: Postgres and other relational databases; Redis and other non-relational data stores; non-Redhat Linux distributions or *BSD operating systems; master-master topologies; multi-instance-per-host setups; etc. We have attempted to design Jetpants in a way that is sufficiently flexible to eventually support a wide range of environments.

Environment¶ ↑

• Ruby 2.3 or higher.

  • If your OS bundles an old non-upgradeable Ruby, we recommend using rvm to manage multiple Ruby versions.

  • To date, Jetpants has only been tested on the MRI/YARV implementation of Ruby. We hope to add JRuby compatibility in the near future.

• Several Ruby gem module dependencies. If you’re new to Ruby, the easiest way to install everything at once is gem install jetpants.

  • Some of these gems require building native extensions, which means they need devel headers in order to build.

  • For example, jetpants requires the mysql2 gem, which in turn needs MySQL development C headers in order to compile.

• MySQL (or compatible, like Percona Server), version 5.1 or higher.

• Required Linux binaries that must be installed and in root’s PATH on all of your database machines:

  • service, a wrapper around init scripts, supporting syntax service mysql start, service mysql status, etc. Some distros include this by default (typically as /sbin/service or /usr/sbin/service) while others offer it as a package. Implementation varies slightly between distros; currently Jetpants expects service mysql status output to include either “not running” (RHEL/Centos) or “stop/waiting” (Ubuntu) in the output if the MySQL server is not running. Additionally, service must pass extra parameters through to the daemon – service mysql start --skip-networking should pass --skip-networking to mysqld.

  • nc, also known as netcat, a tool for piping data to or from a socket.

  • Whichever compression tool you’ve specified in the Jetpants config file for the compress_with and decompress_with options, if any. (if omitted, compression will not be used for file copy operations.)

• InnoDB / Percona XtraDB for storage engine. Jetpants has not been tested with MyISAM, since Jetpants is geared towards huge tables, and MyISAM is generally a bad fit.

• All MySQL instances run on port 3306, with only one instance per logical machine.

  • A plugin could override this easily, but would require you to use the –report-host option on all slaves running MySQL 5.1, so that crawling the replication topology is possible. It would also have to override various methods that specify the MySQL init script location, config file location, data directory, etc.

  • Since there’s no “standard” layout for multi-instance MySQL, this won’t ever be part of the Jetpants core, but we may include one implementation as a bundled plugin in a future release.

• Master-Master replication is not in use. It presents a large number of additional failure states which make automation quite difficult, since transaction IDs are not global in MySQL. This restriction will be lifted if/when we add MySQL 5.6 support.

• Unusual replication topologies – special-purpose slaves, ring topologies, hierarchical replication – are not in use. Plugins could certainly override this by adding additional types of slaves, but there are too many options for this to be part of the Jetpants core.

• You must be running Jetpants as a UNIX user with access to an SSH private key, capable of connecting as root to any host in your database topology. If this key has a nonstandard name or location, you must specify this using the ssh_keys option in the Jetpants configuration file. SSH agent forwarding should work fine automatically, but be careful of its interaction with UNIX tools like screen.

• The root MySQL user must be able to access the database on localhost via MySQL’s UNIX socket. You may specify the root password with Jetpants’ mysql_root_password option in the configuration file. Or if all of your database hosts has a /root/.my.cnf file specifying the root password, you can omit mysql_root_password in the configuration file.

Database topology¶ ↑

We define a database pool as 1 master and 0 or more slaves. Each slave falls into one of three classifications:

• Active slaves: replicas that actively receive reads from your application(s)

• Standby slaves: replicas that exist for high availability, to replace failed nodes or to quickly spin up new standby replacements.

• Backup slaves: replicas that never receive application queries and will never be promoted to another class. For example, might be a different hardware spec.

At Tumblr we ensure that all pools have exactly 2 standby slaves. This establishes a minimum level of redundancy at 3 copies of each row. If a master or active slave fails, we use Jetpants to promote one standby slave in its place, and use the other standby slave to quickly clone a replacement. (Jetpants clones nodes by stopping the MySQL daemon and then copying the data set; this is substantially faster than using a hot-copy solution, especially on very large and/or very actively written data sets.)

You may alter the two-standbys-per-pool requirement by tuning the standby_slaves_per_pool setting in the Jetpants configuration file.

We no longer use dedicated backup slaves at Tumblr, although we still offer support in Jetpants for them.

Sharding scheme¶ ↑

We define a database shard as a specialized type of pool, always consisting of 1 master and 2 standby slaves. All shards contain the same set of tables, but each stores a different partition of the data set, divided by ID ranges of a global sharding key.

For example, say your application’s natural sharding key is user_id. The users table itself might not be shardable (since it may require global lookup patterns like find-by-email), but any other table containing a user_id column could be sharded. One shard might contain all sharded table data for users 1 through 1000000; another shard would contain data for users 1000001 through 2000000; etc. These ranges may be of uneven sizes. When disk usage approaches capacity or I/O utilization gets too high, Jetpants can be used to split the shard into N new “child” shards of equal or unequal ranges.

We prefer this range-based scheme due to its simplicity. The ranges can be expressed in a language-neutral JSON or YAML format, which can be shared between application stacks easily. Jetpants solves all data rebalancing problems for you. There’s no need to “pre-allocate” thousands of tiny shards, nor do you need an external lookup service which would be a bottleneck and a single point of failure.

Under this sharding scheme, there will always be a “last” shard, with a range of X to infinity. All new users (or whatever your sharding key is) will have their data placed on this shard. Jetpants cannot split this last shard, but instead it can truncate its range to no longer have an infinite max_id – ie, truncate its range such that it now handles IDs X through Y, where Y is an ID “in the future” (sufficiently higher than your current max ID such that you won’t hit ID Y for another few days). Jetpants then adds a new last shard which handles IDs (Y+1) to infinity. Please see the jetpants shard_cutover command.

Assumptions for shard-split logic¶ ↑

In order for Jetpants to be able to rebalance your shards efficiently, we also must assume the following:

• No auto_increment columns in any sharded table. Use an external ID generation service instead. Build your own (it’s a hundred lines of C) or use Memcached or Redis or Snowflake or any other number of options.

• Primary key of all sharded tables begins with the sharding key. This takes advantage of InnoDB clustered indexes to allow efficient export/import of data by ranges. This isn’t strictly mandatory, but it makes an absolutely huge performance difference versus using a secondary index.

• Uniform MySQL configuration between masters and slaves: log-slave-updates, unique server-id, generic log-bin and relay-log names, replication user/grants everywhere.

• Redundant rows temporarily on the wrong shard don’t matter to your application. Hopefully these would only matter if you’re doing aggregate queries (like COUNTs) over the whole data set… but if your data set is big enough to shard, these already don’t scale, so it shouldn’t really matter.

By default, the standby_slaves_per_pool config option is set to 2. This means each shard has 3 machines: 1 master and 2 standby slaves. Therefore, splitting a shard into N pieces will require 3*N spare machines, because Jetpants does not split the shard “in-place”. After the process is completely finished, you’ll get back the original shard’s 3 machines to re-use or cancel.

A “spare” machine in Jetpants should be in a clean-slate state: MySQL should be installed and have the proper grants and root password, but there should be no data on these machines, and they should not be slaving. Jetpants will set up replication appropriately when it assigns the nodes to their appropriate pools.

For more information on the shard split process implemented by Jetpants, including diagrams of each stage of the process, please see Evan Elias’s presentation at Percona Live 2013, starting at slide 38.