Jetpants offers an extensible plugin system. Plugins are Ruby code (such as stand-alone gems) that add to Jetpants by supplying callback methods, and/or overriding core methods.
It is highly recommended that you tie Jetpants into your site’s asset tracker (ie, hardware management system) by writing a custom plugin. This will allow Jetpants to automatically know what database pools and shards are present, and to make topological changes immediately be reflected in your site’s configuration.
Other recommended uses of plugins include integration with your site’s monitoring system, trending system, query killers, and environment-specific overrides to various core methods.
We supply a sample plugin called simple_tracker, demonstrating how to go about writing a very basic asset-tracking plugin. This plugin just uses an internal JSON file to keep track of database topology/state, and separately writes app configuration to a YAML file. This isn’t actually suitable for production use, but should provide a reasonable starting point for learning the plugin system and building a real asset tracker.
To use simple_tracker, be sure to define its tracker_data_file_path (where it saves its internal asset-tracking data) and app_config_file_path (where it exports web app config files). If you’re wondering why these are kept separate, it’s to allow you to support pool topology changes that don’t immediately impact your application, in case you’re doing a complex multi-step migration.
For an example of how to configure the plugin, please see the bottom of etc/jetpants.yaml.sample.
When you first start using simple_tracker, there will be no pools, shards, or spare machines. If you’re fluent in Ruby, you can start to add some via jetpants console, or you can write a custom script to import data from another source or config file. Alternatively, you can use these commands that simple_tracker adds to the Jetpants command suite:
-
jetpants add_pool -
jetpants add_shard -
jetpants add_spare
We also supply a plugin called jetpants_collins, offering integration with the Collins asset management software. This is fully intended for production use, and powers our automation at Tumblr. For more information on jetpants_collins, please see jetpants_collins.rdoc (view on GitHub).
If you’re writing your own asset-tracker plugin, you will need to override the following methods:
-
Jetpants::Topology#load_pools
-
Reads data from an asset tracker to initialize all pools and shards.
-
-
Jetpants::Topology#write_config
-
Writes a configuration file for your webapp, or commits configuration changes to a service, depending on how your site handles configuration knowledge
-
-
Jetpants::Topology#claim_spares
-
Obtains spare database nodes and marks them as no longer being spare
-
-
Jetpants::Topology#count_spares
-
Returns a count of spare database nodes
-
-
Jetpants::Pool#sync_configuration
-
Updates the asset tracker with the current status of a pool.
-
This should update the asset tracker’s internal knowledge of the database topology immediately, but not necessarily cause the application’s config file to be regenerated immediately.
-
You may also want to override or implement these, though it’s not strictly mandatory:
-
Jetpants::DB#for_backups?
-
Defines how to tell the different between standby slaves and backup slaves
-
Skip this if your architecture doesn’t include backup slaves, or if the default implementation (hostnames starting with “backup”) is sufficient
-
-
Jetpants::DB#after_probe_slaves
-
In order to support master promotions when a master has failed, you will need to implement this to set a slave list from your asset tracker if a MySQL instance isn’t available
-
-
Jetpants::DB#after_probe_master
-
In order to support operations on a slave that has failed, you will need to implement this to set @master based on information in your asset tracker if a MySQL instance isn’t available
-
-
Jetpants::Pool#after_remove_slave!
-
If your implementation of Jetpants::Pool#sync_configuration works by iterating over the current slaves of the pool’s master, it won’t correctly handle the
jetpants destroy_slavecommand, or anything else that calls Jetpants::DB#disable_replication! which does a RESET SLAVE. You can instead handle that special case here.
-
-
Jetpants::DB#stop_query_kiler and Jetpants::DB#start_query_killer
-
If you use a query killer, you’ll want to override these methods to start and stop it. stop_query_killer is called prior to a long-running query (SELECT … INTO OUTFILE, LOAD DATA INFILE, SELECT COUNT(*), etc) and start_query_killer is called after the operation is complete.
-
-
Jetpants::DB#disable_monitoring and Jetpants::DB#enable_monitoring
-
If you use monitoring software, you can override these methods to prevent alerts from Jetpants operations. Jetpants calls disable_monitoring prior to performing maintenance operations on a node, such as stopping MySQL prior to cloning a slave, splitting a shard, etc. It calls enable_monitoring once the operation is complete.
-
If you define a plugin named “foo” in your Jetpants config file, on startup Jetpants will first attempt to require ‘foo/foo’ (for loading bundled plugins in <tt>jetpants/plugins/<>tt>), if that fails then simply ‘foo’ (for loading a stand-alone gem).
Plugins may be located anywhere on your Ruby load path, so packing them as standard gems should work perfectly. You may want to prefix the gem name with “jetpants_” to avoid conflicts with other gems. Jetpants also adds its plugins directory to its load path automatically, so that any pre-bundled plugins (like simple_tracker) can be loaded easily.
The Jetpants plugin system is designed to be as light and simple as possible. Plugins are just normal Ruby code that gets required after the rest of Jetpants has been initialized. This allows you to override any method in any Jetpants class. Simply re-open the class as normal in Ruby.
The base of your plugin may use the Jetpants::Plugin module as an outer namespace, but doing so is not required.
To make it easier to chain behavior before or after existing Jetpants methods, all core Jetpants classes include a mix-in (Jetpants::CallbackHandler) that adds automatic callback functionality. For example, any time you call method Jetpants::DB#foo, it will magically first call method Jetpants::DB#before_foo if it exists. Similarly, after the foo method finishes, it will magically call method Jetpants::DB#after_foo if it exists.
A particularly nice aspect of callbacks is that they “stack”. For example, if two plugins both implement Jetpants::DB#before_foo, they will both be called before Jetpants::DB#foo. You can even specify the priority of a callback (higher = called first, default is 100) by preceding it with the Jetpants::CallbackHandler.callback_priority decorator. For example:
module Jetpants class DB callback_priority 150 def after_query(*args) puts "After calling DB#query, I am called next due to my high priority" end end end module Jetpants class DB callback_priority 85 def after_query(*args) puts "After calling DB#query, other plugins' callbacks with higher priorities get called, then I get called" end end end
Callbacks receive the same parameter list as the base method was called with. Their return values are ignored.
Callbacks may interrupt the control flow by raising a Jetpants::CallbackAbortError exception:
-
If this is raised in a before_foo method, all lower-priority before_foo stacked callbacks will be skipped, as will the call to foo and any after_foo callbacks. The exception is automatically caught and is not fatal. The return value of foo will be nil.
-
If this is raised in an after_foo method, all subsequent lower-priority after_foo stacked callbacks will be skipped. The return value of foo will be unaffected.
This plugin adds additional Jetpants commands to aid in performing a major MySQL upgrade. For full documentation, please see upgrade_helper.rdoc (view on GitHub).