Skip to content

Tutorial

Jump to bottom
Wenbo Tao edited this page Oct 16, 2020 · 59 revisions

This tutorial serves as quick intro of how to run and develop a kyrix application.

Running Example Kyrix Applications

After following the installation instructions, you will be able to interact with a Kyrix application showing NBA basketball data. Now let's try running another version of this application which has two coordinated views. Open another terminal tab, and run the following commands:

cp docker-scripts/compile.sh compiler/examples/nba_cmv/compile.sh   # copy the compile script 
cd compiler/examples/nba_cmv                                        # go to the app directory
chmod +x compile.sh                                                 # make the compile script executable
sudo ./compile.sh nba_cmv.js                                        # compile the application

Then it's time to see a more different one which shows the crime rates in the United States:

wget https://www.dropbox.com/s/xi7bhcuxza2a0n9/usmap_new.sql         # download data
sudo ./docker-scripts/load-sql.sh usmap_new.sql --dbname usmap       # load data into the database, must be run in root folder
cp docker-scripts/compile.sh compiler/examples/USMap/compile.sh      # copy the compile script
cd compiler/examples/USMap                                           # go to the app directory
chmod +x compile.sh                                                  # make the compile script excutable
sudo ./compile.sh USMap.js                                           # compile the application

More example applications and their data can be found here. See Loading your data section for more instructions on how to load different data sources into the database. You can run ./run-kyrix.sh --help to see more options provided by the script.

Declarative Authoring

One great thing about Kyrix is the declarative authoring in Javascript. Declarative design hides from you complex execution details such as frontend rendering and backend data fetching logic, and allows you to stay focused on designing your application. We describe this declarative model in the following.

Kyrix declarative model has a concise set of abstractions, as shown in the figure above. The most important notion is a canvas, which can be seen as a "zoom level". Canvases are connected by jumps, which enable zooming in/out between canvases.

A canvas is composed of one or more overlaid layers, which can be thought of as layers in PhotoShop or Adobe Illustrator. Each layer is associated with a SQL query, the result of which is passed in the rendering function and placement function of this layer.

A Kyrix visualization has one or more views. A view is effectively a "window" through which you can see canvases. Different views can be linked with some type of coordination, which is also defined using the "jump" notion.

See the API reference for the exact syntax of the declarative model.

An example

This example has two views with coordinated loading and selection, and a couple canvases connected by semantic zooms.

Four files

For this basketball app, we write the spec in four files:

  • nba_cmv.js: this file has definitions of projects, views, canvases, layers and jumps.
  • transforms.js: this file defines all data transform objects.
  • renderers.js: rendering functions of all layers are in this file.
  • placements.js: this file has all placement functions defined.

Examples in the example folder all follow this four-file format, which we believe is good separation of the thinking process when writing a Kyrix app. We have provided templates of these four files here.

Loading Data into Docker Container

To move beyond the example applications, you need to load custom data into the Postgres container (kyrix_db_1). We provide useful scripts that let you load either a csv file or a Postgres SQL dump using one command.

Loading CSV

./docker-scripts/load-csv.sh CSV_FILE [OPTIONS]

Here, CSV_FILE is the name of a csv file. OPTIONS allow you to specify the Postgres database name and table name where your data is loaded, as well as the delimiter of your csv file. Without OPTIONS, both database name and table name will default to the name of your csv file. Make sure that your CSV file contains proper header names that are good for Postgres. An example:

./docker-scripts/load-csv.sh nba_celtics.csv --dbname nba --tablename celtics --delimiter "\t"

will load the data in nba_celtics.csv (which uses tab as delimiter) into table celtics in the database nba.

Note that this script has no type inference and every column is typed TEXT. To do more sophisticated csv loading, consider the following strategy:

  • copy your csv file into docker container using docker cp;
  • create your database tables manually by connecting to the Postgres instance as above;
  • load your csv using Postgres COPY.

Loading SQL Dump

./docker-scripts/load-sql.sh SQL_DUMP_FILE [OPTIONS]

Here, SQL_DUMP_FILE is the name of a Postgres sql dump file. OPTIONS allow you to specify the Postgres database name. Without OPTIONS, the database name will default to the name of your sql dump file. An example:

./docker-scripts/load-sql.sh forest_dump.sql --dbname forest

Inspect what's loaded

It is often useful to connect to the Postgres instance to see what's actually being loaded. To do so, run the following commands to start an interactive Psql shell in container kyrix_db_1 (see here for more details of Kyrix docker configs):

docker exec -it kyrix_db_1 /bin/sh                          # a shell session into the db container
psql postgresql://postgres:kyrixftw@localhost/postgres      # log into the postgres instance
\list                                                       # see the list of databases
\c dbname                                                   # connect to one database
\d                                                          # see the list of tables
\d tablename                                                # see the schema of a table
select * from tablename;                                    # see what's in a table

Typical Workflow to Create a Kyrix App

Developing locally with Docker containers running

We recommend using our docker images to start a Kyrix backend to avoid running into weird errors with manual installation. However, we do not recommend writing Kyrix specifications inside the containers since it's slower and you are most likely more used to your local environment.

To assist you in developing locally, we provide an automated script (docker-scripts/compile.sh) to help you compile your Kyrix specs inside the containers. Simply copy this script to the directory where all your Kyrix specs are in, then run it as the following (you may have to make it executable first, e.g. chmod +x compile.sh):

sudo ./compile.sh app.js [OPTIONS]

app.js is where you construct the project object and call project.saveProject(). This will send a REST request to the Kyrix back-end (in container kyrix_kyrix_1) which will then save the specifications in the database container (kyrix_db_1). OPTIONS is either -f (force computing indexes) or -s (skip index computations).

Note that this script will put your JS specs files under /kyrix/compiler/examples/user_app/ in the kyrix_kyrix_1 container. So make sure that any relative paths to files (e.g. require modules) follow this structure.

When you first start the containers, the Kyrix backend (in kyrix_kyrix_1) is by default serving the example NBA application. After you compile your spec, the backend will start serving your app. To see your app, simply go to localhost:8000.

Debugging

Generally, there are three types of errors: compiler errors, frontend errors and backend errors. Compiler errors can be relatively easily fixed with the help of error messages after running sudo ./compile.sh app.js.

Frontend errors are typically notified by your browser's console. A good iterative way to debug them is to compile the project every time you modify the rendering function and then refresh the browser. Note that only modifying a rendering function will not trigger a recomputation of indexes. Another way would be to embed your rendering function into an HTML file, and use some sample data to see if it looks correct.

There are two ways to use the browse console to deal with front-end errors. First, Kyrix exposes a global variable called globalVar. You can access many useful information by typing this variable in the console. Most frequently, you'll want to inspect the information about a specific view, e.g., globalVar.views[YOUR_VIEW_ID]. Second, you can jump to the source of a rendering function by typing globalVar.views[YOUR_VIEW_ID].curCanvas.layers[LAYER_ID].rendering.parseFunction(), and then double-clicking on the generated function. You can then set breakpoints.

Backend errors are trickier to debug right now. The most common type of backend errors are database-related, e.g., the data isn't loaded, the column names specified do not exist, etc. If there is an error occurred during either indexing or serving, the backend will continue to run, allowing you to trial and error.

Skipping reindexing

Every time you run ./compile.sh app.js, the backend decides if the difference between your new spec and the previous one requires recomputing the DB indexes. Things that can trigger a recomputation include modifying the data transforms, adding or deleting layers, etc.

For large datasets, recomputation can take a fair amount of time. We are working on ways to reduce this turnaround time. Stay tuned. When you know indexing recomputation is unnecessary, run ./compile.sh app.js -s to skip recomputation.

Forcing reindexing

Note that Kyrix is (currently) unable to detect changes in the database. So if the data corresponding to one canvas is updated, you need to force the backend to recompute the index. To do so, run ./compile.sh app.js -f.

A List of Commonly Used Commands

Command Description
docker-compose up Start Kyrix docker containers, run under root directory.
./docker-scripts/load-csv.sh CSV_FILE [OPTIONS] Load a csv file into the Postgres instance. See descriptions above.
./docker-scripts/load-sql.sh SQL_FILE [OPTIONS] Load a Postgres SQL dump file into the Postgres instance. See descriptions above.
./compile.sh app.js Compile your app and send it to the Kyrix backend if there is no error. The Kyrix backend will compare the current specification with the previous one to decide if re-indexing is needed.
./compile.sh app.js -s Compile your app and send it to the Kyrix backend if there is no error. The Kyrix backend will skip re-indexing.
./compile.sh app.js -f Compile your app and send it to the Kyrix backend if there is no error. The Kyrix backend will force re-indexing.
docker-compose build Rebuild docker containers. Run every time there are changes to Kyrix code (e.g. a git pull to fetch latest code). Run under root directory.
docker exec -it kyrix_kyrix_1 /bin/sh Get a shell into the Kyrix backend container.
docker exec -it kyrix_db_1 /bin/sh Get a shell into the Postgres DB container.
docker exec -it kyrix_db_1 /bin/sh -c "psql postgresql://postgres:kyrixftw@localhost/postgres" Get a shell into the Postgres instance.
docker container stop $(docker container ls -a -q) && docker system prune -a -f --volumes Clear everything docker-related (you need to run it twice to remove all volumes, images and networks).
Clone this wiki locally