Getting sphinx/thinking_sphinx to run in a dokku container

[ncri]

Hi,

this is not really an issue with he gem I think, but rather with my app setup: I try to run thinking sphinx (with real time indexes) in an app I have deployed using Dokku - a Heroku style open source PaaS, using docker containers. So far I don't have much luck. I tried 2 ways:

1. Installing Sphinx on the host system and using a shared folder with the deployed docker container. However, this leads to file path issues (due to paths looking different when seen from within the container) and also permission issues.
2. Installing sphinx into the deployed container on each deploy
   I do get thinking sphinx to start searchd and build my indexes, however, when I try to make a search query I get: Can't connect to MySQL server on '127.0.0.1'.
   Interestingly rake ts:regenerate worked fine, starting searchd. However rake ts:start fails with "Failed to start searchd daemon. Check /app/log/staging.searchd.log." However, that log file doesn't exist.

Any advice would be very helpful!

I also have a question on SO: http://stackoverflow.com/questions/36336558/how-to-use-thinking-sphinx-with-a-dokku-deployed-rails-app

Nico

[pat]

Hi Nico

I'm not really across Docker nor Dokku - but if you use absolute paths in the configuration file (like in the documentation, even though that was written with capistrano in mind), surely the paths should be reliably consistent? Certainly I highly recommend putting all those files in a shared folder, rather than something release-specific.

When you're seeing "Failed to start searchd daemon" but there's no log files, it's worth trying to start the Sphinx daemon manually as that may provide more useful feedback: searchd --config path/to/env.searchd.conf --pidfile.

Do either of those points help?

[ncri]

Hi Pat,

I tried to use absolut paths, but I cannot get "out" from inside the deployed release (it's in a docker container). Absolut paths inside that container are only absolute to the container itself, so from inside, folders on the host are not accessible. Unless going through a shared folder. But that shared folder's path is different from inside the container than when seen from the host.

When I use option 2 (running sphinx from inside the deployed container) and I try to start searchd from within the release directly, like you suggest, I get this:

using config file '/app/config/staging.sphinx.conf'...
WARNING: compat_sphinxql_magics=1 is deprecated; please update your application and config
listening on 127.0.0.1:9306
precaching index 'apprenticeship_job_core'
WARNING: index 'xyz_core': preload: failed to open /app/shared/db/sphinx/staging/xyz_core.lock: No such file or directory; NOT SERVING
... and 3 more of these warnings ...
precached 4 indexes in 0.004 sec

This is in the log:

[Sun Apr 10 13:30:42.672 2016] [  176] Child process 177 has been forked
[Sun Apr 10 13:30:42.674 2016] [  177] listening on 127.0.0.1:9306
[Sun Apr 10 13:30:42.678 2016] [  177] WARNING: index 'xyz_core': preload: failed to open /app/shared/db/sphinx/staging/xyz_core.lock: No such file or directory; NOT SERVING
... 3 more of those warnings ...
[Sun Apr 10 13:30:42.680 2016] [  177] FATAL: failed to open '/app/shared/tmp/binlog/staging/binlog.lock': 2 'No such file or directory'
[Sun Apr 10 13:30:42.682 2016] [  176] Child process 177 has been finished, exit code 1. Watchdog finishes also. Good bye!

This is the generated config for searchd when I do rake ts:configure:

searchd
{
  listen = 127.0.0.1:9306:mysql41
  log = /app/log/staging.searchd.log
  query_log = /app/shared/staging.searchd.query.log
  pid_file = /app/shared/tmp/searchd.pid
  workers = threads
  binlog_path = /app/shared/binlog
}

[pat]

Do the directories specified exist?

Also, this is perhaps not really part of the solution: but why can you not use absolute paths from within the container's context? Surely nothing that is outside of the container cares about the paths? (Again, I haven't used Docker, so I could very well be missing some critical knowledge here)

[ncri]

Oops, actually /app/shared/tmp/binlog/staging did not exist. I had a binlog folder, but in /app/shared, not in tmp. Looks good now, last line is:

[Sun Apr 10 13:45:22.294 2016] [ 178] accepting connections

Will try now to start using ts:start.

About your question: Inside the container there is no awareness that one is inside a container on another host. It's like the filesystem begins in the container's root.

[ncri]

Awesome, was able to start sphinx with ts:start now. Thanks for the hint.

[pat]

Well that's good progress :)

Is everything operating within the container, though? i.e. you call the rake task inside the container, Sphinx is running inside the container, your Rails app is inside the container. Thus, all within the same file system?

[ncri]

Yes, except the index and those config files are on the host, in the shared folder, as you recommend in your deploy wiki. However, when I try to search from my rails app I still get:

Error connecting to Sphinx via the MySQL protocol. Error connecting to Sphinx via the MySQL protocol. Can't connect to MySQL server on '127.0.0.1' (111)

[pat]

Is there anything you need to do with opening up ports for your app to communicate with Sphinx?

[ncri]

Hm, well, not that i know of. Sphinx runs with the same user the app runs, that looks good too. Running rake ts:regenerate currently. Will take a while. Do I have to restart the app possibly?

[pat]

I'm pretty sure a restart wouldn't change anything… so I'm at a bit of a loss as to what the cause of this new problem is! ¯\_(ツ)_/¯

[ncri]

Yeah, me too. I will continue to work on it, although probably not today. ;-) Thanks for your help so far!

[ncri]

Uhh, have to go out now, so just quickly posting this.
After a redeploy of the container I now get this:

dokku run yousty-stage-ch ts:start
setuidgid: fatal: unable to run ts:start: file does not exist

dokku run yousty-stage-ch ts:stop
setuidgid: fatal: unable to run ts:stop: file does not exist

[ncri]

ooops, forgot the rake ;-)

[pat]

So did you end up getting this working? :)

[ncri]

Unfortunately not yet, but I will have to. :) So far it is more a side project, so I rarely have time to look into it, but I will next week.

[pat]

Rightio :) I'm going to close this issue just so I know it's not an ongoing problem I need to work on, but you're certainly welcome to keep posting feedback and questions here!

[ncri]

Thanks :)

[gingerlime]

@ncri also looking into running sphinx / thinking-sphinx / rails inside docker containers. Curious to hear if you figured the best way to put it all together.

I'm hoping to run sphinx inside a container as well, rather than on the host, but any working setup would be a good starting point. Hope you can share from your experience!

[ncri]

We are working on a Dokku plugin for sphinx, however, it is is on hold currently as we have some higher priority stuff we are working on. Anyway, the plugin is already almost finished I think, so if there is enough interest we could possibly open source what we have done so far.

[gingerlime]

Thanks @ncri ! Anything you can share would be great. I couldn't find anything on running thinking-sphinx with docker.

[ncri]

https:/yousty/dokku-sphinx

The working commands so far are create, start, stop and destroy. Linking is not implemented yet.

[gingerlime]

Thanks for sharing this @ncri.

I'm still a little confused on how things like rake ts:index, rake ts:rebuild work between the rails app and the thinking-sphinx container though... From the little I could gather, those tasks launch the sphinx binary... but if the rails app and sphinx live in different containers - then how does this work then?

[mateusz-useo]

@gingerlime It should actually work through linking. Every plugin creates its own sandbox and makes services available through network ports. So create sets up the environment and starts the service since dokku checks if a service is available after the creation. And Dokku checks that by trying to connect to the container through specified ports. Now, next step is to link the container to the app which means to create a tunnel from plugin to app (app then receives all environment variables that are present inside the plugin container and is able to connect). But I still have to note that I'm not an expert on that field and might be wrong.

[gingerlime]

I'm no expert either, but my understanding is that linking containers simply allow you to reach them via the (overlay?) network. It sets some env vars and dns records (previously /etc/hosts) to allow you to reach one container from the other.

So the link would allow you to connect via the mysql41 network interface. What I don't quite know/sure about is how to trigger those rebuilds/index operations, which as far as I understand happen by calling the sphinx binaries (indexer, searchd etc). If those binaries "live" on a different container, how do you actually run them across containers??

Please forgive me if I'm just showing my ignorance as far as how sphinx/thinking-sphinx/docker work.

[mateusz-useo]

That's a good question, I guess it can be done by writing another command using docker exec and (if possible) running it from within an app, but I haven't tried that yet, maybe there's another, simpler way.

https://docs.docker.com/engine/reference/commandline/exec/

[aivils]

Nobody tells whole story.

My project has Procfile

web:         bundle exec passenger start -p 3000 --max-pool-size 10
worker:      env QUEUES=default,paperclip,mailers bundle exec rake jobs:work
search:      env NODETACH=true bundle exec rake ts:index ts:start

I run application foo in the dokku

dokku apps:create foo

My project deploys fine. Web container cannot reach search container.

Solution:

Crucial is add user-defined network to the docker

docker network create foo-net
dokku docker-options:add foo deploy,run "--network foo-net"

where foo - application title

Project config/thinking_sphinx.yml must be fixed:

production:
  address: <%= if ENV['DYNO'] == 'search.1' then '0.0.0.0' else 'foo.search.1' end %>

Where
ENV['DYNO'] is dokku specific , contains container name from Procfile
'0.0.0.0' - search daemon listens docker subnet not accessible from world
'foo.search.1' - hostname for web server search. name taken from "docker ps"

Now web server container can reach search container via docker built-in subnet

[xtyrrell]

Just to update @aivils's excellent answer:

Using Dokku v0.20, there is a Dokku plugin network, which we can use to connect Sphinx and web application, using Dokku-native networking commands.

First, create a dokku network for your app, and set your app to attach to it post-deploy:

dokku network:create some-app-network
dokku network:set some-app attach-post-deploy some-app-network

Now, from dokku networking docs:

When a container created for a deployment is being attached to a network, [...] a network alias of the pattern APP.PROC_TYPE will be added to all containers

If Sphinx is running on the search process (like in @aivils's Procfile above), this means that any process can connect to Sphinx at the host: some-app.search.1.

Second, set the SPHINX_HOST to this hostname. We can use Dokku's environment variables for this purpose:

dokku config:set --no-restart $APP SPHINX_HOST=$APP.search.1

where $APP is your app name.

Then update your Sphinx config to take in this environment variable:

config/thinking_sphinx.yml:

  address: <%= if ENV['DYNO'] == 'search.1' then '0.0.0.0' else ENV['SPHINX_HOST'] end || '' %>

Hope this helps someone!