Parallel Elixir Run

Last time we’ve managed to access a worker from multiple processes. This time, we’ll try to distribute worker’s load across several processes. The problem we’re trying to solve is to calculate occurrence of letters in given list of texts. There is no need for shared state, we just want to distribute the load. Hence, the Agent is not really needed. Task supports exactly what we need, the async call.

It works like this:

  • invoke a job, asynchronously
  • wait for job to finish and return results

And you have the corresponding methods:

Or, taken from the documentation:

task = Task.async(fn -> do_some_work() end)
res  = do_some_other_work()
res + Task.await(task)

Here, the important thing to notice is that you need to await for the result, and that both processes are linked so any crashes on either side will cause the other to fail as well. This suits our purpose quite nicely, so no worries there.

The above example shows how to spawn and await a single process. But, we’d like to spawn more, and distribute an unknown load evenly across those processes / workers, later collecting all of their results. And, funny enough, Enum.map/2 to the rescue!

@spec frequency([String.t], pos_integer) :: Dict.t
def frequency(texts, workers) do
  texts
    |> group_by(workers)
    |> Enum.map(fn(texts_for_a_worker) ->
      Task.async(fn -> count_frequencies(texts_for_a_worker) end)
    end)
    |> Enum.map(&(Task.await(&1, 10000)))
    |> merge_results
end

So, the actual magic happens with first Enum.map/2, which spawns a new worker and delivers it the texts to process and in the second Enum.map/2, which awaits each worker, mapping it’s results. The complete example is a bit more elaborate and can be found at Github.

So how much speed we’ve gained? To get a rough estimate, I’ve created a list of ten thousand short poems (through List.duplicate/2 on some national anthems, I’m not that eager to write poems yet!) and put it through the loops, while changing the number of workers. Here are some very much informal results:

  • 1 worker – 12 seconds
  • 4 workers – 5 seconds
  • 16 workers – 5 seconds
  • 32 workers – 5 seconds
  • 128 workers – 5 seconds

I didn’t go so far as to inspect the reasons behind the top off, that might be an interesting exercise for later time. The take away is that we do have some significant gain compared to a single process, with little more code than usual, so it’s a win in my book 🙂

Advertisements
Tagged , , ,

4 thoughts on “Parallel Elixir Run

  1. José Valim says:

    You probably do not see an improvement beyond 4 because you likely have 4 cores and, because those tasks are CPU bound, you are not going to be any faster then the amount of cores you have. 😀

    • elvanja says:

      Yeah, makes sense, and I do have 4 cores 🙂
      Thank you!

      • ejstembler says:

        I chunk my tasks taking into account the number of processors like this:

        logical_processors = :erlang.system_info(:logical_processors)
        min_simultaneous_exports = Application.get_env(:exposures_report, :min_simultaneous_exports)
        n_and_steps = Enum.max([logical_processors, min_simultaneous_exports])

        Enum.chunk(missing_mams, n_and_steps, n_and_steps, [])

      • elvanja says:

        Mmm, nice one, thanks!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: