Long running jobs with Laravel Horizon

This post was originally shared on Medium.com

Laravel Horizon is an Official Package for Laravel that sits above and monitors Supervisor and helps to manage jobs queued on a Redis queue.If you’re using Laravel queues and the Redis queue driver, but not using Horizon then check it out here: https://horizon.laravel.com/

Horizon is clever and has several features built-in to manage jobs that fail, run too long and so on. However this built-in failure management, if misconfigured can cause jobs that you expect to run for a long time to fail mid-processing with little in the way of feedback. To say that is tedious is a bit of an understatement and I would hope this to be resolved in the future or at least documented officially.

After some Googling I came across a few issues on Github that also experienced this and I’m sharing my configuration here in case it helps anyone else with their long running jobs.

There are multiple settings for Horizon/Queues that will affect the timeout of your jobs, I’m not sure exactly what the default is but it’s ~90 seconds. If you have a job that needs to run longer than that you’ll need to do some configuration of your setup to ensure it works as expected.

The way I’ve configured the below is to make use of a completely separate queue and Horizon supervisor instance to ensure that jobs that shouldn’t need to take a long time to execute still utilise the existing timeout/retry times.

Our config/horizon.php file looks like this, within our production environment we’ve created a separate supervisor supervisor-long-running that manages our long-running jobs. It utilises a separate queue (one in this case):

'environments' => [
    'production' => [
        'supervisor-1' => [
            'connection' => 'redis',
            'queue' => 'default',
            'balance' => 'simple',
            'processes' => 5,
            'tries' => 3,
            'timeout' => 60 // Timeout after 60 seconds
        ],
        'supervisor-long-running' => [
            'connection' => 'redis-long-running',
            'queue' => 'default_long',
            'balance' => 'simple',
            'processes' => 2,
            'tries' => 2,
            'timeout' => 900 // Timeout after 15 minutes
        ]
    ],
]

The above references a separate redis queue connection redis-long-running this needs to be configured within your config/queue.php file as well:

'connections' => [

    'redis' => [
        'driver' => 'redis',
        'connection' => 'default',
        'queue' => 'default',
        'retry_after' => 90, // Run for 1.5 minutes
        'block_for' => null,
    ],

    'redis-long-running' => [
        'driver' => 'redis',
        'connection' => 'default',
        'queue' => 'default_long',
        'retry_after' => 1200, // Run for max 20 minutes
        'block_for' => null,
    ],

When dispatching your long-running jobs you’ll need to define the queue they are dispatched on to your new long queue: MyLongRunningJob::dispatch()->onQueue('default_long');

And that’s it, your long running jobs should be able to run for 15 minutes before timing out (and being removed) and your normal jobs will continue to timeout after (in this case) 60 seconds, which is usually ample time.

Note: Importantly, in the above you’ll note that our connections are defined to retry_after a longer period than the timeout defined in the Horizon file. The retry_after releases a job back onto the queue after this much time has elapsed, if your job is allowed to run longer than this then you’ll find your jobs run more than once. The timeout will kill your job if it runs longer than it should, this is why retry_after needs to be longer (01–12–2018: this was incorrectly “shorter”, edited to amend) than timeout you want your job to be killed before Horizon attempts to re-process the job — otherwise you could end up with the same job running twice!