Faster PHP apps using Queues and Workers
•
4 j'aime•8,052 vues
PHP apps typically perform tasks in a synchronous manner; Resizing an image or sending a push notification. For most applications this works well, but as apps grow or experience increased traffic, each task adds extra milliseconds to a request, leaving users waiting. A common solution is to defer these tasks to the background using a cron task. However, there is a better way. Job queues not only help to decouple your application and improve resilience but will also cut request times. In this talk I we’ll explore some common queue systems; the features and tradeoffs of each solution, what to queue, refactoring existing code into jobs, and running workers. By the end you’ll be ready to build your next app one job at a time.
Recommandé
Contenu connexe
Tendances
Tendances (20)
En vedette
En vedette (20)
Similaire à Faster PHP apps using Queues and Workers
Similaire à Faster PHP apps using Queues and Workers (20)
Dernier
Dernier (12)
Faster PHP apps using Queues and Workers
- 1. QUEUES & WORKERS FASTER PHP APPS USING PHP DORSET 6/6/16
- 4. BACKGROUND MOBILE API ▸ 140+ apps ▸ ~4.5m users ▸ REST/JSON ▸ Apache + PHP + ElasticSearch ▸ 20+ servers / various tasks ▸ Hosted on AWS
- 7. ELIMINATE EXPENSIVE, TIME CONSUMING OPERATIONS AND CALLS TO EXTERNAL SERVICES
- 8. USE MYSQL
- 9. WHAT TO QUEUE MYSQL AS A QUEUE job_id job_name data status 1 sendEmail {payload} completed 2 sendPush {payload} processing 3 requestThing {payload} waiting 4 resizeSelfie {payload} waiting
- 10. WHAT TO QUEUE WHY NOT USE TRANSACTIONS TO OBTAIN A LOCK?
- 11. THE DATABASE IS NOT A QUEUE. THE DATABASE IS NOT A QUEUE. Stephen Corona WHAT TO QUEUE https://www.scalingphpbook.com
- 12. USE A QUEUE!
- 13. QUEUES & WORKERS MESSAGE QUEUES VS JOB QUEUES ▸ Kind of similar ▸ Most job queues built upon some kind of message queue ▸ Broker messages between systems ▸ Provide transport, storage and protocol ▸ Job queues abstract the lower level message component ▸ Integrate with most applications fairly easily
- 14. QUEUES & WORKERS JOBS VS SCHEDULED TASKS ▸ Run at a predefined point in time ⏰ ▸ May repeat at regular intervals or according to calendar 📅 ▸ Typically triggered by cron or other scheduler 🔫 ▸ Scheduled tasks can trigger jobs! 💥
- 15. CHOOSE A QUEUE WITH FEATURES BEST SUITED TO YOUR APPLICATION
- 16. CHOOSING A JOB QUEUE CONSIDERATIONS ▸ Job priority & time sensitivity ▸ Job ordering and consistency (FIFO) ▸ Payload size limits ▸ Message data type & protocol ▸ Support for other languages / client libraries ▸ Failure management / retry policy ▸ Fault tolerance & redundancy ▸ One-time delivery guarantee ▸ Monitoring & statistics ▸ Distribution by task to specific workers e.g. Video encoding
- 17. CHOOSING A JOB QUEUE BEANSTALKD ▸ Protocol similar to Memcached ▸ Clients need to know about all Beanstalkd servers (like memcached!) ▸ Beanstalkd servers can persist jobs, handle restarts without losing jobs ▸ Uses “tubes” to differentiate different queues ▸ Supports job TTR. Failed/hung jobs get put back into queue. ▸ Supports blocking. Client connects and waits for a new job. ▸ Requires setup and maintenance ▸ Loads of client libraries http://kr.github.io/beanstalkd/
- 18. CHOOSING A JOB QUEUE AMAZON SQS ▸ SAAS - Already using AWS, literally no setup ▸ Massively redundant, cheap, maintenance free ▸ HTTP/JSON under the hood, simple ▸ Supports long-polling. ▸ Best effort FIFO (no guarantees) ▸ No concept of job priority. Use different queues. ▸ Retry policy allows jobs to reappear in queue if not completed in specified time ▸ Configurable number of retries ▸ Queue stats and alarms integrate with autoscaling ▸ Scale worker instances based on queue length/backlog/rate https://aws.amazon.com/sqs/
- 19. CHOOSING A JOB QUEUE OTHER POPULAR QUEUES ▸ Celery (backed by RabbitMQ) - http://www.celeryproject.org ▸ php-resque (backed by Redis) -https://github.com/ chrisboulton/php-resque ▸ Kafka - http://kafka.apache.org ▸ Gearman - http://gearman.org ▸ Iron.io (SAAS) - https://www.iron.io ▸ Loads more - www.queues.io
- 20. PROCESSING JOBS
- 21. PROCESSING JOBS WORKER PROCESS ▸ Essentially an infinite loop ▸ Executed on command line ▸ Asks queue for new job ▸ Resolves job method ▸ Execute with payload ▸ Delete job from queue ▸ Repeat
- 22. PROCESSING JOBS <?php $queue = new Queue(); while(true) { $job = $queue->pop('queue-name'); try { if ($job->execute()) { $job->delete(); } else { $job->release(); } } catch (Exception $e) { $job->release(); } }
- 25. PROCESSING JOBS PROCESS CONTROL EXTENSIONS (PCNTL) ▸ Respond to unix process signals ▸ Gracefully stop worker processes ▸ Complete current job before exiting ▸ Careful if using Apache mod_php on same server ▸ http://php.net/manual/en/book.pcntl.php
- 26. IMPROVED WORKER <?php namespace Demo; use DemoQueueQueueInterface; class Worker { protected $shouldRun = true; protected $queue; public function __construct(QueueInterface $queue) { declare(ticks = 1); $this->queue = $queue; pcntl_signal(SIGTERM, [$this, 'signalHandler']); pcntl_signal(SIGINT, [$this, 'signalHandler']); pcntl_signal(SIGQUIT, [$this, 'signalHandler']); }
- 27. public function run($queueName) { echo "Starting worker on queue '{$queueName}' n"; while ($this->shouldRun) { $job = $this->queue->pop($queueName); try { if ($job->execute()) { $job->delete(); } else { $job->release(); } } catch (Exception $e) { $job->release(); error_log($e->getTraceAsString()); } pcntl_signal_dispatch(); } } public function signalHandler($signal) { switch ($signal) { case SIGTERM: case SIGINT: case SIGQUIT: echo "Job completed. Exiting... n"; $this->shouldRun = false; break; } } }
- 28. IMPROVED WORKER WTF IS DECLARE(TICKS = 1);? ▸ Officially deprecated ▸ Triggered after php has executed a certain number of statements ▸ Interacts with pcntl_signal_dispatch() ▸ I admit i’ve not fully tested this with PHP7.0
- 29. PROCESSING JOBS KEEPING WORKERS RUNNING
- 30. PROCESSING JOBS SUPERVISOR ▸ Process manager in similar vein to forever, pm2, php-fpm ▸ Runs as service ▸ Starts and restarts php worker processes ▸ Has CLI client (supervisorctl) ▸ Web interface ▸ Easy to install and configure http://supervisord.org
- 31. PROCESSING JOBS SUPERVISOR CONFIG [program:alertworker] command = /usr/bin/php /path/to/queueRunner.php -q=prod-alerts autorestart = true autostart = true directory = /path/to/scripts environment = DEPLOYMENT='production' exitcodes = 0,2 numprocs = 1 numprocs_start = 0 priority = 999 startretries = 3 startsecs = 4 stderr_capture_maxbytes = 1MB stderr_events_enabled = false stderr_logfile = AUTO stderr_logfile_backups = 10 stderr_logfile_maxbytes = 50MB stderr_syslog = false stdout_capture_maxbytes = 1MB stdout_events_enabled = true stdout_logfile = AUTO stdout_logfile_backups = 10 stdout_logfile_maxbytes = 40MB stdout_syslog = false stopsignal = TERM stopwaitsecs = 10 umask = 022 user = worker
- 32. DEMO TIME