Integrating Guzzle 6 Asynchronous Requests with ReactPHP

So, I built a slack bot nopolabs/yabot in php.

Yabot’s basic architecture looks like this:

Yabot opens a websocket connection to Slack’s RTM API and then runs a React event loop that receives messages and dispatches them to plugins.

The event loop is single threaded and plugins handle messages (or not) as they please. This means that if a plugin blocks on an operation the whole event loop comes to a standstill. Now I’m not the first person to have run into this, in fact the author of sagebind/slack-client, the Slack client library used by Yabot, wrote a blog post about it.

The gist of the matter is that Guzzle’s CurlMultiHandler has a tick() method that will check for completed requests and resolve their promises.

So I wrapped GuzzleHttp\Client in a class: Guzzle.php that creates a timer to call tick() periodically:

private function scheduleProcessing()
{
    if ($this->timer === null) {
        $self = & $this;
        $this->timer = $this->eventloop->addPeriodicTimer(0, \Closure::bind(function() use (&$self) {
            $this->tick();
            // Stop the timer when there are no more requests
            if (empty($this->handles) && queue()->isEmpty()) {
                $self->timer->cancel();
                $self->timer = null;
            }
        }, $this->handler, $this->handler));
    }
}

That’s pretty cool, but GuzzleHttp\Client accepts a handler stack as an option in the constructor, and CurlMultiHandler accepts a CurlFactory as an option in the constructor, so I wrote a library: nopolabs/react-aware-guzzle-client

This is how it is used:

public function newClient(
    LoopInterface $eventLoop,
    array $config = [],
    CurlFactory $curlFactory = null,
    LoggerInterface $logger = null) : Client
{
    $clientFactory = new ReactAwareGuzzleClientFactory();
    return $clientFactory->createGuzzleClient($eventLoop, $config, $curlFactory, $logger);
}

Posted in Uncategorized | Comments Off on Integrating Guzzle 6 Asynchronous Requests with ReactPHP

xdebug bash script improved

I use a bash script to help me manage enabling and disabling xdebug (gist).

This version works on both my OSX laptop and my Centos virtual machine.

When using the script to toggle xdebug on or off you need to source it in order for it to be able to update XDEBUG_CONFIG in your current shell environment.

The function restart-services is called to restart php-fpm when xdebug is toggled on or off so that it will use the updated php.ini file.

Usage:
    source xdebug --on   Enable XDebug in php.ini file.
    source xdebug --off  Disable XDebug in php.ini file.
    xdebug --status      Report if XDebug is enabled or disabled.
    xdebug php-file      Run a php script with XDebug enabled.
#!/bin/bash

PHP=${PHP:-`which php`}
XDEBUG_INI=${XDEBUG_INI:-`${PHP} --ini | awk 'BEGIN { FS=","; } /xdebug.ini/ { print $1; }'`}
XDEBUG_HOST=${XDEBUG_HOST:-1.2.3.1}
XDEBUG_PORT=${XDEBUG_PORT:-9001}
IDE_KEY=${IDE_KEY:-PHPSTORM}

if [ "$1" = "--help" -o "$1" = "" ]
then
    echo "Usage:"
    echo "  source xdebug --on   Enable XDebug in php.ini file."
    echo "  source xdebug --off  Disable XDebug in php.ini file."
    echo "  xdebug --status      Report if XDebug is enabled or disabled and if ."
    echo "  xdebug php-file      Run a php script with XDebug enabled."
    exit 0
fi

OS=`uname`
if [ "$OS" = "Darwin" ]
then
    SED=(sed -i "''")
else
    SED=(sed -i)
fi

restart-services() {
    echo "Restarting services ..."
    if [ "$OS" = "Darwin" ]
    then
        valet restart
    else
        service php-fpm restart
    fi
}

status() {
    grep "^zend_extension" ${XDEBUG_INI} &> /dev/null
    if [ "$?" -eq 0 ]
    then
        echo XDebug enabled in ${XDEBUG_INI}
        echo XDEBUG_CONFIG=${XDEBUG_CONFIG}
    else
        echo XDebug disabled in ${XDEBUG_INI}
        echo XDEBUG_CONFIG=${XDEBUG_CONFIG}
    fi
}

check() {
    # check for a process is listening at host:port
    nc -z ${XDEBUG_HOST} ${XDEBUG_PORT} &> /dev/null

    if [ "$?" -ne 0 ]
    then
        echo "No listener at ${XDEBUG_HOST} ${XDEBUG_PORT}"
        exit 1
    fi

    echo "Found listener at ${XDEBUG_HOST} ${XDEBUG_PORT}"
}

if [ "$1" = "--on" ]
then
    ${SED[@]} 's/;zend_extension/zend_extension/' ${XDEBUG_INI}
    export XDEBUG_CONFIG="idekey=${IDE_KEY} remote_host=${XDEBUG_HOST} remote_port=${XDEBUG_PORT}"
    status
    check
    restart-services

elif [ "$1" = "--off" ]
then
    ${SED[@]} 's/^zend_extension/;zend_extension/' ${XDEBUG_INI}
    unset XDEBUG_CONFIG
    status
    restart-services

elif [ "$1" = "--status" ]
then
    status
    check

else
    check

    XDEBUG_SO=$(awk 'BEGIN { FS="="; } /^;?zend_extension/ { print $NF; }' ${XDEBUG_INI})

    XDEBUG_CONFIG="idekey=${IDE_KEY} remote_host=${XDEBUG_HOST} remote_port=${XDEBUG_PORT}" \
        ${PHP} \
            -didekey=${IDE_KEY} \
            -dzend_extension=${XDEBUG_SO} \
            -dxdebug.remote_enable=1 \
            -dxdebug.remote_mode=req \
            -dxdebug.remote_port=${XDEBUG_PORT} \
            -dxdebug.remote_host=${XDEBUG_HOST} \
              "$@"
fi
Posted in Uncategorized | Comments Off on xdebug bash script improved

A bash script to run php commands with XDebug enabled.

I’m a big fan of XDebug. But it really slows down the PHP native tools I frequently use: Composer, Symfony console commands, phpunit, etc. So, I don’t leave it installed all the time on my development system. Instead I use this bash script to enable XDebug just when I want to debug:

#!/bin/bash
(
    PHP=${PHP:-/usr/bin/php}
    XDEBUG_SO=${XDEBUG_SO:-/usr/lib64/php/modules/xdebug.so}
    IDE_KEY=${IDE_KEY:-PHPSTORM}
    XDEBUG_HOST=${XDEBUG_HOST:-1.2.3.1}
    XDEBUG_PORT=${XDEBUG_PORT:-9001}

    nc -z ${XDEBUG_HOST} ${XDEBUG_PORT}

    if [ "$?" -ne 0 ]
    then
    echo "No listener at ${XDEBUG_HOST} ${XDEBUG_PORT}"
        exit
    fi

    export XDEBUG_CONFIG="idekey=${IDE_KEY} remote_host=${XDEBUG_HOST} remote_port=${XDEBUG_PORT}"

    ${PHP} \
        -didekey=${IDE_KEY} \
        -dzend_extension=${XDEBUG_SO} \
        -dxdebug.remote_enable=1 \
        -dxdebug.remote_mode=req \
        -dxdebug.remote_port=${XDEBUG_PORT} \
        -dxdebug.remote_host=${XDEBUG_HOST} \
          "$@"
)

For example:

xdebug yabot.php

The script also uses netcat (nc) to check that I have something listening at XDEBUG_HOST:XDEBUG_PORT and warns if I haven’t started my debug client (e.g. phpstorm) listening for connections.

There is one case where this approach doesn’t work: when php is being run by a web server (e.g. httpd or php-fpm).

My earlier blog post Laravel, Valet, and XDebug shows a quick work around for that case..

Posted in Uncategorized | Comments Off on A bash script to run php commands with XDebug enabled.

Laravel, Valet, and XDebug

I’ve been exploring Laravel. Valet made it easy to get up and running quickly on my MacBook and I wrote a simple RESTful api (reminder).

Then I wanted to debug one of the endpoints…

Cool, that’s a job for XDebug. XDebug is an important part of my php toolset but I don’t leave it installed all the time because it can really slow down other php based tools that I use, like Composer. I have a couple of functions in my .zshrc that let me enable and disable XDebug as needed:

xdon() {
 sed -i '' 's/;zend_extension/zend_extension/' /usr/local/etc/php/7.1/conf.d/ext-xdebug.ini
 grep zend_extension /usr/local/etc/php/7.1/conf.d/ext-xdebug.ini
 export XDEBUG_CONFIG="idekey=PHPSTORM remote_host=127.0.0.1 remote_port=9001"
}

xdoff() {
 sed -i '' 's/^zend_extension/;zend_extension/' /usr/local/etc/php/7.1/conf.d/ext-xdebug.ini
 grep zend_extension /usr/local/etc/php/7.1/conf.d/ext-xdebug.ini
 unset XDEBUG_CONFIG
}

I’m not going to go in depth about what these functions do (that’s another blog post). But I expected to be able to set a breakpoint in PhpStorm and then, in a shell, type:

xdon
curl 'http://reminder.dev/api?XDEBUG_SESSION_START=PHPSTORM'

Nope, didn’t work. So I did the dance of checking all the things:

  • PhpStorm listening for debug? check
  • Using port 9001? check
  • PhpStorm directory mappings? check
  • search the web… this

Long story short: Valet sets your Mac up to be an nginx webserver delegating web requests to php-fpm. But php-fpm read my php.ini config files long before shell command edited them to enable XDebug. Quick solution:

valet restart

And I’m a happy man 🙂

Posted in Uncategorized | Comments Off on Laravel, Valet, and XDebug

Making Latkes

I spend my days making great software with the team at OpenSky.com. At the end of the day the value of what we do is not measured by how many tasks we complete, how many lines of code we write, or even how many features we deliver, but by happy customers and successful merchants. On the last day of Hannukah and I found myself making latkes and thinking about the parallels between my experience as a cook and as a software developer.

Latkes are little potato pancakes, fried in oil and served with applesauce and sour cream. They are really tasty hot from the pan, but you can also make latkes ahead of time and reheat them so they can be served all at once. My plan was to start cooking at noon and to be done by 2pm so that I would have time to go for a swim and still arrive on time at the party with my latkes at 4pm.

When I set to work at noon I decided to make the latkes in three batches because my mixing bowl wasn’t big enough to hold all the potatoes at once. This worked out serendipitously because when 2pm rolled around I had only just finished the second batch. I could have soldiered on to make the third batch and arrived at the party tired and grumpy at having missed my swim, but I realized that I had made 24 latkes and that was enough. Since I hadn’t grated the last batch of potatoes I skipped making the third batch completely. Aha! I thought, this is the value of delivering work in smaller batches: it makes it possible to meet my schedule by reducing my scope.

My pan is big enough to fit four latkes at a time and in an effort to maximize throughput I started out frying the latkes four at a time. I quickly learned that when I did that quality suffered for two reasons: 1) the temperature of the oil dropped – the latkes did not cook as quickly and soaked up more oil – the recipes warned of this, but apparently I have to learn by doing, and 2) having only 3 latkes in the pan left me some room to maneuver – I could slide the pancakes around to brown them more evenly. Maybe this is why I only finished two batches instead of three, but I focused on the value of the delivered product: two dozen crispy delicious potato pancakes – they were a hit 🙂

Lots of people contributed to the party – each did their bit, and it all came together on schedule for a great evening.

Happy making!

Using scala implicits to make my code more readable.

Scala implicits can be very frustrating to new users of the language, they were for me.

Their operation seems magical because the compiler chooses when and where to apply them.

A simple use case for implicits is providing type conversions that would otherwise cause a lot of boilerplate code clutter.

Here’s an example adapted from some tests I wrote to check payment calculations for an e-commerce web site.

Payments are represented by a case class like this:

case class Payment(date: DateTime, amount: BigDecimal, note: Option[String])

Here’s how I created a testPayment to compare with my calculations:

val date = ISODateTimeFormat.date.parseDateTime("2014-04-01")
val testPayment = Payment(date, BigDecimal(100.00), Some("comment"))

There’s a lot of boilerplate to get the right types.

I cleaned things up a bit by introducing some implicits in the body of my test:

  
implicit def String2DateTime(value: String): DateTime = ISODateTimeFormat.date.parseDateTime(value)
implicit def String2OptionString(value: String): Option[String] = Some(value)
implicit def Double2BigDecimal(value: Double): BigDecimal = BigDecimal(value)

Once these implicits were in scope my testPayment creation became much easier to read:

val testPayment = Payment("2014-04-01", 100.00, "comment")

Posted in Uncategorized | Comments Off on Using scala implicits to make my code more readable.

Riak 1.3.0 dev cluster on Ubuntu 12.x

Riak 1.3.0 is out and I decided that I would rather have my dev cluster on a stable linux box rather than an OS/x laptop.

I took https://github.com/xing/riak-dev-cluster as a starting point. This repo provides a way to install a riak dev cluster on OS/x.

My modifications to build and run a dev cluster on Ubuntu 12.x are at https://github.com/nopolabs/riak-dev-cluster

I added a couple rake tasks:
rake bind_all
to bind to all IPs (0.0.0.0) and
rake bind_local
to bind to localhost (127.0.0.1), convenient for me since I still want to do client development on my laptop.

I needed to edit the app.config files for the riak servers to change the IP bindings. These files contain Erlang terms, and I found a script that could be used to update them (http://adamschepis.com/blog/2011/08/25/a-script-to-update-riak-config-files/). But I ended up using sed and regular expressions to get the job done. Learning Erlang is still somewhere in the future for me 😉

You might find https://github.com/hectcastro/vagrant-riak-cluster an interesting alternative. It uses Vagrant to a riak cluster on virtualboxes.

You will need to have an erlang development installed to build riak:

sudo apt-get install erlang erlang-base erlang-tools erlang-appmon erlang-pman erlang-observer erlang-ic erlang-parsetools erlang-toolbar erlang-et erlang-mnesia erlang-debugger erlang-percept

Posted in Uncategorized | Comments Off on Riak 1.3.0 dev cluster on Ubuntu 12.x