Author: naderman

Restarting supervisord processes in parallel

At Private Packagist we manage some of our background processes with Supervisor. It comes with a command line interface supervisorctl which has an option to restart a group of processes. The way this works is by sending the respective signal to all processes, waiting for them to terminate and then starting them all up again. The way we called this so far is as follows:

supervisorctl signal INT packagist:*
supervisorctl restart packagist:*

As a consequence if you have some processes which take a while before they shut down all other process are already shut down and won’t get started up again yet. For us this happens because some processes may be in the middle of some work we don’t want to interrupt. So instead we want all processes to start up again immediately after they are stopped. My solution uses GNU parallel to run supervisorctl restart in parallel for each of our processes:

supervisorctl signal INT packagist:*
supervisorctl avail | cut -f 1 -d " "|grep packagist:| parallel supervisorctl restart {}

Hope this can be of help to someone looking for a solution to the same problem!

Advertisement

Composer: Replace, Conflict & Forks Explained

Recently there has been an increase of cases in which Composer installs a fork of a package instead of the package the user expects. Most frequently these are forks of packages using a “replace” statement in their composer.json. These forks are usually meant for private use only but are still published on Packagist.

Developers: composer update. Automated systems: composer install.

First of all, this behavior is not a security issue in Composer. While the behavior is unintuitive, it will not result in malicious code being used if you use Composer correctly. Most importantly you should only ever run composer update yourself manually on your development machine. You should read its output, verify it installed and updated packages as expected (use --dry-run to check what would happen without installing anything). You should then commit the generated composer.lock file to your version control system. Continous integration, deployment tools and other automated systems should always run composer install. A composer install run will always use the exact packages that were installed by composer update which was used to generate the lock file – no surprises possible.

Replace identifies which original versions a fork is API compliant with

Secondly it is not wrong to publish forks on Packagist. While we discourage the publication of forks intended solely for private use, you are encouraged to publish forks of open source projects which you intend to maintain for others on Packagist. You may use the replace key to specify which versions of the original package your fork is an API compliant replacement for. You should avoid using too generic version constraints (e.g. * or >=1.0.0) with replacements as with all other dependencies.

Dependency Resolution with Replace, Conflict & Provide

After downloading the list of available packages Composer tries to find a set of packages that satisfies all version constraints specified by all dependencies and their dependencies recursively. Package dependencies are specified as a tuple of name and version constraint. Only such packages will be considered that match the name and version constraint, or replace or provide the specified name with a version constraint matching the given version constraint, and do not match a name and version constraint of a conflict specified in any other package about to be installed. This means that a conflict resulting in a package being excluded, alternatives which replace or provide that name will be considered.

Debugging unexpected fork installations

If your dependencies lead to a conflict with a package Composer may decide to install a fork instead which does not have the same conflict. If you notice that an unexpected fork is installed when running composer update you can debug the dependency problem that lead to the fork installation. Use the conflict key in your composer.json to blacklist the fork. Its structure matches that of the require key. If you run composer update again you will now receive an error message explaining the problem, as the fork is no longer available as an alternative solution to the dependency issue.

{
    "require": {
        "my-favorite/framework": "1.0.0",
        "cool/library-which-conflicts": "0.0.1"
    }
    "conflict": {
        "fork-of/my-favorite-framework": "*"
    }
}

Planned changes to improve usability

We understand that Composer’s behaviour regarding forks and package replacements is unintuitive. So I’ve proposed a number of changes yet to be implemented to the handling of replace & provide on our issue tracker at https://github.com/composer/composer/issues/2690.

TL;DR

Replace is not a bug. Don’t run composer update in automated systems. Forks are allowed on Packagist. Don’t be an idiot when publishing a fork. Got an unexpected fork on update? Your dependencies conflict with the original package. Use conflict (syntax like require) in your composer.json to blacklist the fork and see an explanation of the dependency issue.

Need Composer Consulting? Contact me at composer-consulting@forumatic.com.

Composer: Skipping a dependency

Today Henrik Bjørnskov asked me if it was possible to prevent Composer from installing a package which is a dependency of another package you wish to install. The answer is yes, and this is how:

Let’s say we’re installing bakery/blueberry-cupcake 1.0 which depends on farm/blueberries 1.0. However we’d rather just have the cupcake and there’s no alternative cupcake package. So we’ll tell composer there’s an alternative to farm/blueberries 1.0 called imaginary/blueberries which aren’t blueberries at all.

{
    "repositories": [{
        "type": "package",
        "package": {
            "name": "imaginary/blueberries",
            "type": "metapackage",
            "version": "1.0",
            "replace": {"farm/blueberries": "*"}
        }
    }],
    "require": {
        "bakery/blueberry-cupcake": "1.0",
        "imaginary/blueberries": "1.0"
    }
}

Metapackage is a package type which lets composer know that there are no actual source files associated with the package and it exists solely for composer.

After running composer update to update the lock file and install packages composer show --installed informs us that we have successfully installed bakery/blueberry-cupcake with imaginary/blueberries:

$ composer show --installed
bakery/blueberry-cupcake 1.0
imaginary/blueberries 1.0

A simpler solution without a metapackage is available in this situation as well. Your project can directly replace the farm/blueberries package too.

{
    "require": {
        "bakery/blueberry-cupcake": "1.0"
    },
    "replace": {
        "farm/blueberries": "*"
    }

However this solution will lead to problems if you wish to depend on this package from other packages which want to keep the original dependency.

Need Composer Consulting? Contact me at composer-consulting@forumatic.com.