Marginalia

Julia: Package Workflow

(Draft)

  1. Intro
  2. Starting a package
    1. Starting a package as part of an application project
    2. Starting a package repository
  3. Tests
  4. Continuous Integration
  5. Providing
    1. Using a package under development
    2. Registration of a new package or a new version in a local registry
    3. Without write access to the registry
    4. With write access to this repository
    5. Using the General registry
    6. Naming and versioning
  6. Other useful introductions

Intro

This post is work in progress. May be one of the other useful introductions will be more useful at the moment.

For general terminology, see the package manager glossary. According to this, here, we assume that a package is Julia code which is supposed to be used by other projects, applications or packages.

When developing Julia packages it is advisable to set the environment variable JULIA_PKG_DEVDIR to a reasonable path. All Julia packages under development will reside there, and for a particular package, we refer to its corresponding package_dir during this post.

Starting a package

See also the Pkg documentation

Starting a package as part of an application project

Often, packages evolve during the development of larger application, when it becomes clear that some part of it provides some well defined functionality which can be formulated with its own API. Local packages within an application project allow to start the package development process in such a situation. For this structure, see my previous post.

Starting a package repository

Typically, each Julia package resides in its own git repository (other configurations are possible).

Create an empty repository NewPackageName.jl on a server with git_url

$ cd $JULIA_PKG_DEVDIR
$ julia
pkg> generate NewPackageName
julia> exit()
cd NewPackageName
git init
git add remote origin git_url
git add .

Julia packages have a fixed structure which includes

A Manifest.toml file is created as soon as the package environment is instantiated. It contains the full metadata of all package dependencies. This file should not be tracked by git. Instead, the Project.toml shall contain a [compat] section which inform the package resolver about the compatibility with different versions of dependencies.

Instead of invoking pkg> generate NewPackageName, a more comprehensive initial package structure can be be created by cloning Example.jl or with PkgTemplates.jl. These also create the necessary files for controlling automatic CI tests on github/gitlab.

Tests

See also the Pkg documentation.

The files in test provide the environment and the code to run tests of the functions implemented in the package. While it is hard to have 100% test coverage of the package functionality, one should strive to achieve this goal. Besides of the obvious reasons of reproducibility, peer pressure etc., sufficient test coverage helps to keep a package author sane. Imagine the need to change something in the package three years after the last serious work on it...

It is quite possible that tests have more dependencies than the package itself. Therefore Julia packages can have their own test environment in test/Project.toml. The runtests.jl file contains the code for the tests. All code called from the @test macro is part of the test coverage. A simple file runtests.jl contains e.g.:

using Test

@test 1+1 == 2

Tests code can be called in different ways. In the package root environment:

julia> Pkg.test()

or from the Pkg prompt:

] test

The seame test can be invoked also in any environment with the package added: 

julia> Pkg.test("MyPackageName")

or

] test MyPackageName

Continuous Integration

Your package tests sucessfully run on your linux laptop. But how can you be sure that they also run on Mac and Windows ? And, you want to show off with a good test coverage helping to build up trust in your work.

The solution ? Run your tests on different systems via github actions and show off the result.

In order to achieve this, it is sufficient to add a file

.github/workflows/ci.yml

Alternatively, this can be done on gitlab as well.

Providing

Using a package under development

While the package is unregistered you can add it to the environment of a project:

$ cd project_dir
$ julia --project=@.
pkg> add git_url

In this case, the version in the git repository on the server is used via a shallow clone (clone without versioning information) created in $HOME/.julia/packages. If you use

pkg> dev git_url

instead, a full clone is created in $JULIA_PKG_DEVDIR and used. If the corresponding subdirectory in JULIA_PKG_DEVDIR does not exist, it will be checked out from git_url. The same would happen with registered packages added by name.

Registration of a new package or a new version in a local registry

We assume to speak about MyRegistry as an example, but generally this is true for all registries.

Without write access to the registry

Send an email to one of the admins containing the git url of the corresponding repository singnaling the advance of a new version. The Project.toml file already should contain the new version number.

With write access to this repository

Add the LocalRegistry.jl package to your environment

pkg>  add LocalRegistry

For a new package X.jl or a new version thereof, first obtain a clone, either by checking it out via its url, or by adding it to the environment, getting into develop mode and updating it:

pkg> add X
pkg> develop X
pkg> update X # not clear if really necessary

Then, you will be able to register a new version based on the updated version number in Project.toml:

julia> using X
julia> using LocalRegistry
julia> LocalRegistry.register(X,"MyRegistry",push=true)

This assumes that the remote origin of the local clone in .julia/registries/MyRegistry has been modified to git@github.com:me/MyRegistry. The push=true can be omitted, in this case, LocalRegistry.Register results in a commit to the local copy which you can push later.

After this, do not forget to create a tag in your repository marking the version x.y.z of the package:

$ git tag vx.y.z

Using the General registry

Registring packages in the general registry assumes full compliance to a number of rules:

All of this is automatically handeled by the JuliaRegistrator

It needs to be installed as a github app in for the package repository.

Naming and versioning

Other useful introductions

Update history

© Jürgen Fuhrmann. License: CC-BY-SA 4.0. Last modified: October 17, 2023. Website built with Franklin.jl and the Julia programming language. Upon finding factual errors, typos etc. raise an issue.