Marginalia

Julia: Package Workflow

(Draft, work in progress)

  1. Resources
  2. Starting a package
    1. Where do packages under development reside ?
    2. Starting a package repository
  3. Tests
  4. Documentation
    1. Generation using Documenter.jl
    2. Local preview using LiveServer.jl
  5. Running things on github
  6. Providing packages to others
    1. Using a package under development
    2. Registration of a new package or a new version in a local registry
    3. Using the General registry

Resources

Starting a package

Where do packages under development reside ?

When developing Julia packages it is advisable to set the environment variable JULIA_PKG_DEVDIR to a path convenient for you. By default (if JULIA_DEPOT_PATH was not changed) it is .julia/dev. Julia packages checked out for development via Pkg.develop() will reside there, and it is convenient to have all packages under development in this path.

Starting a package repository

The creating packges chapter of the package manager documentation explains how to create a package with a minimal functional structure. Enabling important aspects of package workflow like continuous integration, documentation generation or git integration would involve the addition of all of this tooling by hand.

The package PkgTemplates.jl provides standard templates for creating Julia packages:

$ cd $JULIA_PKG_DEVDIR
$ julia
julia> using PkgTemplates
julia> t = Template(;
    user="UserName",
    dir=".",
    julia=v"1.10",
    plugins=[
    Git(; manifest=false, ssh=true),
    GitHubActions(),
    Codecov(),
    Documenter{GitHubActions}(),
    ],
    )

julia> t("MyPkg")

This creates a standard Julia package structure enhanced with tooling for CI, documentation, git etc., which is ready to be pushed to a corresponding github repository:

MyPkg
├── .git
│  ...
├── .github
│  ├── dependabot.yml
│  └── workflows
│     ├── CI.yml
│     ├── CompatHelper.yml
│     └── TagBot.yml
├── .gitignore
├── docs
│  ├── make.jl
│  ├── Manifest.toml
│  ├── Project.toml
│  └── src
│     └── index.md
├── LICENSE
├── Manifest.toml
├── Project.toml
├── README.md
├── src
│  └── MyPkg.jl
└── test
   └── runtests.jl

Tests

See also the Pkg documentation.

The files in test provide the environment and the code to run tests of the functions implemented in the package. While it is hard to have 100% test coverage of the package functionality, one should strive to achieve this goal. Besides of the obvious reasons of reproducibility, peer pressure etc., sufficient test coverage helps to keep a package author sane. Imagine the need to change something in the package three years after the last serious work on it...

It is quite possible that tests have more dependencies than the package itself. Therefore Julia packages can have their own test environment in test/Project.toml. Alternatively, the test dependencies can be listed in an "[extras]" section of MyPkg/Project.toml. The test/runtests.jl file contains the code for the tests. All code called from the @test macro is part of the test coverage. A simple file runtests.jl contains e.g.:

using Test

@test 1+1 == 2

Tests code can be called in different ways. In the package root environment:

julia> Pkg.test()

or from the Pkg prompt:

] test

The seame test can be invoked also in any environment with the package added:

julia> Pkg.test("MyPkg")

or

] test MyPkg

Documentation

Generation using Documenter.jl

Documenter collects the docstrings of functions of a package and builds a documentation website. If MyPkg contains code with docstrings like

"""
    mult(x,y)

Calculates the product of `x` and `y`
"""
function mult(x,y)
    return x*y
end

In a documentation markdown file this can be referred to in a "@docs block".

Documentation is built using the script docs/make.jl which needs to be started in the environmet defined in docs:

$ julia --project=docs
julia> include("docs/make.jl")

Local preview using LiveServer.jl

By default, locally built documentation is created in the docs/build subdirectory. Preview can be performed using a local webserver pointing to this directory.

LiveServer.jl provides one:

julia> using LiveServer
julia> serve(dir="docs/build", port=8001)

This starts a web browser pointing to http://localhost:8100, allowing to navigate documentations locally.

Running things on github

For public repositories, github provides unlimited access to Github actions. This means that the scripts under .github/workflows are active and triggered at the events described therein. Among others, the CI.yml triggers running of tests and documentation at every push to the main branch.

The script tries to deploy the documentation to a separate branch which publishes the documentation under https://github.io/UserName/MyPkg. For this purpose, github actions need to authenticate theselves with the repository. For this purpose, a DOCUMENTER_KEY ssh key needs to be stored in the repository. The setup is described in the Documenter.jl documentation.

Alternatively, this can be done for gitlab as well.

Providing packages to others

Using a package under development

While the package is unregistered you can add it to the environment of a project:

$ cd project_dir
$ julia --project=@.
pkg> add git_url

In this case, the version in the git repository on the server is used via a shallow clone (clone without versioning information) created in $HOME/.julia/packages. If you use

pkg> dev git_url

instead, a full clone is created in $JULIA_PKG_DEVDIR and used. If the corresponding subdirectory in JULIA_PKG_DEVDIR does not exist, it will be checked out from git_url. The same would happen with registered packages added by name.

Registration of a new package or a new version in a local registry

A registry is a git repository containing package information which allows to find the package source code given the package name. By default, julia packages are found in the General registry, wich is also installed by default. However, it is possible to have multiple registries, and to maintain your own registry.

The LocalRegistry.jl allows for an easy workflow.

We assume to speak about MyRegistry as an example, but generally this is true for all registries. Assuming write access to the repository containing MyRegistry, the process is the following:

Registring a new version of MyPkg goes as simple as follows:

MyPkg$ julia --project
julia> using MyPkg
julia> using LocalRegistry
julia> LocalRegistry.register(X,"MyRegistry", push=true)

After this, do not forget to create a tag in your repository marking the version x.y.z of the package:

$ git tag vx.y.z

If users want to access MyPkg via this registry, they first need to add the registry to their julia installation:

julia> Pkg.Registry.add("https://github.com/UserName/MyRegistry")

After that, they can add the package as usual:

julia> Pkg.add("MyPackage")

Using the General registry

Registering and updating packages in the general registry is automatically handeled by the JuliaRegistrator github app.

It assumes compliance to a number of heuristic rules concerning package names, Project.toml content etc.

If these are followed, just comment the most recent commit of your package with

@JuliaRegistrator register

and the process starts. JuliaRegistrator checks compliance to the rules, and finally merges the updated information into the General registry

It needs to be installed as a github app in for the package repository.

Update history

© Jürgen Fuhrmann. License: CC-BY-SA 4.0. Last modified: November 14, 2024. Website built with Franklin.jl and the Julia programming language. Upon finding factual errors, typos etc. raise an issue.