Developing Python and Rust projects on NixOS using IntelliJ IDEA and PyCharm

on 2022-08-31 by Olivier under Programming with tags: Nix NixOS Rust Python

Context🔗

I recently gave up on using Ubuntu on my desktop because I needed to repartition and reinstall, but the installer was giving me lots of trouble. It was easier to change distro, so I bit the bullet and gave NixOS a try on my desktop (having started using it again on my server for a few months).

Problem Statement🔗

I need to:

  • be able to develop Python and Rust projects
    • which have special dependencies
  • use the IDEs which I'm used to — IntelliJ IDEA and PyCharm from JetBrains
  • not have any Nix rebuilds inbetween me updating the code and running it
    • (this notably means that my solutions so far for packaging projects using poetry2nix and naersk don't apply here.)

Note: this is not about packaging, purely about development. I've had comfortable success using poetry2nix (Python + Poetry) and naersk (Rust) for packaging my projects as flakes, but that's out of scope in this document.

Tools🔗

nix-shell + shell.nix? nix develop + Flakes?🔗

Some preliminary research pointed to nix-shell and nix develop as being plausible tools for doing this.

  • nix-shell lets you write a shell.nix file in order to provide a bash shell environment.
  • nix develop does the same, except it's experimental and appears to be intended for use with flakes.

I went with nix-shell because it appeared more widely used at the time and because nix develop uses flakes. The problem with using flakes is that they are supposed to be pure — they are not meant to have access to the source code's location on disk. This then means that using them with Python editable installs (which I find useful for development) is not entirely straightforward; you have to opt-in to using impure flakes. See poetry2nix issue 425 where this issue is discussed and notably where we are reminded that flakes are an experimental feature.

direnv (optional)🔗

There's nothing wrong with activating the shell.nix environments using nix-shell by hand, but it can soon become tedious, much like activating Python virtualenvs by hand does.

direnv, a tool which was useful for automatically enabling Python virtualenvs or Poetry shells, can also be used to activate Nix shells. Direnv includes built-in support for Nix, but there are actually at least 6(!) approaches of using Nix with direnv. From some hunch of simplicity, performance and flexibility, I chose to install Nix-direnv (I used the instructions for installing it into my NixOS configuration.nix file).

From now on, I just need to write

use nix

into .envrc in a directory and then simply cding into that directory will activate the shell.nix within.

For Poetry-defined Python projects, I installed the 'Poetry layout' into my direnv configuration, by following the wiki. I can now write:

use nix
layout poetry

into .envrc to both activate the Nix shell and activate the Poetry environment.

Solution (Python)🔗

(Quite minimal for now, so I'll revisit this when I have a more complicated project to develop.)

Minimal example🔗

This is enough to get you able to develop a Poetry-based project and to install pure Python packages.

In the future, it may be looking at whether poetry2nix offers anything more for shell.nix.

{ pkgs ? import <nixpkgs> {} }:

pkgs.mkShell {
  buildInputs = [
    pkgs.python3
    pkgs.poetry
  ];
}

If you need any C libraries or other programs for running your software, adding those as buildInputs seems to work.

I would note that it's not uncommon for packages that depend on dynamic libraries (.so shared object files) to not work, because find_library is not useful on Nix platforms. Nix issue 7307 has more information about this. I believe using poetry2nix would help, because they maintain overrides/patches for some of these cases (and it is likely possible to contribute more) — see this override for the file-magic library for an example.

Solution (Rust)🔗

I will show some shell.nix files that I think are useful. I'll start with a minimal example and then expand by giving something a bit more monstruous that got a more complicated project working.

Minimal example🔗

Just adding pkgs.cargo as a build input ought to be enough to get you able to run cargo build and therefore able to build projects.

You might want to add other tools like clippy or rustfmt (which includes cargo fmt); just add these to your buildInputs as shown.

shell.nix:

{ pkgs ? import <nixpkgs> {} }:

pkgs.mkShell {
  buildInputs = [
    pkgs.cargo
    pkgs.clippy
    pkgs.rustfmt
  ];
}

Monstruous example🔗

I use this monstruous example for developing a Tauri-based app. Notable points I had to bear in mind:

  • I need to add a package (cargo-tauri) from the unstable channel of Nixpkgs. This is because it's not in the current stable channel (22.05). This won't be a problem for you if you're not developing with Tauri, but the general idea may help you out of another pickle, so I include it here anyway.
    • I first had to add the unstable channel using nix-channel --add https://nixos.org/channels/nixpkgs-unstable nixpkgs-unstable. I then added export NIX_PATH=$NIX_PATH:$HOME/.nix-defexpr/channels at the end of my ~/.bashrc, according to nix issue 2033, to make <nixpkgs-unstable> usable in my shell.nix file.
  • I use some libraries that make use of bindgen to bind to C libraries (avahi in my case here). This can be challenging because it uses libclang and it needs to be able to access the source headers for the library you're binding to, as well as the standard C library.
    • I cargo culted and adapted some flags from the NixOS wiki's page on Rust — notably LIBCLANG_PATH and BINDGEN_EXTRA_CLANG_ARGS.
    • Having pkgs.pkg-config as a buildInputs input is useful if you need to bind to any libraries, as well as including the libraries themselves as nativeBuildInputs. Bear in mind that NixOS/Nixpkgs doesn't always make it obvious which package includes a given library...
  • I needed yarn for the JavaScript portion of my Tauri-based app. It's unlikely to be useful for you if you don't have any JavaScript in your project :-).

shell.nix:

{ pkgs ? import <nixpkgs> {} }:

let
  # We need some packages from nixpkgs-unstable
  unstable = import <nixpkgs-unstable> {};
in

pkgs.mkShell {

  buildInputs = [
    pkgs.cargo

    # not in 22.05 pkgs.cargo-tauri
    unstable.cargo-tauri

    pkgs.pkg-config
    pkgs.yarn
  ];

  nativeBuildInputs = [
    pkgs.dbus
    pkgs.openssl
    pkgs.avahi
    pkgs.gobject-introspection
    pkgs.gtk3 # for gdk3 (!) gdk-pixbuf does not do
    pkgs.libsoup
    pkgs.webkitgtk # javascriptcore-gtk4 ..
  ];

  # Needed for bindgen when binding to avahi
  LIBCLANG_PATH="${pkgs.llvmPackages_latest.libclang.lib}/lib";

  # Cargo culted:
  # Add to rustc search path (I didn't need!)
  #RUSTFLAGS = (builtins.map (a: ''-L ${a}/lib'') [
  #]);
  # Add to bindgen search path
  BINDGEN_EXTRA_CLANG_ARGS =
    # Includes with normal include path
    (builtins.map (a: ''-I"${a}/include"'') [
      pkgs.avahi
      pkgs.glibc.dev # very important otherwise the #include_nexts fail in clang's headers
    ])
    # Includes with special directory paths
    ++ [
      ''-I"${pkgs.llvmPackages_latest.libclang.lib}/lib/clang/${pkgs.llvmPackages_latest.libclang.version}/include"''
    ];
}

For searchability, I will note that adding pkgs.glibc.dev to the bindgen includes list is what solved the following confusing error:

/nix/store/yq5jvp7dvgxfcna6zmsbas24ns099ld2-clang-13.0.1-lib/lib/clang/13.0.1/include/inttypes.h:21:15: fatal error: 'inttypes.h' file not found, err: true

(This error occurs because clang's inttypes.h uses #include_next to fall back to/extend the normal inttypes.h file, so you still need the one from the standard C library.)

Using JetBrains IDEs (IntelliJ IDEA, PyCharm)🔗

nix-shell is all well and good, but I use an IDE! I need to make the tools (including compilers and libraries required to build my project) available to my IDE!

There are a few options here:

  • launch the IDE from the directory, once the shell has been activated (either manually or automatically using direnv).
    • After installing the IDE into my NixOS configuration.nix, this just means typing pycharm-community or idea-community in the right shell! The IDE then is capable of using the provided development tools. The IDE also seems to directly open the current project, so it's not too inconvenient.
  • (untested) use an out-of-date, unmaintained IDE extension that can import the environment from shell.nix: Enter Nix Shell
    • the extension repository says this is incompatible with the latest versions of the JetBrains IDEs, so I haven't tried it.
    • It may or may not be fine if you can force install it; though experience suggests that IDE extensions rot fairly quickly so I wouldn't get my hopes up.
  • (untested) use an IDE extension that can import the environment from direnv: intellij-direnv
    • Of course, this means you need to have your direnv setup to use nix-shell (as described above) too.
    • I haven't tested this yet, but it looks actively maintained — latest commit within this month (at time of writing).

I don't yet know how this all works if you want to open multiple projects in an IDE; I'm not sure if they will open separate processes or not — presumably you'd need to if you wanted each project to have its own environment.

Remarks🔗

These steps worked on the publication date, 2022-08-31. If they're broken, clarification is needed, or there are other worthwhile resources including on this page, please feel free to send me a message — see the 'Contact' section of my website.