Let me start with a bit of a narrative first:
Around a year ago, I released a C#/.NET Core library called Bassoon. I was looking for a cross platform (Windows, OS X, and Linux) audio` playback library for C#, but I couldn’t find one that was suitable. So I did what any normal software developer would do: make your own. Instead of going full C# with it, I opted to take some off the shelf C libraries and use P/Invoke to chat with them. It uses libsndfile for decoding audio formats (sans MP3, but that might change soon). And PortAudio for making speakers make noise.
If you look at the repo’s README’s
Developing
section, you might notice that I’m not telling anyone one to do a
sudo apt install libsndfile libportaudio (or some other package
manager command for another OS). I’m not the biggest fan of baked
dev environments. They can be a pain to reproduce for others. I
like to have my dependencies per project instead of being installed
system wide if I can help it.
The only downside is that you need to
then create some (semi-) automated way for others to set up a dev
environment for the project. E.g. all that “Download package from
here then tar xzf, cd ..., ./configure, make, make install”
nonsense. At first, I tried to make a simple bash script, but that
got kinda ugly pretty quickly. I’m not the best shell programmer
nor am I too fond of the syntax. There was a consideration for
Python too, but I assumed that it could get a bit long and verbose.
I found out about CMake’s
ExternalProject_Add
feature and set off to make one
surely disgusting CMakeLists.txt file. After a lot of
pain and anguish, I got it to work cross platform and generate all of
the native DLLs that I desired. Some things that stick out in my
mind are:
- having to also run the
autogen.shin some cases - needing to rename DLLs on Windows
- finding the correct
./configureoptions to use on OS X
These all reduced the elegance/simplicity. But hey, it works!... Until about a month ago…
While it was still good on Linux to set up a clean build environment. After some updates on OS X, it stopped building. Same for Windows/MSYS2 as well. This has happened before for me with MSYS2 updates (on other projects) and I waslooking for an alternative solution.
C++ specific package managers are a tad bit of a new thing. I remember hearing about Conan and vcpkg when they first dropped. After doing a little research, I opted to use the Microsoft made option. While it was yet another piece of software to install, it seemed quite straightforward and easy to set up. PortAudio and libsndfile was in the repo as well. After testing it could build those libraries for all three platforms (which it did), I was sold on using it instead. There were a few caveats, but well worth it for my situation:
- Dynamic libraries were automatically built on Windows, but I needed to specify 64 bit. It was building 32 bit by default
- For Linux and OS X, static libraries are built by default. If you want the dynamic ones all you have to do is something called overlaying tripplets
-
The generated file names of the
DLLs were not always what I needed them to be. For example, in my
C# code I have
[DllImport(“sndfile”)]to make a P/Invoked function. On Windows, the DLL name must besndfile.dll, Mac OS islibsndfile.dylib, finally Linux islibsndfile.so. On Windows I getlibsndfile-1.dllbuilt by default. Linux nets melibsndfile-shared.so. For these a simple file renaming works. OS X is a bit of a different story:
You see, every operating
system has their own personality quirks. The Apple one is no
exception. When I tried renaming libsndfile-shared.dylib to
libsndfile.dylib, dotnet run crashed saying it couldn’t find
the library. I know that I had all of the path & file locations
correct, as the previous CMake built libraries worked. I was kinda
of stumped...
After setting DYLD_PRINT_LIBRARIES=1 and
trying another run I got a little hint. libsndfile.dylib was
being loaded and then unloaded almost as soon as it was
called:
dyld: loaded: /Users/ben/Desktop/Bassoon/third_party/lib//libsndfile.dylib dyld: unloaded: /Users/ben/Desktop/Bassoon/third_party/lib//libsndfile.dylib
It also should be loading up libogg.dylib, libFLAC.dylib,
libvorbis.dylib, etc. but that wasn’t happening. Looking at the
vcpkg generated libs, running otool -L (OS X’s version of ldd),
I got the reason why things weren’t the way I expected:
$ otool -L *
libFLAC.dylib:
@rpath/libFLAC.dylib (compatibility version 0.0.0, current version 0.0.0)
@rpath/libogg.0.dylib (compatibility version 0.0.0, current version 0.8.4)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.250.1)
libogg.dylib:
@rpath/libogg.0.dylib (compatibility version 0.0.0, current version 0.8.4)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.250.1)
libsndfile.dylib:
@rpath/libsndfile-shared.1.dylib (compatibility version 1.0.0, current version 1.0.29)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.250.1)
@rpath/libogg.0.dylib (compatibility version 0.0.0, current version 0.8.4)
@rpath/libvorbisfile.3.3.7.dylib (compatibility version 3.3.7, current version 0.0.0)
@rpath/libvorbis.0.4.8.dylib (compatibility version 0.4.8, current version 0.0.0)
@rpath/libvorbisenc.2.0.11.dylib (compatibility version 2.0.11, current version 0.0.0)
@rpath/libFLAC.dylib (compatibility version 0.0.0, current version 0.0.0)
libvorbis.dylib:
@rpath/libvorbis.0.4.8.dylib (compatibility version 0.4.8, current version 0.0.0)
@rpath/libogg.0.dylib (compatibility version 0.0.0, current version 0.8.4)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.250.1)
libvorbisenc.dylib:
@rpath/libvorbisenc.2.0.11.dylib (compatibility version 2.0.11, current version 0.0.0)
@rpath/libogg.0.dylib (compatibility version 0.0.0, current version 0.8.4)
@rpath/libvorbis.0.4.8.dylib (compatibility version 0.4.8, current version 0.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.250.1)
libvorbisfile.dylib:
@rpath/libvorbisfile.3.3.7.dylib (compatibility version 3.3.7, current version 0.0.0)
@rpath/libogg.0.dylib (compatibility version 0.0.0, current version 0.8.4)
@rpath/libvorbis.0.4.8.dylib (compatibility version 0.4.8, current version 0.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.250.1)
From this, I was able to identify two problems:
- The “id” of a dylib didn’t
match it’s filename. E.g.
libvorbis.dylib’s id was set tolibvorbisfile.3.3.7.dylib - The dylibs were looking for
non-existent dylibs. E.g.
libvorbisenc.dylibwas looking forlibogg.0.dylib.
As to why this
wasn’t happening with the previously CMake build native libs, it’s
because they were configured/compiled with --disable-rpath. With
vcpkg, I wasn’t able to set this when building libsndfile. The
OS X toolkit does have a utility to fix the rpaths;
install_name_tool:
install_name_tool -id "@rpath/<dylib_file>" <dylib_file>is used to set the id we wantinstall_name_tool -change "@rpath/<bad_dylib_path>" "@rpath/<good_dylib_path>" <dylib_file>can fix an incorrect rpath
Since I wanted the setup process to be
fire and forget I still needed to write a script to automate all of
this. At first I considered bash again, but then I thought “I
don’t want to force someone to install the entire MSYS2 ecosystem
for Windows. What else can I use?...” Python came to mind.
Any developer is bound to have Python on their machine. I know
that’s what I tried to avoid in the first place, but looking at the
built in libraries for Python 3.x (.e.g shutil, subproccess,
pathlib, etc) it was a better choice IMO. I also like the syntax
more; I’ll always trade simple and easy to understand code any
day over something that’s complex and shorter. For an example,
here is how I have the dylibs for OS X fixed up:
To run this 3rd party dependency setup script, all you need to do is set an environment variable telling it where vcpkg is installed and then it will take care of the rest!
Now that all of the native library dependencies have been automated away, the next challenge was packaging them for NuGet. Before, I told my users to “clone the repo and run the CMake setup command yourself”. That wasn’t good for many reasons. A big one being that no one could easily make their own program using Bassoon and easily distribute it. I know that I needed to also have the native libs also put inside the NuGet package, but what to do…
If you search for “nuget packaging
native libraries” into Goggle you get a slew of results telling
you what to do; all of it can seem overwhelming from a quick glance.
“Do I use dotnet pack or nuget pack? Do I need to
make a separate .nuspec file? But wait, dotnet pack does that
for me already… What is a .targets file? What is a
.props file? How many of those do I need? What is this whole
native/libs/* tree structure? Oh man, all that XML looks
complicated and scary. I have no idea what I’m reading.”
Throwing in cross platform native libraries adds a whole other level
of trouble too. Most tutorials are only written for Windows and for
use within Visual Studio. Not my situation, which was all three
major platforms. Even peeking into other cross platform projects
(e.g. SkiaSharp)
to see how they did it, makes it look even more confusing. Too many
configuration files to make sense of.
Then I found NativeLibraryManager.
It has a much more simpler method to solve this
problem: embed your native libraries inside of your generated .NET
DLL and extract them at runtime. I don’t want to copy what it says
in it’s README, so go read that. But I’ll summarize that I only
had to add one line for each native library to the .csproj (for
embedding). Then for extracting at runtime, a little bit of code.
For people who want to use PortAudioSharp or libsndfileSharp
directly, they only need to call the function LoadNativeLibrary()
before doing anything else. And as for the nature of Bassoon’s
initialization, they don’t have to do anything!
I cannot thank @olegtarasov enough for creating this. I’m a programmer. I like to write code; not configuration and settings files.
At the time of writing,
libsndfileSharp package is partially broken for OS X due to a bug
in NativeLibraryManager. But
a ticket has been filed explaining what’s wrong and
most likely what needs to be fixed. It should be good soon :P
If anyone wants to help out with Basson (e.g. adding multi-channel support) or the lower level libraries (adding more bindings to libsndfile and PortAudio), I do all of the development over on GitLab.
I’d like to mention that I’m a little less employed than I would like to be; I need a job. My strongest skills are in C/C++, C#/.NET, Python, Qt, OpenGL, Computer Graphics, game technologies, and low level hardware optimizations. I currently live in the Boston area, so I’m looking for something around there. Or a company that lets me work remotely is fine too. I’m also open to part time, contract, and contract-to-hire situations. If you send me an email about your open positions, I’ll respond with a full resume and portfolio if I’m interested.
Please; I’m the sole provider for a cat who’s love is motivated by food. Kibble ain’t free.
