I've always been wanting to run applications on a ESXi console.
These ambitions grew as I started learning more and more about AWS over the last 10 years.
I've dreamt of building an environment that I could use to learn about how AWS does what it does.
vim-cmd was my catalyst.
This was the key to being able to automate VM deployments within my homelab.
It's annoying to SSH in to call this command. I've locked myself out of my ESXi plenty of times because X number of wrong password attempts via SSH.
I needed something else…
Go binaries are statically built (with exceptions) that gave me hope of running applications within the ESXi console.
It's been a long time ago, but my small experiments never resulted in anything runnable within the console. I ended up giving up on
Go for this.
Learning and contributing to https://github.com/josenk/terraform-provider-esxi was something that got me closer to my goal, but was too far away from the metal for my liking. This filled my need for 95% of the things that I wanted to do within my homelab. Terraform is another beast though and this isn't a perfect combo.
While yearning for the goal of creating a micro AWS environment for learning purposes have sided, I took an opportunity to try building/running a custom Rust binary on the console in between projects.
The trick is simple, but you need to understand a little bit about dynamically linked binaries. I'm not doing a deep dive in the subject, but Rust will link against the
libc version that is local. You can see this via:
❯ ldd target/debug/rust_esxi linux-vdso.so.1 (0x00007ffc1ffae000) libdl.so.2 => /home/silas/.linuxbrew/lib/libdl.so.2 (0x00007fa37bd7e000) librt.so.1 => /home/silas/.linuxbrew/lib/librt.so.1 (0x00007fa37bb76000) libpthread.so.0 => /home/silas/.linuxbrew/lib/libpthread.so.0 (0x00007fa37b959000) libgcc_s.so.1 => /home/silas/.linuxbrew/lib/libgcc_s.so.1 (0x00007fa37b742000) libc.so.6 => /home/silas/.linuxbrew/lib/libc.so.6 (0x00007fa37b3a2000) /home/silas/.linuxbrew/lib/ld.so (0x00007fa37bf82000)
This is a pretty basic hello world binary and your results may vary. My system is setup with
linuxbrew but you can see that
libc is being linked here.
The problem is when moving the binary to another machine you need to verify that the machine has a similar version of a
libc present. This is not the case with ESXi which appears to be built with a very old libc from
CentOS. So to make dynamcally linked binaries work, we'd need to be building our Rust binaries within that old OS that contained a similar version of
libc. This isn't fun.
The trick…. it's simple, just build the binary with
musl support which is a different
libc implementation that works nicely when building static binaries.
You can build non-trivial Rust applications that run on ESXi console.
❯ ldd target/x86_64-unknown-linux-musl/debug/rust_esxi not a dynamic executable
See, static binary.
❯ scp ./target/x86_64-unknown-linux-musl/release/rust_esxi [email protected]:/tmp ❯ ssh [email protected] [[email protected]:/tmp] cd /tmp [[email protected]:/tmp] ./rust_esxi
Now in a browser tab you can hit 10.5.5.5:8090 and get "hello world" back.
Neat. A binary running on ESXi console that is using
actix web framework.
For this example to work you'll while building
active-web I needed to disable all features via
default-features = false to avoid
I'm sure this makes sense as some libraries might use highly specific features from a certain
musl might not provide.
I didn't get a chance to dive in much farther, but my experiment worked and I'm happy.
It's not perfect and there are tradeoffs.
musl key princibles to learn about them.
File size is one of them:
❯ du -hs target/debug/rust_esxi 2.8M target/debug/rust_esxi ❯ du -hs target/x86_64-unknown-linux-musl/debug/rust_esxi 82M target/x86_64-unknown-linux-musl/debug/rust_esxi
Release sizes are much better, but stil quite bigger than their dynamically linked cousins.
A fun hack and might be good for some projects as long as you are using crates that support
Will this rekindle my desire to do something around my original goal, maybe.