I've always been wanting to run applications on a ESXi console.
These ambitions grew as I started learning more and more about AWS over the last 10 years.
I've dreamt of building an environment that I could use to learn about how AWS does what it does.
Learning about vim-cmd was my catalyst.
This was the key to being able to automate VM deployments within my homelab.
It's annoying to SSH in to call this command. I've locked myself out of my ESXi plenty of times because X number of wrong password attempts via SSH.
I needed something else…
Since Go binaries are statically built (with exceptions) that gave me hope of running applications within the ESXi console.
It's been a long time ago, but my small experiments never resulted in anything runnable within the console. I ended up giving up on Go for this.
Learning and contributing to https://github.com/josenk/terraform-provider-esxi was something that got me closer to my goal, but was too far away from the metal for my liking. This filled my need for 95% of the things that I wanted to do within my homelab. Terraform is another beast though and this isn't a perfect combo.
While yearning for the goal of creating a micro AWS environment for learning purposes have sided, I took an opportunity to try building/running a custom Rust binary on the console in between projects.
The trick is simple, but you need to understand a little bit about dynamically linked binaries. I'm not doing a deep dive in the subject, but Rust will link against the libc version that is local. You can see this via:
❯ ldd target/debug/rust_esxi
linux-vdso.so.1 (0x00007ffc1ffae000)
libdl.so.2 => /home/silas/.linuxbrew/lib/libdl.so.2 (0x00007fa37bd7e000)
librt.so.1 => /home/silas/.linuxbrew/lib/librt.so.1 (0x00007fa37bb76000)
libpthread.so.0 => /home/silas/.linuxbrew/lib/libpthread.so.0 (0x00007fa37b959000)
libgcc_s.so.1 => /home/silas/.linuxbrew/lib/libgcc_s.so.1 (0x00007fa37b742000)
libc.so.6 => /home/silas/.linuxbrew/lib/libc.so.6 (0x00007fa37b3a2000)
/home/silas/.linuxbrew/lib/ld.so (0x00007fa37bf82000)
This is a pretty basic hello world binary and your results may vary. My system is setup with linuxbrew but you can see that libc is being linked here.
The problem is when moving the binary to another machine you need to verify that the machine has a similar version of a libc present. This is not the case with ESXi which appears to be built with a very old libc from CentOS. So to make dynamcally linked binaries work, we'd need to be building our Rust binaries within that old OS that contained a similar version of libc. This isn't fun.
The trick…. it's simple, just build the binary with musl support which is a different libc implementation that works nicely when building static binaries.
After following this: https://doc.rust-lang.org/edition-guide/rust-2018/platform-and-target-support/musl-support-for-fully-static-binaries.html
You can build non-trivial Rust applications that run on ESXi console.
❯ ldd target/x86_64-unknown-linux-musl/debug/rust_esxi
not a dynamic executable
See, static binary.
❯ scp ./target/x86_64-unknown-linux-musl/release/rust_esxi [email protected]:/tmp
❯ ssh [email protected]
[root@localhost:/tmp] cd /tmp
[root@localhost:/tmp] ./rust_esxi
Now in a browser tab you can hit 10.5.5.5:8090 and get "hello world" back.
Neat. A binary running on ESXi console that is using actix web framework.
For this example to work you'll while building active-web I needed to disable all features via default-features = false to avoid brotli-sys compiling.
I'm sure this makes sense as some libraries might use highly specific features from a certain libc that musl might not provide.
I didn't get a chance to dive in much farther, but my experiment worked and I'm happy.
It's not perfect and there are tradeoffs.
Read musl key princibles to learn about them.
File size is one of them:
❯ du -hs target/debug/rust_esxi
2.8M target/debug/rust_esxi
❯ du -hs target/x86_64-unknown-linux-musl/debug/rust_esxi
82M target/x86_64-unknown-linux-musl/debug/rust_esxi
Release sizes are much better, but stil quite bigger than their dynamically linked cousins.
A fun hack and might be good for some projects as long as you are using crates that support musl.
Will this rekindle my desire to do something around my original goal, maybe.