CI: test debian-13-armhf

[bastelfreak]

No description provided.

[bastelfreak]

@Gitii I tested a build on Debian 13 armhf. It fails to compile Ruby 3.2.9. Any chance you can take a look at it? https://github.com/OpenVoxProject/puppet-runtime/actions/runs/21296496686/job/61303836044 (let me know if the link doesn't work)

[Gitii]

@bastelfreak Sure, I will look into it. It looks like arm64 code is being assembled. That is not correct. I will analyze the build trace and try to identify why it tries to build anything for arm64 at all...

[bastelfreak]

The pipeline has an Ubuntu 24.04 arm64 (provided by github) and then starts a Debian 13 armhf container (I think). I don't know how big the differences between arm64 and armhf are / if we can configure anything in the container / the ruby build to get this working. Github doesn't provide armhf runners.

[Gitii]

@bastelfreak The runners are fine, the docker container is using the linux/arm/v7 arch. That is all working. The build process starts fine and then suddenly starts ... switching host architectures:

From the github runner:

more information, such as the ld(1) and ld.so(8) manual pages.
----------------------------------------------------------------------
make[4]: Leaving directory '/var/tmp/tmp.cltzMdTsbs/libffi-3.5.2/armv8l-unknown-linux-gnueabihf'
make[3]: Leaving directory '/var/tmp/tmp.cltzMdTsbs/libffi-3.5.2/armv8l-unknown-linux-gnueabihf'
make[2]: Leaving directory '/var/tmp/tmp.cltzMdTsbs/libffi-3.5.2/armv8l-unknown-linux-gnueabihf'
make[1]: Leaving directory '/var/tmp/tmp.cltzMdTsbs/libffi-3.5.2'
touch libffi-install
export LDFLAGS="-L/opt/puppetlabs/puppet/lib -Wl,-rpath=/opt/puppetlabs/puppet/lib,-z,relro,-z,now" && \
export CFLAGS="-fstack-protector-strong -fno-plt -O2" && \
gunzip -c "yaml-0.2.5.tar.gz" | tar xf -
touch libyaml-unpack
touch libyaml-patch
[ -d yaml-0.2.5 ] || mkdir -p yaml-0.2.5
export LDFLAGS="-L/opt/puppetlabs/puppet/lib -Wl,-rpath=/opt/puppetlabs/puppet/lib,-z,relro,-z,now" && \
export CFLAGS="-fstack-protector-strong -fno-plt -O2" && \
cd yaml-0.2.5 && \
./configure --prefix=/opt/puppetlabs/puppet --sbindir=/opt/puppetlabs/puppet/bin --libexecdir=/opt/puppetlabs/puppet/lib/libyaml 
[...]
checking dependency style of gcc... gcc3
checking how to run the C preprocessor... gcc -E
checking whether ln -s works... yes
checking whether make sets $(MAKE)... (cached) yes
checking build system type... aarch64-unknown-linux-gnu
checking host system type... aarch64-unknown-linux-gnu
checking how to print strings... printf
checking for a sed that does not truncate output... /usr/bin/sed

When running locally (my machine):

more information, such as the ld(1) and ld.so(8) manual pages.
----------------------------------------------------------------------
make[4]: Leaving directory '/var/tmp/tmp.pBqwwPbGaP/libffi-3.5.2/armv7l-unknown-linux-gnueabihf'
make[3]: Leaving directory '/var/tmp/tmp.pBqwwPbGaP/libffi-3.5.2/armv7l-unknown-linux-gnueabihf'
make[2]: Leaving directory '/var/tmp/tmp.pBqwwPbGaP/libffi-3.5.2/armv7l-unknown-linux-gnueabihf'
make[1]: Leaving directory '/var/tmp/tmp.pBqwwPbGaP/libffi-3.5.2'
touch libffi-install
export LDFLAGS="-L/opt/puppetlabs/puppet/lib -Wl,-rpath=/opt/puppetlabs/puppet/lib,-z,relro,-z,now" && \
export CFLAGS="-fstack-protector-strong -fno-plt -O2" && \
gunzip -c "yaml-0.2.5.tar.gz" | tar xf -
touch libyaml-unpack
touch libyaml-patch
[ -d yaml-0.2.5 ] || mkdir -p yaml-0.2.5
export LDFLAGS="-L/opt/puppetlabs/puppet/lib -Wl,-rpath=/opt/puppetlabs/puppet/lib,-z,relro,-z,now" && \
export CFLAGS="-fstack-protector-strong -fno-plt -O2" && \
cd yaml-0.2.5 && \
./configure --prefix=/opt/puppetlabs/puppet --sbindir=/opt/puppetlabs/puppet/bin --libexecdir=/opt/puppetlabs/puppet/lib/libyaml 
[...]
checking dependency style of gcc... gcc3
checking how to run the C preprocessor... gcc -E
checking whether ln -s works... yes
checking whether make sets $(MAKE)... (cached) yes
checking build system type... armv7l-unknown-linux-gnueabihf
checking host system type... armv7l-unknown-linux-gnueabihf
checking how to print strings... printf
checking for a sed that does not truncate output... /usr/bin/sed

libffi is already build for armv8l on github and it continues to build for aarch64. That is all wrong and not what I am expecting at all (and in fact, not what I am getting when building locally). Even more interesting: At the beginning, the correct arch is being targeted.
And I am using the same commands (bundle exec build agent-runtime-main debian-13-armhf --engine docker)

As far as I can tell, you are using the armhf branch of shared-actions. That doesn't contain the docker/setup-qemu-action action.
Can you please add these lines to the workflow? It's a shot in the dark but maybe it fixes it.

[bastelfreak]

I updated the workflow, but we still get the same error: https://github.com/OpenVoxProject/puppet-runtime/actions/runs/21296496686/job/61315115215

[bastelfreak]

I think qemu isn't used if the host and container image are both arm, no matter if it is arm64 and armhf. I'm testing it now with an x86 host: https://github.com/OpenVoxProject/puppet-runtime/actions/runs/21300920352/job/61318167563

[Gitii]

This is wild 😅...
I have a theory (ehm, ok, claude opus has one...):
We are running a 32bit arm process on a arm64 host. That can be done natively but because it is happening in user space, kernel commands still report the actual host architecture. Switching to "32 bit" mode, can partially solve that: Instead of arm64, armv8 will be reported (arm64 in 32bit mode).

That explains the armv8l entries in the logs. And some build scripts might map armv8l to aarch64 (arm64).
That does imply, that docker on arm64 doesn't use qemu when targeting arm32. Even when it's available.

So the solution is: Just build it on a x64 host and use full emulation!

Can you please update the workflow:

• Keep the docker/setup-qemu-action
• change to arm64 linux runner

Something like this:

              *-aarch64 | *-arm64) # <- remove armhf here
                runner="ubuntu-24.04-arm"
                shell="bash"
                ;;
              *-armhf) # add whole new switch statement or ...
                runner="ubuntu-24.04" # arm32 on x64 with full emulation to avoid 32bit userspace on arm64
                shell="bash"
                ;;
              *)
                runner="ubuntu-24.04" # ... fall through and use default
                shell="bash"
                ;;

[Gitii]

@bastelfreak You were faster 👍

[bastelfreak]

not sure if it's a good or bad sign that claude had the same idea as me :D

[bastelfreak]

based on the incredible slow performance we now know qemu is involved. It feels a bit weird to run this on X86, when an ARM runner would be available. with the qemu software emulation it's now even slower than building on MacOS & Windows 😞 . let's see if it passes.