[OpenWrt Wiki] Building OpenWrt ON OpenWrt (2024)

Why Build OpenWrt on OpenWrt?

1. Becoming self-hosting is an important milestone for any OS

2. It makes building OpenWrt more accessible for those with less resources.

3. Only building on a mono-culture (x86-based full blown Linux systems) lets a lot of bugs slide under the radar. Building on a different system can (and did) expose a lot of issues that would inevitably crop up later

4. For the educational value and bragging rights of knowing that your device's OS was built on your device.

This has so far been tested on armv7l and aarch64 devices, but will likely work on most if not all supported OpenWrt architectures. However the device's hardware does need to meet some minimum requirements:

• 1GHz or faster CPU, multi-core is an asset.

• 512MiB of physical RAM per core you intend to use in the build for 32-bit devices. For 64-bit devices you need double this.¹

• 2GiB of swap space. This should be on a physical hard drive, or a high quality external device (meaning not the device's internal eMMC).²

• 1GiB of storage space on your root device for all the OpenWrt packages you will need.

• ~26GiB of storage space for the build

Note¹: The initial toolchain build (especially of llvm-bpf) is quite RAM-hungry. The given memory requirements should be considered a minimum if you are building your own llvm-bpf. If you are able to use a pre-built llvm-bpf, then you can get away with less RAM-per-used-core

Note²: Your swap space (and to a lesser extent, the storage space for the build itself) should be on a physical hard drive, or on a high quality solid state (USB, SSD) storage device. It should not under any circ*mstances be on the device's own internal eMMC storage. Certain steps in this process can be somewhat heavy on swapping. If you're not using a mechanical hard drive, then a good quality storage device will handle the wear better. Also, the larger your storage device is the more the wear will be spread.

First of all you need a whole lot of OpenWrt's own packages:

• The development tools themselves:
  opkg install pkg-config make gcc diffutils autoconf automake check git git-http patch libtool-bin

• Miscelaneous tools used in the build system:
  opkg install grep rsync tar python3 getopt procps-ng-ps gawk sed xz unzip gzip bzip2 flock wget-ssl

• Perl³ and Python:
  opkg install perl perlbase-findbin perlbase-pod perlbase-storable perlbase-feature perlbase-b perlbase-ipc perlbase-module perlbase-extutils python3

• A bunch of coreutils, since busybox versions aren't good enough in some cases, or aren't there:
  opkg install coreutils-nohup coreutils-install coreutils-sort coreutils-ls coreutils-realpath coreutils-stat coreutils-nproc coreutils-od coreutils-mkdir coreutils-date coreutils-comm coreutils-printf coreutils-ln coreutils-cp coreutils-split coreutils-csplit coreutils-cksum coreutils-expr coreutils-tr coreutils-test coreutils-uniq coreutils-join

• A few libraries:
  opkg install libncurses-dev zlib-dev musl-fts libzstd
ln -s libncursesw.a /usr/lib/libncurses.a

  See Also

  [OpenWrt Wiki] How much time is needed for compiling?

• Some good-to-have packages to make life easier (instructions below will assume you have these):
  opkg install joe joe-extras bash htop whereis less file findutils findutils-locate chattr lsattr xxd

Note³: The test systems this was performed on had quite a bit of perl pre-installed, so there may be more perl packages required.

Before we can go further, we have our first bug workaround:

OpenWrt Bug: OpenWrt doesn't put execute permissions on the scripts for the automake, autoconf, and libtool packages. This prevents a lot of build systems (including OpenWrt's own) from working correctly. To fix:
chmod +x /usr/share/automake-1.16;chmod -x /usr/share/automake-1.16/COPYING /usr/share/automake-1.16/INSTALL
chmod +x /usr/share/autoconf/Autom4te/*
chmod +x /usr/share/libtool/build-aux/*

• libargp: A GNU GLIBC extension used by elflibs

  git clone https://github.com/xhebox/libuargp.gitcd libuargpmakemake prefix=/usr installln -s libargp.so /usr/lib/libargp.so.0cd ..

• libfts: Another GNU GLIBC extension needed for elflib.
  We installed the non-dev version above already, so this will overwrite it with one that can be compiled against. We installed the non-dev version first so that it doesn't get installed later and overwrite your dev version with a lobotomized one:

  git clone https://github.com/void-linux/musl-fts.gitcd musl-fts./bootstrap.sh./configure --prefix=/usrmakemake installcd ..

• libobstack: The final GNU GLIBC extension needed for elflib

  git clone https://github.com/void-linux/musl-obstack.gitcd musl-obstack./bootstrap.sh./configure --prefix=/usrmakemake installcd ..

• libzstd: A compression library now needed by llvm-bpf.
  Like musl-fts above, OpenWrt does support this library, but by default ships a lobotomized version that can't be built against. We install the OpenWrt version above so it won't be later installed and overwrite the proper version we're making below:

  wget https://github.com/facebook/zstd/releases/download/v1.5.2/zstd-1.5.2.tar.gztar -xvzf zstd-1.5.2.tar.gzcd zstd-1.5.2/libmakemake prefix=/usr installcd ..

• libdl, librt, libresolv, and libpthread:
  These are libraries for functions which are actually included in musl libc directly. However, since some tools try and link explicitly against these libs, you will run into “file not found” errors. To get around that, just make stub libraries for them:

  cat <<EOF > /usr/lib/libdl.a!<arch>EOFcp -a /usr/lib/libdl.a /usr/lib/librt.acp -a /usr/lib/libdl.a /usr/lib/libpthread.acp -a /usr/lib/libdl.a /usr/lib/libresolv.a

This is based on the build system instructions, with a bunch of deviations to fix various bugs along the way:

Make sure you're in your devel directory from before. The following assume you are building SNAPSHOT, but can be (with appropriate changes) be used for release versions as well.

1. Get OpenWrt master and associated packages:

   git clone https://git.openwrt.org/openwrt/openwrt.gitcd openwrtgit checkout mastergit pull./scripts/feeds update -a./scripts/feeds install -a

2. Get the config file for your architecture and save it as .config. For example, for a device based on the mt7622 chipset (such as the BPI-R64) it is this:

   wget https://downloads.openwrt.org/snapshots/targets/mediatek/mt7622/config.buildinfo -O .config

3. Expand the configuration:

   make defconfig

4. DECISION POINT: llvm-bpf - build yourself or get pre-built
   llvm-bpf is the the Berkeley Packet Filter bytecode compiler for Linux. It is a tool that is needed in order to build OpenWrt, but also can be built along with the rest of the OpenWrt toolchain below. Compiling llvm is easily the single most time and resource intensive step in building OpenWrt. If there is a pre-made one for your platform of the correct version for the version of OpenWrt you are building, then it is recommended to use it. Check the resources below. If you are using a pre-build one, then get it now and untar it into your openwrt working folder, so it will be there when you configure OpenWrt. For example:

   wget https://va1der.ca/~public/openwrt/llvm/llvm-bpf-15.0.7.Linux-armv7l.tar.xztar -xvaf llvm-bpf-15.0.7.Linux-armv7l.tar.xz

5. Configure OpenWrt:
   make menuconfig
   Things to remember when you configure the build: Change “Target Profile” to build for only your platform (since by default it will want to build for all platforms using a given chipset). Also, if you obtained a pre-build llvm-bpf then change “Advanced Configuration Options → BPF Toolchain” to use the pre-built binary you obtained. Make any other tweaks you normally do, then save.

6. If you are using pre-made llvm from step 4, then skip all of this step (read the NOTE at the end though).
   If you were unable or didn't want a pre-built llvm-bpf, then now is when you will build it. But first...
   llvm-bpf bugs: There are two bugs in llvm's build system. One that causes a built failure due to libatomic not being linked, and one that causes a build failure when gcc is used instead of g++ at the linking stage. We will copy patches for both into the OpenWrt build system:

   mkdir tools/llvm-bpf/patcheswget https://va1der.ca/~public/openwrt/patches/llvm-bpf/901-fix-fuzzer-linking.patch -O tools/llvm-bpf/patches/901-fix-fuzzer-linking.patchwget https://va1der.ca/~public/openwrt/patches/llvm-bpf/902-use-libatomic-as-required.patch -O tools/llvm-bpf/patches/902-use-libatomic-as-required.patch

   After musl-1.2.4 there were changes, such as lseek64 missing, and replaced with lseek, so llvm fails to build. To fix this, you need to make a minor change to llvm-bpf's Makefile, just under include $(INCLUDE_DIR)/cmake.mk add following “HOST_CFLAGS+= -D_LARGEFILE64_SOURCE”, so it will look a bit like this:

   ...CMAKE_SOURCE_SUBDIR := llvminclude $(INCLUDE_DIR)/host-build.mkinclude $(INCLUDE_DIR)/cmake.mkHOST_CFLAGS+= -D_LARGEFILE64_SOURCELLVM_BPF_PREFIX = llvm-bpf-$(PKG_VERSION).$(HOST_OS)-$(HOST_ARCH)...

   Once those patches are copied and Makefile is fixed, you can build llvm. Caution! This is memory intensive, and a multi-core build on a system with marginal memory can push the system so far into memory-debt swapping that it can damage solid-state storage. A single-core build will take longer, but is safer. This is going to take a long time. Note that a drawback of OpenWrt's build system is that any time the build of any one package is interrupted, it starts that package from the beginning the next time. You don't want compilation to stop on a random net blip after ten hours, so use nohup. And start getting into the habit of using V=sc and logging everything, because this whole process is still experimental:
   

   nohup nice make -j 1 V=sc tools/llvm-bpf/compile > ~/llvmcomp01.out &

   Periodically check htop to make sure it is going well, and check the log when it stops. Hopefully there's no error, but if there is, welcome to the experiment. Debug, rinse and repeat and report it here for the rest of us.

   NOTE: Once llvm-bpf is out of the way, if you have multiple cores and enough RAM per core, then you should be ok to use multi-core compiling. Always keep a periodic eye on your build, though and htop is your friend. For all examples throughout these instructions, -j 1 will be supplied to make for pasted commands for safety. But feel free to increase the core count at your discretion.

7. Now build the rest of the toolchain:

   nohup nice make -j 1 V=sc toolchain/install > ~/toolchain01.out &

   As with llvm, and all else, periodically check htop and the output log, debug, and repeat.

8. With the toolchain complete, you can do a make download. It had to wait until now because a bug in qosify's makefile prevents make download from working before the toolchain is complete in some cases.

   make download

9. Build the the one missing tool noted above: rev. OpenWrt doesn't make a package out of it, even though it's built by default when OpenWrt builts packages out of its own util-linux. So we're going to build OpenWrt's util-linux then copy rev out. This is quick, probably no need to log it:

   make -j 1 package/util-linux/compilecp $(find . -wholename \*ipkg-install/usr/bin/rev) /usr/bin/rev

10. perl bugs: Here we fix three bugs in perl. One which causes a problem with perl on musl, one where recent compilers cause a seg fault, and one in Perl's Configure script. Fort these we need two patches. The first patch we will apply directly to OpenWrt's perl makefile:

    ( cd feeds/packages/lang/perl && wget https://va1der.ca/~public/openwrt/patches/perl/perl_fix_memmem_and_segfault.patch -O - 2> /dev/null | patch -p 1 )

    That fixes the first two bugs. This next fixes perl's configure bug, and we're just going to download and store it in the OpenWrt build system and let OpenWrt sic it on Perl when it builds it:

    wget https://va1der.ca/~public/openwrt/patches/perl/997-fix-Configure-gcc-parse.patch -O feeds/packages/lang/perl/patches/997-fix-Configure-gcc-parse.patch

11. DECISION POINT: perl bug - Test to see if your the OpenWrt device you are building on is affected by one of the above bugs. Some are, and if it is, you need to build perl right now with the above patch to fix your existing system. This is because the host system's perl is used when OpenWrt builds openssl. Download and run the script:

    wget https://va1der.ca/~public/openwrt/patches/perl/test_index.perl -O ~/test_index.plchmod +x ~/test_index.pl~/test_index.pl

    This will tell you if your perl is affected by the index() bug. If it is then build perl with the above bug patch above and copy libperl.so over top of of your system's buggy one:
    

    make -j 1 package/perl/compilecp $(find staging_dir -name libperl.so) /usr/lib/perl5/5.28/CORE/~/test_index.pl

    The test should now show the index() bug is fixed

12. That was the last bug fix - you can now proceed with the rest of the build. This is likely to take a while (not as long as LLVM though). Continue to use nohup, tell make to give verbose output with V=sc, and continue to redirect that output to a log:

    nohup nice make -j 1 V=sc world > world01.out &

• llvm-bpf-14.0.6.Linux-armv7l.tar.xz

• llvm-bpf-15.0.7.Linux-armv7l.tar.xz

• llvm-bpf-14.0.6.Linux-aarch64.tar.xz

• llvm-bpf-15.0.7.Linux-aarch64.tar.xz

• llvm-bpf-13.0.0.Linux-x86_64.tar.xz

• llvm-bpf-15.0.7.Linux-x86_64.tar.xz