dlopen() on the main binary loads it a second time and re-runs its initialisers (breaks non-PIE executables, e.g. all Delphi/Linux apps)

[delphiprofi]

Summary

When a non-PIE x86_64 executable calls dlopen() on its own path, box64 loads the file a
second time and runs its initialisers again. On native Linux glibc simply returns NULL for that
call and nothing is loaded twice.

The consequences are easy to overlook in a small program but fatal in a real one: the second copy
cannot be placed at the executable's fixed load address, so it is relocated — yet its absolute
addresses still point into the first image. Its initialisers therefore run against the running
program's data. A Delphi application dies with SIGSEGV shortly after.

This is not an exotic pattern. Every application built with Embarcadero Delphi for Linux does
this during startup (the RTL calls dladdr() and then dlopen() on the module's own file), and
Delphi always produces non-PIE executables — so currently no Delphi/Linux application runs under
box64.

Environment

• box64 fbbb054 (v0.4.5), -DRPI3ARM64=1 -DCMAKE_BUILD_TYPE=RelWithDebInfo,
  cross-compiled with aarch64-linux-gnu-gcc 13.3.0
• Raspberry Pi 3 Model B Rev 1.2, Raspberry Pi OS Trixie arm64, kernel 6.18.50+rpt-rpi-v8, 4K pages
• box64 banner: Dynarec for ARM64, with extension: ASIMD CRC32,
  Didn't detect 48bits of address space, considering it's 39bits

Reproducer

/* build on x86_64:  gcc -no-pie -o selfopen2 selfopen2.c -ldl
   The constructor makes the double load visible: it must run exactly once. */
#include <stdio.h>
#include <dlfcn.h>
#include <limits.h>
#include <unistd.h>

static int counter = 0;

__attribute__((constructor))
static void ctor(void)
{
    counter++;
    printf("  ctor runs: counter at %p is now %d\n", (void*)&counter, counter);
    fflush(stdout);
}

int main(void)
{
    char self[PATH_MAX];
    ssize_t n = readlink("/proc/self/exe", self, sizeof(self) - 1);
    if (n < 0) { perror("readlink"); return 1; }
    self[n] = '\0';

    printf("main: counter at %p is %d\n", (void*)&counter, counter);
    printf("opening myself: %s\n", self);
    fflush(stdout);

    void *h = dlopen(self, RTLD_LAZY);
    printf("dlopen -> %p\n", h);
    printf("main: counter at %p is %d  (expected: still 1)\n", (void*)&counter, counter);
    fflush(stdout);
    return 0;
}

-no-pie matters: the binary must have a fixed load address (here 0x400000).

native x86_64 Linux (Ubuntu 24.04, glibc) — reference

  ctor runs: counter at 0x40404c is now 1
main: counter at 0x40404c is 1
opening myself: /tmp/selfopen2
dlopen -> (nil)
main: counter at 0x40404c is 1  (expected: still 1)

glibc refuses to dlopen an executable and returns NULL. The constructor runs once.

box64 on the Pi

  ctor runs: counter at 0x40404c is now 1
main: counter at 0x40404c is 1
opening myself: /tmp/selfopen2
[BOX64] Warning: cannot create memory map (@0x400000 0x4050) for elf "/tmp/selfopen2" got 0x7fa53db000
  ctor runs: counter at 0x40404c is now 2      <-- second load, initialiser runs again
dlopen -> 0x4
main: counter at 0x40404c is 2  (expected: still 1)

Note the address: the relocated second copy writes into the running program's counter.

What it looks like in a real application

The Delphi HTTP server I was porting starts, reaches its entry point, and then dies:

[BOX64] Warning: cannot create memory map (@0x10000000 0x4108c0) for elf "…/VeGA_Server" got 0x7fb877c000
[BOX64] Pre-allocated 0x4108c0 byte at 0x7fb877c000 for …/VeGA_Server
[BOX64] Delta of 0x7fa877c000 (vaddr=0x10000000) for Elf "…/VeGA_Server"
[BOX64] Adding "…/VeGA_Server" as #2 in elf collection
…
[BOX64] Warning, calling Signal 11 function handler SIG_DFL

The warning is misleading in a way that costs hours: it reads like an address conflict, so one
starts looking at ASLR, mmap_min_addr, or the load address. None of that is the cause — the kernel
hands out exactly these hints without complaint (checked with a small C program calling mmap()
with the same address, size and flags), and relinking the application to a different base
(--image-base=0x10000000) only moves the same failure, because the range is occupied by the first
(correct) load of the very same binary.

Where it comes from

In src/wrapped/wrappedlibdl.c, my_dlopen_internal() first walks the already-opened libraries
(IsSameLib(dl->dllibs[i].lib, rfilename)) and then calls GetLibInternal(rfilename). Both only
know libraries; the main executable is not among them, so lib == NULL, dlopened becomes 1, and
AddNeededLib() loads the file again.

dlopen(NULL, …) is handled correctly a few lines below — it looks for an existing entry with
is_self and recycles it. Only the "open myself by name" spelling misses that path.

Two possible fixes

(a) Match glibc and return NULL when the path resolves to the main executable. Most faithful
to native behaviour; applications that do this already cope with NULL, since that is what they get
on real Linux.

(b) Return the is_self handle — more forgiving, and what I used locally:

        // dlopen() on the main binary itself: hand out the "self" handle instead of loading
        // the file a second time. A non-PIE executable cannot be mapped twice - the second
        // copy lands elsewhere and its initialisers run against the first image's data.
        if(my_context->fullpath && !strcmp(rfilename, my_context->fullpath)) {
            for (size_t i=MIN_NLIB; i<dl->lib_sz; ++i) {
                if(dl->dllibs[i].full && dl->dllibs[i].is_self) {
                    ++dl->dllibs[i].count;
                    return (void*)(i+1);
                }
            }
            if(dl->lib_sz == dl->lib_cap) {
                dl->lib_cap += 4;
                dl->dllibs = (dllib_t*)box_realloc(dl->dllibs, sizeof(dllib_t)*dl->lib_cap);
                memset(dl->dllibs+dl->lib_sz, 0, (dl->lib_cap-dl->lib_sz)*sizeof(dllib_t));
                if(!dl->lib_sz)
                    dl->lib_sz = MIN_NLIB;
            }
            intptr_t idx_self = dl->lib_sz++;
            dl->dllibs[idx_self].lib = NULL;
            ++dl->dllibs[idx_self].count;
            dl->dllibs[idx_self].dlopened = 0;
            dl->dllibs[idx_self].is_self = 1;
            dl->dllibs[idx_self].full = 1;
            return (void*)(idx_self+1);
        }

With (b) the reproducer's constructor runs once, counter stays 1, and the Delphi server starts and
serves requests on the Pi (0.89 s to first listen, ~20 ms per request, well usable).

A proper fix should probably compare resolved paths rather than the raw string (symlinks, ./name,
/proc/self/exe) and may want the same treatment in elfloader32.c. I kept the change minimal on
purpose. Happy to send this as a pull request in whichever of the two shapes you prefer.

Side note, in case it helps someone searching

Two further things were needed before the Delphi application ran, both unrelated to the bug above:
its embedded resources are not reachable under box64 (resourcestrings come out empty, .dfm
resources are not found), and ICU must be shipped as x86_64 libraries of exactly the version the
target carries — box64's wrapped native ICU is missing symbols the Delphi RTL looks up, ending in
Ask to run at NULL, will segfault. BOX64_EMULATED_LIBS only helps when the emulated files are
actually present; otherwise library.c falls back to the native wrapper.

[ptitSeb]

So, I pushed something (and forgot to tag the ticket): it should not be possible now to open elf file as library if they are not of type ET_DYN

There is still an issue if you open a self program that is PIE via /proc/self/exe for now (it will load the program again instead of using the already loaded copy). Should this also be fixed?

[delphiprofi]

Follow-up, and a correction to my own report: Option A above does not actually fix this. I implemented and measured all three variants; here is what came out.

Option A (return NULL, like glibc) is not sufficient

glibc's behaviour is what I described — verified with a non-PIE reproducer on native x86-64:

dlopen("./glibc-probe") = (nil)
  dlerror: ./glibc-probe: cannot dynamically load executable
dlsym(NULL, "marke") = 0x404038      <- RTLD_DEFAULT does reach the executable's own symbol

So returning NULL is faithful. It is still not enough for a Delphi application. Measured on a
Raspberry Pi 3 with box64 v0.4.5, same binary in all three runs, a minimal test program that
reads a resourcestring, instantiates a TDataModule from its embedded .dfm, and opens the
same resource through TResourceStream:

variant	HInstance	resourcestring	.dfm
unpatched	—	crash	crash
A — dlopen returns NULL	0	works	fails
A + dlsym(NULL) also searching the main ELF	0	works	fails
B — hand out a "self" handle	4	works	works

The .dfm failure under A is not a symbol lookup problem. With BOX64_DLSYM_ERROR=1 every
dlsym(NULL, "__rstr_…") resolves to a proper address — but there is not a single
__rsrc_… lookup in the whole run. With HInstance = 0 the RTL never gets as far as
FindResource: InitInheritedComponent goes through FindClassHInstance /
FindResourceHInstance first, and that path gives up before the lookup happens. Extending
box64 so that dlsym(NULL, …) also searches my_context->elfs[0] therefore changes nothing —
the gap is earlier. (That extension may still be worth having on its own: RTLD_DEFAULT is
supposed to cover the executable's exported symbols, and today it only walks the loaded libraries.)

Option B works, but it needs a second change — in dlclose

Handing out a "self" handle makes the application run, and I originally stopped there. That
turned out to hide a second, silent failure: all embedded resources stayed unreachable,
resourcestrings came back empty and .dfms were not found. No error anywhere — a Delphi
service just runs mute.

The cause shows up in two lines of box64's own log:

dlopen: New self handle 0x4 (/path/to/app)
Call to dlclose(0x4)

Delphi's RTL closes the handle immediately and keeps using it afterwards — FindResource
in System.pas does dlsym(ModuleHandle, "__rsrc_N6RCDATA…E"), and ModuleHandle is that
handle. Under glibc this is harmless: a handle there is a pointer to a link_map, the
executable is never unloaded, and the pointer stays usable. box64 keeps handles as an index
into dl->dllibs[] and invalidates the entry on close, so every later dlsym ends in
Bad handle (already closed).

Keeping the "self" entry alive makes all of it work — same behaviour as the native run,
resourcestring, .dfm and TResourceStream all fine, with no measurable change to the
application otherwise (19.70 ms vs 19.50 ms per request):

    // in my_dlclose_internal(), before the refcount is decremented
    // dlclose() on the "self" handle: glibc never unloads the executable, and its handle
    // stays usable afterwards - callers rely on that.
    if(dl->dllibs[nlib].is_self && !dl->dllibs[nlib].lib) {
        printf_dlsym(LOG_DEBUG, "dlclose: keeping self handle %p alive\n", handle);
        return 0;
    }

This part looks like a defect independent of Delphi: any program that dlopens something,
closes the handle and keeps using it works under glibc for as long as the object stays
loaded, and fails under box64.

I am happy to open a PR with both pieces if that is useful — just say which shape you would
prefer (the is_self route as above, or dlopen returning NULL plus whatever would be
needed to make the RTL path work, which I have not found).

[ptitSeb]

Mmmm, I don't understand. dlsym with a NULL handle already search the main elf. I don't understand what the issue is. Also, I don't understand why you are talking about option B. the point of box64 is to have the same behaviour as glibc.