[#1092] Probe the Docker container's health without binding as the root user

[vharseko]

Fixes #1092

Problem

The image HEALTHCHECK bound as ROOT_USER_DN with ROOT_PASSWORD, which is only the initial root password. Once an operator changed the root password, every probe failed with 49 (Invalid Credentials) and the container stayed unhealthy for good while the server kept serving. The same happens to a new container started over an existing volume without the original ROOT_PASSWORD: run.sh does not use the password on an existing instance, but the probe did. Every probe also put the password on a command line, readable from /proc/<pid>/cmdline while it ran, and so did every tool bootstrap/setup.sh ran during the bootstrap, where the ps of the Docker host lists it to every user of the host.

Change

• healthcheck.sh (new, next to run.sh): tests the $BOOTSTRAP_COMPLETE marker first, as before (Report the OpenDJ container healthy only once its bootstrap has succeeded #898), then reads the root DSE with the attribute list 1.1 over LDAPS anonymously, with --noPropertiesFile so that a tools.properties in the image user's home cannot turn it into a bind. It sits next to run.sh rather than in bootstrap/ because bootstrap/ is the directory users mount their own content over. Any failure exits with 1, since Docker reserves 2.
• Instances that reject unauthenticated requests: with reject-unauthenticated-requests:true the server refuses the anonymous search with 53 (Unwilling to Perform) before any ACI is evaluated (DirectoryServer.checkCanEnqueueRequest). For such an instance, HEALTHCHECK_BIND_DN names the account the probe binds as, and the probe reads its password from HEALTHCHECK_BIND_PASSWORD_FILE with --bindPasswordFile, so it never shows on a command line. Neither variable is set by default.
• bootstrap/setup.sh: writes ROOT_PASSWORD to a file readable by its owner only, on /dev/shm where there is one, removed by an EXIT trap, as replicate.sh does since [#1084] Keep the root password out of the log and off the command line when a Docker container joins replication #1094; setup, dsconfig, import-ldif and ldapmodify read it with --rootUserPasswordFile / --bindPasswordFile. ldapmodify used to get it unquoted, so a password with a space in it broke the loading of bootstrap/schema and bootstrap/data.
• run.sh: removes a password file a killed setup.sh leaves in /dev/shm, next to the one of replicate.sh.
• Dockerfile, Dockerfile-alpine: copy the script with --chown=$OPENDJ_USER:0, like the other scripts since [#1088] Let the Docker image start under an arbitrary uid in group 0 #1091, and set HEALTHCHECK ... CMD ["/opt/opendj/healthcheck.sh"]. Interval, timeout, start period and retries are unchanged.
• pom.xml: Dockerfile.zip, which both CI and the release build the image from, lists its files explicitly, so healthcheck.sh is added there.
• README.md: the Health check section explains why the probe does not bind as root and how to use the two new variables; both are added to the variable table. It also says that an instance which rejects unauthenticated requests was healthy on images that probed as root, and has to set both variables before it starts on this one.
• CI: Docker test health check is added to both build-docker and build-docker-alpine, with --health-interval=5s:
  • "stays healthy" means the status reads healthy with a failing streak of 0 every 2 s for 45 s, so a single failed probe in that window fails the step;
  • a container started with ROOT_PASSWORD=initial_password turns healthy, its root password is changed with ldappasswordmodify, bind settings with a wrong password are written to /home/opendj/.opendj/tools.properties, and it stays healthy;
  • a container started with a root password of its own, HEALTHCHECK_BIND_DN and a mounted password file turns healthy, reject-unauthenticated-requests is switched on, and an anonymous search is checked to fail with 53 (so the setting has taken effect). On that instance healthcheck.sh must exit 1 with HEALTHCHECK_BIND_DN cleared, and exit 1 reporting an unreadable password file, both for a missing one and for one with mode 000. dsconfig reads the password from the same mounted file. The container must stay healthy, and the command lines of every process in it are sampled for 12 s to check that the password never appears on one.
  • both containers run without --rm, so one whose bootstrap failed is still there for the ERR trap to print; the trap and the end of the step remove them.
• CI, Docker test replication (from [#1084] Keep the root password out of the log and off the command line when a Docker container joins replication #1094): the check that no tool gets the password on its command line covers setup.sh too, --rootUserPassword included; the check that nothing is left in /tmp or /dev/shm covers the master as well, and no longer skips /tmp/hsperfdata_*, which only the old probe needed.

Verification

Images built locally from release 5.1.2, with the CI step run as a script:

• openidentityplatform/opendj:latest (the old probe): the rotation case fails; after the password change every probe logs The LDAP search request failed: 49 (Invalid Credentials);
• this branch, default and Alpine images: both cases pass.

The review round was checked the same way, with healthcheck.sh mounted over the image's copy: the branch passes on both images, and the step fails for each of these mutants of the probe: || exit 1 → || true, --bindPasswordFile → --bindPassword "$(cat …)", the unreadable-file guard removed, and a probe that fails every fourth call. With bind settings in /home/opendj/.opendj/tools.properties, the probe without --noPropertiesFile fails with 49, and with it passes.

The bootstrap: every process's command line was sampled from the container start until the bootstrap marker, with bootstrap/schema and bootstrap/data mounted and a root password with a space in it. With this branch's setup.sh: no hits on either image, both LDIFs loaded, /dev/shm empty. With master's: the password shows on the command lines of setup, its JVMs, import-ldif, dsconfig and ldapmodify, and the LDIFs are not loaded.

On thin images over 5.1.2 carrying this branch's scripts, Docker test replication and Docker test health check pass on both images as extracted from build.yml (only the waits relaxed for a slower host). The health check step fails without --noPropertiesFile and with the unreadable-file guard weakened to -e.

Notes

• Kubernetes ignores the image HEALTHCHECK; the Helm chart discussed in Add an official Helm chart so OpenDJ can be deployed on any Kubernetes cluster #1079 should use the same probe.
• [#1088] Let the Docker image start under an arbitrary uid in group 0 #1091 and [#1084] Keep the root password out of the log and off the command line when a Docker container joins replication #1094 are merged, and this branch is rebased onto them. Its Docker test arbitrary uid step waits for healthy, so it also runs this probe under an arbitrary uid in group 0. The probe only reads, so it needs nothing more than the group the image gives it.
• [#1093] Hand the root password to the Docker bootstrap tools in a file #1099 and [#1087] Copy the secret volume on every start of the Docker image, and run the server as PID 1 after the bootstrap #1100 add CI steps at the same place in build.yml, and [#1093] Hand the root password to the Docker bootstrap tools in a file #1099 also changes bootstrap/setup.sh, so whichever merges later needs a rebase.

[vharseko]

@maximthomas, rebased onto the current master (2ba918a) as f5f095f.

The merge of #1091 made this PR conflict in three files:

• Dockerfile, Dockerfile-alpine: [#1088] Let the Docker image start under an arbitrary uid in group 0 #1091 moved the COPY of the scripts to --chown=$OPENDJ_USER:0. The new line copies healthcheck.sh the same way: COPY --chown=$OPENDJ_USER:0 healthcheck.sh /opt/opendj/healthcheck.sh. This is the only change to what the commit does. The script only reads, so under an arbitrary uid it needs the group 0 and the +x it already gets, nothing more.
• build.yml: both PRs add a step after Docker test custom password in each docker job. Both steps are kept, Docker test arbitrary uid first and then Docker test health check. The arbitrary uid step waits for healthy, so from now on it also exercises healthcheck.sh under a uid only in group 0.

The description is updated to match.

[maximthomas]

praise: The probe now checks what it should without holding the root password.

• healthcheck.sh:39 passes the password with --bindPasswordFile, so it no longer shows up in /proc/<pid>/cmdline.
• The rc=53 check at build.yml:585-587 proves that reject-unauthenticated-requests has taken effect before stays_healthy runs, so the bind case cannot pass without actually testing anything.
• pom.xml adds healthcheck.sh to Dockerfile.zip, which both CI and the release build the image from.

---

suggestion (non-blocking): Say in the README that existing instances which reject unauthenticated requests need HEALTHCHECK_BIND_DN before they upgrade.

opendj-packages/opendj-docker/healthcheck.sh:42-43, opendj-packages/opendj-docker/README.md:33-46

Take an instance with reject-unauthenticated-requests:true in its volume config whose root password is still ROOT_PASSWORD. It is healthy on 5.1.2, because the old probe bound as root. On this image, run.sh:44-48 upgrades it and touches the marker. The anonymous probe then gets 53, and after three retries the container is unhealthy even though it is serving. Swarm replaces it, and compose services that depend on service_healthy never start. The README explains the variables for such instances, but it does not say that an instance which is already running has to set them before it upgrades.

Upgrading from an image that probed as the root user: an instance with
`reject-unauthenticated-requests:true` is now probed anonymously and turns `unhealthy`
unless `HEALTHCHECK_BIND_DN` and `HEALTHCHECK_BIND_PASSWORD_FILE` are set before it starts
on this image.

---

suggestion (non-blocking): Pass --noPropertiesFile so a tools.properties in the image user's home cannot turn the anonymous probe into a bind.

opendj-packages/opendj-docker/healthcheck.sh:42

/opt/opendj/bin/ldapsearch is the toolkit LDAPSearch. Unless it gets --noPropertiesFile, it loads ~/.opendj/tools.properties (ArgumentParser.java:383-403, LDAPSearch.java:263-269), and the image creates /home/opendj with useradd -m. Suppose an operator puts ldapsearch.bindDN/bindPassword there so they can use the CLI through docker exec. The probe then binds with those credentials, and rotating that password makes the container unhealthy again: the #1092 failure through a different path. The old probe passed both bind arguments itself, so a properties file could not change how it bound.

/opt/opendj/bin/ldapsearch --noPropertiesFile --hostname localhost --port "${LDAPS_PORT:-1636}" --useSsl --trustAll \
  "${BIND_ARGS[@]}" --baseDN "" --searchScope base "(objectClass=*)" 1.1 || exit 1

---

suggestion (non-blocking): No CI case expects the probe to fail, so a probe that always exits 0 would pass CI.

.github/workflows/build.yml:559-589, :756-786, opendj-packages/opendj-docker/healthcheck.sh:43

Both new cases, and every older docker step, only assert that a container becomes or stays healthy. The rc=53 check runs its own ldapsearch rather than the probe. If || exit 1 becomes || true, or the ldapsearch is replaced by exit 0, both jobs stay green. That is the opposite of #1092: the container reports healthy while the server refuses the probe.

          test "$rc" = 53
          # the probe itself reports the refusal
          rc=0
          docker exec -e HEALTHCHECK_BIND_DN= test_health_bind /opt/opendj/healthcheck.sh || rc=$?
          test "$rc" = 1

Pin: the || true and exit 0 mutants both make the final test "$rc" = 1 fail. Add the same lines to the alpine job.

---

suggestion (non-blocking): stays_healthy checks the status once after 45 s, which does not show that every probe since then passed.

.github/workflows/build.yml:566-571, :763-768

A successful probe resets FailingStreak to 0 and sets the status back to healthy. So a full unhealthy episode that recovers (fail, fail, fail, pass) still reads back as healthy 0, and so do alternating failures. The regressions this PR targets fail on every probe, so they are still caught. What slips through is intermittent failures. Either keep the check and reword the comment to "no failing streak at the end of the window", or check every probe:

          stays_healthy() {
            # Docker keeps the last 5 probes (>= 20 s at a 5 s interval): every one must have passed
            for _ in 1 2 3; do
              sleep 15
              test -z "$(docker inspect --format='{{range .State.Health.Log}}{{if ne .ExitCode 0}}x{{end}}{{end}}' "$1")"
            done
            test "$(docker inspect --format='{{.State.Health.Status}}' "$1")" = healthy
          }

Pin: a probe that fails once in the middle of the window makes the check fail.

---

suggestion (non-blocking): Nothing tests that the password never appears on a command line.

opendj-packages/opendj-docker/healthcheck.sh:39

Suppose BIND_ARGS used --bindPassword "$(cat "${HEALTHCHECK_BIND_PASSWORD_FILE}")" instead. It binds with the same password, both cases stay green, and the /proc/<pid>/cmdline exposure this PR removes would be back.

          docker exec test_health_bind sh -c 'end=$(($(date +%s)+12)); while [ "$(date +%s)" -lt "$end" ]; do for f in /proc/[0-9]*/cmdline; do tr "\0" " " < "$f" 2>/dev/null; echo; done | grep -q -- "--bindPassword[ ]" && exit 1; sleep 0.2; done; exit 0'

Pin: this samples the command lines for two probe intervals, and the mutant's probe JVM shows --bindPassword password. The [ ] keeps the check from matching its own command line.

---

suggestion (non-blocking): No CI case runs the guard for an unreadable password file.

opendj-packages/opendj-docker/healthcheck.sh:35-38

test_health leaves HEALTHCHECK_BIND_DN unset, and test_health_bind mounts a readable file, so line 36 never runs. If the guard is deleted, the health status does not change, because ldapsearch rejects the file itself. Only the explanatory message in .State.Health.Log is lost.

          rc=0
          out=$(docker exec -e HEALTHCHECK_BIND_PASSWORD_FILE=/nonexistent test_health_bind /opt/opendj/healthcheck.sh) || rc=$?
          test "$rc" = 1
          grep -q 'is not a readable file' <<< "$out"

---

issue (non-blocking): A single probe that fails just before the 45 s check fails the job, even though the container is healthy.

.github/workflows/build.yml:568-571, :573, :581

The status must be exactly healthy 0 at one moment, while the probe starts a second JVM every 5 s next to the server in a 512m container. One failure that lands just before the check leaves healthy 1. I found nothing that would cause such a failure. Not run: that needs repeated runs of the step on ubuntu-latest, and the docker jobs have not reported on this head yet. If it does flake, the ERR trap already prints .State.Health for both containers.

[vharseko]

@maximthomas, thanks. Round 1 is in 1c49ed2. Six of the seven points are taken, #4 and #5 in a slightly different form; #7 is left as it is.

1. README, upgrade – taken. The Health check section now says that images before this one probed as the root user, so an instance that rejects unauthenticated requests has to set HEALTHCHECK_BIND_DN and HEALTHCHECK_BIND_PASSWORD_FILE before it starts on this one, or it turns unhealthy while it serves.
2. --noPropertiesFile – taken, with a comment in healthcheck.sh saying why. Checked in a container: with bindDN/bindPassword=wrong_password in /home/opendj/.opendj/tools.properties, the probe without the flag fails with 49 (Invalid Credentials), and with it exits 0.
3. A case where the probe has to fail – taken as suggested, in both jobs: healthcheck.sh run with HEALTHCHECK_BIND_DN cleared on the instance that rejects unauthenticated requests must exit 1. The || true mutant fails the step there.
4. stays_healthy – taken, but not with the .State.Health.Log check. That log holds the last five probes including the failed ones from the start period, while the marker is still missing. The first check would come about 15 s after the container turned healthy, so the log would usually still hold some of those failures, and the step would fail with a correct probe. In the local mutant runs the log read back at the end of the step still held two ExitCode: 1 entries from the bootstrap while the status was healthy 0. Instead, stays_healthy now reads Status FailingStreak every 2 s for 45 s and requires healthy 0 every time. A failed probe keeps the streak at 1 or more until the next probe passes, at least 5 s later, so any single failure in the window is seen, and the start period does not matter. The comment above the function says so. Pin: a probe that fails every fourth call fails the step at the first stays_healthy.
5. Password on a command line – taken, with a different match. --bindPassword[ ] misses -w <pwd> and --bindPassword=<pwd>. The bind case now runs with a root password of its own (hc_secret_1092, also in the password file and used by dsconfig), and the command lines of every process in the container are searched for that value for 12 s ([h]c_secret_1092 keeps the check from matching itself). The sampling runs after stays_healthy, long after the dsconfig that takes the password on its own command line has exited. Pin: the --bindPassword "$(cat …)" mutant fails there.
6. Unreadable password file – taken as suggested. Pin: with the guard removed, ldapsearch rejects /nonexistent itself, the exit code is still 1, and the step fails on the missing message.
7. One failed probe just before the check – left as it is. The step is meant to fail on any failed probe in the window, and after restore backup from OpenDJ 2.4 to 4.1.5 An error occurred while executing the task defined in entry ds-task-id=20180328163614145,cn=Scheduled Tasks,cn=Tasks: NullPointerException (TaskUtils.java:74 RestoreTask.java:247 Task.java:965 TaskThread.java:179)" #4 it is even stricter. As you say, nothing points to such a failure, and none showed up in the runs below. If it does happen, the ERR trap prints .State.Health for both containers, and that will show what failed.

Verification: the CI step was extracted from build.yml and run locally against 5.1.2 images, with healthcheck.sh mounted over the image's copy:

Run	Result
this branch, Debian image	passes
this branch, Alpine image	passes
|| exit 1 → || true	fails: the probe with HEALTHCHECK_BIND_DN cleared exits 0
--bindPasswordFile → --bindPassword "$(cat …)"	fails: the command line check finds the password
guard for the unreadable file removed	fails: no is not a readable file in the output
probe that fails every fourth call	fails: the first stays_healthy

[maximthomas]

praise: Round 1 is taken in full and the new checks are stronger than the ones suggested.

• opendj-packages/opendj-docker/healthcheck.sh:43 passes --noPropertiesFile, so a tools.properties in /home/opendj/.opendj can no longer turn the anonymous probe into a bind.
• The command-line check (.github/workflows/build.yml:603) searches for the password value itself ([h]c_secret_1092), so it catches -w, --bindPassword= and --bindPassword "$(cat …)" alike.
• set -o errtrace (.github/workflows/build.yml:563) makes the ERR trap fire for a check that fails inside stays_healthy.

---

suggestion (non-blocking): No CI case pins --noPropertiesFile; removing it leaves both docker jobs green.

opendj-packages/opendj-docker/healthcheck.sh:43, .github/workflows/build.yml:578-579, :790-791

The toolkit parser reads ~/.opendj/tools.properties only when the file exists. Nothing in the image or in the "Docker test health check" steps creates /home/opendj/.opendj/tools.properties, so without the flag every case runs with the same arguments and the mutant survives both jobs. You checked this by hand (reply, point 2). Writing the same file in the rotation case turns that check into a CI pin.

docker exec test_health /opt/opendj/bin/ldappasswordmodify --hostname localhost --port 1636 --useSsl --trustAll --bindDN "cn=Directory Manager" --bindPassword initial_password --currentPassword initial_password --newPassword rotated_password
docker exec test_health sh -c 'mkdir -p /home/opendj/.opendj && printf "bindDN=cn=Directory Manager\nbindPassword=wrong_password\n" > /home/opendj/.opendj/tools.properties'
stays_healthy test_health

Pin: without --noPropertiesFile, every probe after the new line fails with 49 (Invalid Credentials) and stays_healthy test_health goes red. With the flag, the case stays green. Add the same line to the Alpine job.

---

suggestion (non-blocking): The unreadable-password-file case only uses a path that does not exist, so a guard weakened from -r to -e or -f still passes.

opendj-packages/opendj-docker/healthcheck.sh:36, .github/workflows/build.yml:597-600, :809-812

/nonexistent fails -r, -e and -f alike, so the weakened guard prints the same message and exits 1. The case the message describes, a file that exists but that the probe user cannot read, never runs: the mounted file is 644 (:584). Your pin (reply, point 6) catches a removed guard, not a weakened one. Container health is the same either way, so only the message is unpinned.

docker exec test_health_bind sh -c 'touch /tmp/unreadable_password && chmod 000 /tmp/unreadable_password'
rc=0
out=$(docker exec -e HEALTHCHECK_BIND_PASSWORD_FILE=/tmp/unreadable_password test_health_bind /opt/opendj/healthcheck.sh) || rc=$?
test "$rc" = 1
grep -q 'is not a readable file' <<< "$out"

Pin: the image runs as opendj (uid 1001), not as root, so mode 000 is unreadable to the probe. Under -e or -f the file passes the guard, ldapsearch fails on it with its own error on stderr, and the grep fails.

---

suggestion (non-blocking): The two health-check containers start with --rm, so the ERR trap has nothing to print when a bootstrap fails.

.github/workflows/build.yml:576, :585, :788, :797

If setup fails before it starts the server, run.sh:106 execs start-ds --nodetach, which exits when there is no usable config (#1088: exit 98). PID 1 exits and --rm removes the container. The wait loop then runs until timeout 3m, and the trap (:564) prints only "No such container" for docker logs and .State.Health. A check that fails after the container is healthy still gets its logs. The earlier docker steps (:526, :541, :551) are built the same way, so this repeats an existing pattern.

docker run -it -d --memory="512m" --health-interval=5s -e ROOT_PASSWORD=initial_password --name=test_health $IMAGE

Or: also leave the wait loop once {{.State.Running}} is false, so the step fails at once instead of after 3 minutes. Runners are ephemeral, and the docker kill at the end stays.

[vharseko]

@maximthomas, thanks. Round 2 is in 4621a5c. Your review covered 1c49ed2; since then the branch has also been rebased and has one more commit, described first.

Rebased onto master (d30ff78). The merge of #1094 made build.yml conflict: both PRs add a step in the same place in each docker job. Both steps are kept, Docker test replication first and then Docker test health check. Nothing else changed in the two earlier commits.

b272e53 – the bootstrap keeps the root password off the command line as well. In round 1 dsconfig in the bind case still took the password on its command line. Moving it to --bindPasswordFile led on to bootstrap/setup.sh, which gave the root password to setup, dsconfig, import-ldif and ldapmodify on their command lines for the whole bootstrap, where the ps of the Docker host lists it to every user of the host.

• setup.sh now follows replicate.sh from [#1084] Keep the root password out of the log and off the command line when a Docker container joins replication #1094: mktemp -p /dev/shm opendj-setup.XXXXXX (owner-only, falls back to mktemp), removed by an EXIT trap, and read by --rootUserPasswordFile / --bindPasswordFile. run.sh runs it with sh, so it stays POSIX.
• run.sh removes /dev/shm/opendj-setup.* a killed setup.sh leaves behind, next to opendj-replicate.*.
• ldapmodify used to get -w $ROOT_PASSWORD unquoted, so a password with a space in it broke the loading of bootstrap/schema and bootstrap/data, and the container never turned healthy. That is fixed as a side effect.
• CI, in the replication step: the grep for a password on a command line covers setup.sh too, --rootUserPassword included; the check that nothing is left in /tmp or /dev/shm covers the master as well; and it no longer skips /tmp/hsperfdata_*. That exclusion was there because the old HEALTHCHECK bound as root, and after this PR no process left running has the root password on its command line.

Verified locally on 5.1.2 images with the branch's scripts. Every process's command line was sampled from the container start until the bootstrap marker, with bootstrap/schema and bootstrap/data mounted and the password setup secret 1092: 0 hits on Debian and Alpine, both LDIFs loaded, /dev/shm empty. With master's setup.sh the password shows on the command lines of setup, its JVMs, import-ldif, dsconfig and ldapmodify, and the grep in CI finds seven lines. Both Docker test replication and Docker test health check were run as extracted from build.yml, with only the waits relaxed for a slower host, and pass on both images.

Round 2 – all three points taken, in both jobs:

1. --noPropertiesFile in CI – taken as suggested: after the password change, the rotation case writes bindDN/bindPassword=wrong_password to /home/opendj/.opendj/tools.properties before stays_healthy. Pin: without the flag the probe fails right after the file is written and stays_healthy test_health goes red.
2. Unreadable password file – taken: the case now runs for /nonexistent and for a file that exists with mode 000. Pin: with the guard weakened to -e, the 000 file passes the guard, ldapsearch fails on it with FileNotFoundException(... (Permission denied)), and the grep for the message fails.
3. --rm – taken with the first option: the two containers run without --rm, and the ERR trap and the end of the step remove them with docker rm -f. I did not change the wait loop. Checked with a setup.sh that fails at once: the trap prints the container's log, mutant: bootstrap fails and setup.sh failed, this container will not report itself healthy included. In that run the container did not exit (run.sh keeps a container whose bootstrap failed running, to be looked at), so it would have been there under --rm too. For a container that does exit, keeping it is what makes the difference.

Run (health check step)	Result
this branch, Debian / Alpine	passes
--noPropertiesFile removed	fails: first stays_healthy
guard -r → -e	fails: no is not a readable file for the 000 file
setup.sh failing at once	fails at the wait; the trap prints the container's log