Summary
The HTCondor ClassAd parsing in docker/prepare-env/prepare-env-cc-analysis.sh
works, but it's fragile. It extracts attributes with grep <attr> | awk '{print $NF}',
has no fallbacks when an attribute is missing, and formats the memory limit with a
string hack ($MEMORY".00MB"). This is the "careful parsing going forward" cleanup
flagged back in #8 — the functional part (cores/memory/name are forwarded to
dask-worker) already landed; this issue is purely about hardening it.
Why
- Fragile extraction.
grep "DaskWorkerName " (trailing space) and awk '{print $NF}'
rely on whitespace/field positioning and trailing-space tricks to avoid prefix
collisions. A line-anchored read of Attr = value is sturdier.
- No empty-value handling. If an attribute is missing, the script emits a broken
flag (e.g. --nthreads with no value) and the worker dies with a cryptic error
instead of a clear message.
- Memory unit hack.
MEMORY_MB_FORMATTED=$MEMORY".00MB" assumes RequestMemory
is always a clean integer MB value and concatenates a unit string. The attribute
intended for this is DaskWorkerMemory (bytes, set by dask-jobqueue), which
dask-worker --memory-limit accepts directly.
Proposed changes
Out of scope
DaskWorkerDisk handling (pointing --local-directory at writable scratch) is related
but tracked separately in #421.
References
Summary
The HTCondor ClassAd parsing in
docker/prepare-env/prepare-env-cc-analysis.shworks, but it's fragile. It extracts attributes with
grep <attr> | awk '{print $NF}',has no fallbacks when an attribute is missing, and formats the memory limit with a
string hack (
$MEMORY".00MB"). This is the "careful parsing going forward" cleanupflagged back in #8 — the functional part (cores/memory/name are forwarded to
dask-worker) already landed; this issue is purely about hardening it.Why
grep "DaskWorkerName "(trailing space) andawk '{print $NF}'rely on whitespace/field positioning and trailing-space tricks to avoid prefix
collisions. A line-anchored read of
Attr = valueis sturdier.flag (e.g.
--nthreadswith no value) and the worker dies with a cryptic errorinstead of a clear message.
MEMORY_MB_FORMATTED=$MEMORY".00MB"assumesRequestMemoryis always a clean integer MB value and concatenates a unit string. The attribute
intended for this is
DaskWorkerMemory(bytes, set by dask-jobqueue), whichdask-worker --memory-limitaccepts directly.Proposed changes
grep | awkattribute extraction with an anchored helper that reads^<attr> = <value>from$_CONDOR_JOB_AD.for the networking attributes the worker can't start without (ports, host).
DaskWorkerMemory(bytes) for--memory-limit, falling back toRequestMemory(MB) only if it's absent; drop the.00MBstring formatting.with the matching
--nthreads/--memory-limit.Out of scope
DaskWorkerDiskhandling (pointing--local-directoryat writable scratch) is relatedbut tracked separately in #421.
References
docker/prepare-env/prepare-env-cc-analysis.sh