oc-url-fetch

A small server-side helper for OpenCloud that downloads a file from a URL and stores it into the caller's space.

The OpenCloud web client cannot do this on its own: a browser cannot read a cross-origin response (CORS), and there is no server-side fetch in the web client. oc-url-fetch performs the download server-to-server and streams the result straight into OpenCloud's normal WebDAV upload pipeline, so ownership, assimilation, quota and postprocessing are handled by OpenCloud rather than by writing to disk behind its back.

How it works

1. A client calls POST /fetch with the source URL and a destination path, authenticating with their own OpenCloud app token (HTTP Basic) or OIDC bearer.
2. The service validates those credentials against OpenCloud (a Depth-0 PROPFIND on the destination) before making any outbound request, so it cannot be used as an open proxy with a bogus token.
3. It downloads the URL (with an SSRF guard, see below) as a background job and returns a job id straight away.
4. It uploads the bytes to the destination as the calling user, forwarding the Authorization header. The service stores no credentials.

Because the transfer runs in the background it survives the client disconnecting, which matters for the multi-gigabyte downloads this is meant for. The client polls GET /jobs/{id} for progress.

If the source advertises a Content-Length, the body is streamed straight through. If not, it is spooled to a temporary file first, because OpenCloud's WebDAV rejects HTTP chunked PUTs and the upload needs a known length.

Usage

go build -o oc-url-fetch .
./oc-url-fetch -host cloud.midvault.se -opencloud http://127.0.0.1:9200

Start a download:

curl -X POST http://127.0.0.1:8080/fetch \
  -u 'username:app-token' \
  -H 'Content-Type: application/json' \
  -d '{"url":"https://example.com/file.iso","target":"Downloads/"}'
# -> {"id":"<job-id>","target":"Downloads/file.iso"}

Poll its progress (same credentials; only the job's creator may read it):

curl -u 'username:app-token' http://127.0.0.1:8080/jobs/<job-id>
# -> {"id":"...","state":"downloading","bytes":12345,"total":67890,"target":"Downloads/file.iso","source":"..."}

state moves through starting -> downloading -> uploading -> done, or error with an error message. total is -1 when the source did not advertise a length. A target ending in / derives the filename from the URL, and missing parent folders are created automatically.

Endpoints

method	path	purpose
POST	/fetch	validate the caller, start a download job, return its id
GET	/jobs/{id}	progress of a job (scoped to the creator's token)
GET	/healthz	liveness check

Flags

flag	default	meaning
-addr	:8080	address to listen on
-base-path	(empty)	URL prefix the routes are mounted under, e.g. /url-fetch, when fronted by a reverse proxy that does not strip the prefix (the OpenCloud proxy does not)
-opencloud	http://127.0.0.1:9200	OpenCloud base URL for the WebDAV upload
-host	(empty)	Host header to send to OpenCloud, e.g. cloud.midvault.se
-tmpdir	(system default)	where to spool downloads of unknown length
-job-ttl	30m	how long a finished job stays queryable before it is forgotten
-job-timeout	6h	hard cap on a single download+upload
-allow-private	false	allow URLs that resolve to private/loopback/link-local addresses

SSRF guard

By default the service refuses to connect to private, loopback, link-local or unspecified addresses. The check runs in the dialer, so it also applies to redirects. Pass -allow-private only in a trusted environment.

Status

Downloads run as background jobs with progress polling, and the caller is verified against OpenCloud before any fetch. Uploads still use a single WebDAV PUT, so very large files go through the public data path; chunked tus uploads are the planned next step so big downloads survive proxy timeouts.

License

AGPL-3.0