Skip to content

Fix GRPC channel leak - #39663

Draft
shunping wants to merge 1 commit into
apache:masterfrom
shunping:fix-channel-leak
Draft

Fix GRPC channel leak#39663
shunping wants to merge 1 commit into
apache:masterfrom
shunping:fix-channel-leak

Conversation

@shunping

@shunping shunping commented Aug 6, 2026

Copy link
Copy Markdown
Collaborator

Description

Problem

When running Portable/YAML pipelines, gRPC outputs severe error logs indicating that ManagedChannel instances were garbage collected without being shut down:

SEVERE: *~*~*~ Previous channel ManagedChannelImpl{logId=..., target=...} was garbage collected without being shut down! ~*~*~*
    Make sure to call shutdown()/shutdownNow()
java.lang.Throwable: java.lang.RuntimeException: ManagedChannel allocation site
    at org.apache.beam.vendor.grpc.v1p69p0.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.<init>(...)

Cause

Three places in the Java Fn Harness allocated gRPC ManagedChannel instances without shutting them down upon task completion or harness shutdown:

  • ExternalWorkerService: A ManagedChannel was created to query the provision service for worker options, but was left unclosed after obtaining provisionInfo.
  • BeamFnDataGrpcClient & BeamFnDataGrpcMultiplexer: BeamFnDataGrpcClient created a ManagedChannel per BeamFnDataGrpcMultiplexer, but neither stored nor shut down the channel when close() was called.
  • FnHarness: The control service ManagedChannel was created during harness startup but was never shut down when FnHarness.main() finished.

outboundObserver.onError(
Status.CANCELLED.withDescription("Multiplexer hanging up").asException());
inboundObserver.onCompleted();
if (channel != null) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. I was hopping the leak could have resolved in #39390, where BeamFnDataGrpcMultiplexer is closed there. Turns out there are still missing piece here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants