airbyte fails to load after airbyte-protocol was updade to 0.14.4

[mannharleen]

import airbyte throws the following error since a few hours ago, when 0.14.4 was released.
It is working fine with 0.14.3!

NameError: name 'StreamDescriptor' is not defined

Stacktrace:
File /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.11/site-packages/airbyte_cdk/models/airbyte_protocol.py:58, in AirbyteStreamState()
     56 @dataclass
     57 class AirbyteStreamState:
---> 58     stream_descriptor: StreamDescriptor  # type: ignore [name-defined]
     59     stream_state: Optional[AirbyteStateBlob] = None

tested with:

• airbyte==0.23.0

[morgan-dgk]

Presumably some breaking change in airbyte_protocol_dataclasses package which this type was previously imported from

Rob Morris (@BrettNWoods) pointed out that #120 coincided with the appearance of this error. Seems like changes could be related?.

edit: have posted in Airbyte slack, looks like file used to generate python dataclasses may have been inadvertently deleted as part of #120.

Bash script used to generate Python dataclasses points to protocol-models/src/main/resources/airbyte_protocol in repo. This was deleted in #120.

[carlesoctav]

i've the same issues

[WojciechSzybaVeVe]

Same here using pyAirbyte along with Salesforce connector

[tcboles]

I am also running into issues with this. Let me know if I can be of help with a fix. Deleting that much code where other packages depend on it should have at the least been a minor version push and not a patch

[WojciechSzybaVeVe]

I was able to fix this by force install explicit older versions of affected modules:

airbyte-protocol-models-dataclasses==0.14.3
airbyte-protocol-models==0.14.3

(Including running pip install airbyte-protocol-models-dataclasses==0.14.3 inside .venv-source-salesforce/bin/python venv)

[davinchia]

Hi, these models were removed as they are part of a now-defunct migration, and I was doing some house keeping to keep things sane. I wasn't aware this could affect our older python connectors - our in house tests work off our CDK - apologises for the hassle!

Let me revert this to unblock ya'll. Can I have the connector version folks are seeing this on so I can better understand how this happened?

[davinchia]

In the meantime, you can resolve this issue by pinning to 0.14.3. I'd also recommend doing this in general.

[davinchia]

I've reverted this change in 0.14.5. Things should be back to normal now. Please let me know otherwise. Thanks and apologies again!

[morgan-dgk]

Thanks Davin Chia (@davinchia), couple of hopefully useful additions.

There's a handful of places in the Python CDK where these dataclasses are still used which affects newer declarative connectors built with the python CDK:

• ./models/well_known_types.py:5:from airbyte_protocol_dataclasses.models.well_known_types import *
• ./models/airbyte_protocol.py:8:from airbyte_protocol_dataclasses.models import *
• ./sources/streams/permissions/identities_stream.py:9:from airbyte_protocol_dataclasses.models import SyncMode
• ./sql/shared/sql_processor.py:16:from airbyte_protocol_dataclasses.models import AirbyteStateMessage

In our case we saw failures when the SPEC operation ran due to classes missing from airbyte_protocol module:

Traceback (most recent call last):
  File "/airbyte/source_declarative_abc/main.py", line 1, in <module>
    from source_declarative_abc.run import run
  File "/airbyte/source_declarative_abc/source_declarative_abc/run.py", line 6, in <module>
    from airbyte_cdk.entrypoint import AirbyteEntrypoint, launch, logger
  File "/usr/local/lib/python3.12/site-packages/airbyte_cdk/__init__.py", line 51, in <module>
    from .config_observation import (
  File "/usr/local/lib/python3.12/site-packages/airbyte_cdk/config_observation.py", line 15, in <module>
    from airbyte_cdk.models import (
  File "/usr/local/lib/python3.12/site-packages/airbyte_cdk/models/__init__.py", line 8, in <module>
    from .airbyte_protocol import (
  File "/usr/local/lib/python3.12/site-packages/airbyte_cdk/models/airbyte_protocol.py", line 57, in <module>
    class AirbyteStreamState:
  File "/usr/local/lib/python3.12/site-packages/airbyte_cdk/models/airbyte_protocol.py", line 58, in AirbyteStreamState
    stream_descriptor: StreamDescriptor  # type: ignore [name-defined]
                       ^^^^^^^^^^^^^^^^
NameError: name 'StreamDescriptor' is not defined

Finally, not sure how easy it is to pin this lib in Airbyte Platform OSS deployment. We've noticed a couple of times that some flows, like setting up a new source connection appear to always pull an image with the latest version of the CDK - regardless of the version specified in the manifest. Some other operations, like CHECK for existing sources do not seem to have this behaviour. Not 100% certain on this last point above though.