Currently, in the ingest pipeline, duplicate batches are checked by querying ingested_batches table in Clickhouse. This can potentially be further optimized by using a in-memory cache, bloom filter or something else to avoid the database query while retaining good enough success rates for duplicate batch detection and falling back to db query only if unavoidable.
Currently, in the ingest pipeline, duplicate batches are checked by querying
ingested_batchestable in Clickhouse. This can potentially be further optimized by using a in-memory cache, bloom filter or something else to avoid the database query while retaining good enough success rates for duplicate batch detection and falling back to db query only if unavoidable.