Skip to content

Improve v2 SDK test robustness and failure diagnostics - #1325

Open
RafaelPChequer wants to merge 7 commits into
main_video_v3from
rpc/apex-16463-e2e-tests-fix-and-improve
Open

Improve v2 SDK test robustness and failure diagnostics#1325
RafaelPChequer wants to merge 7 commits into
main_video_v3from
rpc/apex-16463-e2e-tests-fix-and-improve

Conversation

@RafaelPChequer

Copy link
Copy Markdown
Contributor

Description

Enhanced end-to-end tests for v2 WebRTC in the SDK by introducing retry mechanisms for call creation and active status checks, implementing call stability monitoring to ensure consistent call states over time, and adding fallback validations for audio metrics using audioLevel when totalAudioEnergy is unavailable (common in certain codec scenarios). These improvements boost test robustness, provide better diagnostics for intermittent failures, and reduce flakiness in network-dependent tests like TURN connections.

Type of change

  • Internal refactoring
  • Bug fix (bugfix - non-breaking)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)

Code snippets

In case of new feature or breaking changes, please include code snippets.

// Utility function to create call with better error handling
export const createCallWithRetry = async (resource: string, inlineLaml: string, codecs?: string, maxRetries = 3) => {
  for (let i = 0; i < maxRetries; i++) {
    try {
      const result = await createCallWithCompatibilityApi(resource, inlineLaml, codecs)
      if (result === 201) {
        return result
      }
      console.log(`Attempt ${i + 1} failed with status ${result}`)
    } catch (error) {
      console.log(`Attempt ${i + 1} failed with error:`, error)
    }
    
    if (i < maxRetries - 1) {
      await new Promise(resolve => setTimeout(resolve, 2000))
    }
  }
  throw new Error(`Failed to create call after ${maxRetries} attempts`)
}
// Utility function to wait for call stability
export const waitForCallStability = async (page: Page, minDuration = 5000) => {
  const startTime = Date.now()
  let lastStatus = ''
  let stableCount = 0
  
  while (Date.now() - startTime < minDuration) {
    const currentStatus = await page.locator('#callStatus').textContent()
    
    if (currentStatus === lastStatus) {
      stableCount++
    } else {
      stableCount = 0
      lastStatus = currentStatus || ''
    }
    
    // If status has been stable for 2 seconds, consider it stable
    if (stableCount > 2) {
      break
    }
    
    await page.waitForTimeout(1000)
  }
  
  return lastStatus
}
export const waitForCallActive = async (page: Page, timeout = 30000) => {
  const callStatus = page.locator('#callStatus')
  expect(callStatus).not.toBe(null)
  
  try {
    await expect(callStatus).toContainText('-> active', { timeout })
    return true
  } catch (error) {
    const currentStatus = await callStatus.textContent()
    console.log(`Call status check failed. Current status: ${currentStatus}`)
    throw error
  }
}
const expectCallActiveWithRetry = async (page: Page, maxRetries = 3) => {
  for (let i = 0; i < maxRetries; i++) {
    try {
      const callStatus = page.locator('#callStatus')
      await expect(callStatus).toContainText('-> active', { timeout: 15000 })
      return true
    } catch (error) {
      console.log(`Attempt ${i + 1} failed to get active status:`, error)
      if (i === maxRetries - 1) throw error
      await page.waitForTimeout(2000)
    }
  }
  return false
}
const totalAudioEnergy = audioStats['inbound-rtp']['totalAudioEnergy']
const packetsReceived = audioStats['inbound-rtp']['packetsReceived']
const audioLevel = audioStats['inbound-rtp']['audioLevel']

if (totalAudioEnergy !== undefined && totalAudioEnergy !== null ) {
  expect(totalAudioEnergy).toBeGreaterThan(minTotalAudioEnergy)
} else {
  console.log('Warning: totalAudioEnergy was missing from the report!')
  
  if (audioLevel !== undefined && audioLevel !== null) {
    console.log(`Using audioLevel as fallback: ${audioLevel}`)
    expect(audioLevel).toBeGreaterThan(minTotalAudioEnergy / 10) // audioLevel is typically higher than totalAudioEnergy
  } else if (packetsReceived) {
    // We still want the right amount of packets
    expect(packetsReceived).toBeGreaterThan(minPacketsReceived)
  } else {
    throw new Error(`Unable to validate audio. No suitable metrics available.`)
  }
}

@changeset-bot

changeset-bot Bot commented Oct 22, 2025

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 4e54cd2

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@iAmmar7 iAmmar7 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The v2 e2e tests were moved to the rtc-testing repo and they no longer need to be in this repo. If you are making improvements, you probably would want to do it in that repo. Please check with @giavac

Comment thread internal/e2e-js/utils.ts Outdated

if (audioLevel !== undefined && audioLevel !== null) {
console.log(`Using audioLevel as fallback: ${audioLevel}`)
expect(audioLevel).toBeGreaterThan(minTotalAudioEnergy / 10) // audioLevel is typically higher than totalAudioEnergy

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand this part - audioLevel is the current level of the audio. It could be zero if there's silence. Why would we need to expect it higher than 10% of the expected minimum total audio energy?
(also the comment is not consistent with the code)

@RafaelPChequer
RafaelPChequer requested a review from giavac October 29, 2025 15:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants