Inconsistent BEV lane semantics causing incorrect offroad detection (dashed/solid line misclassification)

[moonzerokevin]

#59
Similar question of this issue with further experiments.

Summary

We observe systematic inconsistencies between BEV lane semantics and real-world driving rules in AlpaSim, which lead to incorrect offroad detection. This directly affects evaluation metrics such as avg_dist_between_incidents (AlpaSim score).

The issue is particularly evident in:

1. Rural / narrow roads without explicit centerline markings
2. Incorrect classification of dashed vs solid lane boundaries

---

Observations

We manually inspected several dozen rollout videos and found the following recurring issues:

1. Rural / narrow road scenarios (no centerline in reality)

In many cases:

• The real-world road has no visible centerline
• Vehicles (human drivers) would normally drive near or over the center when necessary (e.g., narrow roads with parked cars)

However, in BEV:

• The road center is interpreted as a non-drivable solid boundary

This leads to frequent false-positive offroad events.

clipgt-clipgt-07981e6a-22dd-4796-ad2f-1252037ecd28_0_97944a4e-24a6-11f1-82ab-4d6b050ee9c2_camera_front_wide_120fov_default.mp4

clipgt-clipgt-048b974e-1546-488a-b8f9-d32bff77f5aa_0_76c908fa-24a5-11f1-82ab-4d6b050ee9c2_camera_front_wide_120fov_default.mp4

---

2. Dashed line incorrectly treated as non-crossable (solid)

We observed multiple cases where:

• A white dashed line (lane divider) is present in the RGB view.
• But in BEV / drivability logic, it is treated as non-crossable (solid boundary)

clipgt-clipgt-05e74bef-0afa-468e-b679-b7f14079be30_0_97d57302-24a6-11f1-82ab-4d6b050ee9c2_camera_front_wide_120fov_default.mp4

---

3. Solid line incorrectly treated as crossable

Conversely, we also observed at least one case where:

• A white solid line appears in the scene
• But BEV treats it as crossable

clipgt-clipgt-0fa7060f-1481-4ff3-977e-510285e2515d_0_dc4b697a-24aa-11f1-82ab-4d6b050ee9c2_camera_front_wide_120fov_default.mp4

---

4. Simple bad BEV.

clipgt-clipgt-559d4ad6-aed7-4065-b25c-cff64aba28c0_0_2a53495c-2489-11f1-8063-e78570d9410b_camera_front_wide_120fov_default.mp4

Impact on Evaluation

This issue has a direct impact on key metrics:

• offroad detection becomes unreliable
• avg_dist_between_incidents (used as AlpaSim score, also reported in Alpamayo paper) becomes biased

Since many policies (including Alpamayo) already achieve low collision rates,
offroad becomes also a significant failure signal, and thus its correctness is important.

[maximilianigl]

Hi, thanks a lot for your effort in finding these examples, having them will be helpful in figuring out the issue!
I won't have time over the weekend but will have a look on Monday to figure out whether this is a data issue or a problem of our implementation in trajdata for reading the files (the problem there is that XODR is a weird format and doesn't really support road boundaries well, so our implementation is a bit hacky).