Feat/keyframe vlm integration - #70
Open
pkhyrn268 wants to merge 17 commits into
Open
Conversation
BGR 프레임 리스트 → (T, 2048) numpy array. fc 레이어 제거, avgpool 출력 사용. batch_size/device config화.
- Qwen2VLProvider: keyframe/inference.py Qwen2VL을 slowfast provider 인터페이스로 포팅 - VLMRefiner: keyframe_selector_input_dim config화 (기존 192 하드코딩 → 2048 기본값) - VLMRefiner: ResNet-50 extractor 로드 추가 - VLMRefiner.score_clip_frames(): ResNet+BiGRU 기반 per-clip 프레임 선택 후 VLM 호출
clip_lookup 파라미터 추가. clip_lookup이 있으면 event의 clip_ids 중 fast_score 상위 max_clips_per_event개를 선택해 clip당 VLM 1회 호출 후 max score로 집계. clip_lookup 없으면 기존 균등 샘플링 fallback.
- level: event 기본값 설정 - keyframe_selector_path/input_dim/device 추가 - frames_per_clip, max_clips_per_event 추가 - resnet_batch_size, peak_score_max 추가
- InternVLVLMProvider: 로컬 repo 의존성 제거, AutoModel/AutoTokenizer로 교체 - exp_mock.yaml: mock provider smoke test 용 - exp_baseline.yaml: InternVL2-8B 균등 샘플링 baseline - exp_A_internvl8b.yaml: InternVL2-8B + ResNet-50 BiGRU keyframe - exp_C_qwen2vl.yaml: Qwen2-VL-7B + ResNet-50 BiGRU keyframe
- Phi35VisionProvider: microsoft/Phi-3.5-vision-instruct - MiniCPMVProvider: openbmb/MiniCPM-V-2_6 - run_vlm_experiments.sh: GPU 메모리 해제 확인 후 순차 실행 (4B → Qwen2VL → MiniCPM → Phi3.5) - exp_B/C/D/E config 추가
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
…cc=89.2%) AI/keyframe/outputs/frame_selector_vlm.pth - 학습 label: InternVL2-4B VLM pseudo-label (train_vlm.json, 41,416 clips) - input_dim=2048 (ResNet-50 features), n_frames=6, epochs=50 - best val acc=89.2% (epoch 50)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
합치면 기존 AI 부분도 같이 수정되는거라, keyframe 부분 수정된 내용을 pr로만 올려놓았습니다.(future work)