Hi,
Thanks for providing this good work. I'm cursious about the stage2 Single MCTP Module Training. In the paper, it said " We initialize the MCTP module using parameters from the final layer of the LLM, with the gradient detached from the LLM ". But the code does not contain this intialization process. So, how to achieve that, because of the dim of MCTP is different with the final layer of the LLM( and i'm not sure what module it denotes, llm_head or decoder layer?).
Hi,
Thanks for providing this good work. I'm cursious about the stage2 Single MCTP Module Training. In the paper, it said " We initialize the MCTP module using parameters from the final layer of the LLM, with the gradient detached from the LLM ". But the code does not contain this intialization process. So, how to achieve that, because of the dim of MCTP is different with the final layer of the LLM( and i'm not sure what module it denotes, llm_head or decoder layer?).