config block_ff_dim wrong
#18
by
florianvoss
- opened
Hey,
i am pretty sure, that there is a slight error in the config.json: "block_ff_dim": 12288,
Looking at the safetensor file the dimension should be 8 192.
Hi thanks for reaching out. The config is correct, but you are also correct that the resulting dimension is 8192. See https://github.com/huggingface/transformers/blob/8cb5963cc22174954e7dca2c0a3320b7dc2f4edc/src/transformers/models/lfm2/modeling_lfm2.py#L109 where the block_auto_adjust_ff_dim=True argument adjusts the dimension.
jimmysmith1919
changed discussion status to
closed