monsoon-nlp/dna-blockdiff

DNA and Block Diffusion

Untrained architecture test using the Block Diffusion architecture and AgroNT's six-nucleotide-length tokens.

Loading model

from transformers import AutoModelForMaskedLM
m = AutoModelForMaskedLM.from_pretrained(
    "monsoon-nlp/dna-blockdiff",
    trust_remote_code=True,
)

Generating text

cd bd3lms
python -u main.py \
    loader.eval_batch_size=1 \
    model=small \
    algo=bd3lm \
    algo.T=900 \
    algo.backbone=hf_dit \
    algo.sampler=analytic \
    data=openwebtext-split \
    model.length=2048 \
    block_size=4 \
    wandb=null \
    mode=sample_eval \
    eval.checkpoint_path="monsoon-nlp/dna-blockdiff" \
    model.attn_backend=sdpa \
    sampling.nucleus_p=0.9 \
    sampling.kv_cache=true \
    sampling.logdir=$PWD/sample_logs/samples_genlen_bd3lm_blocksize4 \
    data.tokenizer_name_or_path="monsoon-nlp/dna-blockdiff"