Young

Hello,I encountered the same problem. But now I find a solution, which was found on the github repository prulong.https://github.com/princeton-pli/PruLong/blob/main/prulong/training/dataset.py.
StreamingDataset provides a replication parameter, which you can set to seq_parallel_size. In this way, you can get the same inputs in the same seq_parrallel_group.