bert-chunker-3.5

GitHub

bert-chunker-3.5 is a text chunker based on BertForTokenClassification to predict the start token of chunks (for use in RAG, etc), and using a sliding window it cuts documents of any size into chunks. We see it as an alternative of Kamradt semantic chunker, but specially, it not only works for the structured texts, but also the unstructured and messy texts.

As a new version of bc3, it has a simpler chunk strategy, less bugs, trained on larger model length, more competitive performance.

Usage

Run the following:

import torch
from transformers import AutoTokenizer, BertForTokenClassification
from collections import deque

model_path =r"tim1900/bert-chunker-3.5"

tokenizer = AutoTokenizer.from_pretrained(
    model_path,
    padding_side="right",
    trust_remote_code=True,
)

device = "cpu"  # or 'cuda'

model = BertForTokenClassification.from_pretrained(model_path).to(device)

def split_text(text, max_tokens_per_chunk):
    MAX_LEN = 512
    MAX_USE_LEN = MAX_LEN-2
    MASK_EDGE =50
    EFFECT_SIZE = MAX_USE_LEN - MASK_EDGE*2
    with torch.no_grad():
        unk_id = tokenizer.unk_token_id
        print("\n>>>>>>>>> Tokenizing text...")
        tokens = tokenizer(text, return_tensors="pt", truncation=False)
        print("\n>>>>>>>>> Chunking text...")
        input_ids = tokens['input_ids'].squeeze()[1:-1]
        len_of_input_ids = len(input_ids)
        left =  EFFECT_SIZE - len_of_input_ids%EFFECT_SIZE 
        full_input_ids = torch.cat([ torch.tensor([unk_id]*MASK_EDGE), input_ids, torch.tensor([unk_id]*(MASK_EDGE + left))])
        prob_list = []
        start_idx = 0
        while(1):
            end_idx = start_idx + MAX_USE_LEN
            window_input_ids =torch.cat([ torch.tensor([tokenizer.cls_token_id]),  full_input_ids[start_idx:end_idx], torch.tensor([tokenizer.sep_token_id])])
            window_input_ids =  window_input_ids.to(model.device)
            output = model(
                input_ids=window_input_ids.unsqueeze(0),
                attention_mask=torch.ones(1, window_input_ids.shape[0], device=model.device),
            )
            logits = output["logits"][:, 1+MASK_EDGE:-1-MASK_EDGE, :]
            logit_diff = logits[:, :, 1] - logits[:, :, 0]
            logit_diff = logit_diff.squeeze().tolist()
            prob_list = prob_list + logit_diff
            start_idx = start_idx + EFFECT_SIZE
            if end_idx==len(full_input_ids):
                break
    prob_list = prob_list[:len_of_input_ids]
    def find_split_points(numbers, m):
        if m >= len(numbers): return [0]
        def sliding_window_max_indices(arr, M):
            dq = deque()
            result = []
            for i in range(len(arr)):
                while dq and dq[0] < i - M + 1:
                    dq.popleft()
                while dq and arr[dq[-1]] <= arr[i]:
                    dq.pop()
                dq.append(i)
                if i >= M - 1:
                    result.append(dq[0])
            return result

        max_pos = sliding_window_max_indices(numbers, m)

        splits = [0]
        st = 1

        while st < len(max_pos):
            split = max_pos[st]
            splits.append(split)
            st = split + 1

        return splits

    
    token_split_points = find_split_points(prob_list, max_tokens_per_chunk)

    str_split_points = [tokens.token_to_chars(pos + 1).start for pos in token_split_points]

    if str_split_points[0] != 0:
        str_split_points[0] = 0

    substrings = [
        text[i:j] for i, j in zip(str_split_points, str_split_points[1:] + [len(text)])
    ]

    return substrings, token_split_points

# chunking 

txt = r"""The causes and effects of dropouts in vocational and professional education are more pressing than ever. A decreasing attractiveness of vocational education, particularly in payment and quality, causes higher dropout rates while hitting ongoing demographic changes resulting in extensive skill shortages for many regions. Therefore, tackling the internationally high dropout rates is of utmost political and scientific interest. This thematic issue contributes to the conceptualization, analysis, and prevention of vocational and professional dropouts by bringing together current research that progresses to a deeper processual understanding and empirical modelling of dropouts. It aims to expand our understanding of how dropout and decision processes leading to dropout can be conceptualized and measured in vocational and professional contexts. Another aim is to gather empirical studies on both predictors and dropout consequences. Based on this knowledge, the thematic issue intends to provide evidence of effective interventions to avoid dropouts and identify promising ways for future dropout research in professional and vocational education to support evidence-based vocational education policy.

We thus welcome research contributions (original empirical and conceptual/measurement-related articles, literature reviews, meta-analyses) on dropouts (e.g., premature terminations, intentions to terminate, vertical and horizontal dropouts) that are situated in vocational and professional education at workplaces, schools, or other tertiary professional education institutions. 


Part 1 of the thematic series outlines central theories and measurement concepts for vocational and professional dropouts. Part 2 outlines measurement approaches for dropout. Part 3 investigates relevant predictors of dropout. Part 4 analyzes the effects of dropout on an individual, organizational, and systemic level. Part 5 deals with programs and interventions for the prevention of dropouts.

We welcome papers that include but are not limited to:

Theoretical papers on the concept and processes of vocational and professional dropout or retention
Measurement approaches to assess dropout or retention
Quantitative and qualitative papers on the causes of dropout or retention
Quantitative and qualitative papers on the effects of dropout or retention on learners, providers/organizations and the (educational) system
Design-based research and experimental papers on dropout prevention programs or retention
Submission instructions
Before submitting your manuscript, please ensure you have carefully read the Instructions for Authors for Empirical Research in Vocational Education and Training. The complete manuscript should be submitted through the Empirical Research in Vocational Education and Training submission system. To ensure that you submit to the correct thematic series please select the appropriate section in the drop-down menu upon submission. In addition, indicate within your cover letter that you wish your manuscript to be considered as part of the thematic series on series title. All submissions will undergo rigorous peer review, and accepted articles will be published within the journal as a collection.

Lead Guest Editor:
Prof. Dr. Viola Deutscher, University of Mannheim
viola.deutscher@uni-mannheim.de

Guest Editors:
Prof. Dr. Stefanie Findeisen, University of Konstanz
stefanie.findeisen@uni-konstanz.de 

Prof. Dr. Christian Michaelis, Georg-August-University of Göttingen
christian.michaelis@wiwi.uni-goettingen.de

Deadline for submission
This Call for Papers is open from now until 29 February 2023. Submitted papers will be reviewed in a timely manner and published directly after acceptance (i.e., without waiting for the accomplishment of all other contributions). Thanks to the Empirical Research in Vocational Education and Training (ERVET) open access policy, the articles published in this thematic issue will have a wide, global audience.

Option of submitting abstracts: Interested authors should submit a letter of intent including a working title for the manuscript, names, affiliations, and contact information for all authors, and an abstract of no more than 500 words to the lead guest editor Viola Deutscher (viola.deutscher@uni-mannheim.de) by July, 31st 2023. Due to technical issues, we also ask authors who already submitted an abstract before May, 30th to send their abstracts again to the address stated above. However, abstract submission is optional and is not mandatory for the full paper submission.

Different dropout directions in vocational education and training: the role of the initiating party and trainees’ reasons for dropping out
The high rates of premature contract termination (PCT) in vocational education and training (VET) programs have led to an increasing number of studies examining the reasons why adolescents drop out. Since adol...

Authors:Christian Michaelis and Stefanie Findeisen
Citation:Empirical Research in Vocational Education and Training 2024 16:15
Content type:Research
Published on: 6 August 2024"
"""
chunks, token_pos = split_text(txt, max_tokens_per_chunk = 400)

# print chunks
for i, (c, t) in enumerate(zip(chunks, token_pos)):
    print(f"========================== CHUNK: {i}  TOKEN_IDX: {t} ==========================")
    print(c)

Evaluation

The following RAG evaluation is done by code from brandonstarxel/chunking_evaluation, most of the following results come from Evaluating Chunking Strategies for Retrieval.

Chunking	Size	Overlap	Recall	Precision	PrecisionΩ	IoU	Time complexity by token number N	Is max chunk size strictly controlable
Recursive	<= 800	400	85.4 ± 34.9	1.5 ± 1.3	6.7 ± 5.2	1.5 ± 1.3	O(N)	Yes
TokenText	800	400	87.9 ± 31.7	1.4 ± 1.1	4.7 ± 3.1	1.4 ± 1.1	O(N)	Yes
Recursive	<= 400	200	88.1 ± 31.6	3.3 ± 2.7	13.9 ± 10.4	3.3 ± 2.7	O(N)	Yes
TokenText	400	200	88.6 ± 29.7	2.7 ± 2.2	8.4 ± 5.1	2.7 ± 2.2	O(N)	Yes
Recursive	<= 400	0	89.5 ± 29.7	3.6 ± 3.2	17.7 ± 14.0	3.6 ± 3.2	O(N)	Yes
TokenText	400	0	89.2 ± 29.2	2.7 ± 2.2	12.5 ± 8.1	2.7 ± 2.2	O(N)	Yes
Recursive	<= 200	0	88.1 ± 30.1	7.0 ± 5.6	29.9 ± 18.4	6.9 ± 5.6	O(N)	Yes
TokenText	200	0	87.0 ± 30.8	5.2 ± 4.1	21.0 ± 11.9	5.1 ± 4.1	O(N)	Yes
Kamradt	N/A (~660)	0	83.6 ± 36.8	1.5 ± 1.6	7.4 ± 10.2	1.5 ± 1.6	O(N)	No
KamradtMod	<= 300	0	87.1 ± 31.9	2.1 ± 2.0	10.5 ± 12.3	2.1 ± 2.0	O(N)	Yes
Cluster	400 (~182)	0	91.3 ± 25.4	4.5 ± 3.4	20.7 ± 14.5	4.5 ± 3.4	O(N2)	No
Cluster	200 (~103)	0	87.3 ± 29.8	8.0 ± 6.0	34.0 ± 19.7	8.0 ± 6.0	O(N2)	No
LLM (GPT4o)	N/A (~240)	0	91.9 ± 26.5	3.9 ± 3.2	19.9 ± 16.3	3.9 ± 3.2	O(N2)	No
semchunk	<=400	0	90.0 ± 29.1	3.6 ± 2.8	17.3 ± 12.6	3.6 ± 2.8	O(N)	Yes
semchunk	<=200	0	89.3 ± 28.7	6.8 ± 5.2	28.9 ± 17.1	6.7 ± 5.1	O(N)	Yes
bert-chunker-3 (experimental, prob_threshold=0.50543)	<= 400	0	91.3 ± 26.6	5.4 ± 4.7	23.1 ± 17.6	5.4 ± 4.7	O(N)	Yes
bert-chunker-3 (experimental, prob_threshold=0.50543)	<= 200	0	89.7 ± 27.9	7.6 ± 6.0	30.9 ± 19.1	7.7 ± 5.8	O(N)	Yes
bert-chunker-3 (prob_threshold=0.50543)	N/A	0	90.4 ± 28.7	3.3 ± 3.1	16.0 ± 17.0	3.3 ± 3.1	O(N)	No
★ bert-chunker-3.5	<= 800	0	89.5 ± 27.7	7.6 ± 5.8	29.2 ± 17.9	7.6 ± 5.8	O(N)	Yes
★ bert-chunker-3.5	<= 400	0	94.1 ± 22.5	4.3 ± 3.5	18.1 ± 13.3	4.3 ± 3.5	O(N)	Yes
★ bert-chunker-3.5	<= 200	0	90.4 ± 26.2	7.7 ± 5.7	29.2 ± 17.9	7.6 ± 5.7	O(N)	Yes

Future

This model is undertrained due to lack of money and laziness. I observed it is still undertrained for two reasons:

• I trained it on 110 K window data for 2~3 epoch and the eval loss kept decreasing.
• The outputs from two non-overlapping windows show poor comparability in split point probabilities, which will undermines the performance when max_tokens_per_chunk is large. I think more data will amplify probability difference and enhance their comparability. This was corroborated by some of my experiments.

So next version will probably just be more data.

Citation

@article{bert-chunker,
  title={bert-chunker: Efficient and Trained Chunking for Unstructured Documents}, 
  author={Yannan Luo},
  year={2024},
  url={https://github.com/jackfsuia/bert-chunker}
}

Base model is from all-MiniLM-L6-v2.