Commit
·
723775e
1
Parent(s):
58bbcf2
Update README
Browse files
README.md
CHANGED
|
@@ -95,33 +95,33 @@ Performance of this model compared to the top base models on the [MTEB leaderboa
|
|
| 95 |
|
| 96 |
The following datasets were used to evaluate model performance.
|
| 97 |
|
| 98 |
-
- [PubMed QA](https://huggingface.co/datasets/
|
| 99 |
- Subset: pqa_labeled, Split: train, Pair: (question, long_answer)
|
| 100 |
-
- [PubMed Subset](https://huggingface.co/datasets/
|
| 101 |
- Split: test, Pair: (title, text)
|
| 102 |
-
- [PubMed Summary](https://huggingface.co/datasets/scientific_papers)
|
| 103 |
- Subset: pubmed, Split: validation, Pair: (article, abstract)
|
| 104 |
|
| 105 |
Evaluation results from the original model are shown below for reference. The [Pearson correlation coefficient](https://en.wikipedia.org/wiki/Pearson_correlation_coefficient) is used as the evaluation metric.
|
| 106 |
|
| 107 |
| Model | PubMed QA | PubMed Subset | PubMed Summary | Average |
|
| 108 |
| ----------------------------------------------------------------------------- | --------- | ------------- | -------------- | --------- |
|
| 109 |
-
| [all-MiniLM-L6-v2](https://hf.co/sentence-transformers/all-MiniLM-L6-v2)
|
| 110 |
-
| [bge-base-en-v1.5](https://hf.co/BAAI/bge-
|
| 111 |
-
| [gte-base](https://hf.co/thenlper/gte-base)
|
| 112 |
-
| [**pubmedbert-base-embeddings**](https://hf.co/neuml/pubmedbert-base-embeddings) | **93.27** | **97.
|
| 113 |
-
| [S-PubMedBert-MS-MARCO](https://hf.co/pritamdeka/S-PubMedBert-MS-MARCO)
|
| 114 |
|
| 115 |
See the table below for evaluation results per dimension for `pubmedbert-base-embeddings-matryoshka`.
|
| 116 |
|
| 117 |
| Model | PubMed QA | PubMed Subset | PubMed Summary | Average |
|
| 118 |
| --------------------| --------- | ------------- | -------------- | --------- |
|
| 119 |
-
| Dimensions = 64 | 92.16 |
|
| 120 |
-
| Dimensions = 128 | 92.80 | 96.
|
| 121 |
-
| Dimensions = 256 | 93.11 | 96.
|
| 122 |
-
| Dimensions = 384 | 93.42 |
|
| 123 |
-
| Dimensions = 512 | 93.37 |
|
| 124 |
-
| **Dimensions = 768** | **93.53** | **
|
| 125 |
|
| 126 |
This model performs slightly better overall compared to the original model.
|
| 127 |
|
|
|
|
| 95 |
|
| 96 |
The following datasets were used to evaluate model performance.
|
| 97 |
|
| 98 |
+
- [PubMed QA](https://huggingface.co/datasets/qiaojin/PubMedQA)
|
| 99 |
- Subset: pqa_labeled, Split: train, Pair: (question, long_answer)
|
| 100 |
+
- [PubMed Subset](https://huggingface.co/datasets/awinml/pubmed_abstract_3_1k)
|
| 101 |
- Split: test, Pair: (title, text)
|
| 102 |
+
- [PubMed Summary](https://huggingface.co/datasets/armanc/scientific_papers)
|
| 103 |
- Subset: pubmed, Split: validation, Pair: (article, abstract)
|
| 104 |
|
| 105 |
Evaluation results from the original model are shown below for reference. The [Pearson correlation coefficient](https://en.wikipedia.org/wiki/Pearson_correlation_coefficient) is used as the evaluation metric.
|
| 106 |
|
| 107 |
| Model | PubMed QA | PubMed Subset | PubMed Summary | Average |
|
| 108 |
| ----------------------------------------------------------------------------- | --------- | ------------- | -------------- | --------- |
|
| 109 |
+
| [all-MiniLM-L6-v2](https://hf.co/sentence-transformers/all-MiniLM-L6-v2) | 90.40 | 95.92 | 94.07 | 93.46 |
|
| 110 |
+
| [bge-base-en-v1.5](https://hf.co/BAAI/bge-base-en-v1.5) | 91.02 | 95.82 | 94.49 | 93.78 |
|
| 111 |
+
| [gte-base](https://hf.co/thenlper/gte-base) | 92.97 | 96.90 | 96.24 | 95.37 |
|
| 112 |
+
| [**pubmedbert-base-embeddings**](https://hf.co/neuml/pubmedbert-base-embeddings) | **93.27** | **97.00** | **96.58** | **95.62** |
|
| 113 |
+
| [S-PubMedBert-MS-MARCO](https://hf.co/pritamdeka/S-PubMedBert-MS-MARCO) | 90.86 | 93.68 | 93.54 | 92.69 |
|
| 114 |
|
| 115 |
See the table below for evaluation results per dimension for `pubmedbert-base-embeddings-matryoshka`.
|
| 116 |
|
| 117 |
| Model | PubMed QA | PubMed Subset | PubMed Summary | Average |
|
| 118 |
| --------------------| --------- | ------------- | -------------- | --------- |
|
| 119 |
+
| Dimensions = 64 | 92.16 | 96.14 | 95.67 | 94.66 |
|
| 120 |
+
| Dimensions = 128 | 92.80 | 96.58 | 96.22 | 95.20 |
|
| 121 |
+
| Dimensions = 256 | 93.11 | 96.82 | 96.53 | 95.49 |
|
| 122 |
+
| Dimensions = 384 | 93.42 | 97.00 | 96.61 | 95.68 |
|
| 123 |
+
| Dimensions = 512 | 93.37 | 97.07 | 96.61 | 95.68 |
|
| 124 |
+
| **Dimensions = 768** | **93.53** | **97.13** | **96.70** | **95.79** |
|
| 125 |
|
| 126 |
This model performs slightly better overall compared to the original model.
|
| 127 |
|