enable task-specified preprocessing

README.md

@@ -2,7 +2,7 @@
 
 Welcome to the [SUPERB Challenge](https://superbbenchmark.org/challenge-slt2022/challenge_overview)! SUPERB is a collection of benchmarking resources to evaluate the capability of a universal shared representation for speech processing. It comes with a benchmark on the publicly available datasets and a challenge on a secret/not released hidden dataset. In SUPERB Challenge, a challenging hidden dataset is newly recorded to evaluate the ultimate generaliziblity across various tasks and data.
 
-You can participate the challenge by simply submitting your self-supervised (SSL) pretrained models (model definition & pretrained weights), and we benchmark it with the hidden dataset. This repository constains useful tools to let you easliy [submit](https://superbbenchmark.org/submit) your models ***privately*** for evaluation to [the challenge hidden-set leaderboard](https://superbbenchmark.org/leaderboard?track=constrained&subset=Hidden+Dev+Set).
+You can participate the challenge by simply submitting your self-supervised (SSL) pretrained models (model definition & pretrained weights), and we benchmark it with the hidden datasets. This repository constains useful tools to let you easliy [submit](https://superbbenchmark.org/submit) your models ***privately*** for evaluation to [the challenge hidden-set leaderboard](https://superbbenchmark.org/leaderboard?track=constrained&subset=Hidden+Dev+Set).
 
 1. Generate a submission template
 2. Validate the format/interface correctness of your model
@@ -32,16 +32,17 @@ Extract features from waveforms.
     BATCH_SIZE = 8
     EXAMPLE_SEC = 10
     wavs = [torch.randn(SAMPLE_RATE * EXAMPLE_SEC).cuda() for _ in range(BATCH_SIZE)]
-    results = upstream(wavs)
     ```
 
-- **Output:** A dictionary with a key for each task. If any task-specific key is not presented, a "hidden_states" key should be provided as the default key. The value for each key is **a list** of padded sequences in the same shape of **(batch_size, max_sequence_length_of_batch, hidden_size)** for weighted-sum to work. It is welcome to perform some preprocessing on the upstream's raw hidden-sets, including upsampling and downsampling. However, all the values must come from **a single upstream model**:
+- **Output:** A dictionary with a key "hidden_states" (for compatiblility with old ver.). The value is **a list** of padded sequences in the same shape of **(batch_size, max_sequence_length_of_batch, hidden_size)** for weighted-sum to work. It is welcome to perform some task-specified / independent pre- / post-processing on the upstream's raw hidden-sets, including upsampling and downsampling. However, all the values must come from **a single upstream model**:
 
     ```python
-    assert isinstance(results, dict)
-    tasks = ["PR", "SID", "ER", "ASR", "ASV", "SD", "QbE", "ST", "SS", "SE"]
+    tasks = ["hidden_states", "PR", "SID", "ER", "ASR", "ASV", "SD", "QbE", "ST", "SS", "SE", "secret"]
     for task in tasks:
-        hidden_states = results.get(task, "hidden_states")
+        # you can do task-specified pre- / post-processing depend on the arg "upstream_feature_selection"
+        results = upstream(wavs, upstream_feature_selection=task)
+        hidden_states = results["hidden_states"]
+        assert isinstance(results, dict)
         assert isinstance(hidden_states, list)
 
         for state in hidden_states:

{{cookiecutter.repo_name}}/README.md

@@ -2,7 +2,7 @@
 
 Welcome to the [SUPERB Challenge](https://superbbenchmark.org/challenge-slt2022/challenge_overview)! SUPERB is a collection of benchmarking resources to evaluate the capability of a universal shared representation for speech processing. It comes with a benchmark on the publicly available datasets and a challenge on a secret/not released hidden dataset. In SUPERB Challenge, a challenging hidden dataset is newly recorded to evaluate the ultimate generaliziblity across various tasks and data.
 
-You can participate the challenge by simply submitting your self-supervised (SSL) pretrained models (model definition & pretrained weights), and we benchmark it with the hidden dataset. This repository constains useful tools to let you easliy [submit](https://superbbenchmark.org/submit) your models ***privately*** for evaluation to [the challenge hidden-set leaderboard](https://superbbenchmark.org/leaderboard?track=constrained&subset=Hidden+Dev+Set).
+You can participate the challenge by simply submitting your self-supervised (SSL) pretrained models (model definition & pretrained weights), and we benchmark it with the hidden datasets. This repository constains useful tools to let you easliy [submit](https://superbbenchmark.org/submit) your models ***privately*** for evaluation to [the challenge hidden-set leaderboard](https://superbbenchmark.org/leaderboard?track=constrained&subset=Hidden+Dev+Set).
 
 1. Generate a submission template
 2. Validate the format/interface correctness of your model
@@ -32,16 +32,17 @@ Extract features from waveforms.
     BATCH_SIZE = 8
     EXAMPLE_SEC = 10
     wavs = [torch.randn(SAMPLE_RATE * EXAMPLE_SEC).cuda() for _ in range(BATCH_SIZE)]
-    results = upstream(wavs)
     ```
 
-- **Output:** A dictionary with a key for each task. If any task-specific key is not presented, a "hidden_states" key should be provided as the default key. The value for each key is **a list** of padded sequences in the same shape of **(batch_size, max_sequence_length_of_batch, hidden_size)** for weighted-sum to work. It is welcome to perform some preprocessing on the upstream's raw hidden-sets, including upsampling and downsampling. However, all the values must come from **a single upstream model**:
+- **Output:** A dictionary with a key "hidden_states" (for compatiblility with old ver.). The value is **a list** of padded sequences in the same shape of **(batch_size, max_sequence_length_of_batch, hidden_size)** for weighted-sum to work. It is welcome to perform some task-specified / independent pre- / post-processing on the upstream's raw hidden-sets, including upsampling and downsampling. However, all the values must come from **a single upstream model**:
 
     ```python
-    assert isinstance(results, dict)
-    tasks = ["PR", "SID", "ER", "ASR", "ASV", "SD", "QbE", "ST", "SS", "SE"]
+    tasks = ["hidden_states", "PR", "SID", "ER", "ASR", "ASV", "SD", "QbE", "ST", "SS", "SE", "secret"]
     for task in tasks:
-        hidden_states = results.get(task, results["hidden_states"])
+        # you can do task-specified pre- / post-processing depend on the arg "upstream_feature_selection"
+        results = upstream(wavs, upstream_feature_selection=task)
+        hidden_states = results["hidden_states"]
+        assert isinstance(results, dict)
         assert isinstance(hidden_states, list)
 
         for state in hidden_states:

{{cookiecutter.repo_name}}/expert.py

@@ -15,7 +15,8 @@ class Model(nn.Module):
         self.model1 = nn.Linear(1, HIDDEN_DIM)
         self.model2 = nn.Linear(HIDDEN_DIM, HIDDEN_DIM)
 
-    def forward(self, wavs):
+    def forward(self, wavs, upstream_feature_selection="hidden_states"):
+        # You can do task-specified pre- / post-processing based on upstream_feature_selection
         hidden = self.model1(wavs)
         # hidden: (batch_size, max_len, hidden_dim)
 
@@ -25,15 +26,24 @@ class Model(nn.Module):
         return [hidden, feature]
 
 class UpstreamExpert(nn.Module):
-    def __init__(self, ckpt: str = "./model.pt", **kwargs):
+    def __init__(
+        self,
+        ckpt: str = "./model.pt",
+        upstream_feature_selection: str = "hidden_states",
+        **kwargs):
         """
         Args:
             ckpt:
                 The checkpoint path for loading your pretrained weights.
                 Should be fixed as model.pt for SUPERB Challenge.
+            upstream_feature_selection:
+                The value could be 
+                'hidden_states', 'PR', 'SID', 'ER', 'ASR', 'QbE', 'ASV', 'SD', 'ST', 'SE', 'SS', 'secret', or others(new tasks).
+                You can use it to control which task-specified pre- / post-processing to do.
         """
         super().__init__()
         self.name = "[Example UpstreamExpert]"
+        self.upstream_feature_selection = upstream_feature_selection
 
         # You can use ckpt to load your pretrained weights
         ckpt = torch.load(ckpt, map_location="cpu")
@@ -57,21 +67,12 @@ class UpstreamExpert(nn.Module):
         wavs = pad_sequence(wavs, batch_first=True).unsqueeze(-1)
         # wavs: (batch_size, max_len, 1)
 
-        hidden_states = self.model(wavs)
+        hidden_states = self.model(wavs, upstream_feature_selection=self.upstream_feature_selection)
 
+        # Deprecated! Do not do any task-specified postprocess below
+        # You can use the init arg "upstream_feature_selection" to control which task-specified pre- / post-processing to do.
         # The "hidden_states" key will be used as default in many cases
         # Others keys in this example are presented for SUPERB Challenge
         return {
             "hidden_states": hidden_states,
-            "PR": hidden_states,
-            "SID": hidden_states,
-            "ER": hidden_states,
-            "ASR": hidden_states,
-            "QbE": hidden_states,
-            "ASV": hidden_states,
-            "SD": hidden_states,
-            "ST": hidden_states,
-            "SE": hidden_states,
-            "SS": hidden_states,
-            "secret": hidden_states,
         }