Update run_pretraining.py and configuration_aragpt2.py

README.md

@@ -86,7 +86,25 @@ python create_pretraining_data.py
 
  Finetuning:
  ```bash
- python3 run_pretraining.py \\r\n --input_file="gs://<GS_BUCKET>/pretraining_data/*" \\r\n --output_dir="gs://<GS_BUCKET>/pretraining_model/" \\r\n --config_file="config/small_hparams.json" \\r\n --batch_size=128 \\r\n --eval_batch_size=8 \\r\n --num_train_steps= \\r\n --num_warmup_steps= \\r\n --learning_rate= \\r\n --save_checkpoints_steps= \\r\n --max_seq_length=1024 \\r\n --max_eval_steps= \\r\n --optimizer="lamb" \\r\n --iterations_per_loop=5000 \\r\n --keep_checkpoint_max=10 \\r\n --use_tpu=True \\r\n --tpu_name=<TPU NAME> \\r\n --do_train=True \\r\n --do_eval=False
+ python3 run_pretraining.py \
+  --input_file="gs://<GS_BUCKET>/pretraining_data/*" \
+  --output_dir="gs://<GS_BUCKET>/pretraining_model/" \
+  --config_file="config/small_hparams.json" \
+  --batch_size=128 \
+  --eval_batch_size=8 \
+  --num_train_steps= \
+  --num_warmup_steps= \
+  --learning_rate= \
+  --save_checkpoints_steps= \
+  --max_seq_length=1024 \
+  --max_eval_steps= \
+  --optimizer="lamb" \
+  --iterations_per_loop=5000 \
+  --keep_checkpoint_max=10 \
+  --use_tpu=True \
+  --tpu_name=<TPU NAME> \
+  --do_train=True \
+  --do_eval=False
  ```
 # Model Sizes
 

configuration_aragpt2.py

@@ -131,7 +131,7 @@ class AraGPT2Config(PretrainedConfig):
         n_layer=12,
         n_head=12,
         n_inner=None,
-        activation_function="gelu",
+        activation_function="gelu_new",
         resid_pdrop=0.1,
         embd_pdrop=0.1,
         attn_pdrop=0.1,