| 01 |
8/30 |
Intro to NLP, Deep Learning Basics [slides] |
|
|
| 02 |
9/1 |
Deep Learning Basics (2) [slides] |
|
|
| 03 |
9/6 |
Deep Learning Basics (3), Text Classification, Tokenization [slides] |
|
MNIST, [Socher et al., 2013] |
| 04 |
9/8 |
Tokenization (2), Word Embedding [slides] |
Assignment 1 is up |
[Sennrich et al., 2015], [Srivastava et al., 2014], [Mikolov et al., 2013] |
| 05 |
9/13 |
Recurrent Neural Networks [slides] |
|
Vanishing Gradients and Fancy RNNs, [Hochreiter & Schmidhuber, 1997] |
| 06 |
9/15 |
Recurrent Neural Networks (2), Token Classification [slides] |
|
[Rajpurkar et al., 2016] |
| |
9/20 |
No lecture (Chuseok) |
|
|
| |
9/22 |
No lecture (Chuseok) |
|
|
| 07 |
9/27 |
Token Classification (2), Retrieval [slides] |
|
[Hermann et al., 2015], [Chen et al., 2016], [Wang & Jiang, 2017], [Seo et al., 2017], [Chen et al., 2017] |
| 08 |
9/29 |
Retrieval (2) [slides] |
Assignment 1 is due, Final Project is up |
faiss, [Malkov & Yashunin, 2016] |
| |
10/4 |
No lecture (National Foundation Day) |
|
|
| 09 |
10/6 |
Text Generation, Encoder-Decoder [slides] |
|
[Cho et al., 2014] |
| |
10/11 |
No lecture (Hangeul Proclamation Day) |
|
|
| 10 |
10/13 |
Attention [slides] |
Final Project Survey is due. Assignment 2 is up. |
[Bahdanau et al., 2015], [Luong et al., 2015] |
| |
10/18 |
No lecture (midterm week) |
|
|
| |
10/20 |
No lecture (midterm week) |
|
|
| 11 |
10/25 |
Transformer [slides] |
|
[Vaswani et al., 2017], Annotated Transformer |
| 12 |
10/27 |
Transformer (2) [slides] |
Assignment 2 is due. |
|
| 13 |
11/1 |
Language Model, Regularization, Decoding [slides] |
|
[Ioffe & Szgedy 2015], [Ba et al., 2016] |
| |
11/3 |
Pre-BERT Paper Discussion [instructions] |
|
|
| |
|
|
|
|
| 14 |
11/8 |
Transfer Learning, Pretrained Language Model [slides] |
Assignment 3 is up. |
[Min et al., 2017], [Peters et al., 2018], [Howard & Ruder, 2018], [Radford et al., 2018] |
| |
11/10 |
No lecture (EMNLP) |
|
|
| 15 |
11/15 |
BERT [slides] |
|
[Devlin et al., 2018], [Yang et al., 2019], [Liu et al., 2019], BERT Slides |
| 16 |
11/17 |
Hugging Face and Final Project Tutorial [slides] |
|
transformers Tutorial |
| |
11/22 |
No lecture |
Assignment 3 is due, Assignment 4 is up. |
|
| 17 |
11/24 |
Pretraining for Generation [slides] |
|
[Radford et al., 2019], [Lewis et al., 2020a], [Raffel et al., 2020], [Roberts et al., 2020], [Lewis et al., 2020b] |
| 18 |
11/29 |
Scaling Laws, In-Context Learning [slides] |
|
[Kaplan et al., 2020], [Brown et al., 2020], [Ramesh et al., 2021] |
| |
12/1 |
Post-BERT Paper Discussion [instructions] |
|
|
| |
12/6 |
Final Project Presentation |
Assignment 4 is due. |
|
| |
12/8 |
Final Project Presentation |
|
|
| |
12/13 |
No lecture (finals week) |
|
|
| |
12/15 |
No lecture (finals week) |
Final Project Report is due |
|