KAIST AI605 - Deep Learning for NLP (Spring 2021)
Links
Time & Location
Monday, Wednesday 2:30pm-3:50pm via Zoom (visit KLMS or email the instructor for the invitation)
Instructor
Minjoon Seo
Office: KAIST Seoul Campus Building 9 Room 9202
Office Hours: Wed 4-5pm via email
TAs
-
Miyoung Ko (Seoul)
Office: KAIST Seoul Campus Building 9
Office Hours: Thu 10:30-11:30am via email
-
Hyeong-Gwon Hong (Daejeon)
Office: KAIST Main Campus N1 Room 214
Office Hours: Mon 10:30-11:30am via email
-
Seokin Seo (Daejeon)
-
Taehyung Kwon (Daejeon)
Office: KAIST Main Campus N26 Room 110
Office Hours: Tue 2-3pm via email
Grading Policy
There is no exam in this class, and the grade will depend on the following criteria:
- Assignments: 60% (4 assignments altogether but the lowest score is dropped)
- Final project: 30%
- Participation (discussions): 10%
The final grade will be determined by the following policy:
- 90% or higher: A or A+ (only few students will get A+)
- 80% to 89%: B, B+, or A-
- 70% to 79%: C, C+, or B-
- 69% or lower: C- or lower (fail)
Late Submission Policy
- We will give you 7 no-penalty late days that can be used across all assignments (but not the final project).
- After you used all late penalty days, there will be -10% penalty for every late day (24 hours). That is, if the assignment or the project is due at 11pm and you submit at 11:30pm on the next day, 20% will be deducted. Note that I will be giving bonus questions for coding assignments, so you can still achieve 100% (or higher) with a late assignment. An assignment that is more than 7 days late will not be accepted and you will receive 0% for that assignment.
Course Description
This course covers recent advances in natural language processing area driven by deep learning. Topics include (but are not limited to)
- Recurrent Neural Networks
- Encoder-Decoder
- Transformer
- Language Model
- Pretrained Language Model (e.g. BERT)
- Text Classification (Sentiment Analysis)
- Sequence Tagging (NER, Question Answering)
- Sequence Generation (Summarization, Machine Translation, Semantic Parsing)
- Zero-shot Learning (e.g. GPT-3)
- NLP Tools (e.g. Hugging Face)
There are four assignments: 3 coding and 1 writing.
There will be three coding assignments that involve training deep learning models for text classification (sentiment classification), token classification (machine reading comprehension), and text generation (machine translation). In the first two assignments, you will be asked to use PyTorch library only. In the third assignment, you will use a popular NLP tool, Hugging Face, to complete the assignment.
In this class, you will also learn how to write an NLP paper by analyzing the structure of recent papers published in NLP (ACL, EMNLP, NAACL) and machine learning (NeurIPS, ICLR, ICML) conferences. We will have in-class discussion where you will analyze frequent argument patterns in these papers, and the assignment will be writing a sample research paper that adopts an interesting pattern you found (with dummy experiments).
The final project is creating an open-domain question answering system on EfficientQA dataset. The final deliverable will be a report. However, if you are working on an NLP-related research project, you are welcome to work on it instead (but please consult with me first).
Textbook
There is no required textbook for this course but I highly recommend Speech & Language Processing, whose pdf version is available for free, for your reference.
Weekly Schedule (tentative)
- Introduction to NLP and and Review of Deep Learning
- Recurrent Neural Networks, Text Classification
- Token Classification
- Encoder-Decoder, Sequence Generation
- Transformer
- NLP paper writing
- Language Model
- Paper analysis presentation
- Pretrained Language Model & Finetuning
- NLP Tools
- Intro to the final project (Open-domain QA)
- Large Language Model
- Generalization & In-context Learning
- Final project presentation