Zheng Zhao
PhD Student
I am a fourth year PhD student with UKRI Centre for Doctoral Training in NLP at the University of Edinburgh, working with Shay Cohen and Bonnie Webber. I am affiliated with ILCC in the School of Informatics. I am also a member of the Cohort and EdinburghNLP group.
My research topic is on analyzing and interpreting neural networks for NLP. I am also interested in large language models, summarization, discourse, and their related topics.
Action required
Problem: The current root path of this site is "baseurl ("_config.yml.
Solution: Please set the
baseurl in _config.yml to "Education
-
University of EdinburghPhD in Natural Language ProcessingSep. 2020 - Present -
University of EdinburghMasters by Research in Natural Language ProcessingSep. 2019 - Aug. 2020
-
University of EdinburghBEng in Artificial Intelligence and Software EngineeringSep. 2015 - Jul. 2019
Experience
-
Amazon AGIApplied Scientist InternSep. 2024 - present -
Amazon Alexa AIApplied Scientist InternJun. 2023 - Nov. 2023
-
Goldman SachsSummer AnalystJun. 2018 - Aug. 2018
News
Selected Publications view all
Layer by Layer: Uncovering Where Multi-Task Learning Happens in Instruction-Tuned Large Language Models
Zheng Zhao, Yftah Ziser, Shay Cohen
EMNLP 2024
This paper investigates how instruction-tuned LLMs internally process different tasks, finding that their layers organize into three functional groups: early layers for general features, middle layers for task-specific transitions, and final layers for refinement.
Spectral Editing of Activations for Large Language Model Alignment
Yifu Qiu, Zheng Zhao, Yftah Ziser, Anna Korhonen, Edoardo Ponti, Shay Cohen
NeurIPS 2024
We introduce Spectral Editing of Activations (SEA), a novel inference-time method to adjust large language models' internal representations, improving truthfulness and reducing bias. SEA projects input representations to align with positive examples while minimizing alignment with negatives, showing superior effectiveness, generalization, and efficiency compared to existing methods with minimal impact on other model capabilities.
Enhancing Contextual Understanding in Large Language Models through Contrastive Decoding
Zheng Zhao, Emilio Monti, Jens Lehmann, Haytham Assem
NAACL 2024 Oral
This work introduces a novel approach integrating contrastive decoding with adversarial irrelevant passages as negative samples to enhance robust context grounding during generation and operates at inference time without requiring further training.
A Joint Matrix Factorization Analysis of Multilingual Representations
Zheng Zhao, Yftah Ziser, Bonnie Webber, Shay Cohen
EMNLP (Findings) 2023
This work presents an analysis tool based on joint matrix factorization for comparing latent representations of multilingual and monolingual models, and finds the factorization outputs exhibit strong associations with performance observed across different cross-lingual tasks.
Understanding Domain Learning in Language Models Through Subpopulation Analysis
Zheng Zhao, Yftah Ziser, Shay Cohen
BlackboxNLP 2022
We examine how different domains are represented in neural network architectures, focusing on the relationship between domains, model size, and training data. Using subpopulation analysis with SVCCA on Transformer-based language models, we compare models trained on multiple domains versus a single domain. Our findings show that increasing model capacity differently affects domain information storage in upper and lower layers, with larger models embedding domain-specific information similarly to separate smaller models.
Reducing Quantity Hallucinations in Abstractive Summarization
Zheng Zhao, Shay Cohen, Bonnie Webber
EMNLP (Findings) 2020
Abstractive summaries often hallucinate unsupported content, but our system, Herman, mitigates this by verifying specific entities like dates and numbers, improving summary accuracy and earning higher ROUGE scores and human preference.