Papers

Xuwei Ding, Skylar Zhai, Linxin Song, Jiate Li, Taiwei Shi, Nicholas Meade, Siva Reddy, Jian Kang, Jieyu Zhao

Computer-use agent safety usually screens prompts. OS-BLIND shows even benign instructions can produce harm via task context or execution; safety alignment only activates in the first few steps, so even Claude 4.5 Sonnet hits 92.7% attack success.

Taiwei Shi, Yiyang Wu, Linxin Song, Tianyi Zhou, Jieyu Zhao

RL finetuning wastes compute on prompts the model already aces. We schedule training data adaptively — pushing the model toward problems just past its current capability.

Haoxuan Li, Mingyu Derek Ma, Jen-tse Huang, Zhaotian Weng, Wei Wang, Jieyu Zhao

An end-to-end multi-agent framework that plans, tools-up, and detects bias in structured datasets automatically — instead of relying on hand-coded, case-by-case bias checks.

Yongkang Du, Jieyu Zhao, Yijun Yang, Tianyi Zhou

Fairness-accuracy trade-offs usually hand you one operating point. CPT lets users specify a preferred balance via reference vectors, navigating the Pareto front with stabilized fairness updates and gradient pruning.

Ziyi Liu, Priyanka Dey, Zhenyu Zhao, Jen-tse Huang, Rahul Gupta, Yang Liu, Jieyu Zhao

Cultural-intelligence benchmark that tests whether LLMs can infer the *implicit* values baked into natural conversation — not just the cultural norms stated outright.

Yongkang Du, Jen-tse Huang, Jieyu Zhao, Lu Lin

Code-generation LLMs leak social bias into the code they write. FairCode measures how, where, and which prompts make it worse.

Zixiang Xu, Yanbo Wang, Yue Huang, Jiayi Ye, Haomin Zhuang, Zirui Song, Lang Gao, Chenxi Wang, Zhaorun Chen, Yujun Zhou, et al.

Benchmark for LLM social reasoning built from structured multi-party scenarios — instead of static QA, agents have to navigate inferences about other agents.

Linxin Song, Jieyu Zhang, Huanxin Sheng, Taiwei Shi, Gupta Rahul, Yang Liu, Ranjay Krishna, Kang Jian, Jieyu Zhao

Computer-use agents leave a video trail of their work. We turn that trail into reward signal — rating trajectories at the pixel level instead of grading only the final outcome.

Lincan Li, Jiaqi Li, Catherine Chen, Fred Gui, Hongjia Yang, Chenxiao Yu, Zhengguang Wang, Jianing Cai, Junlong Aaron Zhou, Bolin Shen, et al.

Survey of how LLMs are being applied in political science — what they can credibly contribute, and the methodological pitfalls researchers keep stepping into.

Jingnong Qu, Liunian Harold Li, Jieyu Zhao, Sunipa Dev, Kai-Wei Chang

Multimodal dataset for detecting memes designed to spread disinformation — image + text, intentionality-labeled, harder than ordinary fake-news detection.

Yijun Pan, Taiwei Shi, Jieyu Zhao, Jiaqi W. Ma

Find which training examples are pushing a model toward unsafe behavior, by attributing model outputs back through denoised representations — and filter them before they do damage.

Ziyi Liu, Bahar Sarrafzadeh, Pei Zhou, Longqi Yang, Jieyu Zhao, Ashish Sharma

Most agent benchmarks are single-turn task completion. ProMediate puts agents in the middle of multi-party negotiations and asks whether they can read the room, mediate, and act proactively.

Taiwei Shi, Zhuoer Wang, Longqi Yang, Ying-Chun Lin, Zexue He, Mengting Wan, Pei Zhou, Sujay Jauhar, Sihao Chen, Shan Xia, Hongfei Zhang, Jieyu Zhao, Xiaofeng Xu, Xia Song, Jennifer Neville

Align LLMs with the messy, in-the-moment feedback users actually leave during real interactions — not the curated preference pairs that look clean on paper.

Taiwei Shi, Sihao Chen, Bowen Jiang, Linxin Song, Longqi Yang, Jieyu Zhao

Treat past episodes as a retrievable experience bank rather than gradient updates only — letting the policy reason about what worked last time before deciding what to try next.

Linxin Song, Yutong Dai, Viraj Prabhu, Jieyu Zhang, Taiwei Shi, Li Li, Junnan Li, Silvio Savarese, Zeyuan Chen, Jieyu Zhao, Ran Xu, Caiming Xiong

A computer-using agent that doesn't just point and click — it writes code as a first-class action, finishing tasks faster and with higher success than GUI-only agents.

Yue Huang, Chujie Gao, ..., Taiwei Shi, ..., Jieyu Zhao, ..., Xiangliang Zhang

A community guideline plus working assessment toolkit for evaluating trustworthiness in generative foundation models across modalities — what to measure, how, and why each axis matters.

Zhaotian Weng, Haoxuan Li, Xin Eric Wang, Kuan-Hao Huang, Jieyu Zhao

VLMs can recognize what happens in a sequence of frames but fail when the task requires reasoning about *why* in causal order — we probe where the gap is.

Priyanka Dey, Daniele Rosa, Wenqing Zheng, Daniel Barcklow, Jieyu Zhao, Emilio Ferrara

Personalized generation usually needs costly human preference data. GRAVITY synthesizes preference pairs grounded in user profiles (cultural dimensions, values, personality), and its outputs beat baselines 86% of the time on Amazon book descriptions.

Jiazhi Li, Mi Zhou, Mahyar Khayatkhoei, Jingyu Shi, Xiang Gao, Jiageng Zhu, Hanchen Xie, Xiyun Song, Zongfang Lin, Heather Yu, Liang Peng, Jieyu Zhao

Text-to-image models trade diversity for fidelity by default. We show you can recover demographic and conceptual diversity without sacrificing how good the images look.

Yang Trista Cao, Anna Sotnikova, Jieyu Zhao, Linda X. Zou, Rachel Rudinger, Hal Daumé III

Stereotypes encoded in one language's training data don't stay there — multilingual LLMs leak them across language boundaries, sometimes amplifying biases in languages they barely saw.

Sizhe Liu, Yizhou Lu, Siyu Chen, Xiyang Hu, Jieyu Zhao, Yingzhou Lu, Yue Zhao

A multi-agent framework that automates ML programming for drug discovery: an LLM Planner pairs with an LLM Instructor to turn high-level ideas into runnable pharmaceutical pipelines.

Zixiang Xu, Yanbo Wang, Yue Huang, Xiuying Chen, Jieyu Zhao, Meng Jiang, Xiangliang Zhang

Automatic probing surfaces concrete spots where multilingual LLMs are weaker in some languages than others — beyond what a single aggregate score reveals.

Priyanka Dey, Aayush Bothra, Yugal Khanter, Emilio Ferrara, Jieyu Zhao

Personality questionnaires translate poorly across cultures. CulturalPersonas tests whether LLMs can express Big-Five traits in culturally appropriate ways instead of defaulting to a WEIRD baseline.

Jingyuan Huang, Jen-tse Huang, Ziyi Liu, Xiaoyuan Liu, Wenxuan Wang, Jieyu Zhao

Vision-language models guess where photos were taken — and they're substantially more accurate for wealthy regions than for the global south. We quantify the disparity and ask why.

Linxin Song*, Taiwei Shi*, Jieyu Zhao

RL finetuning makes models more decisive — including when they shouldn't be. We measure the hallucination tax and show synthetic unanswerable math problems can pay it back.

Huanxin Sheng, Xinyi Liu, Hangfeng He, Jieyu Zhao, Jian Kang

LLM judges return point scores that look authoritative. We wrap them in conformal-prediction intervals so you can tell when the judge is actually confident — and when it's bluffing.

Jen-tse Huang, Jiantong Qin, Jianping Zhang, Youliang Yuan, Wenxuan Wang, Jieyu Zhao

Vision-language models can pass explicit-bias probes while still leaking the same stereotypes implicitly. VISBIAS tests both axes side by side.

Weijia Shi, Jaechan Lee, Yangsibo Huang, Sadhika Malladi, Jieyu Zhao, Ari Holtzman, Daogao Liu, Luke Zettlemoyer, Noah Smith, Chiyuan Zhang

Six-axis evaluation for machine unlearning in language models — testing whether claimed unlearning actually erases the knowledge or just hides it from a few probes.

Linxin Song, Xuwei Ding, Jieyu Zhang, Taiwei Shi, Ryotaro Shimizu, Rahul Gupta, Yang Liu, Jian Kang, Jieyu Zhao

Where exactly do LLMs not know things? SEA searches a massive KB to surface concrete topical blind spots, instead of reporting a single aggregate accuracy number.

Yubo Zhang, Shudi Hou, Mingyu Derek Ma, Wei Wang, Muhao Chen, Jieyu Zhao

Benchmark probing whether LLM medical advice changes with patient demographics — the bias most likely to cause real-world harm if deployed in clinical workflows.

Huy Nghiem, John Prindle, Jieyu Zhao, Hal Daumé III

LLMs reading a candidate's name recommend systematically different jobs based on inferred ethnicity — even when qualifications are identical.

Zhaotian Weng, Zijun Gao, Jerone Andrews, Jieyu Zhao

Localize and mitigate bias *inside* a vision-language model using causal mediation analysis — instead of treating the model as a black box at the output.

Ziyi Liu, Abhishek Anand, Pei Zhou, Jen-tse Huang, Jieyu Zhao

Evaluate LLM social intelligence by putting them inside an interactive game where success depends on inferring other agents' intentions, not on answering static QA.

Ziyi Liu, Isabelle Lee, Yongkang Du, Soumya Sanyal, Jieyu Zhao

Framework for measuring when an LLM's chain-of-thought contradicts itself — and detecting those contradictions automatically across reasoning benchmarks.

Md Khairul Islam, Andrew Wang, Tianhao Wang, Yangfeng Ji, Judy Fox, Jieyu Zhao

Differential privacy and bias mitigation sometimes conflict. We measure exactly when DP training makes downstream bias worse — and which protected groups bear the cost.

Yusen Zhang, Nan Zhang, Yixin Liu, Alexander Fabbri, Junru Liu, Ryo Kamoi, Xiaoxin Lu, Caiming Xiong, Jieyu Zhao, Dragomir Radev, Kathleen McKeown, Rui Zhang

Summarizers compress text — and often compress out minority viewpoints. We measure that loss and propose summaries that preserve the spread of perspectives in the source.

Taiwei Shi, Kai Chen, Jieyu Zhao

Generate the preference data needed for alignment automatically, with guardrails, instead of paying for tens of thousands of human labels.

Yuancheng Xu, Chenghao Deng, Yanchao Sun, Ruijie Zheng, Xiyao Wang, Jieyu Zhao, Furong Huang

Single-step fairness constraints break down when decisions compound over time. We propose Equal Long-term Benefit Rate — fairness evaluated over the trajectory — and bias-mitigation strategies that target it.

Lichao Sun, Yue Huang, Haoran Wang, Siyuan Wu, ..., Jieyu Zhao, ..., Yan Liu, Yanfang Ye, Yinzhi Cao, Yong Chen, Yue Zhao

A consortium-scale benchmark for trustworthiness in LLMs — truthfulness, safety, fairness, robustness, privacy, and ethics — with consistent protocols across 16 mainstream models.

Alexander Kwako, Yixin Wan, Jieyu Zhao, Mark Hansen, Kai-Wei Chang, Li Cai

When BERT is used to score spoken English, does pretraining-era bias make existing gender / L1 disparities in scoring worse?

Sandra Sandoval, Jieyu Zhao, Marine Carpuat, Hal Daumé III

Machine translation systems mishandle names in systematic ways tied to gender and origin — and the cost lands disproportionately on people whose names don't fit Anglocentric defaults.

Yixin Wan, Jieyu Zhao, Aman Chadha, Nanyun Peng, Kai-Wei Chang

Giving a chatbot a persona can amplify the biases it carries. We measure how much, and which personas make things worse.

Ruyuan Zuo, Jieyu Zhao

Reword the prompt and the bias number changes. We show how unstable prompt-based bias measurements are, and what to do instead.

Anaelia Ovalle, Sunipa Dev, Jieyu Zhao, Majid Sarrafzadeh, Kai-Wei Chang

Apply LOGAN-style local-group bias detection to healthcare ML, weighting clusters by the severity of harm — so the audit prioritizes the failures that matter most clinically.

Haozhe An, Zongxia Li, Jieyu Zhao, Rachel Rudinger

Most bias benchmarks pre-specify which axes to test. SODAPOP lets the social-commonsense model itself surface where it falls apart, instead of grading against a hand-built checklist.

Ruijie Zheng, Xiyao Wang, Yanchao Sun, Shuang Ma, Jieyu Zhao, Huazhe Xu, Hal Daumé, Furong Huang

Visual RL agents struggle to learn action-relevant representations from pixels. TACO contrasts state-action pairs across time to pull out what actually moves the world.

Sunipa Dev, Emily Sheng, Jieyu Zhao, Aubrie Amstutz, Jiao Sun, Yu Hou, Mattie Sanseverino, Jiin Kim, Akihiro Nishi, Nanyun Peng, Kai-Wei Chang

An audit of how the NLP community measures bias and harm — what each metric actually captures, what it misses, and where the field has been talking past itself.

Jieyu Zhao, Xuezhi Wang, Yao Qin, Jilin Chen, Kai-Wei Chang

Ensembling can buy robustness — but the gains depend on which kinds of models you mix, not just how many.

Alexander Kwako, Yixin Wan, Jieyu Zhao, Kai-Wei Chang, Li Cai, Mark Hansen

Use item response theory — borrowed from psychometrics — to measure gender and racial bias of BERT-based English speech assessment, beyond simple accuracy gaps.

Chong Zhang, Jieyu Zhao, Huan Zhang, Kai-Wei Chang, Cho-Jui Hsieh

Counterfactual bias evaluations themselves aren't robust — a small perturbation to the test set can flip the conclusion. We diagnose when and offer a fix.

Jieyu Zhao, Daniel Khashabi, Tushar Khot, Ashish Sabharwal, Kai-Wei Chang

If you literally tell a language model to be fair, does it listen? We test natural-language ethical interventions and find that 'taking advice' is harder than it looks.

Zuohui Fu*, Yikun Xian*, Ruoyuan Gao, Jieyu Zhao, Qiaoying Huang, Yingqiang Ge, Shuyuan Xu, Shijie Geng, Chirag Shah, Yongfeng Zhang, Gerard de Melo

Explainable recommendation can quietly bake in unfair preferences. We modify the KG-walking objective so the explanation is fair, not just plausible.

Jieyu Zhao, Kai-Wei Chang

Average-case fairness numbers hide local pockets of severe bias. LOGAN clusters the data to find where a model fails worst, instead of just reporting the mean.

Jieyu Zhao, Subhabrata Mukherjee, Saghar Hosseini, Kai-Wei Chang, Ahmed Hassan Awadallah

Cross-lingual transfer doesn't just move task signal — it moves bias. We trace how gender bias travels across languages in multilingual embeddings.

Shengyu Jia*, Tao Meng*, Jieyu Zhao, Kai-Wei Chang

Models don't just inherit bias from data — they amplify it. Posterior regularization constrains the predicted distribution to stay close to the empirical one, blunting amplification.

Yichao Zhou, Jyun-Yu Jiang, Jieyu Zhao, Kai-Wei Chang, Wei Wang

Puns hinge on sound, not just meaning. A pronunciation-attentive model recognizes them by attending to phonetic neighbors alongside contextualized text.

Andrew Gaut, Tony Sun, Shirlyn Tang, Yuxin Huang, Jing Qian, Mai ElSherief, Jieyu Zhao, Diba Mirza, Elizabeth Belding, Kai-Wei Chang, William Yang Wang

Relation extractors learn that some relations belong to some genders — a problem when the same model is then used to build knowledge graphs.

Tianlu Wang, Jieyu Zhao, Mark Yatskar, Kai-Wei Chang, Vicente Ordonez

Even with a 50/50 dataset, image models still latch onto gendered cues — because the visual features themselves are correlated with gender. Balancing labels isn't enough.

Jieyu Zhao, Tianlu Wang, Mark Yatskar, Ryan Cotterell, Vicente Ordonez, Kai-Wei Chang

ELMo and friends were supposed to fix the bias problems of static embeddings. They didn't — contextualized vectors carry their own, sometimes worse, gendered geometry.

Tony Sun, Andrew Gaut, Shirlyn Tang, Yuxin Huang, Mai ElSherief, Jieyu Zhao, Diba Mirza, Elizabeth Belding, Kai-Wei Chang, William Yang Wang

A field-defining survey of how gender bias enters NLP systems, how it's measured, and what mitigation actually works versus what just looks like it does.

Pei Zhou, Weijia Shi, Jieyu Zhao, Kuan-Hao Huang, Muhao Chen, Ryan Cotterell, Kai-Wei Chang

English-trained bias methods don't transfer cleanly to languages with grammatical gender. We adapt the measurement and find the bias is still there — just shaped differently.

Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, Kai-Wei Chang

WinoBias: a coref benchmark where the gendered pronoun and the stereotypical answer disagree. State-of-the-art systems flip their answers, and we show how to push back.

Jieyu Zhao, Yichao Zhou, Zeyu Li, Wei Wang, Kai-Wei Chang

GN-GloVe: train word embeddings whose gender dimension is isolated by construction, so downstream models can use the semantics without inheriting the gender association.

Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, Kai-Wei Chang

Vision-language models don't just learn that men cook less than women — they amplify the gap. We add corpus-level constraints at inference time to keep model statistics aligned with training statistics.