Holly Huey.

Applied Research Scientist. Evaluating GenAI image & video models by benchmarking human visual content creation.

Hi there! I'm a 5th-year PhD Candidate in Experimental Psychology at the University of California San Diego.
I'm advised by Dr. Judith Fan @ The Cognitive Tools Lab.

I study how people create, interpret, and interact with visual content. By conducting a wide range of custom coding online web experiments, I crowdsource large-scale datasets of up to >90K human responses in order to systematically measuring people's visual content creation. This scale of data enables me to statistically evaluate GenAI text-to-visual systems, heuristic models, and LLMs against human behavior and communication.

My PhD research specializes in understanding user intentions in visual content creation—specifically, how content creators' different goals to make diverse visual content directly impacts:

what their output content will look like

how viewers will interpret that output depending on their own interpretation goals

In my three internships with Adobe Research, I've been evaluating (1) GenAI image output against human expectations of model reasoning and (2) GenAI video creation and editing.

My approach leverages both quantitative & qualitative methods to improve human-centered tools related to: text-to-visual systems, visual design, object identification, semantic segmentation, user intentions

Bio: I grew up on Whidbey Island, WA. As an introverted teen on a rural island, I spent most of my time sketching and hiding away in art studios. After 2 fine arts apprenticeships and double majoring in Philosophy and History of Mathematics & Sciences from St. John's College, I became fascinated by how novel visualizations have catalyzed and served to communicate many major scientific discoveries (e.g., Cartesian coordinate system, periodic table, Vitruvian man). Despite this critical role of visual media in human innovation, relatively little is known about how people communicate in visual form. This question continues to inspire my research.

research domains

VISUALIZATIONS

My PhD research evaluates human visual content creation in the form of drawings, diagrams, and data visualizations to investigate how people interpret the objects and scenes in their environments.

Our NeurIPS 2023 paper collected >90K drawings across 128 object categories and compared how well 17 visions models could recognize the sketched objects by comparison to human viewers.

IMAGES

Text-to-image generation models provide new opportunities for people to quickly & flexibly create images. To systematically evaluate model errors, I developed a series of behavioral tasks to crowdsource human annotators and generate quality scores for image datasets.

VIDEO

At Adobe Research, I've conducted qualitative interviews aimed at discovering user needs & goals and custom coded online experiments for quantitatively evaluating user behaviors related to video production and editing.

Our Creativity & Cognition 2024 paper evaluated how people (N >800) prioritize different visual concepts to illustrate as B-Roll images depending on goals to make entertaining or informative videos, as well as compared their selection behavior against heuristic & GPT models.

I have also explored & published work in other domains: language, social pragmatics (theory of mind), navigation, causal perception, symbolic reasoning, and developmental psychology. See below publications or my Google Scholar for papers in these areas.

publications

journal articles
Huey, H., Leake, M., Aneja, D., Fisher, M., and Fan, J.E. (2024). How do video content creation goals impact which concepts people prioritize for generating B-roll imagery? Creativity and Cognition. Chicago, Illinois: Association for Computing Machinery
Paper Poster Publisher's Page
Long, B., Fan, J., Huey, H., Chai, R., & Frank, M. C. (2024). Parallel developmental changes in children's production and recognition of line drawings of visual concepts. Nature Communications.
Paper Publisher's Page
Huey, H., Lu, X., Walker, C.M., & Fan, J.E. (2023). Explanatory drawings prioritize functional properties at the expense of visual fidelity. Cognition.
Paper Poster Publisher's Page
*Huey, H.,*Jordan, M., & Dillon,. M.R. (2023). Shortest path problems on different geometric surfaces: Reasoning about linearity through development. Developmental Psychology.
Paper Publisher's Page
Aboody, R., Huey, H., & Jara-Ettinger, J. (2022). Preschoolers decide who is knowledgeable, who to inform, and who to trust via a causal understanding of how knowledge relates to action. Cognition.
Paper Publisher's Page
Jara-Ettinger, J., Floyd, S., Huey, H., Tenenbaum, J.B., & Schulz, L. (2019). Social pragmatics: Four and five-year-olds rely on commonsense psychology to resolve referential ambiguities. Child Development.
Paper Publisher's Page

peer-reviewed conference proceedings & posters
*Mukherjee, K., *Huey, H., *Lu, X., Vinker, Y., Aguina-Kang, R., and Fan, J.E. (2023). SEVA: Leveraging sketches to evaluate alignment between human and machine visual abstraction. Advances in Neural Information Processing Systems, Datasets and Benchmarks Track, 2023. New Orleans, Lousiana: NeurIPS
Paper Github
*Huey, H., *Oey, L.A., Lloyd, H.S., and Fan, J.E. (2023). How do communicative goals guide which data visualizations people think are effective? Proceedings of the 45th Annual Conference of the Cognitive Science Society. Sydney, Australia: Cognitive Science Society.
Paper Poster
*Mukherjee, K., *Huey, H., *Lu, X., Vinker, Y., Aguina-Kang, R., and Fan, J.E. (2023). Evaluating machine comprehension of sketch meaning at different levels of abstraction. Proceedings of the 45th Annual Conference of the Cognitive Science Society. Sydney, Australia: Cognitive Science Society.
Paper Poster
Lloyd, H.S., Huey, H., Brockbank, E., Padilla, L., and Fan, J.E. (2023). What is graph comprehension and how do you measure it? Proceedings of the 45th Annual Conference of the Cognitive Science Society. Sydney, Australia: Cognitive Science Society.
Abstract Poster
*Huey, H., *Long, B., Yang, J., George, K., and Fan, J.E. (2022). Developmental changes in the semantic part structure of drawn objects. Proceedings of the 44th Annual Conference of the Cognitive Science Society. Toronto, Canada: Cognitive Science Society.
Paper Poster
Nagabandi, M., Yang, J., Huey, H., Fan, J.E. (2022). Decomposing objects into parts from vision and language. Proceedings of the 44th Annual Conference of the Cognitive Science Society. Toronto, Canada: Cognitive Science Society.
Abstract
Huey, H., Walker, C.M., & Fan, J.E. (2021). How do the semantic properties of visual explanations guide causal inference? Proceedings of the 43rd Annual Conference of the Cognitive Science Society. (Virtual Meeting) Vienna, Austria: Cognitive Science Society.
Paper Poster Video
Huey, H., Loncar, N., Jordan, M., & Dillon, M.R. (2019). A tale of paths between two points: Children's identification of linearity on different geometric surfaces. Poster presented at the Biennial Meeting of the Society for Research in Child Development 2019.
Loncar, N., Huey, H., & Dillon, M.R. (2019). Infants fail to categorize forms by kinds. Poster presented at the Biennial Meeting of the Society for Research in Child Development 2019.
Aboody, R., Huey, H., & Jara-Ettinger, J. (2018). Success does not imply knowledge: Preschoolers believe that accurate predictions reveal prior knowledge, but accurate observations do not. Proceedings of the 40th Annual Conference of the Cognitive Science Society.
Paper
Aboody, R., Huey, H., & Jara-Ettinger, J. (2017). Success does not imply knowledge: Preschoolers believe that accurate predictions reveal prior knowledge, but accurate observations do not. Poster presented at the Cognitive Development Society's Bi-Annual Meeting 2017.
Poster

workshop presentations
Fan, J.E., Mukherjee, K., Huey, H., Hebart, M.A., & Bainbridge. W.A. (2023). THINGS-drawings: A large-scale dataset containing human sketches of 1,854 object concepts. Journal of Vision. St. Pete Beach, FL: Vision Sciences Society
Talk abstract
Mukherjee, K., Huey, H., Rogers, T., & Fan. J.E. (2022). From Images to Symbols: Drawing as a Window into the Mind. Proceedings of the 44th Annual Conference of the Cognitive Science Society. Toronto, Canada: Cognitive Science Society. Co-organizer of workshop & co-designer of website.
Website Workshop Overview
Pitt, B., Huey, H., Jordan, M., Hart, Y., Dillon, M.R., Bottini, R., Carstensen, A., Boni, I., Piantadosi, S., Gibson, E., Marghetis, T., Holmes, K.J., Star-Lack, M., & Chacon, S., (2022). Dimensions of Diversity in Spatial Cognition: Culture, Context, Age, and Ability. Proceedings of the 44th Annual Conference of the Cognitive Science Society. Toronto, Canada: Cognitive Science Society. Invited speaker.
Workshop Overview