Revealing the multidimensional mental representations of natural objects underlying human similarity judgements

Objects can be characterized according to a vast number of possible criteria (such as animacy, shape, colour and function), but some dimensions are more useful than others for making sense of the objects around us. To identify these core dimensions of object representations, we developed a data-driven computational model of similarity judgements for real-world images of 1,854 objects. The model captured most explainable variance in similarity judgements and produced 49 highly reproducible and meaningful object dimensions that reflect various conceptual and perceptual properties of those objects. These dimensions predicted external categorization behaviour and reflected typicality judgements of those categories. Furthermore, humans can accurately rate objects along these dimensions, highlighting their interpretability and opening up a way to generate similarity estimates from object dimensions alone. Collectively, these results demonstrate that human similarity judgements can be captured by a fairly low-dimensional, interpretable embedding that generalizes to external behaviour.

This is a preview of subscription content, access via your institution

Access options

Access Nature and 54 other Nature Portfolio journals

Get Nature+, our best-value online-access subscription

cancel any time

Subscribe to this journal

Receive 12 digital issues and online access to articles

133,45 € per year

only 11,12 € per issue

Buy this article

Prices may be subject to local taxes which are calculated during checkout

Similar content being viewed by others

Distributed representations of behaviour-derived object dimensions in the human visual system

Article Open access 09 September 2024

A data-driven investigation of human action representations

Article Open access 30 March 2023

The role of semantics in the perceptual organization of shape

Article Open access 17 December 2020

Data availability

The learned embedding, triplet odd-one-out behavioural data for testing model performance, typicality scores, participant-generated dimension labels and dimension ratings are available at https://osf.io/z2784. The behavioural data used for training the model are available from the corresponding author upon request.

Code availability

To reproduce the relevant analyses and figures, the relevant MATLAB scripts and functions are available at https://osf.io/z2784. The computational modelling code to create an embedding is available from the corresponding author upon request.

References

  1. Biederman, I. Recognition-by-components: a theory of human image understanding. Psychol. Rev.94, 115–147 (1987). ArticleGoogle Scholar
  2. Edelman, S. Representation is representation of similarities. Behav. Brain Sci.21, 449–467 (1998). ArticleCASGoogle Scholar
  3. Nosofsky, R. M. Attention, similarity, and the identification–categorization relationship. J. Exp. Psychol. Gen.115, 39–57 (1986). ArticleCASGoogle Scholar
  4. Goldstone, R. L. The role of similarity in categorization: providing a groundwork. Cognition52, 125–157 (1994). ArticleCASGoogle Scholar
  5. Rosch, E., Mervis, C. B., Gray, W. D., Johnson, D. M. & Boyes-Braem, P. Basic objects in natural categories. Cognit. Psychol.8, 382–439 (1976). ArticleGoogle Scholar
  6. Hahn, U. & Chater, N. in Knowledge, Concepts and Categories (eds Lamberts, K. & Shanks, D.) 43–92 (Psychology Press, 1997).
  7. Rips, L. J., Smith, E. E. & Medin, D. L. in The Oxford Handbook of Thinking and Reasoning (eds Holyoak, K. J. & Morrison, R. G.) 177–209 (Oxford Univ. Press, 2012).
  8. Rogers, T. T. & McClelland, J. L. Semantic Cognition: A Parallel Distributed Processing Approach (MIT Press, 2004).
  9. Goldstone, R. L. & Son, J. Y. in The Oxford Handbook of Thinking and Reasoning (eds Holyoak, K. J. & Morrison, R. G.) 155–176 (Oxford Univ. Press, 2012).
  10. Kriegeskorte, N. & Kievit, R. A. Representational geometry: integrating cognition, computation, and the brain. Trends Cogn. Sci.17, 401–412 (2013). ArticleGoogle Scholar
  11. Caramazza, A. & Shelton, J. R. Domain-specific knowledge systems in the brain: the animate–inanimate distinction. J. Cogn. Neurosci.10, 1–34 (1998). ArticleCASGoogle Scholar
  12. Chao, L. L., Haxby, J. V. & Martin, A. Attribute-based neural substrates in temporal cortex for perceiving and knowing about objects. Nat. Neurosci.2, 913–919 (1999). ArticleCASGoogle Scholar
  13. Konkle, T. & Oliva, A. Canonical visual size for real-world objects. J. Exp. Psychol. Hum. Percept. Perform.37, 23–37 (2011). ArticleGoogle Scholar
  14. Murphy, G. The Big Book of Concepts (MIT Press, 2004).
  15. McRae, K., Cree, G. S., Seidenberg, M. S. & McNorgan, C. Semantic feature production norms for a large set of living and nonliving things. Behav. Res. Methods37, 547–559 (2005). ArticleGoogle Scholar
  16. Devereux, B. J., Tyler, L. K., Geertzen, J. & Randall, B. The Centre for Speech, Language and the Brain (CSLB) concept property norms. Behav. Res. Methods46, 1119–1127 (2014). ArticleGoogle Scholar
  17. Hebart, M. N. et al. THINGS: a database of 1,854 object concepts and more than 26,000 naturalistic object images. PLoS ONE14, e0223792 (2019). ArticleCASGoogle Scholar
  18. Tversky, A. Features of similarity. Psychol. Rev.84, 327–352 (1977). ArticleGoogle Scholar
  19. Barsalou, L. W. Context-independent and context-dependent information in concepts. Mem. Cognit.10, 82–93 (1982). ArticleCASGoogle Scholar
  20. Maddox, W. T. & Ashby, F. G. Comparing decision bound and exemplar models of categorization. Percept. Psychophys.53, 49–70 (1993). ArticleCASGoogle Scholar
  21. Hoyer, P. O. Modeling receptive fields with non-negative sparse coding. Neurocomputing52, 547–552 (2003). ArticleGoogle Scholar
  22. Murphy, B., Talukdar, P. & Mitchell, T. Learning effective and interpretable semantic models using non-negative sparse embedding. In Proc. of COLING 2012 1933–1950 (2012).
  23. Shepard, R. N. Stimulus and response generalization: a stochastic model relating generalization to distance in psychological space. Psychometrika22, 325–345 (1957). ArticleGoogle Scholar
  24. Kobak, D. & Berens, P. The art of using t-SNE for single-cell transcriptomics. Nat. Commun.10, 5416 (2019). ArticleGoogle Scholar
  25. Shelton, J. R., Fouch, E. & Caramazza, A. The selective sparing of body part knowledge: a case study. Neurocase4, 339–351 (1998). ArticleGoogle Scholar
  26. Pedersen, T., Patwardhan, S. & Michelizzi, J. WordNet::Similarity—measuring the relatedness of concepts. In HLT-NAACL 2004: Demonstration Papers (eds Dumais, S. et al.) 38–41 (ACL Press, 2004).
  27. Warrington, E. K. & Shallice, T. Category specific semantic impairments. Brain107, 829–853 (1984). ArticleGoogle Scholar
  28. Rips, L. J. in Similarity and Analogical Reasoning (eds Vosniadou, S. & Ortony, A.) 21–59 (Cambridge Univ. Press, 1989).
  29. Smith, E. E. & Sloman, S. A. Similarity- versus rule-based categorization. Mem. Cognit.22, 377–386 (1994). ArticleCASGoogle Scholar
  30. Pilehvar, M. T. & Collier, N. De-conflated semantic representations. In 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP) 1680–1690 (2016).
  31. Nosofsky, R. M., Sanders, C. A., Meagher, B. J. & Douglas, B. J. Toward the development of a feature-space representation for a complex natural category domain. Behav. Res. Methods50, 530–556 (2018). ArticleGoogle Scholar
  32. Nosofsky, R. M., Sanders, C. A., Meagher, B. J. & Douglas, B. J. Search for the missing dimensions: building a feature-space representation for a natural-science category domain. Comput. Brain Behav.3, 13–33 (2020). ArticleGoogle Scholar
  33. Keil, F. C. Constraints on knowledge and cognitive development. Psychol. Rev.88, 187–227 (1981). ArticleGoogle Scholar
  34. Shepard, R. N. The analysis of proximities: multidimensional scaling with an unknown distance function. Psychometrika27, 125–140 (1962). ArticleGoogle Scholar
  35. Torgerson, W. S. Multidimensional scaling: I. Theory and method. Psychometrika17, 401–419 (1952). ArticleGoogle Scholar
  36. Thurstone, L. L. Multiple factor analysis. Psychol. Rev.38, 406–427 (1931). ArticleGoogle Scholar
  37. Tranel, D., Logan, C. G., Frank, R. J. & Damasio, A. R. Explaining category-related effects in the retrieval of conceptual and lexical knowledge for concrete entities: operationalization and analysis of factors. Neuropsychologia35, 1329–1339 (1997). ArticleCASGoogle Scholar
  38. Shepard, R. N. & Arabie, P. Additive clustering: representation of similarities as combinations of discrete overlapping properties. Psychol. Rev.86, 87–123 (1979). ArticleGoogle Scholar
  39. Navarro, D. J. & Lee, M. D. Common and distinctive features in stimulus similarity: a modified version of the contrast model. Psychon. Bull. Rev.11, 961–974 (2004). ArticleGoogle Scholar
  40. Carlson, T. A., Ritchie, J. B., Kriegeskorte, N., Durvasula, S. & Ma, J. Reaction time for object categorization is predicted by representational distance. J. Cogn. Neurosci.26, 132–142 (2013). ArticleGoogle Scholar
  41. Yee, E. & Thompson-Schill, S. L. Putting concepts into context. Psychon. Bull. Rev.23, 1015–1027 (2016). ArticleGoogle Scholar
  42. Charest, I., Kievit, R. A., Schmitz, T. W., Deca, D. & Kriegeskorte, N. Unique semantic space in the brain of each beholder predicts perceived similarity. Proc. Natl Acad. Sci. USA111, 14565–14570 (2014). ArticleCASGoogle Scholar
  43. De Haas, B., Iakovidis, A. L., Schwarzkopf, D. S. & Gegenfurtner, K. R. Individual differences in visual salience vary along semantic dimensions. Proc. Natl Acad. Sci. USA116, 11687–11692 (2019). Google Scholar
  44. Peterson, J. C., Abbott, J. T. & Griffiths, T. L. Evaluating (and improving) the correspondence between deep neural networks and human representations. Cogn. Sci.42, 2648–2669 (2018). ArticleGoogle Scholar
  45. Rajalingham, R. et al. Large-scale, high-resolution comparison of the core visual object recognition behavior of humans, monkeys, and state-of-the-art deep artificial neural networks. J. Neurosci.38, 7255–7269 (2018). ArticleCASGoogle Scholar
  46. Jozwik, K. M., Kriegeskorte, N., Storrs, K. R. & Mur, M. Deep convolutional neural networks outperform feature-based but not categorical models in explaining object similarity judgments. Front. Psychol.8, 1726 (2017). ArticleGoogle Scholar
  47. Iordan, M. C., Giallanza, T., Ellis, C. T., Beckage, N. & Cohen, J. D. Context matters: recovering human semantic structure from machine learning analysis of large-scale text corpora. Preprint at arXivhttps://arxiv.org/abs/1910.06954 (2019).
  48. Bauer, A. J. & Just, M. A. in The Oxford Handbook of Neurolinguistics (eds de Zubicaray, G. I. & Schiller, N. O.) 518–547 (Oxford Univ. Press, 2019).
  49. Binder, J. R. et al. Toward a brain-based componential semantic representation. Cogn. Neuropsychol.33, 130–174 (2016). ArticleGoogle Scholar
  50. Huth, A. G., Nishimoto, S., Vu, A. T. & Gallant, J. L. A continuous semantic space describes the representation of thousands of object and action categories across the human brain. Neuron76, 1210–1224 (2012). ArticleCASGoogle Scholar
  51. Cichy, R. M., Kriegeskorte, N., Jozwik, K. M., van den Bosch, J. J. & Charest, I. The spatiotemporal neural dynamics underlying perceived similarity for real-world objects. NeuroImage194, 12–24 (2019). ArticleGoogle Scholar
  52. Bankson, B. B., Hebart, M. N., Groen, I. I. A. & Baker, C. I. The temporal evolution of conceptual object representations revealed through models of behavior, semantics and deep neural networks. NeuroImage178, 172–182 (2018). ArticleCASGoogle Scholar
  53. Zheng, C. Y., Pereira, F., Baker, C. I. & Hebart, M. N. Revealing interpretable object representations from human behavior. Preprint at arXivhttps://arxiv.org/abs/1901.02915 (2019).
  54. Abadi, M. et al. TensorFlow: a system for large-scale machine learning. In 12th Symposium on Operating Systems Design and Implementation 265–283 (2016).
  55. Kingma, D. P. & Ba, J. Adam: a method for stochastic optimization. Preprint at arXivhttps://arxiv.org/abs/1412.6980 (2015).

Acknowledgements

We thank A. Corriveau for help with the data collection in the laboratory experiment, L. Stoinksi and J. Perkuhn for help with finding public-domain images for this publication, and I. Charest, B. Love, A. Martin and P. McClure for helpful discussions and/or comments on the manuscript. This research was supported by the Intramural Research Program of the National Institutes of Health (grant nos ZIA-MH-002909 and ZIC-MH002968), under National Institute of Mental Health Clinical Study Protocol 93-M-1070 (NCT00001360), and by a research group grant awarded to M.N.H. by the Max Planck Society. The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript.

Author information

Authors and Affiliations

  1. Laboratory of Brain and Cognition, National Institute of Mental Health, National Institutes of Health, Bethesda, MD, USA Martin N. Hebart & Chris I. Baker
  2. Vision and Computational Cognition Group, Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany Martin N. Hebart
  3. Machine Learning Core, National Institute of Mental Health, National Institutes of Health, Bethesda, MD, USA Charles Y. Zheng & Francisco Pereira
  1. Martin N. Hebart