SciXa Research Desk · Cluster Article 44 of 80 · AI + Animal Communication Series
← Series hub: AI and Animal Communication — Decoding the Languages of Nature
Building Multimodal Models of Animal Behavior
Multimodal AI is graduating from detecting animal signals to predicting the behavior of whole animals — and the binding constraint shifts from raw accuracy to proving what the resulting models actually measure.
By Ashraf A. Humam · SciXa Research Desk
Published September 27, 2026 · Updated September 27, 2026 · Estimated reading time: about 71 minutes · 33 cited sources
Audience: researchers, AI/ML engineers, computational biologists, bioacousticians, cognitive scientists, students, and scientifically curious readers
What You Will Learn
- The target variable changes everything. Every model earlier in this series detected, classified, or embedded recordings of communication. A behavior model instead predicts what an animal does, did, or will do — and communication signals become one observable channel among several feeding that prediction.
- Representation precedes and constrains discovery. Ethograms, discrete event sequences, continuous pose trajectories, and space-use tracks each preserve a different slice of behavior and silently delete the rest. The representation you choose is a hypothesis about what matters, and it predetermines which findings are even expressible.
- Ground truth for behavior is manufactured, not found. Labels come from human observers whose agreement is imperfect, whose categories are theory-laden, and whose time is the scarcest resource in the pipeline. The annotation bottleneck — not architecture — usually caps what a behavior model can learn.
- Predictive validity is not construct validity. A model can hit high accuracy on its labels while measuring handler identity, cage position, or recording artifacts instead of the behavior it is credited with. Clever Hans (Pfungst, 1911) remains the founding parable of exactly this failure.
- Working behavior models exist, and their limits are the lesson. Mouse vocalization-to-behavior prediction, valence classification across 7,414 pig calls, machine-decoded waggle dances validated by a dancing robot, and context-annotated bat corpora define today’s achievable envelope — single-setting, correlational, and construct-limited.
- The characteristic error of this field is the over-read. Predicting that grooming will follow a call pattern is not discovering that the animal anticipates grooming. Model outputs are forecasts of observable categories; mental-state and communication claims require experiments, not better fits.
The short answer, stated up front: a multimodal model of animal behavior is a system that consumes several observation channels — sound, posture and movement, position, environmental context — and outputs predictions or explanations of organism-level behavior rather than properties of a recording. This article argues that the endgame of multimodal AI in animal communication research is exactly such models, and that the binding constraint on what they mean is construct validity, not predictive accuracy: the discipline of showing that a model’s inputs, labels, and outputs correspond to the behavioral construct it is claimed to measure. Accuracy is now the easy part; measurement claims are where behavior modeling either succeeds or quietly fails. Throughout, we keep the series’ evidentiary discipline in view — correlations, however predictive, do not establish causation or communication, and the only validated route from pattern to knowledge still runs through experiments, above all playback (Kroodsma, 1989; Rendall, Owren, & Ryan, 2009).
Confidence labels: claims in this article carry HIGH (replicated), MEDIUM (single-study or contested), or LOW / OPEN (hypothesis) grades.
1. Introduction: from signal pipelines to behavior models
The first forty-three articles in this series have mostly been about recordings. Machine learning now detects vocalizations across entire soundscapes, classifies them by species and call type, embeds them in searchable vector spaces, and joins them with movement and environmental context — the scale problem that once defined bioacoustics has been largely solved at the level of the signal (Kahl et al., 2021; Stowell, 2022). That assessment is the first thesis of the series’ pillar article, and it is worth pausing on what it does and does not say. It says detection, classification, embedding, and monitoring are industrialized. It says nothing about whether any of those systems know what an animal is doing. Confidence: HIGH
This article is about the next target, and it closes the series’ multimodal group by inverting its lens. A multimodal model of animal behavior is a system whose output is not a label on a recording but a prediction, description, or explanation of what an animal does, did, or will do — groom, forage, retreat, recruit, call-and-respond — inferred from several observation channels at once. In such a model, communication is demoted from the object of study to one feature source among several. Sound becomes an input like any other, feeding a system whose target variable lives at the level of the whole organism. That inversion is the whole game: once the target is behavior, the interesting difficulties stop being acoustic and start being conceptual.
Three things change when the target variable moves from signal to organism. First, the epistemic stakes rise: a claim about a recording can be wrong without consequences, but a claim about an animal — that it is in pain, that it will attack, that its welfare is compromised — invites action, and actions have welfare, economic, and ethical costs. Second, the gap between measurement and meaning widens: a spectrogram is two seconds from the sound it summarizes, while a “grooming” prediction is an inference across a chain of representation, labeling, and modeling decisions, each of which can fail invisibly. Third, evaluation becomes the hard problem: for signals, ground truth is mechanical (a call is where the detector says it is, or it is not); for behavior, ground truth is itself a human judgment with contested categories.
Ethology has been here before, which is why its old discipline travels so well. Tinbergen’s (1963) four questions — causation, function, ontogeny, and evolution — remain the cleanest map of what a behavior claim can even mean, and every behavior model implicitly answers a subset of them, usually the proximate ones, usually implicitly. A model that predicts attack from posture is a claim about proximate causation at the observational level. A model that tracks how vocal behavior changes across development, as the Egyptian fruit bat corpora do across vocal ontogeny (Prat, Taub, Matosevich, & Yovel, 2017), brushes against ontogeny. The problem is that models rarely announce which question they are answering, and readers routinely hear an answer to a stronger question than the one that was asked.
Which brings us to the thesis. The endgame of multimodal AI in this field is not better detection — detection is done — but predictive and explanatory models of whole-animal behavior, in which communication is one observable channel among several. The binding constraint on those models is not accuracy. Modern learning systems reach impressive accuracy on almost any dataset with enough labels. The binding constraint is construct validity: whether the quantity a model optimizes corresponds to the behavioral construct it is credited with measuring. Clever Hans performed his arithmetic with genuine reliability; Pfungst’s (1911) investigation found the construct was not arithmetic at all but human cue-reading. Every behavior model now being built faces the same examination, and Section 5 argues that most of the field’s current instruments would not yet pass it cleanly.
A standing caveat frames everything that follows. Behavior models learn statistical dependencies in observational data; those dependencies are correlations. A model that predicts grooming from a vocal pattern has found that the pattern and the behavior co-occur, not that one causes, signals, or anticipates the other — common causes, observer structure, and recording artifacts produce the same fit. The series’ third thesis holds here with special force: the only validated route from pattern to knowledge runs through experiments, and behavior models are best understood as machines for proposing them (Kroodsma, 1989). The evidence question for prediction-from-communication specifically — how strong those predictions actually are, study by study — is treated in depth by a later article in this series; here, prediction is handled as one evaluation mode among several.
The plan: Sections 2 through 4 cover what behavior models are made of — representation, architecture, and ground truth. Section 5 develops the evaluation framework, and Section 6 audits the case studies that actually exist: mice, pigs, bees, and bats. Sections 7 and 8 ask what models can do beyond their training data — simulate, transfer. Sections 9 and 10 examine the two places behavior models meet the world: interpretation and welfare. Sections 11 and 12 take stock. The through-line is a single question, asked at increasing depth: what would it take for a model of behavior to be an instrument rather than an anecdote?
Read more about how scientists study animal communication today for the field-practice baseline these modeling ambitions extend, and about the road from animal signals to machine learning models for the pipeline stages this article treats as given.
2. Representing behavior: ethograms, sequences, trajectories, and tracks
Before any architecture is chosen, a more consequential decision has already been made: the formal structure in which behavior will be stored. Everything a model can subsequently find is carved out of that structure, and everything the structure omits is invisible forever — not missing data, but a missing world. This section surveys the four representation families that dominate animal behavior modeling and argues that the choice among them is not administrative but theoretical.
Read more about How Animals Communicate Across Different Species elsewhere in this series.
The four families
Ethograms are the oldest and most human of the four: a fixed catalog of named behavior categories — approach, groom, flee, mount, investigate — with rules for assigning observed behavior to categories. Their virtues are comparability and interpretability; a century of ethological literature speaks in their vocabulary. Their costs are less advertised. Category boundaries import observer theory: where one annotator draws the line between “investigation” and “aggression,” another draws it elsewhere, and the ethogram silently takes a side. Graded behavior is forced into bins, and borderline bouts — often the scientifically interesting ones — get filed by convention. When Hobaiter and Byrne (2014) attached candidate meanings to chimpanzee gesture types, the catalog itself was the discovery instrument; the point generalizes, because an ethogram is always a hypothesis about the natural kinds of a species’ behavior, and hypotheses can be wrong.
Discrete event sequences keep the ethogram’s categories but add what ethograms flatten: order and timing. Behavior becomes a stream of timestamped events, and questions become questions about structure — what follows what, with what probability, over what lag. This is the representation that sequence models consume, and it carries a known warning label: animal event streams have repeatedly turned out to carry structure beyond first-order transitions, in vocal sequences explicitly (Kershenbaum et al., 2016), and there is no reason to expect behavioral sequences to be simpler. A representation that records events but a model that assumes memorylessness will systematically underfit the animal.
Continuous pose trajectories abandon categories for geometry. Markerless pose estimation turns video into per-frame skeletal keypoints (Mathis et al., 2018); derivatives give velocity, acceleration, joint angles, and body curvature as continuous signals; automated identity tracking keeps those trajectories attached to the right animal in a group (Pérez-Escudero et al., 2014). What the continuous family preserves — graded dynamics, sub-second structure, individual movement signatures — is exactly what ethograms delete. What it loses is meaning: a pose trajectory is a curve, not a behavior, and to say anything behavioral you must re-segment it, at which point you have built a new ethogram in disguise. Confidence: HIGH The measurement substrate itself is now mature: pose estimation and identity tracking are replicated, widely adopted instruments in laboratory and an expanding range of field settings.
Space-use tracks compress behavior into geography: position over time, home ranges, proximity events, movement paths. The aggregation layer on top is the animal social network, whose construction and interpretation have a mature methodological literature (Farine & Whitehead, 2015). Space-use is the representation of choice for ecology-level questions — association patterns, habitat selection, range shifts — and it is almost blind to mechanism: two animals whose proximity event reflects courtship and whose proximity event reflects shared shade are, in this representation, the same datum.
Why the choice predetermines the finding
Take one interaction between two mice and store it four ways. In an ethogram it is an attack event, duration-coded, category-assigned. As an event sequence it is one symbol in a stream whose local statistics can be measured. As pose trajectories it is a pair of curves with a convergent approach, a velocity spike, a postural change. As proximity data it is one strong edge in a network that persists for weeks. None of these is the interaction; each is a projection of it, and the projections disagree about what happened. A model built on any one of them can only discover structure that survives that projection — the ethogram model cannot find the graded escalation that the trajectory preserves, and the trajectory model cannot find the categorical transition the ethogram encodes. When a study reports that machine learning “discovered” a behavioral structure, the honest footnote is that the representation co-discovered it, or rather pre-constrained it.
The discipline that follows is to treat representation choice as a reported, justified decision rather than a default. Concretely: state which distinctions the representation can and cannot express; report inter-observer agreement for any human-assigned categories (Section 4); and where findings matter, cross-check across families — does the structure found in trajectories correspond to the structure found in events? The human action-recognition literature is a cautionary mirror here: video understanding only consolidated after large annotated corpora defined shared targets (Carreira & Zisserman, 2017), and animal behavior has no equivalent corpus scale, which is precisely why representation choices still vary lab to lab and why findings resist comparison.
Hybrid representations deserve mention because practice is drifting toward them. Event streams can carry continuous attributes — each symbol annotated with duration, intensity, or trajectory shape — and pose trajectories can be segmented into candidate events by change-point detection, giving the sequence model discrete tokens without a human ever naming them. The hybrids inherit both families’ virtues and both families’ failure modes: an automatically segmented event stream is still a segmentation, with thresholds someone chose, and those thresholds are representational decisions masquerading as preprocessing. The cleanest practice treats representation choice as a robustness exercise rather than a commitment: when a finding survives recoding into a second family — when the structure found in trajectories also appears in the event stream an independent segmentation would produce — it is substantially more credible than a finding that lives in one representation alone.
Identity deserves a final word because it is representational too. A behavior model without individual identity averages over animals, confounding between-individual and within-individual variance — the difference between “mice attack after these calls” and “this mouse attacks after these calls” is the difference between a population statistic and a model of an organism (Pérez-Escudero et al., 2014). Since the series’ social-identity articles treat individual recognition in depth, we flag the point here only as it bears on modeling: identity is not metadata; it is part of the representation that determines what kind of claim the model can support.
Read more about computer vision for animal communication research for the pose-estimation and tracking substrate continuous representations are built on, about AI analysis of animal gestures and movements for how discrete movement units are segmented and cataloged, and about how AI interprets animal body language for the channel-specific version of the representation question.
Read more about combining sound, motion and environmental context for the three-stream join that supplies multimodal behavior models with their aligned inputs, and about modeling animal communication as social networks for the network-level representation this section compresses into one family.
3. Architecture: what kind of model is a behavior model?
Given a representation, what model class consumes it? The honest summary is that no architecture family has consolidated as the standard for animal behavior — the field remains an engineer’s bazaar, and the choice still tracks species, question, and dataset size more than any settled theory (Goodfellow, Bengio, & Courville, 2016). This section sketches the four families in enough detail to see what each assumes about behavior, and defers the deep treatments: transformer sequence modeling to the series’ audio-side article, and fusion architecture to the multimodal foundations article. What cannot be deferred is the structural point this article owns: representation, architecture, evaluation, and use form a stack in which each layer’s choices constrain all layers above it (Figure 1).
Sequence models over events treat behavior as language-like data: a stream of discrete symbols to be embedded, contextualized, and predicted. Next-event prediction is the natural self-supervised objective — hide the last event and ask the model to guess it — and it requires no labels at all, which matters enormously given Section 4’s bottleneck. The family’s known blind spot is structural: if behavior has long-range or hierarchical dependencies, a model must be given the capacity to express them, and the vocal-sequence evidence that animal streams exceed first-order structure (Kershenbaum et al., 2016) suggests behavioral streams will not be trivial either. The transformer machinery that dominates audio belongs here, and the series’ dedicated article covers it; the behavior-specific question is only what the tokens are.
State-space models make behavior’s hidden structure explicit. The animal is assumed to occupy a latent state — foraging, vigilant, engaged — that emits observable signals probabilistically; learning means recovering the state space and its transitions from observations. This family includes classical hidden Markov and Kalman-modeling practice in movement ecology as well as its modern neural descendants. Its virtue is interpretability: states can be inspected, named, and compared across conditions. Its vice is that the state structure is an assumption, and a misspecified latent space produces confident, inspectable nonsense — the model will happily report transitions between states that do not correspond to anything in the animal.
Neural dynamics models — world models in the modern ML sense — learn both a state and a transition function, so the model can be rolled forward: given the current state, predict the animal’s trajectory of observable behavior. This is the family that makes simulation possible, which is why Section 7 returns to it. The learned dynamics are only as good as the coverage of the training data, a constraint that will dominate the simulation discussion: a dynamics model is an interpolation engine over experienced situations, and counterfactuals are, by definition, situations the animal’s data never contained.
A dimension the four families understate is hierarchy. Behavior is organized on nested timescales: postures compose into actions, actions into bouts, bouts into sessions, and sessions sit inside circadian and developmental rhythms that dwarf them all. A model that treats events as a flat sequence sees mixture where the animal has structure, and its “discoveries” then partly re-derive the hierarchy it was denied — a familiar outcome in which unsupervised clustering of behavior recovers coarse timescale bands rather than action categories. Architectures respond with mechanisms for hierarchy, and the choice among them is again an assumption about the animal: fixed multi-scale structure hard-codes the nesting, learned hierarchy lets the data propose it, and both struggle when the relevant timescale differs between contexts — a grooming bout and a territorial patrol do not share a clock.
Cross-modal attention over multimodal streams, finally, handles the fusion problem: aligning sound, pose, and context channels that sample the world at different rates and with different noise. The architectural options — early fusion into a shared representation, late fusion of per-channel predictions, attention-mediated alignment — carry the same content here as in the series’ dedicated treatments, and we defer to them. One behavior-specific note survives the deferral: audio embeddings have become the de facto interface for the sound channel (Ghani, Denton, Kahl, & Klinck, 2023), so the fusion layer of a behavior model typically consumes vector summaries whose geometry was shaped by acoustic, not behavioral, objectives — a small construct-validity leak built into the plumbing.
The stack constraint in Figure 1 is the section’s argument compressed. Discrete representations make sequence models natural and dynamics models awkward; continuous representations do the reverse. Latent-state architectures support interpretability checks that blackboxes do not, so architecture choice pre-decides part of what evaluation can even examine; and evaluation, in turn, decides what uses are defensible — a model with no construct validation has no business informing a welfare decision, however accurate. The contrast with the audio side of the field is instructive: there, a foundation-model moment has consolidated practice around shared pretrained trunks (Bommasani et al., 2021; Ghani et al., 2025). Behavior has no such moment. Whether a general, cross-species foundation model of behavior is even possible — given that ethograms do not compose across species the way spectrograms do — is an open question, and we grade it accordingly. Confidence: LOW
Read more about multimodal AI for understanding animal communication for the full fusion-architecture treatment this section compresses, about transformers for animal bioacoustic analysis for the sequence-modeling depth deferred here, and about audio embeddings for animal communication research for how the sound channel becomes a model-ready interface.
Read more about whether AI can discover hidden structure in animal communication for the unsupervised-structure companion to the architectural choices above.
4. Ground truth: the annotation bottleneck for behavior
Ask a behavior-modeling lab what limits them and the answer is rarely compute. Signals can now be detected automatically — ultrasonic vocalization detectors such as DeepSqueak removed much of the listening burden for rodent audio (Coffey, Marx, & Neumaier, 2019) — but deciding what the animal was doing still consumes expert human time at a rate that dwarfs any other cost in the pipeline. Behavior annotation is slow, ambiguous, and observer-dependent, and every label carries all three properties into the model that learns from it. This section is about that bottleneck and the strategies for living with it: none of them solves it, and each trades a known cost for a quieter one.
Read more about How AI Analyzes Animal Sounds at Scale elsewhere in this series.
Inter-observer reliability: the label’s own error bar
The classic discipline is inter-observer reliability: multiple trained annotators label the same recordings independently, and their agreement is measured before any model sees the data. The logic is unforgiving. If two experts agree imperfectly about which bouts are “aggression,” then the category itself is not stable at that granularity, and a model trained on it inherits a ceiling: it cannot be more right than its labels, and reporting accuracy above the reliability ceiling is a tell that something else — usually leakage — is being measured instead. Reliability varies systematically across behavior kinds: sharp kinematic events with clear onsets and offsets agree well; graded social states, intent-adjacent categories, and blends agree poorly. The mature practices follow directly: report reliability per category, collapse or merge categories that fail to agree, and treat borderline bouts as an explicit class rather than forcing them by convention.
The harder truth is that labels are theory-laden all the way down. An ethogram is not a neutral recording of nature; it is a hypothesis about the natural kinds of a species’ behavior, and the annotator is a model — a human one — whose outputs drift with experience, expectation, and fatigue. Expectation is the dangerous one, and it previews Section 5’s parable: an annotator who believes an interaction is heading toward aggression starts seeing aggression earlier, not from dishonesty but from ordinary perceptual priming. Annotation drift across months of labeling sessions is the quiet, unmodeled version of the same effect, and studies that label at scale across time inherit it silently.
Weak supervision and the context shortcut
Because hand labels are expensive, the field increasingly borrows them from circumstances. Weak supervision from context uses recording-session metadata as a noisy label source: sessions with a known handling event, a feeding schedule, an introduced stimulus, or a recording context become pseudo-labels for the behavior presumed to dominate them. The economics are irresistible and the epistemics are treacherous. A feeding context does not mean every vocalization in the session is a feeding vocalization; it means the session-level prior shifted. Models trained on context-derived labels learn, at minimum, to recognize context — and context recognition is behavior prediction only if the context-to-behavior mapping is tight, which is precisely what the construct-validity analysis of Section 5 exists to check. The Egyptian fruit bat corpus shows the pattern done carefully: vocalizations annotated across varying contexts and vocal ontogeny, with the context structure itself a documented object of study rather than an assumed label (Prat, Taub, Matosevich, & Yovel, 2017).
Active labeling attacks the bottleneck from the demand side: a model-in-the-loop selects which clips a human should annotate next — the ones where its own uncertainty or an ensemble’s disagreement is highest — so that each hour of expert time buys maximal information. The efficiency logic is sound; the sampling-bias risk is under-discussed. A dataset built entirely from a model’s confusion regions is a mirror of that model, and its errors propagate as curriculum: the next model inherits a data distribution shaped by the previous one’s ignorance. The remedy is to reserve a random-annotation fraction alongside the actively selected one, so the dataset keeps an unbiased core against which the active-selection gain can itself be measured.
Multimodality multiplies the bottleneck in a way that is easy to underestimate from outside. A single-channel annotation pass — watch video, label behavior — already costs expert time at a large multiple of real time; a multimodal pass that must also hear, place, and contextualize each event costs more, and the interfaces that would let an annotator move fluidly across synchronized channels are still mostly bespoke per lab. The practical consequence is that multimodal ground truth tends to be thin where it matters most: exactly the rare, brief, multi-channel events — a call answered by a gesture, an approach answered by a retreat — that carry the richest information are the ones a fixed annotation budget samples worst. Studies that report dense single-channel labels and sparse multi-channel ones are not being sloppy; they are being honest about an economics problem the field has not solved.
Identity assignment belongs in this section because it is labeling too. Deciding which track, which call, which proximity event belongs to which animal is a classification problem with its own error rate, and behavior models built on identity-resolved data inherit that error silently (Pérez-Escudero et al., 2014). Automated identification of unmarked animals makes this tractable at group scale, but the honest reporting practice is the same as for any label: state the assignment method, state its error characteristics, and where identification confidence varies, propagate it rather than thresholding it away.
The section closes on the bottleneck’s deepest layer, because it is the one the field talks about least. For behavior categories, imperfect humans are the best available ground truth. For internal states — affect, motivation, intent — there is no direct ground truth at all, only proxies: production contexts assumed to carry valence, physiological correlates, subsequent behavior. The pig valence program of Section 6 is exactly this: valence labels derived from the production context in which calls were recorded (Briefer et al., 2022). That is a legitimate and productive design, and it is also the first link of a construct-validity chain whose remaining links — do the context-derived labels track the state, and does the state mean what we say — can never be closed by adding more data of the same kind. Where the chain cannot be closed, the honest output is not a confident label but a calibrated proxy, and the field’s vocabulary should say so.
Read more about combining animal sounds and behavioral data for the join-level mechanics of aligning labels with signals, and about training AI models on unlabeled animal sounds for the self-supervised strategies that reduce dependence on hand labels.
Read more about tracking who communicates with whom using AI for the identity-assignment machinery behavior models inherit as labeling.
5. Evaluation: predictive validity is not construct validity
Two words that get used interchangeably in casual coverage name the difference between a behavior model that is an instrument and one that is an anecdote. Predictive validity asks whether the model predicts its labeled target on data it has not seen. Construct validity asks whether that target — as operationalized by the representation, the labels, and the model — corresponds to the behavioral construct the model is credited with measuring. The first is a property of a model against its dataset. The second is a property of the whole measurement claim against the world. Behavior models routinely have the first and lack the second, and nothing about improving the first repairs the second: a model can become arbitrarily accurate at predicting the wrong thing.
Read more about Can AI Discover Animal Communication Rules? elsewhere in this series.
The founding parable is a horse. In early twentieth-century Berlin, Clever Hans answered arithmetic and calendar questions by tapping, and did so with a reliability that impressed a public and a scientific commission alike. Pfungst (1911) then did the thing that separates evaluation from demonstration: he varied the conditions. When the questioner could not see the answer plate, or did not know the answer, or was positioned out of the horse’s sight, performance collapsed. The horse’s arithmetic was cue-reading — involuntary postural signals from the humans present, given as the answer approached and stopped at the right count. The detail worth internalizing is that the horse’s predictive performance was real. Hans genuinely predicted the tapping stop-point from available cues, reliably, under the conditions he was tested in. The failure was not in the accuracy; it was in what everyone decided the accuracy measured. Every behavior model with a shortcut feature — handler identity, cage position, recorder artifacts, time-of-day structure — is Clever Hans with a loss curve.
Modern evaluation has a name for the mechanisms that manufacture false accuracy. Leakage is any path by which evaluation information reaches the model. The most common form in behavior modeling is innocent-looking and devastating: random splits that place the same animal, the same session, or the same recording batch in both training and evaluation. The model then solves a recognition problem — whose voice, which cage, what rig — instead of a behavior problem, and scores beautifully. The minimum credible discipline is group-level splitting: hold out entire individuals, sessions, or cohorts, and report the gap between random-split and group-split performance; a large gap is not an embarrassment to be buried but the measurement of how much of the headline number was identity in disguise. Confidence: HIGH
Baseline discipline is the second leg. A behavior model earns its complexity only by beating trivial alternatives under identical splits: the persistence baseline (predict the current behavior continues), the majority-class baseline (predict the most frequent category), and a simple-feature baseline (logistic regression on hand-crafted descriptors). Persistence baselines are brutal for behavior data because behavior is autocorrelated — the best single predictor of what an animal does next is usually what it is doing now — and models that impress against random splits can fail to beat “nothing changes” under honest ones. Reporting baselines is not a formality; it is the only way a reader can compute how much of the reported performance is behavior and how much is clockwork.
Construct validity itself is assessed by pattern, not by a single test, because there is no gold standard for most behavioral constructs. Convergent validity asks whether the model’s output correlates with independent measures of the same putative construct — does the learned “aggression” score track independent aggression indicators, and not merely the features used to train it? Discriminant validity asks the mirror question: does the score fail to correlate with things it should not measure, such as cage location or handler identity? Intervention checks are the strongest evidence available short of full experiments: perturb the world in a way that should change the construct and see whether the score moves. The classic meaning-testing literature in animal communication — where playback experiments earned the vervet alarm call its semantic reading (Seyfarth, Cheney, & Marler, 1980) — is the standard this machinery borrows, and the series’ meaning-testing article develops it in full.
A final framing note before the case studies, because it organizes everything after it. Evaluation in this field is not a gate that papers pass or fail; it is a characterization that papers owe their readers. A behavior model with honest group-split accuracy against a documented baseline, a stated reliability ceiling, and one convergent-validity check is a modest but real instrument. A model with a large random-split number and none of the rest is a rumor with a decimal point. Table 2 summarizes the four criteria as they look in practice.
| Criterion | The question it answers | Example practice | Failure mode when absent |
|---|---|---|---|
| Predictive validity | Does the model predict its labeled target on unseen data? | Group-level splits by animal, session, and cohort; held-out sites; uncertainty reported | Overfit performance that collapses on the next cohort or enclosure |
| Construct validity | Does the target correspond to the behavioral construct claimed? | Convergent and discriminant checks; intervention tests; preregistered predictions | An accurate model of the wrong thing — the Clever Hans outcome (Pfungst, 1911) |
| Leakage controls | Is evaluation information absent from training? | Individual-aware splitting; documented recording provenance; preprocessing audit | Inflated accuracy from shared animals, sessions, or artifacts |
| Baseline discipline | Does the model beat trivial alternatives? | Persistence, majority-class, and simple-feature baselines under identical splits | Architectural complexity mistaken for behavioral signal |
Read more about whether AI can predict animal behavior from communication for the evidence audit of prediction claims this section treats as one evaluation mode, and about how scientists test meaning in animal signals for the experimental standards construct validation ultimately borrows.
Read more about the role of context in understanding animal signals for why context collapse corrupts labels and predictions alike.
6. Case studies: behavior models that exist today
It is one thing to argue that behavior models are hard and another to examine the ones that actually exist. Four research programs anchor this section — mice, pigs, bees, and bats — chosen because together they span the modeling ladder from attribute classification through internal-state proxies to signal-to-behavior chains closed by intervention. Each is treated the same way: what the model takes in, what it predicts, how it was validated, and what the construct analysis says the prediction is actually worth. The summary table collects the inventory.
Read more about Can AI Understand Dolphin Communication? elsewhere in this series.
7,414
pig calls classified by emotional-valence proxy and production context (Briefer et al., 2022)
411
individual pigs recorded from birth to slaughter (Briefer et al., 2022)
8,719
sperm whale codas analyzed for contextual and combinatorial structure (Sharma et al., 2024)
14,597
species in Perch 2.0 pretraining — detection scale that behavior modeling has not approached (Ghani et al., 2025)
Mice: from caller attributes to behavior prediction
The rodent ladder has two visible rungs. On the lower one, Ivanenko et al. (2020) trained deep networks to classify the sex and strain of the caller from mouse ultrasonic vocalizations — a demonstration that caller attributes are encoded in call acoustics recoverable by learning systems. Strictly speaking this is attribute classification, not behavior modeling: the target is who the animal is, not what it does. But the rung matters because it establishes the inference direction a behavior model depends on — from signal properties to organism-level facts — under controlled conditions where the ground truth is mechanical (sex and strain are known, not judged).
The higher rung is more directly this article’s subject. Gnutti et al. (2026) built a deep neural network that predicts mouse behavior from ultrasonic vocalizations — the closest thing in the verified literature to a true communication-to-behavior model. The honest reading keeps all the caveats visible at once: single species, laboratory conditions, correlational, and bounded by the ethogram its labels came from. What it demonstrates is that the information channel exists — vocalizations carry enough signal to support above-noise prediction of behavioral categories in this setting. What it does not demonstrate is mechanism, meaning, or transfer, and the study does not claim to. Confidence: MEDIUM
Pigs: an internal-state construct at scale
The pig program is the field’s most instructive exercise in modeling an internal state. Briefer et al. (2022) recorded 7,414 calls from 411 pigs across the production cycle — birth to slaughter — and classified calls by emotional valence and production context. The scale and design are exemplary: many individuals, many contexts, the full life span. The construct analysis is equally instructive. “Valence” here is proxied by the production context — situations assumed to be positive or negative for the animal — so the model learns the acoustic correlates of context-assumed valence. That is a legitimate and carefully stated operationalization, and it is also a textbook construct-validity chain with open links: from call acoustics to context recognition the chain is testable; from context to valence it rests on assumptions about pig affect that no classifier can close from the same data. The study says what it did with unusual clarity, which is exactly why it teaches. Confidence: MEDIUM
Bees: the one chain closed by intervention
The bee chain deserves its spotlight because of what it proves about the general architecture of progress. Every other case in this section stops at correlational validation — the model predicts labels, the labels proxy the construct. The bee program crosses into causal territory because a robot can intervene: it produces a controlled version of the signal and observes whether the predicted behavior follows. That is the pattern-to-playback loop of the series’ pillar, executed with a machine in the loop. The caveats belong in the same paragraph as the praise: the robot elicits a partial version of natural dance-following, not the full behavioral response, and the system works because bee dances are a comparatively closed, stereotyped channel — a luxury no mammalian communication system in this article offers.
Bats: the ground-truth corpus as a strategic asset
The Egyptian fruit bat program contributes a different kind of case: not a single model but the conditions under which context models become possible. Prat, Taub, Matosevich, and Yovel (2017) published a large annotated dataset of vocalizations across varying contexts and vocal ontogeny — the labeled, context-resolved corpus that Section 4 identified as the scarce resource. Around it sits a validated experimental core: isolation and playback experiments demonstrating vocal learning in a social mammal (Prat, Taub, & Yovel, 2015), and later work establishing the role of auditory feedback in vocal production (Elie et al., 2024). The lesson for behavior modeling is infrastructural: the bat program can ask context questions because it invested in ground truth first, and its developmental dimension — vocalizations annotated across ontogeny — is precisely the axis that static datasets elsewhere ignore.
Table 1 collects the inventory. Read it as a maturity map rather than a leaderboard: each program is honest about which rung it stands on, and the field’s collective position — attributes and contexts modeled, one intervention-validated chain in an insect, nothing yet that would survive transfer to a second species — is the realistic baseline against which the next decade’s claims should be judged.
| Program | Model inputs | Predicted target | Validation approach | Key limitation |
|---|---|---|---|---|
| Mouse caller attributes (Ivanenko et al., 2020) | Ultrasonic vocalizations from laboratory mice | Caller sex and strain | Held-out classification against known attributes | Attribute targets, not behavior; laboratory conditions only |
| Mouse behavior prediction (Gnutti et al., 2026) | Ultrasonic vocalizations | Behavioral categories of the vocalizing mouse | Prediction against independently annotated behavior | Single species and setting; correlational; bounded by its ethogram |
| Pig valence (Briefer et al., 2022) | 7,414 calls from 411 pigs across production contexts | Emotional valence proxy and production context | Held-out classification across contexts and recording batches | Valence proxied from context, not measured feeling; production settings |
| Bee dance decoding (Wario et al., 2017; Dormagen et al., 2023; Landgraf et al., 2018) | Video of the dance floor | Dance vector to the resource; recruitment outcomes | Robot-performed dances with observed recruitment, plus field decoding | Robot elicits partial natural response; robot study documented as preprint |
| Bat context corpora (Prat et al., 2015, 2017; Elie et al., 2024) | Context-annotated vocalizations across ontogeny | Context of production; developmental stage | Isolation and playback experiments; auditory-feedback manipulation | Laboratory colony; context categories coarse relative to field social life |
Read more about AI and honey bee communication for the full dance-decoding program this case compresses, and about AI for understanding bat vocalizations and echolocation for the species-level companion to the bat case.
7. Simulation and counterfactuals: asking the model what-if
A model that only scores its labels is a classifier; a model that can be rolled forward is a laboratory. Once a behavior model learns a transition structure — how states follow states, how observable behavior unfolds given context — it can be run in silico: perturb an input, replay a scenario, ask what would have happened had the animal encountered a different signal, a different partner, a different environment. This section is about what those counterfactual queries are worth. The short version: as planning instruments, a great deal; as sources of findings, almost nothing — and the boundary between the two is where the discipline lives.
Read more about AI Systems for Two-Way Animal Communication elsewhere in this series.
Start with the legitimate uses, because they are real and under-exploited. Experiment planning is the most defensible. Playback experiments are expensive, slow, and ethically constrained; a behavior model can be used the way a power analysis is — to estimate, before committing animals and field seasons, which candidate interventions carry the most information, which conditions are likely to produce interpretable results, and which are likely to be swamped by variance. Kroodsma’s (1989) discipline for playback design — careful stimulus construction, appropriate controls, forethought about what a response even looks like — gains a computational counterpart when a model can cheaply screen the design space first. The simulated screen does not license the conclusion; it ranks the candidates for the real test.
Surrogate stress-testing is a second legitimate use with a welfare dimension. Before a housing change, an enrichment intervention, or a husbandry schedule modification is tried on live animals, a behavior model can be asked to forecast the behavioral consequences under uncertainty. Used this way — as a red-flag generator that prompts closer monitoring, never as an approval mechanism — simulation aligns with the reduce-and-refine logic of animal research ethics: fewer animals exposed to the informative-but-risky conditions first, because the model narrowed the search. The key word is first. The model ranks; the world decides.
Now the boundary, which is a theorem-shaped fact about what learned models are. A behavior model is an estimator of conditional structure within the distribution it was trained on. Counterfactual queries are, almost by definition, questions about points off that distribution — signals the animal never gave, encounters that never occurred, interventions never performed. The model will still answer, because answering is what it does; the answer is an interpolation at best and a confident fabrication at worst, and nothing in the output distinguishes the two. This is not a limitation that more data gently erodes. It is the difference between the inside and the outside of a distribution, and the counterfactual lives outside by construction.
Animals compound the problem, because they are not static systems. The bat program demonstrates the point empirically: vocal production depends on what the animal hears, with auditory feedback shaping vocal output in the Egyptian fruit bat (Elie et al., 2024), and social isolation and playback conditions altering vocal development (Prat, Taub, & Yovel, 2015). A behavior model trained on an animal’s history encodes that history as a fixed coefficient; the living animal treats its own history as a cause of its next change. Simulation assumes the second animal is the first one. Learning, habituation, sensitization, and individual differences are all mechanisms by which that assumption fails quietly — the simulated animal never adapts, and adaptation is what the real one does.
The bee robot also marks the productive middle ground between open-loop simulation and full field experiment. A robot performing a dance is a behavior model’s output made physical: the decoded signal is re-synthesized, played into the world, and the world votes. Interactive designs of this kind — model-in-the-loop experiments where the animal’s response updates the next stimulus — are the natural evolution of simulated playbacks, and the series’ article on interactive experiments treats them in depth. What the robot case adds to the simulation discussion is the reminder that the conversion step is itself an experiment with its own validity conditions: the robot’s dance is not the bee’s dance, and recruitment to a robot is evidence about the decoded channel, not a full behavioral equivalence.
A final scope note. Counterfactual reasoning about behavior is, at bottom, a causal claim — “had X been different, behavior would have differed” — and the machinery for grounding causal claims in observational plus interventional data is a subject of its own. The series’ causation article carries that treatment. What this section claims is narrower and does not depend on it: simulated counterfactuals from behavior models are hypothesis generators with a known failure signature (confidence without coverage), and their proper output format is a prioritized experiment queue, not a conclusions section. We grade the stronger claim — that simulated playbacks can substitute for a portion of real playback experiments as planning instruments at scale — as an open hypothesis. Confidence: LOW
Read more about the road from correlation to causation in animal communication AI for the inferential machinery counterfactual claims require, about interactive AI experiments with communicating animals for the closed-loop designs that convert simulated interventions into real ones, and about testing AI-discovered animal communication patterns for the experiment-side discipline simulation planning should feed.
8. Transfer: moving behavior models across species and settings
Every model in Section 6 works somewhere. None of them works everywhere — and the two axes along which “everywhere” fails are different enough to treat separately. Setting transfer asks whether a model trained in one environment survives another: laboratory to farm, farm to field, one enclosure to the next. Species transfer asks whether a model’s learned structure is reusable for an animal it was never trained on. The field’s progress on the two axes is strikingly asymmetric, and the asymmetry itself teaches the construct lesson of this article better than any argument could.
Setting shift is the everyday failure mode, and it operates on every channel at once. Vision models meet new lighting, new camera geometry, and new occlusion regimes; pose estimators meet substrates and enclosure furniture they never saw; audio models meet microphone placements, background regimes, and signal-to-noise conditions that shift the very features they depend on; tracking meets densities and group sizes outside training range. On top of the channel-level shifts sits a deeper one: ethogram drift. The behavior categories themselves — their frequencies, their boundaries, even their meaningfulness — differ across settings. A category structure elicited in a controlled laboratory interaction may simply not carve a field population’s behavior at its joints, and a model transferred into that setting is solving a subtly different task wearing the same label vocabulary. The consumer-facing symptom is performance decay that no single channel explains, because the shift is distributed across all of them.
Species transfer is where the asymmetry lives, and it is best seen through the contrast the field itself provides. On the audio side, detection and classification transfer at industrial scale: the Perch lineage progressed from a 984-species classifier (Kahl et al., 2021) through global birdsong embeddings (Ghani, Denton, Kahl, & Klinck, 2023) to pretraining across 14,597 species (Ghani et al., 2025). Behavior has nothing remotely comparable — and the reason is not engineering effort but construct depth. Species-level sound detection is a shallow construct: the target is “which species produced this sound,” and while the acoustics vary across species, the task is invariant, so shared structure exists to transfer. Behavior is a deep construct: the categories that organize a mouse’s social life do not organize a bat’s, and an ethogram is not a spectrogram — there is no shared representational substrate on which cross-species behavior transfer can currently ride. The rule of thumb this suggests: the shallower the construct, the better it transfers, and behavior is close to the deep end.
Partial exceptions mark out where transfer does work, and they confirm the rule. Pose estimation transfers across species better than behavior semantics do, because skeletal geometry is conserved in a way that ethograms are not: a model that finds body parts finds body parts, whatever the animal does with them. At the far end, automated detection of rare vocalizations — the series’ detection articles — transfers well because it, too, is shallow. Between those poles sits the open problem this section names: behavior-model portability. What would it even mean for behavior structure to transfer? A speculative but disciplined answer: perhaps at the level of movement building blocks — approach and avoid, orient and investigate, vigilance postures — which recur across taxa while their compositions and meanings diverge. Whether such shared latent structure is real, learnable, and useful is untested, and we label it speculation rather than forecast.
The two axes also interact, and most real deployments change both at once — which is why lab-to-application transfer fails more often than either axis alone predicts. A model moved to a new species in a familiar setting, or an old species in a new setting, degrades along one distribution shift that the re-baselining discipline of this section can isolate. A model moved to a new species in a new setting meets correlated shifts: unfamiliar acoustics, unfamiliar posture vocabulary, unfamiliar context structure, and unfamiliar label semantics, all simultaneously, with no held-out condition in which any single shift can be measured separately. The engineering consequence is that transfer claims should be decomposed where possible — validate the representation shift and the setting shift against anchor tasks one at a time — and the epistemic consequence is that an undecomposed transfer claim is unfalsifiable in practice: when it fails, nothing in the report says which failure occurred.
Practical guidance for anyone deploying behavior models across the gap is mostly calibration discipline. Re-baseline at the boundary: before trusting a transferred model, measure it against persistence and majority baselines in the new setting, not the old one. Re-measure the label layer: have local annotators label a sample and compare agreement with the source setting, because ethogram drift announces itself as reliability collapse. Run a verification loop on a small random sample — human checks against model outputs — for long enough to estimate the new error rates rather than assuming the old ones. And report shift honestly: a model that performs identically in a new setting is suspicious, not impressive, and the report should say which performance is being carried by which assumptions. Confidence: LOW The strong claim — that behavior models can be made portable across species the way detection models are — is an open problem, not a roadmap.
Scope boundaries, stated as promised. The full mechanics of cross-species model reuse — what transfers, what breaks, what the foundation-model era changes — are the dedicated subject of the series’ transfer-learning article, and the question of a single model spanning many species gets its own treatment; this section deliberately restricted itself to the behavior-specific question of what portability would require. The zero-labels extreme of the transfer spectrum belongs with them.
Read more about transfer learning across animal species for the full treatment of cross-species model reuse, about whether one AI model can understand many animal species for the foundation-model version of the portability question, and about zero-shot learning for animal communication for the no-labels extreme of the transfer spectrum.
9. The over-read hazard: outputs are not mental states
Every instrument eventually meets its characteristic misreading, and for behavior models the characteristic misreading is a category error: reading a prediction about observable behavior as a discovery about an animal’s mind. “The model predicts grooming after this call pattern, therefore the animal anticipates grooming.” The first clause is a statistical statement about co-occurrence in data. The second is a claim about a mental state — expectation, foresight, possibly communication of intent. Nothing in the first entails the second, and the habit of sliding from one to the other is common enough, and corrosive enough, that this section exists to dissect it.
The slide is seductive because predictive fit feels like understanding — and there is an instructive parallel in the machine-learning literature on large language models. Bender, Gebru, McMillan-Major, and Shmitchell (2021) argued that fluent output can be produced by systems that manipulate form without grounding in meaning — the “stochastic parrot” critique. The parallel must be drawn carefully, as a parallel about inference and not about machines or animals: in the language-model case, form (text statistics) was mistaken for meaning (grounded communication); in the behavior-model case, form (observable co-occurrence) is mistaken for meaning (anticipation, intent, communicative function). In both cases, the output is real and the attribution is unearned. The lesson transfers precisely because it is methodological: what a system does with its inputs is a fact about the system and the data; what the output means is a claim that requires independent warrant.
Unpack the grooming example and at least three mechanisms generate the same fit without any anticipation on the animal’s part. First, the common-cause structure: an underlying state — hunger, arousal, hormonal cycle, developmental stage — produces both the vocal pattern and the grooming behavior; the signal predicts the behavior the way smoke predicts fire, as an effect of a shared cause, not a message about it. Second, the signal-as-effect structure: the vocal pattern may itself be produced by a state that also organizes the subsequent behavior — the call is a symptom, not a forecast. Third, observer-mediated structure: the labeling protocol, the housing schedule, or the recording design may couple the two — sessions that include post-event grooming also include the call pattern, and the model learns the session, not the animal. Genuine anticipatory signaling is a fourth possibility — animals do plan, and some signals do function prospectively — but it is one hypothesis among four, and choosing among them is what experiments are for. The fit itself is silent on the choice.
The field’s own gold standard shows what the alternative looks like. The vervet alarm call earned its semantic reading not because the calls correlate with predator presence — they do, but so do countless other events — but because playback experiments showed that hearing the call alone, with no predator, produces the appropriate escape behavior in recipients (Seyfarth, Cheney, & Marler, 1980). The correlation suggested the hypothesis; the intervention tested it. Rendall, Owren, and Ryan (2009) generalize the caution: attributing meaning to signal structure without such tests imports assumptions about function and mechanism that the signal data cannot carry. Behavior models inherit the entire problem — and add a new surface for it, because a model’s confident number attached to a mental-sounding label (“anticipation index,” “stress score”) makes the unearned attribution look measured.
The reporting dimension deserves its own paragraph, because the over-read hazard is amplified in transit. A paper that states its claims carefully — “vocal features predict subsequent behavioral category under laboratory conditions” — becomes, two links down the chain, “AI reads mouse minds from their squeaks.” The series’ fourth thesis documents this upgrade pattern across the field, and behavior models are its next feeding ground precisely because their outputs come pre-dressed in psychological vocabulary. The defensive writing practices are boringly effective: name the observable target, not the mental state (“grooming-bout onset,” not “anticipation”); state the mechanisms that could produce the fit; and mark explicitly which claims are correlational. Reviewers can do one thing almost nothing else does: ask, of every mental-state word in a behavior-model paper, what evidence would distinguish the claimed state from its three rivals. If the paper cannot answer, the word is decoration.
One boundary statement, to keep this section honest in both directions. Nothing here says animals lack minds, plans, or communicative intents; the ethological and cognitive literature gives rich reasons to expect all three in many taxa. The claim is strictly epistemic: behavior models, as measurement instruments, do not deliver those states as outputs, and describing their outputs in mental vocabulary borrows credit the instruments have not earned. Whether AI-mediated evidence can ever support meaning-level claims — and what it would take — is the dedicated subject of later articles in this series.
Read more about why AI predictions are not proof of animal language for the full claim-inflation pattern this section dissects at the model level, about whether AI can discover meaning in animal vocalizations for the meaning-inference question the over-read corrupts, and about animal-to-human translation as science or science fiction for where the over-read hazard ends up when left unchecked.
10. Welfare and enrichment: acting on model outputs
Welfare is where behavior models stop being an epistemics problem and become a governance problem. The same instrument that is merely over-interpreted in a journal article becomes, in a barn or a shelter, a trigger for interventions — husbandry changes, veterinary attention, enrichment decisions, culling judgments in the hardest cases. This section keeps its claims deliberately qualitative and its register careful, because the deployment evidence is thinner than the promise, and because the costs of getting it wrong fall on animals that cannot audit the model.
Read more about AI Models for Animal Social Interaction elsewhere in this series.
The promise is real and worth stating precisely. Human welfare observation is periodic and attention-limited: a stockperson sees each animal for minutes a day, a welfare assessment is conducted at inspection intervals, and the events that matter most — pain onset, fear responses, developing abnormal behavior — unfold in the gaps. A continuous, non-invasive behavior model changes the observation regime itself: every hour monitored, every individual tracked, deviations flagged for review rather than discovered at inspection. The pig valence program (Briefer et al., 2022) is the clearest existing prototype of what such systems would consume — per-call classification of emotional-valence proxies at the scale of a production cycle — and expression-based pain assessment has a validated human-coding foundation in tools like the Horse Grimace Scale (Dalla Costa et al., 2014), whose automation is the natural next step for the vision side.
Between prototype and deployment sits every problem this article has developed. Construct validity becomes load-bearing: a monitoring model that actually keys on cage position, handler identity, or microphone placement will misdirect welfare attention toward artifacts — and the errors are asymmetric. A false negative is a welfare event that happened unseen, which is the status quo risk; a false positive is an intervention performed on a healthy animal, a cost paid in resources and possibly in the animal’s experience. There is no setting of the threshold that removes both, so the choice of operating point is an ethical decision dressed as an engineering parameter, and it should be made as deliberately as one — explicitly, with the error asymmetry stated and the affected parties’ interests represented in the decision.
Model drift is the quiet governance problem. A welfare monitor trained on one housing system, one genetics line, one season silently degrades as any of those change, and degradation in monitoring is itself a welfare hazard: the farm believes it is watching. The discipline follows from Sections 5 and 8 — re-baseline at every boundary, maintain a human verification loop on a random sample, audit the flags the model is not raising — but welfare adds a structural requirement the earlier sections did not: someone must own the failure modes. Who reviews flags, how fast, with what authority to act and what authority to override the model, and what happens to the record when the model is wrong — these are organizational commitments without which the monitoring system is a dashboard, not a welfare intervention.
The public-expectation layer cannot be ignored, and here the documented survey evidence is sobering. The Earth Species Project’s first global survey on AI and animal communication gathered 1,057 respondents across 67 countries (Earth Species Project, 2025), and its very existence signals how much public attention is arriving at this field. Attention of that scale is a two-edged instrument for welfare applications: it can fund and legitimize genuine monitoring advances, and it can also convert a valence-proxy classifier into a headline about machines that know what animals feel — precisely the over-read of Section 9, now with commercial and regulatory consequences. Welfare deployment done carelessly does not merely fail; it poisons the well for careful versions, because the backlash punishes the category. Confidence: MEDIUM Prototype welfare-relevant models have been demonstrated in controlled settings; evidence under routine farm and shelter conditions remains limited, and the governance layer is mostly unbuilt.
Enrichment evaluation is a quieter application with arguably the best risk profile of anything in this section. Enrichment interventions — objects, structures, foraging modifications, social arrangements — are routinely adopted on intuition and assessed, if at all, by coarse welfare indicators; a validated behavior model offers a sharper instrument for the same question, comparing behavioral time budgets and transition structure before and after an intervention, per individual rather than per pen. The design is the classical one — baseline period, intervention period, ideally a reversal — and the model’s contribution is resolution and continuity, not a new kind of claim. The construct burden is correspondingly lighter: the question is not “is the animal happier” but “did the distribution of observable behavior change in the direction our construct analysis has characterized,” which is exactly the question a well-instrumented behavior model is built to answer.
A closing distinction keeps the section honest. Everything above concerns models of behavior used for welfare — monitoring, early warning, triage. A distinct and more speculative use is AI-mediated intervention in the animals’ own communication — systems that generate signals to soothe, guide, or manage — which raises questions of manipulation and consent that this article’s scope excludes and the series’ ethics articles treat directly. The boundary matters because the technologies overlap: a model good enough to predict distress is, in principle, good enough to learn what signal reduces it, and the step from prediction to influence is a step this article flags and does not take.
Read more about AI recognition of animal facial expressions for the expression-channel constructs welfare models consume, about AI for detecting rare animal vocalizations for the detection layer early-warning systems are built from, and about the ethics of AI-mediated animal communication for the governance framework acting on model outputs demands.
11. Limitations, open questions, and what we do not know
This section inventories what behavior modeling cannot currently do, separating limits of technique — which effort and engineering erode — from limits of principle — which no quantity of data touches. The distinction matters because the field’s public narrative routinely converts the second kind into the first, and the conversion is where disappointment is manufactured.
Partial observability is a limit of principle. Behavior is the observable fraction of an animal’s state; the states themselves — motivational, affective, cognitive — are only ever inferred. The inference is underdetermined: many distinct internal configurations produce identical observables, so no labeling volume, however large, can recover the hidden variable from the visible one. What partial observability licenses is exactly what Section 5 described — convergent evidence from multiple imperfect windows, interventions that shift the state and observe the observables move — and what it forbids is the fantasy of a model whose latent space is the animal’s internal state rather than a statistical shadow of it.
Context collapse is a limit of instrumentation. Every recording thins the world. The olfactory channel is usually absent, the tactile channel almost always, the social audience before the recording started and after it ended is invisible, and the events that primed the interaction are unrecorded by construction. The context article in this series develops what is lost; the modeling consequence is that behavior models are fit to a systematically impoverished projection of the situations that produced the behavior, and their predictions inherit the impoverishment. This limit is technically tractable — more channels, longer windows, richer metadata — but the tractability is asymptotic: the world is always bigger than the sensor suite, and models are always fitting the suite.
Static datasets meet developing animals. A behavior model is a snapshot, and its subject ages. The bat corpus is again the instructive case: vocalizations were annotated across vocal ontogeny precisely because juvenile bat calls differ from adult ones in ways that matter (Prat, Taub, Matosevich, & Yovel, 2017), and the same developmental logic applies to behavior broadly — the categories, frequencies, and transitions of adolescence are not those of adulthood. A model trained on adults mischaracterizes juveniles; a model trained on a population drifts as the population changes through turnover, habituation, and social reorganization. The engineering responses exist — scheduled retraining, age-stratified models, drift monitors — but they presuppose the ongoing data collection that most studies, structured as bounded projects, do not fund.
Evaluation without ground truth is the standing open problem. For the constructs the field most wants to model — affect, intent, welfare state — there is no independent gold standard, and Section 4’s proxies are the best available. The strategies are convergent validity, interventional checks, and preregistered predictions; their limits are honesty itself. Some questions may remain open not because the models are weak but because the validation targets are not independently measurable, and recognizing that outcome is a scientific result, not a failure to be papered over with a leaderboard number.
Evaluation infrastructure is missing where it is needed most. Audio has shared benchmarks — the BEANS benchmark and the BirdCLEF challenge series documented in this series’ earlier articles — which is a large part of why detection progress is comparable across labs. Behavior modeling has no equivalent: no shared cross-species behavioral corpora with common ethograms, no standard evaluation protocol for behavior prediction, and therefore no way to know whether one lab’s model is better than another’s or merely tested on a friendlier slice of the world. The obstacle is principled, not merely organizational — ethograms do not compose across species, so a shared benchmark would need shared representations first — but the consequence is a field in which every performance number is denominated in a different currency, and where the benchmark-overfitting caution of Section 5 coexists with its mirror image: too little shared evaluation to overfit to.
The wild frontier is the sharpest test of all of it. The most ambitious current attempt to build behavior-adjacent models for a wild, long-lived, pelagic species is the sperm whale program: 8,719 codas analyzed for contextual and combinatorial structure (Sharma et al., 2024), tagging infrastructure of the DTAG kind, and the Project CETI effort’s documented modeling ambitions. Everything this article has developed applies to that program with the dials turned up — partial observability at ocean scale, context collapse across dive cycles, evaluation without ground truth for any construct richer than call identity and timing. If behavior modeling succeeds there, it will be because the validity discipline traveled with the hardware; if it produces headlines instead, it will be because it did not. What sound-only modeling can and cannot deliver, independently of the behavioral channel, is the companion question the series treats elsewhere.
Read more about Project CETI and the search for whale communication for the most ambitious current attempt to build behavior-adjacent models for a wild cetacean, and about what AI can and cannot learn from animal sounds for the sound-only ceiling on models that lack the behavioral channel.
12. Conclusion: behavior models as instruments
A thermometer is not useful because it is accurate; it is useful because its error characteristics are characterized — because someone established what it measures, over what range, with what systematic bias, and when it fails. Instruments earn their place in science through characterized validity, not raw performance. The argument of this article is that behavior models will earn theirs the same way or not at all: the question for the coming decade is not whether we can build them — Section 6’s inventory shows we demonstrably can — but whether we can say what they measure.
The multimodal group of this series now closes, and its arc is worth one paragraph of recap. It began with fusion concepts and the sound-plus-behavior join, moved through the individual channels — vision, expression, body language, gesture — and arrived at the three-stream contextual join that supplies aligned multimodal inputs. This article added the layer above all of them: the model of the animal itself, in which communication is one observable channel among several. The inversion is the group’s quiet conclusion: studied long enough, communication stops being the object and becomes a window — and what the window opens onto is behavior, the integrated activity of a whole organism in a world.
Every methodological claim of the article compresses into the stack of Figure 1. Representation decides what behavior is for the model, and its choices are hypotheses, not defaults. Architecture consumes the representation and adds assumptions of its own, about state and sequence and dynamics. Evaluation is where the whole structure meets the truth, and predictive accuracy there is the cheap part — construct validity, leakage discipline, and baseline honesty are what separate instruments from anecdotes. Use, finally, is where the bills arrive: welfare decisions made on unvalidated models are not preliminary science; they are negligence with a dashboard. Clever Hans is a century old, and the examination he failed has not changed — vary the conditions and see what the performance is made of (Pfungst, 1911).
Consistency with the series’ larger frame is the final test, and the article passes it deliberately. The pillar’s theses read, from here, as design constraints: scale is solved for signals and not for behavior; meaning remains unclaimed by any model in this inventory; and the pattern-to-playback loop remains the only validated path from pattern to knowledge — behavior models belong on the pattern side of that loop, sharpening it, proposing the experiments, and sometimes, as the bee robot showed, becoming the playback. Prediction is one evaluation mode of a behavior model, not a proof of understanding. The models are coming regardless; the choice the field has is whether they arrive as instruments with characterized error, or as mirrors with confident faces. The discipline that decides it is the one this article has tried to make routine.
Read more about the future of AI and animal communication research for where the series goes after the multimodal group closes.
Frequently Asked Questions
Q: What exactly is a multimodal model of animal behavior?
A: It is a system that consumes several observation channels at once — sound, posture and movement from video, position, environmental context — and outputs predictions or explanations of organism-level behavior: what the animal is doing, did, or will do. The defining feature is the target variable. A detector outputs labels about a recording; a behavior model outputs claims about an animal. Communication signals, in such a model, are simply one informative input stream among several, which is a deliberate demotion from their usual starring role in this series.
Q: How is this different from the sound-plus-behavior joins covered earlier in the series?
A: The join is the data layer; the behavior model is the science layer. Joining sounds with behavioral data means aligning, timestamping, and identity-matching multiple streams so they can be analyzed together — it produces a table. Building a behavior model means choosing a representation of behavior, an architecture, a ground-truth strategy, and an evaluation regime — it produces a claim. This article assumes the join exists and asks what can responsibly be built on top of it, which is why the hardest sections here are about representation and validity rather than alignment.
Q: Can these models tell us what an animal is feeling?
A: Not directly, and the honest version of the answer is structural rather than temporary. Feelings are constructs without a gold-standard measurement: models can classify proxies — the pig program classifies calls by a valence proxy derived from production context (Briefer et al., 2022) — and those proxies are scientifically useful. But moving from “context-assumed valence” to “what the animal feels” requires convergent evidence from independent measures and interventions, which no classifier trained on the same proxy data can supply. A well-validated valence model is a calibrated instrument for a proxy, and that is a genuinely valuable thing to own.
Q: What does “construct validity” mean in plain language?
A: It means measuring what you claim to measure, as opposed to measuring what is merely easy to measure. Predictive validity asks whether your model is good at its target; construct validity asks whether the target is the thing you named. A bathroom scale has predictive validity for weight and no construct validity for intelligence, no matter how precisely it reads. In behavior modeling, construct failures look like a “stress model” that has actually learned to recognize the recording rig, or an “aggression model” that has learned cage position. The model works; the name is wrong.
Q: If a model accurately predicts an animal’s behavior, doesn’t that prove the animal is communicating?
A: No, and the gap is not a technicality. Predictive fit is equally consistent with at least four mechanisms: the signal communicates intent; a shared internal state causes both signal and behavior; the signal is a symptom rather than a message; or the recording and labeling structure coupled the two without the animal doing anything meaningful. Distinguishing these requires interventions — above all, playback experiments of the kind that earned the vervet alarm call its interpretation (Seyfarth, Cheney, & Marler, 1980). The fit is the hypothesis, not the proof.
Q: What was Clever Hans, and why does AI research still cite a horse from 1911?
A: Clever Hans was a horse who answered arithmetic questions by tapping, with reliability that convinced audiences a mathematical mind was at work. Pfungst’s (1911) systematic investigation found the horse was reading involuntary cues from the questioner — subtle posture changes that appeared as the correct count approached and vanished when the questioner could not see or did not know the answer. The parable survives because the horse’s predictive performance was genuine; only the interpretation was wrong. Every machine-learning system that solves a task through an unintended shortcut is Hans with a loss curve, which is why leakage controls and condition-variation belong in every evaluation.
Q: Can a behavior model trained in a lab be used on a farm or in the wild?
A: Only with revalidation at the boundary, and often not then. Setting shift degrades every channel at once — lighting, acoustics, densities, and the behavior categories themselves drift between environments — and the discipline that keeps the transfer honest is unglamorous: re-measure performance against simple baselines in the new setting, re-check inter-observer agreement with local annotators, run a human verification loop long enough to estimate new error rates, and treat identical performance in a new setting as a warning sign rather than a comfort. The strong version — lab-to-wild portability as a solved problem — is nowhere near true today.
Q: What should a credible behavior-model paper report?
A: At minimum: the representation and why it was chosen, including what it cannot express; inter-observer reliability for every human-labeled category, per category rather than averaged; group-level evaluation splits — held-out animals, sessions, and cohorts — with the random-split number reported alongside for contrast; comparisons against persistence, majority-class, and simple-feature baselines under identical splits; a construct-validity discussion naming what the target operationalizes and what could produce the same fit; and an explicit statement of which claims are correlational. A paper with all of that and modest numbers is worth more than a paper with a record accuracy and none of it.
Glossary
Behavior model
A system whose output is a prediction, description, or explanation of organism-level behavior rather than a property of a recording; communication signals serve as inputs among other observable channels.
Ethogram
A fixed catalog of named behavior categories for a species, with rules for assigning observed behavior to categories; functionally a hypothesis about the natural kinds of that species’ behavior.
Construct validity
The degree to which a model’s target, as operationalized by representation, labels, and learning, corresponds to the behavioral construct it is claimed to measure — as distinct from being merely predictable.
Predictive validity
How well a model predicts its labeled target on unseen data; a property of the model against its dataset, and silent on whether the target means what its name suggests.
Ground truth
The reference labels a model is trained and evaluated against. For behavior, ground truth is a human judgment with measured reliability; for internal states, it exists only as proxies.
Inter-observer reliability
Measured agreement between independent annotators labeling the same data; the error bar on the labels themselves and a hard ceiling on what any model trained on them can honestly claim.
Pose trajectory
A continuous time series of skeletal keypoint positions produced by markerless pose estimation from video, with derivatives such as velocity and joint angles; preserves graded dynamics but carries no categories.
State-space model
A model family that assumes a latent behavioral state emits observable signals probabilistically; interpretable when the state structure is right, confidently wrong when it is misspecified.
Weak supervision
Training labels derived from circumstances rather than direct annotation — session context, schedules, events — cheap and biased: a model learns to recognize context, which is behavior only if the mapping is tight.
Active learning
A model-in-the-loop labeling strategy that selects which examples a human should annotate next, typically where model uncertainty is highest; efficient, with a known risk of datasets shaped by the model’s own confusion.
Data leakage
Any path by which evaluation information reaches a model during training — most commonly the same animal, session, or recording batch appearing in both — manufacturing accuracy that measures recognition rather than behavior.
Shortcut feature
An input cue that predicts the label without carrying the construct — handler identity, cage position, recorder artifacts — the machine-learning descendant of the human cues Clever Hans read (Pfungst, 1911).
Counterfactual query
A what-if question put to a behavior model — what would the animal have done had the signal differed — which by construction leaves the training distribution and returns interpolation at best, confident fabrication at worst.
Over-read
The category error of treating a prediction about observable behavior as a discovery about mental states — “predicts grooming” read as “anticipates grooming” — when common causes and observer structure produce the same fit.
Partial observability
The structural fact that behavior is the observable fraction of an animal’s state; internal states are underdetermined by observables, so models recover statistical shadows of them, not the states themselves.
Ethogram drift
The shift in behavior categories, their boundaries, and their frequencies across settings or over development — the reason a transferred behavior model may be solving a different task under the same label vocabulary.
References and Further Reading
Citations follow an APA-style author–date format. Entries were drawn from the series’ verified citation pool; where a work is documented only as a preprint or an open resource, it is listed as such and should be cited accordingly. Documented dataset and coverage figures (7,414 calls from 411 pigs; 8,719 codas; 984 and 14,597 species) are the artifacts’ documented scopes, not performance claims.
Peer-Reviewed Research and Scholarly Books
- Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big? Proceedings of the ACM Conference on Fairness, Accountability, and Transparency (FAccT 2021), 610–623.
- Briefer, E. F., et al. (2022). Classification of pig calls produced from birth to slaughter according to their emotional valence and context of production. Scientific Reports, 12, 3409.
- Carreira, J., & Zisserman, A. (2017). Quo vadis, action recognition? A new model and the Kinetics dataset. Proceedings of CVPR 2017.
- Coffey, K. R., Marx, R. G., & Neumaier, J. F. (2019). DeepSqueak: A deep learning-based system for detection and analysis of ultrasonic vocalizations. Neuropsychopharmacology, 44, 859–868.
- Dalla Costa, E., Minero, M., Lebelt, D., Stucke, D., Canali, E., & Leach, M. C. (2014). Development of the Horse Grimace Scale (HGS) as a pain assessment tool in horses undergoing routine castration. PLOS ONE, 9.
- Dormagen, D. M., et al. (2023). Machine learning reveals the waggle drift’s role in the honey bee waggle dance. PNAS Nexus, 2(9), pgad275.
- Elie, J. E., et al. (2024). Role of auditory feedback for vocal production learning in the Egyptian fruit bat. Current Biology, 34.
- Farine, D. R., & Whitehead, H. (2015). Constructing, conducting and interpreting animal social network analysis. Journal of Animal Ecology, 84, 1144–1163.
- Ghani, B., Denton, T., Kahl, S., & Klinck, H. (2023). Global birdsong embeddings enable superior transfer learning for bioacoustic classification. Scientific Reports, 13. https://doi.org/10.1038/s41598-023-49989-z
- Gnutti, A., et al. (2026). A deep neural network for automatic prediction of mouse behavior from ultrasonic vocalizations. Scientific Reports.
- Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep learning. MIT Press.
- Hobaiter, C., & Byrne, R. W. (2014). The meanings of chimpanzee gestures. Current Biology, 24, 1596–1601.
- Ivanenko, A., et al. (2020). Classifying sex and strain from mouse ultrasonic vocalizations using deep neural networks. PLOS Computational Biology, 16, e1007918.
- Kahl, S., Wood, C. M., Eibl, M., & Klinke, H. (2021). BirdNET: A deep learning solution for 984 bird species. Ecological Informatics, 61, 101254.
- Kershenbaum, A., Blumstein, D. T., et al. (2016). Animal vocal sequences: Not the Markov chains we thought they were. Proceedings of the Royal Society B, 283, 20152470.
- Kroodsma, D. E. (1989). Suggested experimental designs for song playbacks. Animal Behaviour, 38.
- Mathis, A., et al. (2018). DeepLabCut: Markerless pose estimation of user-defined body parts with deep learning. Nature Neuroscience, 21, 1281–1289.
- Pérez-Escudero, A., et al. (2014). idtracker: Tracking individuals in a group by automatic identification of unmarked animals. Nature Methods, 11, 743–748.
- Pfungst, O. (1911). Clever Hans. Henry Holt.
- Prat, Y., Taub, M., & Yovel, Y. (2015). Vocal learning in a social mammal: Demonstrated by isolation and playback experiments in bats. Science Advances, 1, e1500019.
- Prat, Y., Taub, M., Matosevich, N., & Yovel, Y. (2017). An annotated dataset of Egyptian fruit bat vocalizations across varying contexts and during vocal ontogeny. Scientific Reports, 7.
- Rendall, D., Owren, M. J., & Ryan, M. J. (2009). What do animal signals mean? Animal Behaviour, 78, 233–240.
- Seyfarth, R. M., Cheney, D. L., & Marler, P. (1980). Vervet monkey alarm calls: Semantic communication in a free-ranging primate. Animal Behaviour, 28, 1070–1094.
- Sharma, P., et al. (2024). Contextual and combinatorial structure in sperm whale vocalisations. Nature Communications, 15, 3617. https://doi.org/10.1038/s41467-024-47221-8
- Stowell, D. (2022). Computational bioacoustics with deep learning: A review and roadmap. PeerJ, 10, e13152.
- Tinbergen, N. (1963). On aims and methods of ethology. Zeitschrift für Tierpsychologie, 20.
- von Frisch, K. (1967). The dance language and orientation of bees. Harvard University Press.
- Wario, F., et al. (2017). Automatic detection and decoding of honey bee waggle dances. PLOS ONE, 12, e0188626.
Preprints and Open-Source Releases
- Bommasani, R., et al. (2021). On the opportunities and risks of foundation models. arXiv:2108.07258.
- Ghani, B., et al. (2025). Perch 2.0: The Bittern Lesson for bioacoustics. arXiv:2508.04665. (14,597 species)
- Landgraf, T., et al. (2018). Dancing honey bee robot elicits dance-following and recruits foragers. arXiv:1803.07126.
Documented Reporting and Open Resources
- Earth Species Project. (2025). What the world thinks about AI and animal communication: Findings from our first global survey. (1,057 respondents, 67 countries.)
- Project CETI documentation and publications. projectceti.org
Explore the Full Series
This article is part of SciXa’s AI and Animal Communication pillar guide — a research-grade synthesis of what machine learning has and has not achieved in the study of animal signals, linking 80 cluster articles across the field.
Foundations of Animal Communication
- What Is Animal Communication? A Scientific Introduction
- How Animals Communicate Across Different Species
- Animal Communication vs Human Language: Key Differences
- Do Animal Signals Have Grammar-Like Structure?
- How Scientists Study Animal Communication Today
- How Animal Communication Changes Across Generations
- Why Animal Communication Is Difficult to Decode
- The Role of Context in Understanding Animal Signals
The Big Questions and the Research Field
- Can Artificial Intelligence Understand Animal Communication?
- The Future of AI and Animal Communication Research
- AI and the Evolution of Animal Communication
- From Animal Signals to Machine Learning Models
- What AI Can and Cannot Learn From Animal Sounds
- The Science Behind AI-Based Animal Communication Research
Bioacoustics and Machine Learning Methods
- What Is Bioacoustics and Why Does AI Matter?
- How AI Analyzes Animal Sounds at Scale
- Deep Learning for Animal Vocalization Recognition
- Machine Learning for Animal Sound Classification
- AI for Detecting Rare Animal Vocalizations
- Self-Supervised Learning for Animal Sounds
- Audio Embeddings for Animal Communication Research
- Transformers for Animal Bioacoustic Analysis
- Training AI Models on Unlabeled Animal Sounds
- Zero-Shot Learning for Animal Communication
- Transfer Learning Across Animal Species
- How AI Separates Animal Sounds From Background Noise
- Biodenoising: Cleaning Animal Recordings With AI
Animal Language Processing and Foundation Models
- What Is Animal Language Processing? A New AI Field
- How Animal Language Processing Differs From NLP
- Can AI Discover Hidden Structure in Animal Communication?
- Foundation Models for Animal Communication
- Large Audio Models for Animal Vocalizations
- NatureLM-audio and the Future of Animal AI
- Can One AI Model Understand Many Animal Species?
Multimodal AI and Behavioral Signal Analysis
- Multimodal AI for Understanding Animal Communication
- Combining Animal Sounds and Behavioral Data
- Computer Vision for Animal Communication Research
- AI Recognition of Animal Facial Expressions
- How AI Interprets Animal Body Language
- AI Analysis of Animal Gestures and Movements
- Combining Sound, Motion and Environmental Context
- Building Multimodal Models of Animal Behavior You are here
Social Networks and Individual Identity
- AI Models for Animal Social Interaction
- Tracking Who Communicates With Whom Using AI
- How AI Identifies Individual Animals by Their Voices
- Animal Voiceprints: Can AI Recognize Individual Animals?
- AI for Mapping Animal Communication Networks
- Who Is Talking to Whom? AI and Animal Social Groups
- AI Detection of Animal Dialects Across Populations
- Animal Personalities and Individual Communication Signals
- AI for Studying Social Information in Animal Calls
- Modeling Animal Communication as Social Networks
Species Case Studies
- AI and Whale Communication: Can We Decode Whales?
- Project CETI and the Search for Whale Communication
- AI and Sperm Whale Vocalizations Explained
- Can AI Understand Dolphin Communication?
- AI and Elephant Communication: Names and Social Calls
- AI for Understanding Birdsong and Bird Communication
- AI and Crow Communication: What Can We Learn?
- AI and Primate Communication and Social Signals
- AI for Understanding Bat Vocalizations and Echolocation
- AI and Honey Bee Communication: From Hives to Data
Meaning, Rules and Validation
- Can AI Discover Meaning in Animal Vocalizations?
- How Scientists Test Meaning in Animal Signals
- From Correlation to Causation in Animal Communication AI
- Can AI Discover Animal Communication Rules?
- Can AI Detect Syntax-Like Patterns in Animal Calls?
- Compositionality in Animal Communication and AI
- Can AI Predict Behavior From Communication?
- Testing AI-Discovered Animal Communication Patterns
- Why AI Predictions Are Not Proof of Animal Language
Translation and Two-Way Communication
- Can AI Translate Animal Communication?
- Animal-to-Human Translation: Science or Science Fiction?
- Can Humans Communicate With Animals Using AI?
- AI Systems for Two-Way Animal Communication
- Interactive AI Experiments With Communicating Animals
- Could AI Build a Universal Animal Communication Model?
- What Would Interspecies Communication Change?