Xinxu Cloud Brain AI is designed to understand emotional cues in text, speech, and drawings that users voluntarily provide, enabling more empathetic human-computer interaction.
The challenge is not feeding three inputs into one large model. Each modality contains different evidence; emotions can coexist and change with context; and a model output can only express a hypothesis from available evidence. It cannot replace self-report or become a psychological diagnosis.
The system therefore follows three principles: preserve modality-specific evidence, model uncertainty explicitly, and leave final interpretation with the user.
Define the task before choosing a model
Happy, sad, angry, fearful, surprised, disgusted, and neutral can remain useful presentation labels, but a single seven-way Softmax is too restrictive for training. Expressions can contain sadness and anger together, while “neutral” may simply mean insufficient evidence.
A useful output combines multi-label probabilities, continuous valence and arousal, calibrated uncertainty, modality-specific evidence, and a policy decision:
{
"emotions": [
{"label": "sadness", "probability": 0.68},
{"label": "anxiety", "probability": 0.31}
],
"valence": -0.62,
"arousal": 0.48,
"uncertainty": 0.27,
"decision": "ask_for_confirmation"
}
Intensity must come from a trained and calibrated regression head, not an arbitrary number generated by a language model.
Preserve three kinds of evidence
The text encoder handles semantics, negation, irony, context, and conversation history. ASR transcripts enter this branch with timestamps, confidence, and language metadata, clearly separated from user-written text.
Speech must not be reduced to a transcript. Pitch, energy, rate, pauses, voice quality, and rhythm carry information that words discard. A speech encoder should process waveform or acoustic representations alongside ASR semantics. Audio normalization, voice activity detection, segmentation, and quality scoring belong in preprocessing; low-quality audio should reduce modality weight.
Drawings are highly personal. Color, line, and composition can provide context, but there is no universal mapping such as black meaning sadness. The vision branch models content and structure. Stroke trajectory and drawing duration should be used only with explicit consent. Augmentations must preserve task semantics: strong color jitter and cropping can change or delete the very cue being modeled. Synthetic images may support pretraining or rare-pattern augmentation, but must be provenance-tagged and excluded from validation and test sets.
Why one VLM is not the entire architecture
A vision-language model can jointly understand images and text, but an ordinary VLM does not consume raw audio and may not learn emotion boundaries for the target population and culture. Converting speech only to text discards acoustic evidence.
Xinxu uses modality-specific text, speech, and vision encoders. Adapters map their outputs to a shared space, followed by gated fusion or a cross-modal transformer. A VLM can act as a visual-semantic teacher, weak labeler, or explanation generator; supervised heads own classification, dimensional regression, and uncertainty.
This design also works when a user supplies only one or two modalities.
Missing modalities are the default
Users may type only text, decline audio, upload a damaged image, or encounter an ASR failure. Train and evaluate every available modality combination.
Use modality dropout during training, pass presence and quality signals into fusion, and retain an auxiliary loss for every unimodal branch. At serving time:
Availability → Unimodal encoding → Quality estimation → Dynamic fusion
→ Multi-task prediction → Calibration → Policy decision
Report full-modality, unimodal, and missing-modality performance separately. A single Macro-F1 score with every channel present hides operational risk.
Data: self-report before observer inference
Emotion labels are subjective. Majority votes from external annotators describe how observers interpret an expression, not necessarily how the person feels.
Store self-reported labels and valence/arousal as primary supervision, observer-label distributions as soft labels, context, modality quality, consent scope, and provenance. Split train, validation, and test by user or speaker. Adjacent segments from one person, conversation, or source video must not cross splits.
Public datasets are useful for pretraining and baselines, but acted or scripted MELD and IEMOCAP samples differ from real Chinese users. The final test set must represent the target population, devices, languages, and missing-modality distribution with appropriate authorization.
Training in evidence-driven stages
- Establish text, speech, and image unimodal baselines.
- Freeze most encoders and train adapters, fusion, and multi-task heads.
- Add cross-modal attention, modality dropout, soft labels, and imbalance-aware loss.
- Apply LoRA/QLoRA or staged unfreezing only when evaluation justifies it.
LoRA rank, target layers, learning rate, and quantization are experiment variables, not fixed recipes. Small data can easily damage foundation capabilities if every encoder is tuned indiscriminately.
L = λ1 · MultiLabelEmotionLoss
+ λ2 · ValenceArousalRegressionLoss
+ λ3 · UnimodalAuxiliaryLoss
+ λ4 · CrossModalConsistencyLoss
+ λ5 · CalibrationLoss
Track dataset and split versions, seeds, preprocessing, base models, code commits, hyperparameters, and hardware. The registry should contain weights, thresholds, calibrators, and a Model Card—not just a merged checkpoint.
Evaluation beyond accuracy
Report Macro-F1, per-class precision and recall, confusion matrices, and multi-label metrics; MAE and CCC for valence/arousal; and ECE, Brier score, and selective-risk curves for confidence.
Evaluate modality ablations, damaged and missing inputs, demographic and device slices, cross-dataset/OOD transfer, calibration and abstention, and safety behavior. A headline accuracy without split details, confidence intervals, class distribution, and independent repetition is not meaningful.
Serving needs policy before and after the model
Consent and purpose check
→ File safety, type, and size limits
→ Modality quality and preprocessing
→ Parallel encoders, fusion, and multi-task prediction
→ Calibration, thresholds, and OOD detection
→ Safety policy and response generation
→ User confirmation or correction
Validate output against a strict schema. Parse failure must return an error or explicit degraded result, never silently default to neutral. Apply deadlines, batching limits, GPU admission control, and circuit breakers, and observe ASR, encoder, fusion, and generation latency separately.
LoRA adapters do not always need merging. Merging fits a single low-overhead deployment; separate adapters fit multiple variants and fast rollback. Verify that the chosen inference engine actually supports the target multimodal architecture and adapter configuration.
Safety, privacy, and product boundaries
Voice and images can contain biometric and sensitive information. Minimize collection, encrypt transit and storage, shorten retention, support deletion, isolate purposes, and audit access. Low-confidence samples must never enter a training pool without separate, explicit training consent.
Use language such as “emotional cues” or “model hypothesis,” not claims of reading inner states. Do not use the system for employment, performance, education, insurance, credit, or law-enforcement decisions, and do not present it as a mental-health diagnostic tool. The EU AI Act restricts emotion inference in workplaces and educational institutions and highlights limitations in reliability, specificity, and generalization.
Self-harm or immediate-danger language requires a separately designed safety path, not a conclusion from an emotion label: supportive language, encouragement to contact local emergency services or a trusted person, and human escalation where lawful and consented.
Monitor model behavior, not people's emotions
Monitor modality availability and quality, prediction entropy, calibration error, abstention, user correction, slice performance, drift, tail latency, and resource saturation. Fixed targets such as neutral under 40% or average confidence above 0.75 are not universal health indicators. Confidence can be wrong, and real population distributions change.
Retraining should follow evidence: degradation on a stable target set, widening group gaps, new devices or languages, or a changed label definition. Release through offline gates, shadow traffic, and a small canary with rapid rollback.
The architectural conclusion
Xinxu Cloud Brain AI is not merely Qwen plus Whisper plus a classifier. It is a multimodal learning system centered on evidence quality, consent, and uncertainty.
Text says what was expressed; speech contributes how it was expressed; drawings add personal visual context. Fusion combines evidence without forcing certainty. A reliable system lowers confidence when signals conflict, degrades safely when modalities are missing, respects user corrections, and never confuses an affective cue with a diagnosis.