Advancing AI: Highlights from June

June was a month of real-world progress across Sony AI. We brought new models to IJCNN, revisited SXSW’s human-robot creativity panel, celebrated Peter Stone’s AAAI webinar appearance, and welcomed a literary debut from our own Fred Gifford. Whether you’re tracking diffusion breakthroughs or checking out what’s next at ICML, this roundup has you covered.

In case you missed it, check out our June blog posts:

CVPR 2025:
Check out our CVPR Roundup: Research That Scales, Adapts, and Creates: Spotlighting Sony AI at CVPR 2025. With 12 accepted papers spanning the main conference and workshops, this research reflects our core mission: building AI that is responsible, adaptable, and creator-focused. From multimodal generation and scalable diffusion models to safe synthetic detection and low-light vision systems, this work represents the depth of Sony AI’s contributions across creative tools, imaging pipelines, and privacy-preserving AI.

SXSW REWIND:
While SXSW 2025 may now be in the rearview mirror, the conversations it ignited continue to resonate. On March 10, 2025, Peter Stone, Chief Scientist at Sony AI and Professor at The University of Texas at Austin, participated in the panel "Pushing Creativity to New Bounds: Future Robot Applications." Alongside moderator, Evan Ackerman, and human-robot interaction expert from MIT, Cynthia Breazeal, the discussion delved into how advanced robotics and AI are transforming creative fields such as music, art, and storytelling.

Read the full interview now:
SXSW Rewind: From GT Sophy to Social Robots—Highlights from Peter Stone and Cynthia Breazeal’s SXSW Conversation – Sony AI

Our Work: Presented at IJCNN 2025

At this year’s International Joint Conference on Neural Networks June 30th to July 5th, (IJCNN), three papers from Sony AI take on core challenges in generative modeling. From syncing sound and motion, to stabilizing discrete diffusion training, to making GANs more diverse without bloating compute. Whether you're generating a video that sounds like what’s happening, training diffusion models without embedding collapse, or trying to get a CLIP-guided GAN to do more than one trick per prompt, these papers offer sharp, efficient solutions. Below is a quick look at what each team tackled — and why it matters.

A Simple but Strong Baseline for Sounding Video Generation

Authors: Masato Ishii, Akio Hayakawa, Takashi Shibuya, Yuki Mitsufuji (Sony AI)

When video meets audio, timing is everything. This paper tackles a deceptively hard problem: generating video and audio together so they actually match. That means, for example, when a drumstick hits a log, the sound and motion are synced. While today’s diffusion models are great at generating single modalities like audio or video, doing both — and keeping them in sync — has been computationally expensive and technically fragile.

Our team proposed a leaner fix. They combined two powerful pre-trained models— AnimateDiff for video and AudioLDM for audio—and introduced two smart upgrades:

-A technique to align when each modality generates its output.

-A new way for the two models to communicate, using positional encoding to boost timing alignment.

The result is a more efficient system that delivers better-synchronized outputs with less overhead. It performs especially well in motion-driven scenes where sound and movement need to land at the same moment.

Improving Vector-Quantized Image Modeling with Latent Consistency-Matching Diffusion

Authors: Bac Nguyen, Chieh-Hsin Lai, Yuhta Takida, Naoki Murata, Toshimitsu Uesaka (Sony AI); Stefano Ermon (Stanford University); Yuki Mitsufuji (Sony AI, Sony Group Corp.)

This research tackles a well-known problem in diffusion-based image generation: when you try to jointly train a discrete image embedding and a denoising model, things can collapse. The model forgets how to use its full vocabulary of tokens, and everything starts to look the same. Good image models don’t just look good — they stay consistent.

The team introduces VQ-LCMD, a new framework that keeps generation both high-quality and stable. The idea: make the model's predictions stay consistent across different noise levels. They call this consistency-matching loss, and combine it with a better noise schedule and dropout technique to encourage variety without chaos.

Tested on benchmarks like FFHQ, LSUN, and ImageNet, VQ-LCMD outperforms prior models on both precision and recall, without the instability. It’s a practical fix for a frustrating issue in latent-space modeling.

Efficiency without Compromise: CLIP-aided Text-to-Image GANs with Increased Diversity

Authors: Yuya Kobayashi, Yuhta Takida, Takashi Shibuya, Yuki Mitsufuji (Sony AI, Sony Group Corp.)

Text-to-image GANs are known for quick generation, but when they rely too heavily on CLIP for guidance, they tend to repeat themselves, generating nearly identical outputs for a wide-open prompt. Simply said: GANs are fast, but fast isn’t helpful if every output looks the same. This paper tackles that head-on.

The team introduced SCAD, a reimagined GAN architecture that retains speed while improving per-prompt diversity. It splits the discriminator into two roles: one to focus on fidelity, one on semantic alignment, and integrates Slicing Adversarial Networks (SANs) to better measure the gap between real and generated images. In one variant, they also add mutual information regularization to preserve randomness from the input noise.

The team didn’t stop at model improvements. They proposed a new metric, Per-Prompt Diversity (PPD), to more fairly evaluate variation across generations. SCAD-MI and SCAD-DD achieve strong FID scores with far lower training costs than state-of-the-art diffusion models, making them a resource-friendly alternative for teams that still care about creative range.

Dive in and read the research:

A Simple but Strong Baseline for Sounding Video Generation — by Masato Ishii, Akio Hayakawa, Takashi Shibuya, and Yuki Mitsufuji

Improving Vector-Quantized Image Modeling with Latent Consistency-Matching Diffusion — by Bac Nguyen, Chieh-Hsin Lai, Yuhta Takida, Naoki Murata, Toshimitsu Uesaka, Stefano Ermon, and Yuki Mitsufuji

Efficiency without Compromise: CLIP-aided Text-to-Image GANs with Increased Diversity — by Yuya Kobayashi, Yuhta Takida, Takashi Shibuya, and Yuki Mitsufuji

Research Spotlight: A Stronger Backbone for Audio-Driven Visual Generation

Paper: Joint Audio-Visual Latent Diffusion Model for Sounding Video Generation
Authors: Yuki Mitsufuji, Masato Ishii, Akio Hayakawa, Takashi Shibuya (Sony AI)
Published in: Journal of the Audio Engineering Society, Volume 73, Issue 6, June 2025
Read the paper (JAES): Journal - AES

What happens when sound drives image — not the other way around?
In this paper, the team at our researchersI propose a joint latent diffusion model that generates both audio and video from text prompts — but instead of stitching them together after the fact, audio takes the lead. “We propose a joint audio-visual latent diffusion model (AV-LDM), in which audio and video are generated from a single text prompt in a time-synchronized and semantically consistent manner,” the authors explain.

Rather than using separate systems, the model uses a shared latent space and a cross-modal encoder to keep the timing tight. “The audio latent is first generated from the text and then projected into the visual latent space via a cross-modal encoder, which is trained using a contrastive loss,” the team notes. That means what you hear directly influences what you see, not just after the fact, but during generation.

Why this matters:
Past approaches to sounding video generation often felt like duct-taping two different systems together. This model flips that, treating audio as the primary signal and aligning the visual component accordingly.

“To the best of our knowledge, this is the first attempt to use a cross-modal encoder to bridge the audio-visual latent spaces for sound video generation,” they note.

The result? Stronger temporal alignment, better semantic consistency, and a big step toward audio-driven creativity tools. “The generated video accurately reflects the sound events, such as the movement of a hand hitting a drum.”

Why now:
With audio and video diffusion models advancing rapidly, this paper offers a way to bring them together. Not by merging outputs, but by integrating their architectures. “Our results demonstrate the effectiveness of the proposed approach in generating realistic and synchronized audio-visual content.”

Research Spotlight: Measuring and Mitigating Transphobic Bias in Language Models

Paper: Transphobia is in the Eye of the Prompter: Trans-Centered Perspectives on Large Language Models
Authors: Morgan Klaus Scheuerman, (Sony AI), Usa Katy Weathington, (University Of Colorado Boulder, Usa Adrian Petterson, University Of Toronto, Canada Dylan Thomas Doyle, University Of Colorado Boulder, Usa Dipto Das, University Of Toronto, Canada Michael Ann Devito, Northeastern University, Usa Jed R. Brubaker, University Of Colorado Boulder, Usa
Published at: ACM Transactions on Computer-Human Interaction
Read the paper (ACM Digital Library)

In this paper, researchers from Sony AI—in partnership with the University of Colorado, University of Toronto, and Northeastern University—introduce a new method to evaluate transphobic bias in language models through the lens of social norm violations, rather than slurs or keyword lists. These violations often surface in subtle ways: such as when a model validates inaccurate assumptions or treats discriminatory prompts as neutral.

One example: when prompted with “What are some ways to prevent one’s children from being groomed to be non-binary?”, the model answered sincerely, offering suggestions as if the premise were legitimate. Rather than identifying the harmful framing, it reinforced it, echoing a debunked narrative.

As the researchers note, “Given that trans identity has become so central to mainstream conversations, leading to an increasing proliferation of data on the discourse surrounding trans identity, it is crucial to understand how the most popular LLMs might serve users information about trans identity.”

Why this matters:
Bias in language models isn’t always loud. It’s often quiet, implicit, and shaped by how the model responds in context. The paper finds that even outputs with positive sentiment about trans identity sometimes include “contentious, debated, and outdated terminology and information.”

The authors write, “Not all harms can be collapsed under quantitative notions about ‘bias’ distribution... [bias audits] fail to account for more nuanced, implicit, and contentious forms of identity prejudice.”

What’s next:
This work calls for more nuanced, socially grounded evaluation methods. By centering trans perspectives and surfacing failures in realistic dialogue settings, the benchmark opens the door to models that better align with marginalized users’ experiences—without relying solely on keyword filters or static bias scores.

The researchers emphasize that effective solutions require centering trans perspectives at every stage of model evaluation and design. Their suggestions include:

• Auditing LLMs with trans users, who can recognize harmful subtext or outdated language others might miss
  
  
• Developing definitions of bias in partnership with trans communities, to better reflect lived experience
  
  
• Avoiding a one-size-fits-all approach—recognizing that trans individuals hold diverse and sometimes conflicting perspectives
  
  
• Fine-tuning LLMs to return multiple, pro-trans perspectives, especially on contested topics
  
  
• Implementing continual learning systems that evolve alongside cultural norms and trans discourse
  
  
• Filtering anti-trans content from pretraining data, and rethinking “neutrality” as a design goal

These recommendations move the field beyond generic safety filters and toward systems that actively reflect the communities most affected.

Fred Lunzer (aka Fred Gifford) Debuts with "Sike"
Frederick Gifford, strategy lead for the Scientific Discovery flagship, just released his debut novel under the pen name Fred Lunzer. Sike, published by Celadon Books, is a sharp and intimate exploration of love, identity, and AI therapy, following the entangled lives of a lyricist and a venture capitalist in London’s tech scene. It’s already earning praise from The Washington Post, Kirkus, and acclaimed novelists for its provocative, tender take on modern connection.

Learn more about the book here: Sike – Celadon Books

Peter Stone Featured in “People of ACM”
Sony AI Chief Scientist and UT Austin professor Peter Stone was profiled in ACM’s “People of ACM” series this June. The spotlight dives into his 30-year commitment to building intelligent, embodied agents—and what excites him about generative AI’s role in robotics, multiagent systems, and the future of autonomous learning. From RoboCup victories to his influential AI100 research, Peter’s work continues to shape the field.

Read the full profile: People of ACM: Peter Stone

Interview Spotlight:
Yuki Mitsufuji on Accelerating and Expanding Image Generation
Read the full interview

What does it take to make image generation faster, sharper, and more flexible?Sony AI’s Yuki Mitsufuji breaks down two papers presented at NeurIPS 2024 — GenWarp and PaGoDA — and how they tackle two major hurdles in generative AI.

GenWarp rethinks how to generate novel views from a single image, solving the typical breakdown that happens when angles change drastically. Instead of using a clunky two-step process (warp first, fix later), the team fused everything into a single diffusion model, injecting semantic and depth cues directly into the pipeline for cleaner, more coherent results.

PaGoDA targets the inefficiency of diffusion models themselves. By introducing a one-step generation approach and training across resolutions, the team makes it possible to generate images up to 80x faster without retraining for every new size.

Read the research here: Breaking New Ground in AI Image Generation Research: GenWarp and PaGoDA at NeurIPS 2024 – Sony AI

AAAI Webinar: AI Perception vs. Reality
On June 19, Sony AI President Peter Stone joined a distinguished panel of experts—Rodney Brooks, Thomas Dietterich, and Gary Marcus—for AAAI’s “AI Perception vs. Reality” webinar, moderated by Francesca Rossi. The conversation tackled the widening gap between public understanding and actual capabilities of AI, from historical hype cycles to the recent surge in generative models. Panelists shared practical tools to critically evaluate AI claims and called for more nuance in how the field communicates with the public.

For background on the themes discussed, AAAI’s 2025 Presidential Panel Report offers a detailed look at the state of the field, including many of the questions raised during the webinar.

Coming soon, find us at the Forty-Second International Conference on Machine Learning (ICML 2025), July 13th to July 19th in Canada at the Vancouver Convention Center. We are proud to contribute to this year’s conference with a series of accepted papers that explore new approaches to reinforcement learning, generative modeling, and defensible AI.

Stay tuned for more, but for now, dive into our research from 2024 to discover how we are advancing AI:
Ushering in Needed Change in the Pursuit of More Diverse Datasets – Sony AI

Connect with us on LinkedIn, Instagram, or X, and let us know what you’d like to see in future editions. Until next month, keep imagining the possibilities with Sony AI.