About
I build World Action Models that adapt generative video models into robot manipulation policies, adding 3D, tactile, and force sensing so they reason about geometry and contact, stay robust under occlusion, and react to contact events in fine-grained manipulation.
At RoboForce I lead physical foundation AI modeling for WAM and VLA, end to end: cross-embodiment multimodal mid-training, tactile and force sensing, open-loop, closed-loop, and simulation-based evaluation, test-time improvement, failure recovery, and real-robot deployment. Before that I was the post-training tech lead at Amazon AGI for video generation and multimodal reasoning, and I shipped on-device vision-language models on the Astro home robot.
Research focus
Highlighted streams add touch and force to what cameras see.
WAM backbone mid-training
Adapting pretrained video diffusion backbones into World Action Models on multimodal robot data, robust to missing or noisy sensors, with temporal memory and VLM-grounded visual hints for long-horizon, fine-grained tasks.
Tactile & force sensing
Training on tactile and force data alongside vision, proprioception, and 3D, so policies sense contact, slip, and force that cameras miss, time grasps and insertions, keep working when the gripper blocks the view, and catch execution errors early.
Evaluation & test-time improvement
Open-loop, closed-loop, and simulation-based evaluation for VLA and WAM, plus an OmniJev-style test-time improvement harness that raised real-robot manipulation success and added failure recovery without retraining: a VLM judge triggers retries and replanning, and review agents turn failed rollouts into targeted fixes.
Failure recovery
Recovery data collection, curation, and training so policies notice execution errors and perturbations and get back on track.
Experience
Member of Technical Staff · Physical Foundation AI Modeling Lead (WAM & VLA)
- Lead WAM and VLA modeling from data and mid-training to evaluation and real-robot deployment: set technical direction, drive integration on in-house robots, and mentor the team.
- Designed mid- and post-training recipes, including cross-embodiment multimodal mid-training, that adapt pretrained diffusion backbones (Cosmos3, Wan2.2, FLUX) into World Action Models that stay robust to missing or noisy sensors, trained on multi-view RGB, 3D geometry, tactile, force, proprioception, and language.
- Built open-loop replay, simulation-based, and real-robot closed-loop evaluation, plus an OmniJev-style test-time improvement harness (VLM judge and review agents) that raised real-robot manipulation success and added failure recovery without retraining.
- Optimized WAM training and inference for real-robot deployment (kernel fusion, quantization, FSDP, KV caching, compilation).
- Built multimodal data pipelines for filtering, cleaning, and curation (Qwen3-VL visual-quality review, video–action consistency checks, metric-depth labeling), and led recovery data collection and recovery-aware fine-tuning.
Senior Applied Scientist, Post-training Tech Lead
- Tech lead for multimodal post-training (video generation and multimodal reasoning): owned recipes, data strategy, and evaluation; hands-on in RL and DPO.
- Optimized GRPO-based RL recipes with tailored reward verifiers and reward models for multimodal reasoning in thinking and non-thinking modes, improving accuracy and robustness on multimodal benchmarks; built a unified text + multimodal RL recipe across verticals (STEM, document understanding).
- Built a DPO + SFT recipe for video generation (T2V & I2V) that shipped in product, improving temporal consistency, motion quality, text–video alignment, and visual quality.
- Led curation of large-scale human-preference data, owned the I2V post-training data strategy, and built the video-generation evaluation system (automated metrics and human evaluation).
Applied Scientist
- Built and compressed vision-language models for the Astro home robot (2.4 GB to 30.8 MB, 176 ms on device).
- Video generation data pipelines and sampling; human-feedback-tuned text-to-image for Fire TV.
Research Scientist
- Production language and speech models: BERT-based intent recognition for a voice-bot platform and grapheme-to-phoneme modeling for text-to-speech.
Selected publications
- ICLR 2025T2V-Turbo-v2: Enhancing Video Generation Model Post-Training through Data, Reward, and Conditional Guidance DesignJ. Li, Q. Long, J. Zheng, X. Gao, R. Piramuthu, W. Chen, W.Y. Wang
- TMLR 2025FlexEControl: Flexible and Efficient Multimodal Control for Text-to-Image GenerationX. He, J. Zheng, J.Z. Fang, R. Piramuthu, M. Bansal, V. Ordonez, G.A. Sigurdsson, N. Peng, X.E. Wang
- ACM MM 2024Zero-Shot Controllable Image-to-Video Animation via Motion DecompositionS. Yu, J.Z. Fang, J. Zheng, G. Sigurdsson, V. Ordonez, R. Piramuthu, M. Bansal
- Tech report 2024The Amazon Nova Family of Models: Technical Report and Model CardAmazon AGI
- WACV 2024Text-to-Image Editing by Image Information RemovalZ. Zhang, J. Zheng, Z. Fang, B.A. Plummer
- WACV-W 2024E-ViLM: Efficient Video-Language Model via Masked Video Modeling with Semantic Vector-Quantized TokenizerZ. Fang, J. Zheng, V. Sharma, R. Piramuthu
- ECCV 2022Personalized Education: Blind Knowledge DistillationX. Deng, J. Zheng, Z. Zhang
- ICCV 2019Temporal Attentive Alignment for Large-Scale Video Domain AdaptationM.H. Chen, Z. Kira, G. AlRegib, J. Yoo, R. Chen, J. Zheng
- arXiv 2025ProxT2I: Efficient Reward-Guided Text-to-Image Generation via Proximal DiffusionZ. Fang, J. Zheng, Q. Gao, X. Gao, J. Sulam