OpenAI, the research company behind the popular language model ChatGPT, has unveiled its latest creation: Sora, an AI that can generate realistic and imaginative videos from text prompts. Sora is a diffusion-based model that can create videos up to a minute long, with high visual quality and fidelity to the user’s instructions. Sora’s videos have amazed and shocked many creatives, who see both the potential and the challenges of this new technology.
How Sora works
Sora is based on the same transformer architecture as ChatGPT, but instead of generating text, it generates images and videos. Sora works by compressing videos into a lower-dimensional latent space, and then decomposing them into spacetime patches. Patches are like tokens for visual data, and they allow Sora to unify videos of different durations, resolutions, and aspect ratios. When a user sends a text prompt to Sora, it creates a video by stitching together compressed patches of visual data.
Sora uses a diffusion process to generate high-quality images and videos. Diffusion models are named after the physical process of molecules moving from high-concentration to low-concentration zones. In machine learning, these models generate new data by reversing the diffusion process. They add noise to the data and then filter out the noise to recover the original data. Sora uses this technique to create videos that are crisp, clear, and photorealistic.
What Sora can do
Sora can create videos from any text prompt, as long as it is coherent and descriptive. Sora can create videos from images, and also fill in gaps in existing videos. Sora can create videos of various genres, such as animation, documentary, fantasy, horror, and comedy. Sora can create videos of various scenes, such as nature, city, space, and history. Sora can create videos of various characters, such as animals, humans, aliens, and monsters.
Some examples of Sora’s videos are:
- A stylish woman walks down a Tokyo street filled with warm glowing neon and animated city signage. She wears a black leather jacket, a long red dress, and black boots, and carries a black purse. She wears sunglasses and red lipstick. She walks confidently and casually. The street is damp and reflective, creating a mirror effect of the colorful lights. Many pedestrians walk about.
- Several giant wooly mammoths approach treading through a snowy meadow, their long wooly fur lightly blows in the wind as they walk, snow covered trees and dramatic snow capped mountains in the distance, mid afternoon light with wispy clouds and a sun high in the distance creates a warm glow, the low camera view is stunning capturing the large furry mammal with beautiful photography, depth of field.
- A movie trailer featuring the adventures of the 30 year old space man wearing a red wool knitted motorcycle helmet, blue sky, salt desert, cinematic style, shot on 35mm film, vivid colors.
- Drone view of waves crashing against the rugged cliffs along Big Sur’s garay point beach. The crashing blue waters create white-tipped waves, while the golden light of the setting sun illuminates the rocky shore. A small island with a lighthouse sits in the distance, and green shrubbery covers the cliff’s edge. The steep drop from the road down to the beach is a dramatic feat, with the cliff’s edges jutting out over the sea. This is a view that captures the raw beauty of the coast and the rugged landscape of the Pacific Coast Highway.
- Animated scene features a close-up of a short fluffy monster kneeling beside a melting red candle. The art style is 3D and realistic, with a focus on lighting and texture. The mood of the painting is one of wonder and curiosity, as the monster gazes at the flame with wide eyes and open mouth. Its pose and expression convey a sense of innocence and playfulness, as if it is exploring the world around it for the first time. The use of warm colors and dramatic lighting further enhances the cozy atmosphere of the image.
- A gorgeously rendered papercraft world of a coral reef, rife with colorful fish and sea creatures.
How Sora is different from other video generators
Sora is not the first AI model that can create videos from text, but it is the most advanced and impressive one so far. Other video generators, such as Meta’s Make-a-Video and Google’s Lumiere, have limitations in terms of quality, realism, and creativity. Meta’s Make-a-Video uses pre-recorded clips and stock footage to create videos, which results in low resolution, poor lighting, and repetitive scenes. Google’s Lumiere uses a generative adversarial network (GAN) to create videos, which results in blurry, distorted, and unrealistic images. Sora, on the other hand, uses a diffusion model to create videos, which results in high resolution, natural lighting, and original scenes.
Sora’s videos are so realistic and convincing that they can fool the human eye and challenge the human imagination. Sora can create videos that look like they were shot by professional filmmakers, with cinematic effects and artistic styles. Sora can also create videos that look like they were drawn by talented animators, with expressive characters and vibrant colors. Sora can even create videos that look like they were made by magic, with fantastical creatures and surreal landscapes.
What Sora means for the creative industry
Sora’s arrival has sparked mixed reactions from the creative industry, which sees both the potential and the challenges of this new technology. On one hand, Sora can be a powerful tool for creative expression, education, entertainment, and communication. Sora can help artists, filmmakers, storytellers, educators, and journalists to create stunning videos with ease and efficiency. Sora can also help audiences, students, and consumers to enjoy and learn from immersive and engaging videos.
On the other hand, Sora can also be a source of ethical, legal, and social issues, such as plagiarism, misinformation, manipulation, and deception. Sora can create videos that are indistinguishable from reality, which can pose problems for authenticity, credibility, and accountability. Sora can also create videos that are harmful, offensive, or inappropriate, which can pose problems for morality, privacy, and security. Sora can even create videos that are beyond human comprehension, which can pose problems for creativity, originality, and identity.
Sora is a breakthrough in AI and video generation, but it is not ready for mass adoption. OpenAI has not released Sora to the public, and has only shared it with third-party safety testers. OpenAI is concerned about the potential misuses and abuses of Sora, and wants to ensure that it is used responsibly and ethically. OpenAI is also working on improving Sora’s capabilities and limitations, such as increasing its video length, diversity, and coherence.
Sora is a glimpse into the future of AI and video creation, but it is also a reminder of the need for human oversight and guidance. Sora is a testament to the power and beauty of AI, but it is also a challenge to the role and value of human creativity.
