27th Edition: 2/16/24
Google's Gemini 1.5 using "Mixture of Experts", Cohere's Aya LLM can handle 100+ languages, and OpenAI's Sora pioneers text-to-video with surprising accuracy....
The Mixture of Experts architecture is quickly gaining attention, after Mistral’s rapid rise to prominence with their highly-accurate 8x7B model using MoE drives competitors like Google to follow suit. Today, I’ll give an elementary-level rundown of how MoE layers work, and how they impact the LLM.
Let’s get started!
Just for fun: Artist Alicia Framis is set to become the first person to marry an AI-generated hologram, as part of her art project 'Hybrid Couple’
PNW AI News:
Microsoft has developed an AI system called UFO (UI-Focused Agent) that could replace traditional Windows interfaces. Powered by OpenAI's image recognition model, UFO runs on your desktop (rather than browser) and navigate within apps to complete tasks for you
MSFT and OpenAI partnering to thwart state-affiliated online attackers that are using ChatGPT and other AI tools for improving the creativity of their attack vectors
MSFT introduced "Automatic Super Resolution" in Windows 11, an AI-assisted feature designed to upscale video and image quality in games
WA State Senator Joe Nguyen, lead sponsor of WA's AI Task Force Bill, has opened up about his use of ChatGPT on structuring arguments + drafting reports
Key AI Product Updates:
OpenAI launches Sora, an LLM that can generate near-perfect videos in a range of styles at 1080p from text or a still image, and can "extend" existing video clips as well
Enabled by a novel spatiotemporal video encoding scheme fed into GPT4's transformer architecture
In limited release to safety experts and select artists
Google unveils Gemini 1.5, a major upgrade to its LLM using "Mixture of Experts" to combine specializations of several smaller models, with a massive context window and can ingest up to 1M tokens (equivalent to hours of video or thousands of lines of code!!!)
Google follows Mistral 8x7B's path of using MoE to improve performance on complex queries:
Mixture of Experts consists of two main elements:
Sparse MoE layers are used instead of dense feed-forward network (FFN) layers like typical general-purpose LLMs. MoE layers have a certain number of “experts” (e.g. 8 for Mistral's 8x7B model), where each expert is a neural network.
A gate network or router, that determines which tokens are sent to which expert.
Cohere releases Aya, an open-source multilingual LLM that can handle 100+ languages, including underrepresented ones like Azerbaijani and Welsh
Enabled by fine-tuning on a unique dataset of prompt/completion pairs across diverse languages
Nvidia's "Chat with RTX" offers a radical way to interact with file bases. The free, LLM-agnostic 35GB download leverages RAG to transform documents, notes, PDFs, and even video transcripts into a highly personalized AI chatbot
Nvidia unveiled Eos, its newest enterprise AI supercomputer (ranked 9th fastest globally), for advanced AI development and scalability
Eos required 4.7K Nvidia H100 GPUs and 1.2K Intel Xeon Platinum CPUs… whopping compute power
Leaked docs have exposed an internal Google project named "Goose," a LLM aimed at boosting staff productivity by training on Google's engineering knowledge corpus
LangChain announced the public GA launch of its LangSmith platform for LLM app development, also announcing a $25M Series A from Sequoia
OpenAI is developing a new web search service with Bing integration, looking to challenge Google's dominance in search
ElevenLabs now allows actors to monetize the creation of high-quality AI replicas of their voice and earn $ each time their voice is used in the Voice Library
Slack integrated new generative AI features into the platform, including enhanced search, channel recaps, thread summaries, and more
Apple introduces Keyframer, an AI-powered tool that generates animation code from static images and natural language descriptions
Key AI Business/Investment Updates:
YC announces its newest "Request for Startups", outlining their themes of focus, including (unsurprisingly) several leveraging AI/ML:
Applying ML to robotics
Using ML to stimulate the physical world
New space companies using safe AI
Dev tools using AI to extend ability of existing internal tools
Explainable AI
LLMs for manual back office processes in legacy enterprises
AI to build enterprise software
Foundation models for biological systems
Small fine-tuned models (SLMs) as alternative to giant generic LLMs
The US Patent and Trademark Office (USPTO) rejected OpenAI's application to trademark "ChatGPT" and "GPT" per Hacker News, citing non-competitive issues with other LLM operators
The University of Michigan is facing major backlash after a 3rd-party vendor of UM advertised datasets of lectures and papers for licensing to AI firms for training LLMs, without student consent
Magic announced a $117M Series B led by former Github CEO Nat Friedman for its frontier-scale AI code models, seeking to build a full stack ‘AI software engineer’
Protesters rallied outside OpenAI’s offices, demanding they end their Pentagon contract and halt all work on AGI, organized by activist groups Pause AI and No AGI
PNW AI/ML Fundings:
Guardrails (Seattle, WA/San Francisco, CA) raised $7.5M in Seed funding led by Zetta Venture Partners, joined by Github Fund, Pear VC, Factory, Bloomberg Beta, and more
OS platform lets developers build and reuse validation techniques for their AI models, to secure and prevent unintended outputs when working with LLMs - TFTD $7.5M
CEO: Shreya Rajpal
Planette (Seattle, WA) raised $2.4M in Seed funding led by Audacious Ventures, joined by Jetstream, Dash Fund, and Graham & Walker
AI-boosted technology is designed to help businesses plan for weather and climate risks that could impact resources and operations - TFTD $2.4M
CEO: Hansi Singh
SmartApps (Sammamish, WA) raised $820K in Pre-Seed funding from several undisclosed angels
AI to automatically create data models and reduce complex data engineering practices, enabling businesses to be data driven by default - TFTD $820K
CEO: Alekh Jindal
OraQ (Calgary, AB) raised $1.2M in Seed funding from undisclosed investors
A dental AI/ML platform that is redefining how dental professionals engage, diagnose, and treat patients on a holistic level - TFTD
CEO: Amreesh Khanna