[AINews] $1150m for SSI, Sakana, You.com + Claude 500m context • ButtondownTwitterTwitter

The AI Twitter Recap section provides a roundup of key trends in AI research and development. It covers topics such as MoE Models, Challenges with AI Alignment, and Emerging AI Projects. Additionally, it discusses Innovative Tools and APIs for AI Development, highlighting Command and Control in AI, RAG Systems, and GitHub Integration in AI. The section also explores Sectoral Impacts of AI Deployment, focusing on Healthcare Innovations, Educational Outreach, and Geopolitical Dimensions. Lastly, it delves into Humor and Memes in AI Discussion, featuring Coding Lamentations.

/r/LocalLlama Recap

• Theme 1. Benchmarking New AI Models Against Previous Generations: OLMoE, a new open-source language model using sparse Mixture-of-Experts, outperforms models with similar active parameters, sparking discussions on MoE training speed advantages and performance potential as a local assistant.
• Theme 2. Claude-Dev Extension Adds Support for Local LLMs: Claude-Dev version 1.5.19 introduces support for local Language Models through Ollama and OpenAI-compatible servers, receiving positive feedback for compatibility and advanced capabilities.

Other AI Subreddit Recap

• AI and Autonomous Systems: An experiment featuring over 1000 autonomous AI agents in Minecraft and Tesla's improved Smart Summon feature highlight advancements in autonomous systems.
• AI Image Generation and Processing: Demonstrations of real-time AI-powered portrait generation, improved text encoder for Stable Diffusion, and advancements in AI applications like GPT-NEXT announcement and AI vs. Robot speculation.
• AI Development and Future Predictions: A focus on GPT-NEXT teasing for 2024, infographic emphasizing long-term AI progress, and humor with GPT-Hype Meme and AI vs. Robot Speculation meme.

AI Discord Recap

• Llama 3 Models Make a Splash: Meta's Llama 3 family's launch, including Llama 3.1-405B-instruct, attracts users for its advanced capabilities and competitive pricing.
• Optimization Techniques for LLMs: Discussions on low-rank approximations and dynamic expert routing to enhance training efficiency and model flexibility.
• Open Source AI Developments: Tinygrad's affordable cloud service launch and Re-LAION 5B dataset release address user needs for cost-effective AI operations and ethical datasets.
• AI Applications and Industry Impact: GameNGen's DOOM simulation, Meta AI assistant's rapid growth, and community discussions on improving AI project performance and user engagement in AI models.

In this section, discussions cover a range of AI topics from concerns about GPU performance to advancements in AI models and platforms. Members engage in conversations about the evolving capabilities of AI models like Hermes 3 and low-rank approximations' impact on distributed training. Additionally, the section highlights innovative approaches such as the Word Game Bench for evaluating LLMs and the GameNGen neural model's impressive real-time simulation of DOOM. Furthermore, discussions touch on challenges and improvements in AI platforms like LM Studio, Discord servers like Perplexity AI, and the growing impact of AI assistants like Meta's on the market. Participants also delve into the performance of programming languages like Go, engagement levels in collaborations with platforms like OPENSEA, and challenges faced by technology projects such as Mojo in the web3 space.

A member shared challenges in generating suitable images for novel covers, seeking ways to achieve a more comic book or cartoon style. Although efforts with DALL-E were made, they received heavily generated AI pictures instead, illustrating difficulties in achieving intended styles.

This section features engaging discussions on various AI-related topics. Users share insights, raise questions, and explore new developments. These conversations cover areas like AI model performance, cost implications, user experiences with AI tools, and future AI releases. Members discuss personalizing AI models, comparing different models like Gemini and Grok 2, utilizing AI in customer support, evaluating AI coding abilities, and speculating on upcoming AI models from OpenAI and Cerebras. The discussions highlight the complexity and potential of AI applications in diverse fields.

One member suggested utilizing image processing techniques combined with pre-trained models like OpenCV to evaluate document quality in terms of blurriness and darkness. They recommended algorithms like Laplacian variance for detecting blur and consulting CNNs like VGG or ResNet for general image quality features.

The community engaged in discussions about animating fireballs in photos, exploring various techniques and tools like AnimateDiff. Members shared insights on enhancing static images with dynamic effects and highlighted the effectiveness of AnimateDiff for visual enhancements. Additionally, there were conversations about utilizing IP Adapter Plus and SVD as potential solutions for animating fireballs in images, showcasing a collaborative effort to achieve desired animation effects.

M2 Ultra Mac arrives, huge potential for LLMs:

A user sets up their new M2 Ultra Mac with 192 GB Unified Memory and a 2 TB drive, aiming to establish a development environment before exploring LLMs. Pydus is eager to find out how large a model he can load on his new machine.

Discussing Power Limits for GPUs:

With a 96-core EPYC and 4x RTX 4090s, a user noted calculations showing a power consumption limit of 3500W, stressing the need for careful power distribution across multiple outlets. The conversation involved configuring multiple PSUs and ensuring they could handle the load without blowing breakers.

Empirical testing on LLMs for Token Rate:

Mylez_96150 shared that the Llama 3.1 70B model runs at 97 tokens per second using a multi-GPU setup, while another user indicated a 1 token per second rate may have been recorded earlier. The discussions explored various setups including how to optimize performance when splitting model layers across GPUs.

Challenges with Multi-GPU Inference:

Concerns were raised about how to efficiently run LLMs across multiple GPUs, particularly whether the performance improves when using NVLink drivers and how memory sharing impacts speed. Communications highlighted that proper model loading and configurations can potentially increase throughput significantly.

Debating Impact of PCIe Configurations:

A user queried how switching RTX 4090 settings from Gen4 x16 to Gen4 x8 could impact performance when working with a 70B or 405B model. Another user explained that it might

Word Game Bench for Language Models

A new benchmark called Word Game Bench has been developed to evaluate language models on word puzzle games like Wordle and Connections. No model currently scores above 50% average win rate, and the benchmark focuses on interactive testing rather than static responses.

Challenges in Measuring Consistency

A member is exploring how to measure consistency in multiple choice questions when prompts vary slightly. Creating datasets for comparisons and utilizing functions like doc_to_target or doc_to_text are suggested solutions, although they require effort for each model.

In this section, various topics related to AI development and growth are discussed. Codeium secured $150 million in Series C funding, Meta's AI assistant reached impressive user numbers, Google DeepMind introduced customizable Gems, advancements in code generation tools like Claude 3.5 Sonnet and Townie were discussed, and Tome shifted its focus to enterprise AI assistance. These discussions highlight the ongoing advancements and changes in the AI industry, showcasing the rapid evolution and impact of AI technologies.

• LlamaIndex Warning about Valid Config Keys: Users discussed receiving warnings about changed config keys in LlamaIndex V2, specifically 'allow_population_by_field_name' and 'smart_union'.
• Query Engines Deprecation Concerns: Concerns were raised about potential QueryEngines deprecation based on documentation, with references to deprecated methods for RAG workflows.
• Using Llama3 LLM for API Calls: Users inquired about utilizing Llama3 with OpenAI for API calls, seeking clear guidance.
• Handling JSON Data in LLM Workflows: Users shared struggles integrating JSON output from external APIs into the LLM efficiently.
• Issues with Azure OpenAI Integration: Frustrations were expressed with the integration of LlamaIndex and Azure AI, citing issues with citation mismatches in search results.

This section includes discussions from different channels on topics related to AI technology. Members engage in conversations about Lightning Fast AI Model Serving with LitServe and the integration of LitServe with LlamaIndex to enhance AI applications. Another set of discussions revolves around the OpenInterpreter community, where members gather for a House Party, seek advice on terminal applications for KDE, and face issues with the Obsidian OI plugin and GPT-4o. The LAION group discusses topics like Google's GPU acquisition, RunwayML's repo deletions, and the release of Re-LAION-5B dataset. Additionally, the GameNGen neural model for real-time gaming and AgentOps team's future plans are highlighted. The Interconnects channel covers news on OpenAI's funding round, ChatGPT's user base, and chatbot competition excitement. Discussions in Torchtune include topics like QLoRA memory limits and illegal memory access errors. The DSPy community engages in conversations about GitHub repos and LinkedIn Auto Jobs Applier. Lastly, the OpenAccess AI Collective discusses dark mode on Axolotl GitHub documentation and hardware requirements for Llama 70B training with A6000 GPUs.

• Llama 3.1 special tokens: A member inquired about issues with special tokens in Llama 3.1.
  • Another participant suggested a solution to address uninitialized special tokens.
• Introducing fix_untrained_tokens option: A new option, fix_untrained_tokens, has been added to address problems with special tokens initialization.
• Link: Add assistant prefill for chat templates and TextGenerationPipeline by Rocketknight1