[AINews] a calm before the storm • ButtondownTwitterTwitter

Recaps of AI developments and industry updates include OpenAI's release of new models o1 and o1-mini, Alibaba's Qwen2.5 model, Microsoft and Blackrock raising funds for AI data centers, Groq's partnership with Aramco for an AI inference center, Disney Research and ETH Zurich's RobotMDM project, Pudu Robotics' semi-humanoid robot, Slack and Microsoft's new AI-powered features, papers on long-context models and KV cache quantization, and techniques for quantizing cached KV activations in large language models.

Theme 1: Qwen2.5 Emerges as New Open Source SOTA, Replacing Larger Models

• Users are integrating Qwen2.5 (state-of-the-art model) into daily workflows, replacing models like Llama 3.1 70B IQ2_M with Qwen2.5 32B IQ4_XS.
• Experimentation with Qwen2.5 for tasks like article summarization and custom Python setups for content processing.
• Mixed reviews on Qwen2.5's capabilities, with comparisons to Gemma2 2B for YouTube transcripts.

Theme 2: Safe Code Execution in Open WebUI Using gVisor Sandboxing

• Open WebUI implements safe code execution with Docker containers, enhancing security and enabling interactive coding experiences.
• Features two modes: 'Function' for code blocks and 'Tool' for autonomous code execution by LLMs.
• Discussions include extending language support to Go and handling missing dependencies.

Theme 3: NSFW AI Models Optimized for Roleplay Scenarios

• Comparison of small NSFW RP models under 20B parameters, categorizing them as 'Good,' 'Great,' and 'ABSOLUTELY FANTASTIC'.
• Discussion on models like L3-Nymeria-Maid-8B-exl2 and Cydonia 22B for RP scenarios.
• Use cases for uncensored models discussed, including explicit content and non-sexual scenarios involving violence.

Theme 4: Jailbreaking and Censorship Testing of Qwen2.5 Models

• Testing of Qwen2.5 models (72b, 32b, 14b) for censorship using Ollama and Open-webui, bypassing censorship for specific topics like Uyghurs and Hong Kong.
• Debate on responses indicating censorship or bias in the models' training data.
• User tests with prompts about sensitive topics like Tiananmen Square, sparking discussions on model capabilities.

Theme 5: Limited Excitement for New Command-R Model Updates

• Discussion on the Command-R model improvements by Cohere, noting limited public enthusiasm compared to its initial release.
• User comparisons with models like Qwen2.5-32B and feedback on Command-R's performance for storytelling and document chatting.
• Concerns raised on Command-R's non-commercial license impacting adoption and its performance in RP/ERP scenarios.

This section discusses updates and suggestions related to the evaluation processes and function documentation in the TIGER-AI-Lab/MMLU-Pro repository. User feedback prompted changes in the sampling logic to enhance accuracy based on question categories. Additionally, concerns were raised about the discrepancy between the available activation functions listed in the documentation and those present in the codebase, particularly highlighting Swiglu. Discussions also touched upon the need for a reference model for the KTO trainer and reported issues with the Qwen 2.5 model's behavior post-updates. The RAG implementation, SetFit 1.1.0 release, and structured approach for training classifiers were also key points of discussion, emphasizing performance improvements, enhanced training capabilities, and efficient classifier training using the Sentence Transformers Trainer.

The AI community is actively engaging in various discussions and developments across different channels. Some key highlights include enhancing LM clients, improving chat adapters to address repetitive responses, boosting synthetic data generation speeds, leveraging DSPy for specialized domain problems, exploring text classification challenges, and more. Additionally, users are inquiring about CUDA hackathons, CPU offloading optimization, experiencing OOM issues with models, debating batch sizes in model evaluation, and discussing bug fixes for evaluation recipes. The community is also demonstrating interest in AI internship leads, dataset sharing for model training, feedback on summarizer AI, and innovative projects like a playlist generator. Platforms like Discord are facilitating these interactions and collaborations among AI enthusiasts and professionals.

Members of the HuggingFace community share various insights and developments in the AI and Machine Learning space:

• Centroidal Triplet Loss already exists: A member discovered that their 'novel' idea, Centroidal Triplet Loss, has already been developed as Centroid Triplet Loss. They are exploring modifications to enhance the concept.

• Mamba-2 surpasses its predecessor: Researchers introduced Mamba-2, a state space model that outperforms Mamba-1 and Transformer++. It features Structured State Space Duality (SSD) for better handling of information-dense data.

• Exploring the BFGS algorithm: A member is researching the BFGS algorithm and seeking input from others to enhance their understanding.

• Langchain connects LLMs to data sources: A member shared excitement about how Langchain integrates LLMs with databases and APIs for data retrieval.

• 1b FP8 matches bfloat16 precision: A member noted that 1b FP8 achieves loss matching that of bfloat16 mixed precision exactly, implying implications for model training efficiency and performance.

Find more details in the sections on Computer Vision, NLP, and Diffusion Discussions.

Aider, an AI pair programming tool, offers users features to optimize performance and enhance productivity. Some key points include:

• Aider maintains a repository map for code changes and interactions with LLMs.
• Users can control the repository map updates and token limits for better clarity.
• Integration with Markdown documents allows specific file reviews.
• Aider supports various local and external models and setting configurations using .env files.
• It is recommended to configure Aider for different environments using symbolic references for paths in configurations, ensuring portable setups.

Today, a joint blog was released by Eleuther and Cerebras focusing on maximizing the accessibility of μ-parameterization (μP) for the training community. The guide provides step-by-step implementation instructions and offers a simplified approach to understanding μP concepts. Benefits of adopting μP include reduced instabilities during training, lower compute requirements for hyperparameter optimization, and facilitating more robust comparisons between different training methods. The future integration of this simplified μP into the upcoming GPT-NeoX 3.0 release was also discussed, with ongoing updates available in the GPT-NeoX repository.

The ongoing discussion covered the effectiveness of curriculum learning in AI, with skepticism about its impact and best practices. OpenAI's Large Reasoning Model claims were questioned by some members, highlighting potential improvements through existing methods. Additionally, concerns were raised about the shortcomings of interpretability methods in AI, leading to discussions on feature attribution explanations. The conversation also touched on benchmarks comparing AI performance to human abilities and the resource efficiency of data usage in AI training.

The section presents discussions on various topics related to neural network training and model evaluation. It covers issues such as calculating irreducible loss in autoregressive models, optimal token count selection, and usage of sparse feature circuits for interpretability. Additionally, insights on few-shot evaluation of language models, stability in model initialization, and activation functions synchronization are shared. The dialogue highlights ongoing advancements and challenges in training and evaluating neural networks.

The section provides insights into various discussions and solutions related to model fine-tuning and implementation. Discussions include clarifications on KTO trainer usage requiring a reference model for rewards calculation and suggestions on pre-generating responses to save memory. Issues with Qwen Model Fine-tuning, RAG implementation for enhancing model responses, challenges with chat templates in fine-tuned models, and discussions on reflection fine-tune methods are also highlighted. Additionally, the section covers topics like the advantages of soft prompts over model fine-tuning, bias in AI systems, and the significance of AI in healthcare compliance. Members express excitement over the future of AI reasoning capabilities and foresee a rapid evolution towards AGI, emphasizing the need for a diverse set of RL environments for effective training.

The section highlights various discussions related to Cohere AI projects and related topics. Some of the key points discussed include: discussions on API geolocation restrictions by Cohere, changes in embedding calls, the launch of an updated AI model, and enhanced training processes in SetFit. Additionally, there are conversations on field reordering in Mojo for memory optimization, Python library compatibility in Mojo, and struct sizes and bit packing implications in Mojo. The section also mentions upcoming community meetings and feedback opportunities for Magic users.

The recent community discussions revolved around various aspects related to Mojo's compatibility with Python libraries, bit packing, struct sizes, C compatibility, and field reordering. Concerns were raised about potential GIL limitations when using Python libraries with Mojo. Members highlighted the reliance on CPython and its performance limitations. The absence of native bit packing support in Mojo was addressed, with solutions like manual packing suggested. LLVM's potential to handle varying bit widths was discussed. The importance of C compatibility and field reordering in structs to optimize memory usage was emphasized. Suggestions for explicit decorators for flexible struct definitions were made. Additionally, an announcement was shared about the rescheduling of the Community Meeting to facilitate planning, scheduled for Monday, September 30th at 10 AM PT.

The section discusses the outcomes of the GPU MODE community's in-real-life (IRL) meetup, where various achievements and community values were highlighted. Key points include the transition from CUDA MODE to GPU MODE, the success of the CUDA MODE IRL meetup with 150 hackers creating over 40 projects, the growth of open-source projects in the community, showcasing hackathon winners and their diverse projects, and the emphasis on fostering collaboration and community values. The section also mentions the positive reception of the event and the community's commitment to innovation and social engagement.

This section discusses various issues related to GPU compatibility and functionality in the context of CUDA/Torch versions and Bitblas backend usage. Users encountered problems with CUDA errors, particularly with the Tesla T4 GPU limitations and Torch compilation on older GPUs. Recommendations were made to try the bitblas backend for better results. Additionally, discussions on coordinating event attendance, exploring optimization techniques, and creating chatbots for student services were also highlighted.

This section discusses various GPU kernel operations and issues encountered by members. It covers topics such as calculation issues with the KLDivLoss kernel, bugs in RMSNorm and LayerNorm, comparison between KLDivLoss and Cross-Entropy, handling of kernel function reduction, and addressing Triton's limitations. Members shared insights on loop unrolling issues, output shape mismatches, program handling, and grid size limitations. The discussions aim to optimize kernel functions and ensure smooth operation of GPU modes.

The hackathon officially kicked off with participant instructions and information on compute credits and project proposals. Dinner and networking opportunities were provided, and preliminary judging was underway. New features in the dspy.LM method were discussed, including an Adapter layer, with feedback and quick updates expected. Users were reminded to submit project details for judging. In another area, performance differences between MPS and WebGPU, collaboration discussions, and Metal usage for low-intensity tasks were explored. Lastly, information on cloud-based testing, an upcoming advanced usage webinar, model changes in OpenRouter, and concerns over OpenAI account security were presented.

The discussion in this section covers various topics related to AI, including struggles of AI models with music theory, concerns over Bittensor's practices, exploration of byte-level cognitive models, integration of RetNet with transformers, and utilization of the World Sim API. Users engaged in discussions about AI models, distributed training algorithms, innovative training methods, and collaboration opportunities in the AI community. The conversation also touched on the use of APIs, free learning resources, new techniques for model initialization, and sharing of research endeavors and resources.

The section delves into various research papers focusing on advancements in Medical AI technology. It highlights the building of Virtual Cells with Artificial Intelligence, introduces new medical LLMs like GP-GPT and HuatuoGPT-Vision, explores innovative frameworks for medical diagnosis using Chain of Diagnosis (CoD), showcases the transformation of clinical trials with LLMs like AlpaPICO, and discusses the importance of addressing AI cyber threats in healthcare. These developments aim to revolutionize medical AI applications, enhance diagnostic efficiency, streamline clinical processes, and ensure cybersecurity in medical practices.

This section discusses various AI-related projects in different channels.

• Encouragement for Similar Projects: There is an open call for sharing information about similar projects to foster collaboration in alignment development. Members are encouraged to feel free to share insights or reach out privately with relevant information.

• Rope Scaling Confusion for Llama 3.1: A member questioned the necessity of rope_scaling when training Llama 3.1 8B and encountered memory issues when increasing sequence_len beyond 40K tokens. Additionally, a spike during fine-tuning on a 100K row dataset was reported, leading to a request for additional logging output to understand the cause of the spike.

• Cohere API Discussions: Discussions on geolocation restrictions affecting API access and changes in the embedding call requirements were highlighted, urging users to contact support for assistance and clarification from the Cohere team.

• AI-Telegram-Chatbot with Cohere AI: A member shared a GitHub repository for an AI-Telegram-Chatbot using Cohere AI to enhance user interaction, reflecting a broader interest in leveraging Cohere technologies for practical applications.

• Mozilla Launches AI Builders Accelerator: The announcement of Mozilla's AI Builders Accelerator cohort and the initiation of support for innovative AI projects is highlighted, showcasing continuous collaboration with enterprises and open-source contributors.

• SoraSNS: A New Fediverse Client: An ex-Apple Engineer introduced SoraSNS, a Fediverse client that utilizes local AI to learn about user interests and provide a personalized 'For You' timeline.

• Open Source AI to Alleviate Issues: Mark Surman discussed the potential of defining Open Source AI to address challenges in the field, underscoring the relevance of collaboration and defining AI standards.

This section discusses various topics related to AI in the OpenAI community. Members shared insights on O1-mini's performance in creative writing, discussed embedding storage solutions, AI tools for analyzing PDFs, comparative analysis of AI chatbot models, and challenges in using AI for nuanced poetry. The section also covers discussions on GPT-O1-preview quota for enterprise, appealing custom GPT removal, using GPT 4o for advanced math, and issues with ChatGPT in Firefox. Additionally, conversations in OpenAI channels explored prompt sharing, anti-KI detection techniques, and AI tools usage. Further discussions touched on the potential valuation surge of Anthropic, challenges in data annotation for AI models, and Qwen model's impressive performance. The Interconnects section delves into logo redesigns at OpenAI and PayPal, consumer sentiment towards Google products, revelations on training Gemini with Shampoo, and gatekeeping around Shampoo's usage.

Discussion on Optimizer CPU Offloading:

• Query about lack of CPU offloading in the optimizer within the full_finetune_single_device.py recipe, citing potential performance issues.

Exploration of KV Caching Impact:

• OOM issues reported during evaluation with KV caching enabled for the qwen2.5 1.5B model.

Batch Size Performance Insights:

• Inquiry regarding performance differences when increasing batch sizes in model evaluation.

Evaluation Recipe Bug Fix Discussions:

• Pointed to a PR addressing bugs found in the evaluation recipe for group tasks.

Clarifications on Adam Update Process:

• Description of complexities in using optimizer_in_backward and memory copy operations for Adam updates.

A member noted positive feedback for Qwen 2.5 compared to Llama 3.1, with benchmark comparisons showing slight outperformance by Qwen 2.5. Another user expressed interest in benchmark comparisons between Qwen 2.5 7B and Llama 3.1 8B models. Additionally, a member inquired about Axolotl's handling of long conversations exceeding max_seq_len in ShareGPT, showcasing ongoing curiosity about managing context limits in chat models.

• BFCL V3 Launches with New Features: The Berkeley Function-Calling Leaderboard (BFCL) V3 introduces a novel evaluation of multi-turn and multi-step function calling, enhancing agentic systems' capabilities.
  • This version allows models to engage in back-and-forth interactions, critical for assessing LLM functionality under complex conditions.
• State Management is Key: Internal state querying for tasks is crucial for LLMs, emphasizing the importance of probing state through APIs.
• Short Context Models Are Out!: Models relying on short context must adapt for tasks requiring longer context understandings.
• Leaderboards Driving Standards: BFCL V3 sets a gold standard for evaluating LLMs' function invocation abilities with multi-turn interactions, informed by community feedback.
• Find More Details on Performance: A blog post on Berkeley Function Calling Blog discusses evaluation methodology and how models are measured on cost and latency in real-world scenarios.