Steve Witkoff

Shalev Lifshitz, Keiran Paster, Harris Chan, Jimmy Ba, Sheila A. McIlraith

Constructing AI models that respond to text instructions is challenging, especially for sequential decision-making tasks. This work introduces a methodology, inspired by unCLIP, for instruction-tuning generative models of behavior without relying on a large dataset of instruction-labeled trajectories. Using this methodology, we create an instruction-tuned Video Pretraining (VPT) model called STEVE-1, which can follow short-horizon open-ended text and visual instructions in Minecraft. STEVE-1 is trained in two steps: adapting the pretrained VPT model to follow commands in MineCLIP's latent space, then training a prior to predict latent codes from text. This allows us to finetune VPT through self-supervised behavioral cloning and hindsight relabeling, reducing the need for costly human text annotations, and all for only $60 of compute. By leveraging pretrained models like VPT and MineCLIP and employing best practices from text-conditioned image generation, STEVE-1 sets a new bar for open-ended instruction-following in Minecraft with low-level controls (mouse and keyboard) and raw pixel inputs, far outperforming previous baselines and robustly completing 12 of 13 tasks in our early-game evaluation suite. We provide experimental evidence highlighting key factors for downstream performance, including pretraining, classifier-free guidance, and data scaling. All resources, including our model weights, training scripts, and evaluation tools are made available for further research.

Sipeng Zheng, Jiazheng Liu, Yicheng Feng, Zongqing Lu

Recent studies have presented compelling evidence that large language models (LLMs) can equip embodied agents with the self-driven capability to interact with the world, which marks an initial step toward versatile robotics. However, these efforts tend to overlook the visual richness of open worlds, rendering the entire interactive process akin to"a blindfolded text-based game."Consequently, LLM-based agents frequently encounter challenges in intuitively comprehending their surroundings and producing responses that are easy to understand. In this paper, we propose Steve-Eye, an end-to-end trained large multimodal model designed to address this limitation. Steve-Eye integrates the LLM with a visual encoder which enables it to process visual-text inputs and generate multimodal feedback. In addition, we use a semi-automatic strategy to collect an extensive dataset comprising 850K open-world instruction pairs, empowering our model to encompass three essential functions for an agent: multimodal perception, foundational knowledge base, and skill prediction and planning. Lastly, we develop three open-world evaluation benchmarks, then carry out extensive experiments from a wide range of perspectives to validate our model's capability to strategically act and plan. Codes and datasets will be released.

Y. Nishimura, Alan Dyer, Lauri Kangas, E. Donovan, V. Angelopoulos

This paper reviews key properties and major unsolved problems about Strong Thermal Emission Velocity Enhancement (STEVE) and the picket fence. We first introduce the basic characteristics of STEVE and historical observations of STEVE-like emissions, particularly the case on 11 September 1891. Then, we discuss major open questions about STEVE: 1) Why does STEVE preferentially occur in equinoxes? 2) How do the solar wind and storm/substorm conditions control STEVE? 3) Why is STEVE rare, despite that STEVE does not seem to require extreme driving conditions? 4) What are the multi-scale structures of STEVE? 5) What mechanisms determine the properties of the picket fence? 6) What are the chemistry and emission mechanisms of STEVE? 7) What are the impacts of STEVE on the ionosphere−thermosphere system? Also, 8) what is the relation between STEVE, stable auroral red (SAR) arcs, and the subauroral proton aurora? These issues largely concern how STEVE is created as a unique mode of response of the subauroral magnetosphere−ionosphere−thermosphere coupling system. STEVE, SAR arcs, and proton auroras, the three major types of subauroral emissions, require energetic particle injections to the pre-midnight inner magnetosphere and interaction with cold plasma. However, it is not understood why they occur at different times and why they can co-exist and transition from one to another. Strong electron injections into the pre-midnight sector are suggested to be important for driving intense subauroral ion drifts (SAID). A system-level understanding of how the magnetosphere creates distinct injection features, drives subauroral flows, and disturbs the thermosphere to create optical emissions is required to address the key questions about STEVE. The ionosphere−thermosphere modeling that considers the extreme velocity and heating should be conducted to answer what chemical and dynamical processes occur and how much the STEVE luminosity can be explained. Citizen scientist photographs and scientific instruments reveal the evolution of fine-scale structures of STEVE and their connection to the picket fence. Photographs also show the undulation of STEVE and the localized picket fence. High-resolution observations are required to resolve fine-scale structures of STEVE and the picket fence, and such observations are important to understand underlying processes in the ionosphere and thermosphere.

Nicola Guerra

ABSTRACT This study analyses the consequences of the Russian-Ukrainian conflict on the European far right by examining the German Alternative für Deutschland (AfD) party and the Swedish Sverigedemokraterna (SD). The political and geopolitical choices of the AfD and SD are contextualized in the political history of the two parties, in the influences that thinkers such as Steve Bannon and Aleksandr Dugin have on the European right, and in the context of geopolitical change taking place at the international level. A dichotomous reality of the European far right emerges with opposing political and geopolitical visions. On the one hand, the Westernist and Atlanticist extreme right, which is based on white supremacism and the West's indispensability to place itself at the helm of the world order, whose army is identified with NATO, on the other hand, an anti-American, Eurasianist, pro-Russian extreme right and that sees in the end of the American unipolar order an opportunity. Two components of the European far right that are irreconcilable and that focus attention on the inappropriateness of looking at the movements and parties of the European far right as if they were a single ‘black thing’.

Zhonghan Zhao, Wenhao Chai, Xuan Wang, Ke Ma, Kewei Chen, Dongxu Guo, Tianbo Ye, Yanting Zhang, Hongwei Wang, Gaoang Wang

Building an embodied agent system with a large language model (LLM) as its core is a promising direction. Due to the significant costs and uncontrollable factors associated with deploying and training such agents in the real world, we have decided to begin our exploration within the Minecraft environment. Our STEVE Series agents can complete basic tasks in a virtual environment and more challenging tasks such as navigation and even creative tasks, with an efficiency far exceeding previous state-of-the-art methods by a factor of $2.5\times$ to $7.3\times$. We begin our exploration with a vanilla large language model, augmenting it with a vision encoder and an action codebase trained on our collected high-quality dataset STEVE-21K. Subsequently, we enhanced it with a Critic and memory to transform it into a complex system. Finally, we constructed a hierarchical multi-agent system. Our recent work explored how to prune the agent system through knowledge distillation. In the future, we will explore more potential applications of STEVE agents in the real world.