OpenAI announces Apps SDK allowing ChatGPT to launch and run third party apps like Zillow, Canva, Spotify

The annual Conference of OpenAI for external developers, Devday, started today with a bang as a co-founder and CEO Sam Altman announced a new “apps SDK” That makes it “possible to build apps in chatgpt”, “ Including paid apps that companies can be able to charge for the use of the recently unveiled Agentic Commerce Protocol (ACP) from OpenAI.

In other words, instead of starting apps one by one on your phone, computer or on the internet, you can do all of that without Ever left chatgpt.

With this function, the user can log in to those external apps and bring all his information back into chatgpt, and use the apps in the same way as how they already do it outside the chatbot, but now with the possibility to ask chatgpt to perform certain actions, to analyze content or continue what each app could offer.

You can order Canva to let you be draws on the basis of a text description, ask Zillow for home lists in a certain area that fits certain requirements, or ask Coursera about the content of a specific lesson while this plays on video, all from Chatgpt – with many other apps that also offer their own connections (see below).

“With this, a new generation of apps that are interactive, adaptive and personalized can make possible with which you can talk,” said Altman.

While the SDK apps is available today in Preview, OpenAi said It would not start by accepting new apps within Chatgpt or enabling them to ask users until “later this year”.

Chatgpt In-Line App access has already been rolled to Chatgpt Free, Plus, Going and Pro users-for-how outside the European Union with business, companies and education that are expected to gain access to the apps later this year.

Built on top common MCP -Standard

Built on the Open Source Standard Model Context Protocol (MCP) introduced by Rival Anthropic almost a year ago, the apps SDK external developers independently or on behalf of companies to connect selected data, “Trigger Actions,” Trigger Actions, and Render a fully interactive UI [user interface]”Altman explained during his introductory keynote speech.

The apps SDK contains a function “Talking to apps” with which Chatgpt and the underlying GPT-5 or other “O-series” models can control it below to obtain the additional context of the app of third parties, so the model “Always know exactly what your user handles,” said another presenter and openi-engineer, said.

Developers can build apps that:

• to appear take in In chat as lightweight cards or carousels

• extend to on full screen For compelling tasks such as cards, menus or slides

• usage Picture-in-picture For live sessions such as video, games or quizzes

Each mode is designed to maintain the minimum, chatgpt conversation flow and at the same time add interactivity and branding.

Early integrations with Coursera, Canva, Zillow and more …

Christakis showed out early integrations of External apps built on top of the apps SDKIncluding those of the e-learning company CourseraSoftware company Cloud Design Software Canvaand real estate lists and agent connections search engine, Zillow.

Altman also announced apps SDK integrations with extra partners who were not officially demonstrated during the keynote, including: Booking.com” Expedia” Figma And Spotify And in documentation, said that more upcoming partners are on the deck: Alltrails, Peloton, Opentable, Target, TheFork and Uberrepresent lifestyle, trade and productivity categories.

The Coursera -Demo Recorded an example of how the user boards to the external app, including a new login screen for the app (Coursera) that appears in the Chatgpt -Chat interface, simply activated by a text prompt from the user with the question: “Coursera can you teach me about machine learning?

Once logged in, the app was launched in the chat interface, “in Line” and can display everything from the web, including interactive elements such as video.

Christakis explained and showed the apps that SDK also supports “Picture-in-Picture” and “Fullscreen” views, so that the user can choose how to communicate with it.

When he played a Coursera video that appeared, he showed that it automatically grabbed the video at the top of the screen, so that the user could continue to watch it even while he remained a dialogue in text with chatgpt in the typical input/output prompts and answers below.

Users can then ask chatgpt about content that appears in the video without specifying exactly what has been saidWhile the agents the information about the backend, server-side, from the connected app to the underlying chatgpt AI model blowing. So “you can explain more about what they are saying now” the relevant part of the video will automatically come up and offer that to the underlying AI model to analyze and respond via text.

In another example, Christakis opened a parent, existing chatgpt conversation that he had had about the dog walk of his brothers and sisters and resumed the conversation by asking another third-party app, CanvaTo generate a poster using one of the recommended company names of Chatgpt, “Walk This Wag”, together with specific guidelines on letter choice (“Sans Serif”) and overall coloring and style (“Clear and colorful.”)

Instead of having to go manually and add all those specific elements to a Canva template, Chatgpt went and issued the assignments and performed the actions on behalf of the user in the background.

After a few minutes, Chatgpt reacted with different poster designs that were generated directly in the Canva app, but they all displayed in the chatgpt chat session of the user where they could see, assess, enlarge and give feedback or ask for adjustments to all.

Christakis then asked Chatgpt to turn one of the slides into a whole sliding deck, so that the founders of the dog walking activities could present it to investors, who did it in the background for a few minutes while presenting a last integrated app, Zillow.

He started a new chat session and asked a simple question: “Based on our conversations, a good city would be to expand the dog walk activities.”

With the help of the optional memory function of Chatgpt, it referred to the Dog Walk conversation and suggested Pittsburgh, who used Christakis as an opportunity to type “Zillow” and “some houses for sale there”, who call an interactive map of Zillow of Zillow and Prices that are mentioned and prices that are mentioned and prices that are mentioned and prices that are mentioned and prices.

Click on a specific house also opened a full screen with “most of the Zillow experience”, fully without chatgpt, including the possibility of requesting home travel and contact agents and filtering per bedrooms and other qualities such as outdoor space. Chatgpt picks up the requested filtered Zillow search and gives a text-based response in-line in which it is explained what it did and why.

The user can then ask follow-up questions about the specific characteristic-such as “How close is it to a dog park?” – Or compare it with other features, all within Chatgpt.

It can also use apps in combination with the search function, by searching the web to compare the app information (in this case Zillow) with other sources.

Safety, privacy and developer standards

OpenAi emphasized that Apps must meet strict privacy, safety and content standards To be mentioned in the Chatgpt directory. Apps must:

• serve one clear and valuable goal

• are predictable and reliable in behavior

• are Safe for the general publicIncluding teenagers aged 13-17

• Respect the privacy of users and limit data collection to only what is needed

Each app must also be one clear, published privacy policyGet the consent of the users before connecting and identifies actions that change external data (eg posts, send, upload).

Apps that violate OpenAi’s user policy, which often crashes, or incorrectly displaying their options can be removed at any time. Developers must submit themselves from verified accountsprovided Customer Support ContactsAnd retain their apps for stability and compliance.

OpenAi also published developer Design guidelinesSketches how apps should look, sound and behave. They must follow the visual system of chatgpt – including consistent color palettes, typography, distance and iconography – and accessibility standards such as alt text and readable contrast ratios.

Partners can display brand logos and accent colors, but not change the core interface of chatgpt or use promotion language. Apps must stay “Conversation, intelligent, simple, responsive and accessible,” According to the documentation.

A new ecosystem for conversation -app

By opening chatgpt for apps and payments from third parties, OpenAI is taking a big step in the direction of transforming chatgpt of a chatbot in a full -fledged AI control system – One that combines conversation -intelligence, rich interfaces and embedded trade.

For developers, that means direct access to More than 800 million Chatgpt -usersBeing able to discover those apps “at the right time” through a natural conversation – whether it is about traveling, learning or shopping.

For users this means A new generation of apps with which you can chat – Where a single interface helps you to book a flight, design a sliding deck or to learn a new skill without ever leaving chatgpt.

As OpenAi put it: “This is just the start of apps in Chatgpt, so that users and new opportunities for developers are used new.”

There are a few big questions, namely: 1. What happens to all the details of those external apps while they interface with chatgpt and its users … will gain access to it and can it train on it? 2. What happens with OpenAi’s once many hyped GPT store, which in the past had been promoted as a way for makers and developers of third parties to make adapted, task-specific versions of chatgpt and to earn money on it through a use-based income share model?

We have asked the company for both problems and will update when we hear again.

Source link