Skip to main content
This guide explains how dynamic User Interface (UI) tools work in MeshAgent. These elements are client-side tools that run on a user’s browser, desktop, or mobile app. Dynamic UIs enable agents to interact directly with users by presenting responsive, on-demand interfaces as needed. They allow agents to do things like:
  • Collect a document from the user in the middle of a task
  • Pop up a one-off approval dialog
  • Fan out a survey to multiple participants in the room

​
Why Dynamic UI Tools Matter

Problem in multi-user agent appsHow a Dynamic UI Tool solves it
Mid-task input: An agent needs extra input during a task (e.g. “Which PDF should I summarise?”).Pop a file-picker on only the requester’s screen, collect the answer, continue the run.
Room wide polls/surveys: You want a one-off survey of everyone currently in the Room.Allow the agent to fan-out an ask_user dialog to each participant and gather the responses asynchronously.
Targeted notifications: Agents must present notifications, approvals, or error messages only to the relevant user—not the whole Room.Each UI toolkit is automatically scoped to the participant who registered it, so dialogs can’t leak across screens.
Security / Phishing: Users should not be able to show dialogs on another user’s screen just because a tool to show dialogs is exposed.Toolkits are scoped to the participant that registers them. An agent must pass participant_id when it calls invoke_tool, so dialogs can’t leak to other users.

​
How MeshAgent Safely Routes UI Tool calls

Behind the scenes MeshAgent supports private tool registrations. This ensures that dialogues only appear on the intended user’s screen even if several clients register toolkits with the same name. For example, you might have multiple ui toolkits, but one toolkit is has an ask_user tool that shows dialogues on a mobile app, a separate one has an ask_user tool that shows a dialogue in the browser, etc. When a user interface registers a private tool, that tool is accessible only to the registering user. An agent can then invoke this private tool by including the unique identifier of the user (the participant ID) in its call. This ensures that the interface is displayed solely to the intended recipient.

​
Example: Survey Room Participants

Let’s create a tool that surveys participants in the Room. The tool will conduct a survey of the participants, summarize the results, and store both the raw results and the summary to the Room storage. The survey fields are dynamically generated so that we can gather a variety of information using the same UI tools. This means we can use the same tool to conduct a survey where participants respond yes/no to a question, or to provide more detailed feedback on their experience, etc. 
import json
import asyncio
import logging
from meshagent.otel import otel_config
from meshagent.api.services import ServiceHost
from meshagent.tools import Tool, ToolContext, RemoteToolkit
from meshagent.api.messaging import TextResponse

otel_config(service_name="my-service")
log = logging.getLogger("my-service")

service = ServiceHost()


async def save_to_storage(room, path: str, data: bytes):
    handle = await room.storage.open(path=path, overwrite=True)
    await room.storage.write(handle=handle, data=data)
    await room.storage.close(handle=handle)


class Survey(Tool):
    def __init__(self):
        super().__init__(
            name="survey",
            title="survey",
            description="a tool that conducts a survey of the participants",
            input_schema={
                "type": "object",
                "additionalProperties": False,
                "required": ["subject", "description", "name"],
                "properties": {
                    "subject": {
                        "type": "string",
                        "description": "The subject of the form",
                    },
                    "description": {
                        "type": "string",
                        "description": "The content to fill in (e.g. feedback, poll result)",
                    },
                    "name": {
                        "type": "string",
                        "description": "A short name to be used on the form",
                    },
                },
            },
        )

    async def execute(
        self, context: ToolContext, subject: str, description: str, name: str
    ):
        room = context.room
        participants = [
            p for p in room.messaging.remote_participants if p.role == "user"
        ]
        log.info("Starting survey for %d participant(s)", len(participants))

        MAX_ATTEMPTS = 2

        async def ask_participant(p):
            errors = []
            for attempt in range(1, MAX_ATTEMPTS + 1):
                try:
                    log.info("→ ask_user attempt %d --> %s", attempt, p.id)
                    resp = await room.agents.invoke_tool(
                        toolkit="ui",
                        tool="ask_user",
                        participant_id=p.id,
                        arguments={
                            "subject": subject,
                            "description": description,
                            "form": [
                                {
                                    "input": {
                                        "multiline": False,
                                        "name": name,
                                        "description": description,
                                    },
                                },
                            ],
                        },
                    )
                    answer = resp.json.get(name)
                    if answer:
                        log.info("participant_id", p.id, "response", answer)
                        return {"participant_id": p.id, "response": answer}
                    raise RuntimeError("empty or timed-out response")

                except Exception as exc:
                    errors.append(f"attempt {attempt}: {exc}")
                    if attempt < MAX_ATTEMPTS:
                        log.info("Retrying %s after: %s", p.id, exc)
                        await asyncio.sleep(1)  # brief back-off

            # All attempts failed – return aggregated error list
            return {"participant_id": p.id, "errors": errors}

        log.info("Surveying participants")
        tasks = [asyncio.create_task(ask_participant(p)) for p in participants]
        results = await asyncio.gather(*tasks)

        summary = {
            "meta": {  # save the prompt generated for the survey
                "subject": subject,
                "description": description,
                "name": name,
            },
            "success": {},
            "failed": {},
        }
        for item in results:
            pid = item["participant_id"]
            if "response" in item:
                summary["success"][pid] = item["response"]
            else:
                summary["failed"][pid] = item["errors"]

        # write survey results to the room
        log.info("Survey completed, writing raw results to Room storage")
        await save_to_storage(
            room=context.room,
            path=f"survey/{room.room_name}-{name}.json",
            data=json.dumps({"summary": summary}, indent=2).encode("utf-8"),
        )

        # summarize results
        log.info("Summarizing survey results")
        summary_resp = await context.room.agents.ask(
            agent="meshagent.schema_planner",
            arguments={
                "prompt": f"Summarize these survey results:\n{json.dumps(summary)}",
                "output_schema": {
                    "type": "object",
                    "properties": {"summary": {"type": "string"}},
                    "required": ["summary"],
                    "additionalProperties": False,
                },
            },
        )
        summary_text = summary_resp.json["summary"]
        log.info("Saving survey result summary")
        await save_to_storage(
            room=context.room,
            path=f"survey/{room.room_name}-{name}-summary.doc",
            data=summary_text.encode("utf-8"),
        )

        return TextResponse(text=summary_text)


@service.path(path="/survey", identity="survey-toolkit")
class SurveyToolkit(RemoteToolkit):
    def __init__(self):
        super().__init__(
            name="survey-toolkit",
            title="survey-toolkit",
            description="a toolkit for conducting a survey",
            tools=[Survey()],
        )


print(f"running on port {service.port}")
asyncio.run(service.run())
Copy this code and activate your virtual environment, you will need two tabs open in your terminal. In the first tab run:
bash
python survey.py
In the second tab run:
bash
meshagent setup # this will prompt you to authenticate to MeshAgent, select your project and API keys
meshagent call tool --room=surveyroom --url=http://localhost:7777/survey --participant-name=survey
Next go to studio.meshagent.com and click into the surveyroom. Once inside the room, select the menu icon and click “Run Task”, the click “Add Tools” and you will see the survey-toolkit. Click on the invoke button. You will be prompted to fill out information that will be used in the survey. For example you might fill in details stating that this survey is to collect feedback on the employee onboarding experience, and you might ask for specific feedback regarding any questions or concerns the employees have and what they learned from the training, you might title the form something like “Onboarding Feedback July”. Once you fill in these details, the agent will survey other users in the room and collect their feedback asynchronously. Once the results are captured you will receive a summary of the results and both the summary and raw results will be saved to the room database for future use/access.

​
Next Steps: Using Dynamic Tools in Your Own App

MeshAgent automatically provides a ui toolkit which you can use and test immediately in the MeshAgent Studio. If you want to create UI tools that work either inside the Studio or present custom dialogues in your own web or mobile client you can create a toolkit with the same name, ui, and the same tools and schemas as the ones available inside the Studio. When a participant from your app invokes the tool, it will display your application’s custom interface and still be usable from the studio.