Gradio: Hassle-Free Sharing and Testing of ML Models in the Wild

Abubakar Abid * 1 2 Ali Abdalla * 2 Ali Abid * 2 Dawood Khan * 2 Abdulrahman Alfozan 2 James Zou 3

AbstractAccessibility is a major challenge of machinelearning (ML). Typical ML models are built byspecialists and require specialized hardware/soft-ware as well as ML experience to validate. Thismakes it challenging for non-technical collabora-tors and endpoint users (e.g. physicians) to easilyprovide feedback on model development and togain trust in ML. The accessibility challenge alsomakes collaboration more difficult and limits theML researcher’s exposure to realistic data and sce-narios that occur in the wild. To improve accessi-bility and facilitate collaboration, we developedan open-source Python package, Gradio, whichallows researchers to rapidly generate a visual in-terface for their ML models. Gradio makes access-ing any ML model as easy as sharing a URL. Ourdevelopment of Gradio is informed by interviewswith a number of machine learning researcherswho participate in interdisciplinary collaborations.Their feedback identified that Gradio should sup-port a variety of interfaces and frameworks, al-low for easy sharing of the interface, allow forinput manipulation and interactive inference bythe domain expert, as well as allow embeddingthe interface in iPython notebooks. We devel-oped these features and carried out a case studyto understand Gradio’s usefulness and usabilityin the setting of a machine learning collaborationbetween a researcher and a cardiologist.

1. IntroductionMachine learning (ML) researchers are increasingly part ofinterdisciplinary collaborations in which they work closelywith domain experts, such as doctors, physicists, geneticists,and artists (Bhardwaj et al., 2017; Radovic et al., 2018; Zou

*Equal contribution 1Department of Electrical Engineering,Stanford University, Stanford, California, USA 2Gradio Inc.,Mountain View, California, USA 3Department of Biomedical DataScience, Stanford University, Stanford, California, USA. Corre-spondence to: Abubakar Abid <a12d@stanford.edu>.

2019 ICML Workshop on Human in the Loop Learning (HILL2019), Long Beach, USA. Copyright by the author(s).

et al., 2018; Hertzmann, 2018). In a typical work flow, thedomain experts will provide the data sets that the ML re-searcher analyzes, and will provide high-level feedback onthe progress of a project. However, the domain expert isusually very limited in their ability to provide direct feed-back on model performance, since, without a backgroundin ML or coding, they are unable to try out the ML modelsduring development.

This causes several problems during the course of the collab-oration. First, the lack of an accessible model for collabora-tors makes it very difficult for domain experts to understandwhen a model is working well and communicate relevantfeedback to improve model performance. Second, it makesit difficult to build models that will be reliable when de-ployed in the real world, since they were only only trainedon a fixed dataset and not tested with the domain shiftspresent in the real world (“in the wild”). Real-world dataoften includes artifacts that are not present in fixed trainingdata; domain experts are usually aware of such artifacts andif they could access the model, they may be able to exposethe model to such data, and gather additional data as needed(Thiagarajan et al., 2018). Lack of end-user engagementin model testing can lead to models that are biased or par-ticularly inaccurate on certain kinds of samples. Finally,end-user domain experts who have not engaged with themodel as it was being developed tend to exhibit a generaldistrust of the model when it is deployed.

In order to address these issues, we have developed an open-source python package, Gradio1, which allows researchersto rapidly generate a web-based visual interface for their MLmodels. This visual interface allows domain experts to inter-act with the model without writing any code. The packageincludes a library of common interfaces to support a widevariety of models, e.g. image, audio, and text-based mod-els. Additionally, Gradio makes it easy for researchers tosecurely share their public links to their models so that col-laborators can try out the model directly from their browserswithout downloading any software, and also lets the domainexpert provide feedback on individual samples, fully en-abling the feedback loop between domain experts and MLresearchers.

In the rest of this paper, we begin by discussing related

1The name is an abbrevation of gradient input output

arX

iv:1

906.

0256

9v1

[cs

.LG

] 6

Jun

201

9

Gradio: Hassle-Free Sharing and Testing of ML Models in the Wild

Figure 1. An illustration of a web interface generated by Gradio, which allows users to drag and drop their own images (left), and getpredicted labels (right). Gradio can provide an interface wrapper around any machine learning model (InceptionNetv3 is shown in theexample here). The web interface can be shared with others using the share link button (center top), and collaborators can providefeedback by flagging particular input samples (bottom right).

works and their limitations, which led to the developmentof Gradio (Section 2). We then detail the implementationof Gradio in Section 3. We have carried out a preliminarypilot study that includes an ML researcher and a clinicalcollaborator, which we describe in Section 4. We concludewith a discussion of the next steps for Gradio in Section 5.

2. Motivation2.1. Related Works

The usefulness of visual interfaces in interdisciplinary col-laborations has been observed by many prior researchers,who have typically created highly customized tools for spe-cific use cases. For example Xu et al. (2018) created aninteractive dashboard to visualize ECG data and classifyheart beats. The authors found that the visualization signifi-cantly increased adoption of the ML method and improvedclinical effectiveness at detecting arrhythmias.

However, visual interfaces that have been developed byprior researchers have been tightly restricted to a specificmachine learning framework (Klemm et al., 2018) or toa specific application domain (Muthukrishna et al., 2019).When we interviewed our users with regards to such tools,they indicated that the limited scope of these tools wouldmake them unsuitable for their particular work.

2.2. Design Requirements

We interviewed 12 machine learning researchers who par-ticipate in interdisciplinary collaborations. Based on thefeedback gathered during these interviews, we identified thefollowing key design requirements.

R1: Support a variety of interfaces and frameworks.Our users reported working with different kinds of models,where the input (or output) could be: text, image, and evenaudio. To support the majority of models, Gradio must beable to offer developers a range of interfaces to match theirneeds. Each of these interfaces must be intuitive enoughso that domain users can use them without a backgroundin machine learning. In addition, ML researchers did notwant to be restricted in which ML framework to use: Gradioneeded to work with at least Scikit-Learn, TensorFlow, andPyTorch models.

R2: Easily share a machine learning model. Our usersindicated that deploying a model so that it can be usedby domain experts is very difficult. They said that Gradioshould allow developers to easily create a link that can beshared with researchers, domain experts, and peers, ideallywithout having to package the model in a particular way orhaving to upload it to a hosting server.

R3: Manipulate input data. To support exploration andimprovement of models, the domain expert needs the abilityto manipulate the input. For example, the ability to crop animage, occlude certain parts of the image, edit the text, addnoise to an audio recording, or trim a video clip. This helpsthe domain expert detect which features affect the model,and what kind of additional data needs to be collected inorder to increase the robustness of the model.

R4: Running in iPython notebooks & embedding. Fi-nally, our users asked that the interfaces be run from andembedded in Jupyter and Google’s Colab notebooks, as wellas embedded in websites. A use case for many of our re-searchers was to expose machine learning models publicly

Gradio: Hassle-Free Sharing and Testing of ML Models in the Wild

after being trained. This would allow their models to betested by many people e.g. in a citizen data science effort, oras part of a tutorial. They needed Gradio to both allow shar-ing of models directly with collaborators, but also widelywith the general public.

3. ImplementationGradio is implemented as a python library, and can be in-stalled from PyPi2. Once installed, running a Gradio inter-face requires minimal change to a ML developer’s existingworkflow. After the model is trained, the developer createsan Interface object with four required parameters (Fig. 2a).The first and second parameter are inputs and outputs, whichtakes as argument the input/output interface to be used.The developer can choose any of the subclasses of Gra-dio.AbstractInput and Gradio.AbstractOutput, respectively.Currently this includes a library of standard interfaces forhandling image, text, and audio data. The next parameter ismodel type which is a string representing the type of modelbeing passed in; This may be keras, pytorch, or sklearn – orit may be pyfunc, which handles arbitrary python functions.The final parameter is model where the developer passes inthe actual model to use for processing. Due to the commonpractice of pre-processing or post-processing the input andoutput of a specific model, we implemented a feature toinstantiate Gradio.Input/Gradio.Output objects with customparameters or alternatively supply custom pre-processingand post-processing functions.

We give the developer the option of how the interface shouldbe launched. The launch function accepts 4 boolean vari-ables, which allow for displaying the model inbrowser(whether to display model in a new browser window) (Fig.2b), displaying it inline (whether to display model embed-ded in interactive python environment such as Jupyter orColab notebooks), attempting to validate (whether to val-idate the interface-model compatibility before launching),and creating a share link (whether to create a public linkto the model interface). If Gradio creates a share link tothe model, then the model continues running on the hostmachine, and an SSH tunnel is created allowing collabora-tors to pass in data into the model remotely, and observethe output. This allows the user to continue running usingthe same machine, with the same hardware and software de-pendencies. The collaborator does not need any specializedhardware or software: just a browser running on a computeror mobile phone (the user interfaces are mobile-friendly).

The user of the interface can input any data and also ma-nipulate the input by, for example, cropping an image (Fig.2c). The data from the input is encrypted and then is passedsecurely through the SSH tunnel to the developer’s com-

2pip install Gradio

puter, which is actually running the model, and the output ispassed back to the end user to display (www.gradio.appalso serves as a coordinator service between public linksand the SSH tunnels). The amount of time until the end userreceives the output is simply the amount of time it takes formodel inference plus any network latency in sending data.The collaborator can additionally flag data where the outputwas false, which sends the inputs and outputs (along witha message) to the ML researcher’s computer, closing thefeedback loop between researcher and domain expert. (Fig.2d).

4. Pilot Study: Echocardiogram ClassificationWe carried out a user study to understand the usefulness ofGradio in the setting of an ML collaboration. The partici-pants were an ML researcher and a cardiologist who had incollaboration developed an ultrasound classification modelthat could determine whether a pacemaker was present ina patient from a single frame in an ultrasound video. Themodel scored an area under receiver-operating characteristiccurve (AUC) of 0.93 on the binary classification task.

We first asked participants a series of questions to recordtheir typical workflow without using the Gradio library. Wethen taught the participants the Gradio library and let themuse it for collaborative work. After being shown instructionsabout the Gradio library, the ML researcher was able to setup Gradio on a lab server that was running his model. Theprocess of setting up Gradio took about 10 minutes, as someadditional python dependencies needed to be installed onthe lab server. After the installation, the researcher was ableto copy and adapt the standard code from Gradio documen-tation and did not run into any bugs. The ML researcherwas advised to share the model with the cardiologist. Hedid so, and the cardiologist automatically began to test therobustness of the model by inputting his own images andobserving the model’s response. After using Gradio, thecardiologist gained more confidence and trust in the perfor-mance of this particular ML model.

We observed the researchers while they carried out thesetasks, as we sought to answer four research questions:

Q1: How do researchers share data and models withand without Gradio?

Before Gradio, the cardiologist provided the entire datasetof videos to the machine learning researcher in monthlybatches. These batches were the latest set of videos that wereavailable to the cardiologist. The ML researcher would trainthe model on increasingly larger datasets and report metricssuch as classification accuracy and AUC to the cardiologist.Beyond this, there was very little data sharing from thecardiologist and the researcher did not ever share the modelwith the cardiologist.

Gradio: Hassle-Free Sharing and Testing of ML Models in the Wild

Figure 2. A diagram of the steps to share a machine learning model using Gradio. Steps: (a) The machine learning researcher definesthe input and output interface types, and launches the interface either inline or in a new browser tab. (b) The interfaces launches, andoptionally, a public link is created that allows remote collaborators to input their own data into the model. (c) The users of the interfacecan also manipulate the model in natural ways, such as cropping images or obscuring parts of the image. (d) All of the model computationis done by the host (i.e. the computer that called Gradio). The collaborator or user can interact with the model on their browser withoutlocal computation, and can provide real-time feedback (e.g. flagging incorrect answers) which is sent to the host.

With Gradio, the cardiologist opened the link to the modelsent by the ML researcher. Even though it was his firsttime using the model, the cardiologist immediately beganto probe the model by inputting an ultrasound image fromhis desktop into the Gradio model. He chose an imagewhich clearly contained a pacemaker, see Fig. 3(a). Themodel correctly predicted that a pacemaker was present inthe patient. The cardiologist then occluded the pacemakerusing the paint tool built into Gradio, see Fig. 3(b). Aftercompletely occluding the pacemaker, the cardiologist resub-mitted the image; the model switched its prediction to “nopacemaker,” which elicited an audible sigh of relief fromthe ML researcher and cardiologist.

The cardiologist proceeded to choose more difficult images,generally finding that the model correctly determined whena pacemaker was and was not present in the image. He alsooccluded different regions in the image to serve as a compar-ison to occluding the pacemaker. The model performancewas generally found to be accurate and robust; a notableexception was in the case of flipping over the vertical axis,which would generally significantly affect the predictionaccuracy. When this happened, the cardiologist flagged theproblematic images, sending them to the ML researcher’scomputer for further analysis.

Q2: What features of Gradio are most used and mostunused by developers and collaborators?

We found that the machine learning researcher quickly un-derstood the different interfaces available to him for hismodel. He selected the appropriate interface for his model,and set share=True to generate a unique publicly accessiblelink for the model. When he shared the link with the car-diologist, the cardiologist spent a great deal of time tryingdifferent ways to manipulate a few sample images to affectthe model prediction. The cardiologist treated this as a chal-lenge and tried to cause the model to make a mistake in anadversarial manner. The cardiologist also used the flaggingfeature in the cases where the model did make a mistake.

The “share” button which appears at the top of the interfacewas not used; instead, the ML researcher simply copied andpasted the URL to share it with his collaborator. And de-spite the general interest in running model interfaces insideiPython notebooks, our users did not use that feature.

Q3: What kind of model feedback do collaborators pro-vide to developers through the Gradio interface?

The collaborator tested various transformations on the testimages, including changing the orientation of the image tooccluding parts of the image. Whenever this would causethe model to make a mistake on an image, the cardiologistwould flag the image, but would usually pass a blank mes-sage. Thus, it seemed that the collaborator would only sendimages that were misclassified back to the ML researcher.

Gradio: Hassle-Free Sharing and Testing of ML Models in the Wild

(a)

(b)

Figure 3. In our pilot study, the clinician used Gradio to test a model that classified echocardiograms based on the presence of a pacemaker.(a) The clinician submitted his own image of an echocardiagram, similar to the one shown here, and the model correctly predicted that apacemaker was present. (b) The clinician used Gradio’s built-in tools to obscure the pacemaker and the model correctly predicted theabsence of a pacemaker. (The white arrows are included to point out the location of the pacemaker to the reader; they were not present inthe original images).

Q4: What additional features are requested by the de-velopers and collaborators?

Our users verbally requested two features as they were us-ing the model. First, the cardiologist asked if it would bepossible for the ML developer to pre-supply images to theinterface. This way, he would not need to find an ultrasoundimage from his computer, but would be able to choose onefrom a set of images already displayed to him.

Second, the collaborator was used to seeing saliency mapsfor ultrasound images that the ML researcher had gener-ated in previous updates. The collaborator expressed that itwould be very helpful for him to see these saliency maps,especially as he was choosing what areas inside of the imageto occlude.

5. Discussion & Next StepsIn this paper, we describe a Python package that allows ma-chine learning researchers to easily create visual interfacesfor their machine learning models, and share them with col-

laborators. Collaborators are then able to interact with themachine learning models without writing code, and providefeedback to the machine learning researchers. In this way,collaborators and end users can test machine learning mod-els in settings that are realistic and can provide new data tobuild models that work reliably in the wild.

We think this will lower the barrier of accessibility for do-main experts to use machine learning and take a strongerpart in the development cycle of models. At a time whenmachine learning is becoming more and more ubiquitous,the barrier to accessibility is still very high.

We carried out a case study to evaluate the usability and use-fulness of Gradio within an existing collaboration betweenan ML researcher and a cardiologist working on detectingpacemakers in ultrasounds. We were surprised to see thatboth the ML researcher and the domain expert seemed to berelieved when the model worked, as though they expectedit not to. We think because researchers can not manipulateinputs the way a domain expert would, they generally haveless confidence in the model’s robustness in the wild. At

Gradio: Hassle-Free Sharing and Testing of ML Models in the Wild

the same time, because domain experts have not interactedwith the model or used it, they feel the same doubt in itsrobustness. This study was however limited in scope to onepair of users, and a short time. We plan to conduct more us-ability studies and quantitative measures of trust in machinelearning models to get a better holistic view of the usabilityand usefulness of Gradio. Similarly, quantitative measuresof user satisfaction on both the end of machine learningresearcher and domain expert can be used to evaluate theproduct and guide its further development.

The next steps in the development of the package wouldbe creating features for saliency, handling other types ofinputs (ex: tabular data), handling bulk inputs, as well ashelping ML researchers reach domain experts even if theydon’t already have access to them.

Additional documentation about Gradio and example codecan be found at: www.gradio.app.

AcknowledgmentsWe thank all of the machine learning researchers who talkedwith us to help us understand the current difficulties insharing machine learning models with collaborators, andgave us feedback during the development of Gradio. Inparticular, we thank Amirata Ghorbani and David Ouyangfor participating in our pilot study and for sharing theirechocardiogram models using Gradio.

ReferencesBhardwaj, R., Nambiar, A. R., and Dutta, D. A study

of machine learning in healthcare. In 2017 IEEE 41stAnnual Computer Software and Applications Conference(COMPSAC), volume 2, pp. 236–241. IEEE, 2017.

Hertzmann, A. Can computers create art? In Arts, volume 7,pp. 18. Multidisciplinary Digital Publishing Institute,2018.

Klemm, S., Scherzinger, A., Drees, D., and Jiang, X. Barista-a graphical tool for designing and training deep neuralnetworks. arXiv preprint arXiv:1802.04626, 2018.

Muthukrishna, D., Parkinson, D., and Tucker, B. Dash:Deep learning for the automated spectral classifica-tion of supernovae and their hosts. arXiv preprintarXiv:1903.02557, 2019.

Radovic, A., Williams, M., Rousseau, D., Kagan, M., Bona-corsi, D., Himmel, A., Aurisano, A., Terao, K., andWongjirad, T. Machine learning at the energy and in-tensity frontiers of particle physics. Nature, 560(7716):41, 2018.

Thiagarajan, J. J., Rajan, D., and Sattigeri, P. Can deep

clinical models handle real-world domain shifts? arXivpreprint arXiv:1809.07806, 2018.

Xu, K., Guo, S., Cao, N., Gotz, D., Xu, A., Qu, H., Yao, Z.,and Chen, Y. Ecglens: Interactive visual exploration oflarge scale ecg data for arrhythmia detection. In Proceed-ings of the 2018 CHI Conference on Human Factors inComputing Systems, pp. 663. ACM, 2018.

Zou, J., Huss, M., Abid, A., Mohammadi, P., Torkamani, A.,and Telenti, A. A primer on deep learning in genomics.Nature genetics, pp. 1, 2018.