Heartbeat: Measuring Active User Base and Potential User Interest
-
Upload
andrea-wiggins -
Category
Technology
-
view
1.232 -
download
2
description
Transcript of Heartbeat: Measuring Active User Base and Potential User Interest
HeartbeatMeasuring Active User Base
and Potential User Interest
in FLOSS Projects
Andrea Wiggins, James Howison & Kevin Crowston
4 June, 2009
Introduction
• Success measures for FLOSS– Internal versus external - success
according to whom?
• Software usage is a desirable success measure, but difficult to obtain
• Goal: Develop an algorithm to estimate active user base and general interest based on download counts
Measuring Software Use
• Many ways to measure usage– Surveys– Usage reporting agents– Mining online data (downloads)
• Downloads provide a proxy for usage– Must get software before you can use it– Usually FLOSS software is downloaded,
which can be counted
Problems with Downloads
• Downloads often used as direct proxy for usage, but…– Cannot indicate how many downloads
“convert” to actual use– Regular users are counted multiple times
due to release updates– Measures inflated by user experimentation– Only counts one distribution channel– Release rates vary, hard to compare
Hypothesis Development
• Experience-based theory– What is the experience of adopting FLOSS
for end-user applications?– Try it out, adopt it, update it when notified
• H1: There is a relatively constant level of downloads by new users trying out the software
• H2: Regular users respond relatively quickly to new releases
Idealized Release/Download
Grey area: potentialuser downloads
White areas: activeuser downloads
Ideally, we would expect that…- experimentation rate is nearly constant, growing over time- active user base updates after release, growing over time
Data & Analysis
• Daily time series data on package downloadsFLOSSmolehttp://ossmole.sourceforge.net
• Release data for each package
SRDAhttp://zerlot.cse.nd.edu
• Analysis with Taverna
http://taverna.sourceforge.net
Descriptive Results - BibDesk
• Spikes following new releases• Cyclic weekly effects• “Flat” periods between releases• Growth over time in both baseline and spikes
Descriptive Results - SkimApp
• Similar overall patterns• Recently founded, less data• More rapid release cycle than BibDesk• In both projects, occasional non-release spikes
appear - one-time marketing?
Quantifying User Base
• Calculations based on daily downloads for two one-week observation periods centered around release date
• Potential user base: sum of daily downloads before release
• Active user base: sum of daily downloads after release, less the baseline average download rate
Numerical Results - BibDesk
• Consistent baseline experimentation rate• Large variance for installed user base
– Further smoothing might help
• User base may be declining in BibDesk, due to small target audience and competition
Numerical Results - Skim-app
• Stable baseline, but substantial variance in calculated installed base– Big spike in April 2008: first release in 3 months
• Overall trends toward growth in both user base and baseline
Discussion - Limitations
• Download data are problematic for a number of reasons
• Calibrating the measures– Varying the duration of time periods leads
to substantial changes– User response rate varies by project– Very sensitive to release date accuracy– Also difficult to sample releases with
sufficient time in between for baselines
Discussion - Uses
• Generalizability– Assumes swift user response– Different cases for end user versus
enterprise software, varying market sizes
• Use with caution– Examine data for consistent release
response patterns– Either measure can serve as a dependent
variable for project popularity
Future Work
• Compare these findings against more dynamically selected time ranges– e.g. time required to return to a rate close
to the pre-release baseline
• Application to more projects, and comparison against other measures
• Statistical fitting for growth estimates
• May apply to other non-FLOSS downloaded software, e.g. iPhone apps
Conclusions
• Introduced a measure for estimating baseline user interest, and one for active user base in FLOSS projects
• Baseline measure shows good face validity in longitudinal time series
• Active user base measure shows surprising variance
Thanks!
• Questions?
• {awiggins|crowston}@syr.edu, [email protected]• floss.syr.edu• flosshub.org
• Background image derived from photo by Vincent Kaczmarek, http://www.flickr.com/photos/kaczmarekvincent/3263200507/