Scaling up globally:30 years of FOSS4G development

Keynote

Markus Neteler

Fondazione Edmund MachResearch and Innovation CentreDepartment of Biodiversity and Molecular EcologyGIS and Remote Sensing Unit

FOSS4G Central and Eastern Europe 2013

16th - 20th June, National Library of RomaniaBucharest, Romania

PostGISomics

Good afternoon and welcome to my talk with the bold title

Scaling up globally: 30 years of FOSS4G development

I am very happy to be here, and would like to thank the organizers for the kind invitation.

In my presentation I'll briefly review 3 decadesof Open Source GIS development, compressed to 30 minutes! Hence, omissions are unavoidable. BTW: The term FOSS4G was coined only in 2004 but I use it in the title as an hommage to the community and its ideals.

So, let's now look back 30 years back to the 80th...

Img. courtesy: J Westervelt 2006:
Early GRASS Community Views on FOSS

THE 80's the beginning

Well... what's this? Do you recognise it?

THE 80's ...

1980 LAGRID J. Westervelt master's thesis:
GIS software, developed on a mainframe computer,
then ported to Cromemco Z-80. FORTRAN

1982 FHIS (Fort Hood Information System)
Vax 11/780 minicomputer, UNIX, C language. Programmer: L. Van Warren

1983 "GIS Version 1 Reference Manual"
by J. Westervelt and M. O'Shea, 29 July 1983

Included GIS programs:

1983 PROJ4 library development started by Gerald I. Evenden

arctogrip area_stats cell_stats cellmod (grid editor) coin (r.coin) combine (boolean combination) distance (r.buffer) dotmap (graphics on a
dot-matrix printer) erase (d.erase)

griptocell layer_info (r.info) list (g.list) over (d.rast, but for b/w monitor) reclass (r.reclass) sho_over (display images created by over) table (stats associated with over) whats_here (r.info with a mask) window (g.region)

Vax 11/780minicomputer

Hence we start in 1980: A student named Jim Westervelt was completing his Master thesis. He wrote the LAGRID software which became the core of the later GRASS GIS software.In the early 1980s the U.S. Army Corps of Engineers' Construction Engineering Research Laboratory (USA/CERL) in Champaign, Illinois, began to explore the possibilities of using GIS to conduct environmental research, assessments, monitoring and management of lands under the stewardship of the U.S. Department of Defense.

Bill Goran of CERL conducted a survey and discovered that no existing GIS satisfied their needs. Hence CERL hired several programmers, and began to develop a hybrid raster-vector GIS for the VAX UNIX minicomputers (photo). Lynn Van Warren was the founding software architect of GRASS.

Very important: PROJ (today PROJ.4) the Cartographic Projections Library essential for many projects, was started in 1983 by Gerald Evenden

Boolean combination - gridcell and polygon Weighted overlay - gridcell and polygon (reworked later into r.mapcalc) Distance-from (now r.buffer) Isoline generation (grid to poly - now r.contour) Coincident tabulation (now r.coin) Mathematical combination (reworked later into r.mapcalc)THE 80's ...

August, 1983 U. S. Army Corps of Engineers'
Construction Engineering Research Laboratory
(USA/CERL) in Champaign, Illinois

Purchase of first Sun-150 computers (2) specs:$8.900 16 bit, 10Mhz, 256Kb memory, 10Mbps Ethernet,
17" b/w monitor (100*800)

$3.400 1Mbyte extra memory

$2.000 3/4M Fast Sun Memory

$1.590 Barko color monitor

$6.540 80Mbyte hard disk

$1.900 Disk controller board

$5.500 Dot Matrix printer

$1.500 Vanilla UNIX software -or-

$2.000 4.2BSD plus library of graphics software

1983/1984 New analysis capabilities added in GRASS GIS:

Source: http://lists.osgeo.org/pipermail/grass-psc/2012-December/000985.html

33k USD

CERL then upgraded their machines to the latest technology, see what you could get for 33,000 USD.... e.g. including a 80MB disks for 6500 USD.
Around 1984 new analytical functionality was available, some command names at least the GRASS users in this room will be familiar with :-)

Please note that it took still another almost 10 years to have the World Wide Web being invented...

THE 80's ...

1984 GRASS running on SUN-1 and Masscomp

1985 GRASS 1.0, GRASSnet established
(pre-mailing list)

15 March 1985: first commercial Internet domain name registered

1985 Richard Stallman publishes the GNU Manifesto

1987 GRASS 2.0; first issue of GRASSClippings Newsletter GRASS video narrated by William Shatner (Captain Kirk of Star Trek)

1988 GRASS 3.0; Army R&D Achievement Award (Webster, Goran, Shapiro, Westervelt)

In the corner photo you see what data exchange meant in those days. Landsat 5 was launched in 1984, GRASS 1.0 was released a year later.

Of huge importance for the Free Software development was the publication of the GNU manifesto by Richard Stallman (the four freedoms were defined).

In 1987 a GRASS video was produced with William Shatner, known as Captain Kirk of Star Trek, being the speaker.

And 5.25 inch floppies were becoming fashion in these days for tiny data sets!

THE 80's ...

1989 GRASS 3.1
First release available on Internet
(uxc.cso.uiuc.edu)

But how was software developed? Locally!

While Revision Control System (RCS) was available and
sometimes used, yet no server based system like CVS(CVS 1.0 in 1990 today: SVN, git, )

TCP/IP goes global (19892010)

However:
By 1992, still less than 15,000 .com domains registered...

See also: http://grass.osgeo.org/home/history/

http://wiki.osgeo.org/wiki/Open_Source_GIS_History

http://grass.osgeo.org/uploads/grass/history_docs/westervelt2004_GRASS_roots.pdf

And then finally an initial internet release of Open Source GIS software! Still there was almost nobody online in those days, at least not able to download 100 MB of source code...

The big spread of internet happened due to the acceptance of TCP/IP as protocol.Information exchange happened via FTP.

Today all geeks are sharing code via SVN or git or other online code sharing repositories. But back in the 80th there was no online system like this. Either manual management or at most RCS which is file based.

THE 90's visualization
and analysis

Source: Helena Mitasovahttp://skagit.meas.ncsu.edu/~helena/gmslab/gsoils/vizrep2.html

So, the first decade of Open Source GIS concluded with the online publication of software (yet with a tiny user group being also online).

Consider to think back when you went online the first time!

Anyway, let us hop into the next decade:
the 90th!

A lot happened here, new hardware, new software, and way more data!

And especially the advent of the internet society was a social revolution.

THE 90's ...

1990 GRASS 4.0 Letter-dot format adopted for commands (e.g. d.rast and g.region)

1991 First mailing lists

In 1990 GRASS 4.0 came out, a key release which was the foundation for the GRASS' architecture.

On the social side, two mailing lists were started, the grass-user and the grass-dev lists. Those are probably the longest, active mailing lists of the internet with more than 2 decades of archive!

Still no WWW, but information exchange happened FTP and Gopher.

The photo shows core developers:- Michael Shapiro- Jim Westervelt- Bill Goran (the coordinator)

THE 90's ...

1992 New graphics technology became available: openGL

Software distribution:Moon FTP server of CERL

(note: still 2 years to wait for WWW!)

SG3D

SGI Indigo

With the availability of new graphics CPUs a completely new way of rendering became possible. OpenGL was designed with hardware acceleration.

Like this, fly-thoughs as seen in the left figure as well as 3D and even 4D animations came to life (here, water contamination modeled as voxels which are changing in time).

I remember from working as a student at the Hannover CeBIT computer fair, to enjoy the demos of the SGI machines while carrying around newspapers across the fair area.

THE 90's ...

ORGANIZATIONS

1992GRASS Inter-Agency Coordinating Committee Open GRASS Foundation (OGF) OpenGIS Consortium (OGC)

1994 Tim Berners-Lee decided to constitute the
World Wide Web Consortium (W3C)

1994 Open Geospatial Consortium
(OGC) founded

Open GIS Open Geospatial
Consortium (OGC) Consortium (OGC) GRASS Interagency
Steering Commitee1990

1992

2006

Open GRASS
Foundation (OGF)1994

2004

OSGeo

At the level of organizations many changes occurred: the GIS world became more professional and organized.

In 1992 the GRASS Interagency Coordinating Commitee was founded, it was turned into the Open GRASS Foundation which then became the OpenGIS Consortium.

The timeline shows the evolution.

While Tim Berners-Lee constituted the World Wide Web Consortium, the today's OGC was established.

THE 90's ...

http://grass.osgeo.org/home/history/documents/

The rationale behind the establishing of the OGC you find described in an article written by Kenn Gardels. It got published in the GRASSCLIPPINGS issue of Fall 1993.

Here the fact that interoperability is a core issue is pointed out.

BTW: If you are interested, the scan is online, all links in my presentation which I'll upload later, are pointing to the respective documents.

Source:http://news.bbc.co.uk/2/hi/technology/8552410.stm

THE 90's Internet growth as a crucial
precursor to FOSS4G development

This map shows the internet accessibility in the year 1998 people being online in percent.

As you see, in those days internet was yet restricted to a few countries in terms of accessibility.

And, obviously:without internet no distributed source code development nor easy geodata exchange.

Do you remember:
30 slackware disks downloaded,but unfortunately in FTP ASCII
mode (7 bit, not 8)??

THE 90's ...

First Web Applications come up, finally!

1993 Xerox PARC Map Viewer

1994 GNU/Linux 1.0 released

1995 First GRASS 4.1.5 port to Linux
(Andreas Holz, Greifswald, Germany)

1995 UMN MapServer project started

http://wiki.osgeo.org/wiki/Open_Source_GIS_History

1995 Well, and also the first spam email in the GRASS mailing list

In the year 1993 the first Web application came up, the Xerox PARC map viewer
[refer to the previous talk of Maria Brovelli]

In 1994 the 1.0 version of Linux was released and shortly after GRASS GIS ported to it. This was still a tough job since the compiler tools where not as mature as nowadays.

The year 1995 was a key year for the WebGIS community with the start of the Mapserver project.

And... wow, a first spam email reached the ML

THE 90's ...

An interactive GIS via WWW, perhaps the first Web Processing service...

1994-1998 Susan M. Huse's GRASSLinks (PhD thesis at Berkeley)

What happened in the WebGIS sector:

Probably less known is the this first Web Processing style service which was published back in 1994.

It was an interactive GIS controllable through the WWW, developed by Susan Huse based on her PhD thesis.

This system was able to ...

THE 90's ...

GRASSLinks

perform true analysis in a user controllable way using CGI scripts.

Of course it is not comparable with anything you can do today but remember the internet accessibility map of 1998 which I have shown earlier.

We are still in the prototype years and draft WPS specifications were published only almost a decade later.

THE 90's ...

Internet, its tools and geeks spreading!

1996 GeoTools project started

1998 deegree (originally JaGo) development started
with an OGC Simple Features implementation

1998 GDAL/OGR development started

1998 First European GRASS GIS server at ILN, Uni Hannover, Germany

1999 GRASS GIS source code moved from manual management to CVS,
precisely on 29 Dec. 1999 :-)

With the advent of the collaborative internet based development tools, a series of new projects like GeoTools, deegree, GDAL/OGR were started.

In 1998 I started the European GRASS GIS server, completely non-authorized of course (except for the ILN landscape architecture institute at the university)... it turned out to be a good idea to take control.

BTW: Anyone here remembering the year 2000 bug?It stimulated us to move the GRASS code from manual code management via email to something more sophisticated called CVS. This was a server based code repository and checkins became independent of a single person who had to read emails and act upon.

Since 2000...
May the FOSS be with you

Well, we happily survived the Year 2000 bug and reached the next decade...

Now Communities growing together!

This drawing has been done by the older daugther of Venka (Venkatesh Raghavan in Osaka) for the initial 2004 FOSS4G conference in Bangkok which I'll mention shortly

2001 OSSIM initial revision in CVS PostGIS started GeoNetwork opensource started GeoServer started2002 Quantum GIS initial revision in CVS GEOS initial revision in CVS2003 Community MapBuilder started. End of life in 2008 Release of Mapbender under the GNU GPL license gvSIG was started 2004 uDig was started2005 MapGuide Open Source2006 Mapbender gets first bits in CVS OpenLayers Started2007 GeoMoose was open sourced (started 2005)2009 rasdaman was open sourced (started 1995)

Since 2000... growing communities

Thanks to CVS,
SVN and git2006: OSGeo!

http://wiki.osgeo.org/wiki/Open_Source_GIS_History

In just a few years a lot of new software projects were started, I don't have the time to illustrate them in details. You see that all adopted the collaborative tools for development.

From the demand to get coordination among these projects the idea arose to create an umbrella foundation, OSGeo, as illustrated by Jeff McKenna earlier today.

Since 2000...growing communities

Chulalongkorn University,Bangkok 2004:
FOSS4G is born!

Jim Westervelt

Special thanks toVenkatesh Raghavan,Osaca City University

For the term FOSS4G we have to thank Venka once more. He proposed the conference title in 2004 for the conference at Chulalongkorn University.

Also Jim Westervelt came to give us an authentic lecture about the early days at this meeting.

Since 2000...growing communities

QGIS Hackfest Pisa 2010

Mapbender
code sprint 2007

FOSS4G Conferences:Lausanne, Denver,Victoria, Cape Town, Sydney,
Barcelona, Denver, ...

The next slides will be much more familiar for you... code sprint photos

Here the mapbender team in 2007 and the QGIS hackfest in Pisa in 2010.

Community sprints, even more...

For those not familiar with it:- it is a gathering of likeminded people- no need to be a developer- newcomers are there up to core developers

Instead of writing 10 emails to discuss an issue, we just discuss it directly, rarely long with some beer

And: The outcome of such a week is often impressive.

Next chance for you: on Thursday this week!

CommunicationFlow of bug reporting and solution:

2. Developer
detects bug

60%

20%

20%

(Percentages are estimated)

Other
developers

New
release3. New
feature

SVN code
RepositoryReleasecycle

1. User sends bug reportBugtrackerMailing list

20%

80%

Organization of distributed source code
management: Code habitats

Two main types of developers may be identified:generalist

specialist (the majority)

It appears that many developer assign themselves to
code habitats, i.e. their area of expertise (e.g., in GRASS GIS
a selection of libraries or topics which they maintain)

These code habitats remain often stable over years

There are also partially abandoned code areas (~ 10% of the code?)
which are functional but aren't really getting improved

A few garbage collectors (generalists) fix lots of odds 'n ends

Organization of distributed source code
management: Code habitats

http://www.youtube.com/watch?v=suyDqmGXoWk

GRASS GIS 6.4 development visualization from 1999 to 2011 with Gource

Code vetting

Legal aspectsLicense compliance (e.g., GPL)No code copying from books like Numerical Receipes in CEnsure that 3rd party contributions are cleanEmployers must agree that work time is spent

Full transparency and peer review help to minimize the risk.

Apache or OSGeo FoundationIncubation phaseGraduation

http://incubator.apache.org/

http://www.osgeo.org/incubator

Since 2010...

Source: Blog of Arnulf Christl

Emergency support

Haiti, January 2010:
7.0 magnitude earthquake

Available Geodata:
almost absentSee also: http://hot.openstreetmap.org/projects/haiti-2

Haiti Dec 2009

Open Data: OpenStreetMap.org

After these years of community based software development, the emerge of Wikipedia also crowd mapping came up.

No need to explain OpenStreetMap here but for sure the Haiti earthquake example is to be mentioned.

It is illustrating an outstanding effort of people around the globe gathering virtually together and helping in an emergency situation.

The map shows the geodata situation in January 2010 when a magnitude 7 earthquake occurred almost no geodata available which were urgently needed.

A few days later ..: international crowd mapping

See also: http://hot.openstreetmap.org/projects/haiti-2

Haiti Jan 2010

Open Data: OpenStreetMap.org

This is the situation a few days later. Based on high resolution satellite data which were made available by the big data providers, the OSM community was able to generate in an extremely short time a detailed coverage of digital data.

A similar effort was done after the Great East Japan Earthquake 2011, you will remember it.

New: Crowdfunding of development

The increasing demand for geospatial data in the Web browser and increasing interactivity naturally requires the adoption of new emerging technologies.

An interesting new method of fundraining, instead of direct sponsoring by a few is after crowd knowledge (Wikipedia) and crowd mapping (OSM) now crowd funding where till a deadline a certain funding goal should be achieved.

Example: OpenLayers 3 funding, aiming at more than 300,000 USD and they are almost there!

New cool stuff: plotting GIS data

From

Radar

to

voxels

to

3D plots

There are also new possibilities for plotting maps: from 2D to 3D using rapid prototyping technologies which have been directly connected to Open Source GIS.

Here an example of the polar ice cap of Mars (so, no need to restrict yourself to mother Earth):

The processing chain was: from Radar remote sensing data to voxels to 3D plots

New cool stuff:
massive data processing

Since 2005 GRASS GIS is running natively on 64bit CPUs

GRASS GIS 7 also offers Large File Support on 32bit Windows

Installed on Grids and TOP500 supercomputers (AKKA Ume,
ENEA Frascati, Aurel Bratislava, )

Runs on Linux, AIX, Solaris, freeBSD, netBSD, ...

Various ways of parallelization

Since we like to think big, why to getting our tools on Supercomputers running...

Indeed, we are already there:PROJ4, GDAL and GRASS GIS are available even on TOP500 systems.

The harder part was to get it compiled on Non-Linux Supercomputers :-)

For massive data processing currently job managers are supported. Work is underway for GPU based clusters which require a major source code restructuring.

Massive data processing:
also for you

https://plus.google.com/u/0/communities/111147786674687562495

Since the home computer
only has 8 GB of RAM...8.5 billion points...Since not everybody has a supercomputer reachable, a lot of efforts have been done on processing massive data on even consumer hardware.

Here an example from a power user who managed to process 8.5 billion lidar points on his home computer.

With this touching base I would like to conclude my quick ride through 30 years of Open Source GIS development and conclude with...

Markus Neteler

Fondazione E. Mach (FEM)Centro Ricerca e InnovazioneGIS and Remote Sensing Unit38010 S. Michele all'Adige (Trento), Italyhttp://gis.cri.fmach.ithttp://www.osgeo.org

markus.neteler@fmach.itneteler@osgeo.org

Concluding remark:

THANKS TO ALL CONTRIBUTORS!
And more to come...

PostGISomics

Happy birthday, GRASS GIS

And

Thanks to all FOSS4G contributors all over the world.

Thank you for listening

Backup slides following

THE 90's ...

http://gis.cri.fmach.it/markus/grass_web_evolution/

FOSS Software development structures

Organizational structures of development teams

Developers

Head

Power users

Linux Kernel

GRASS Development Team/
GRASS Project Steering Committee

Decision level

+

-

SVN write
access

patch/docscontribution

Developers

User community

GRASS: No true BDFL (Benevolent
Dictator For Life)

http://en.wikipedia.org/wiki/Benevolent_Dictator_for_Life

http://producingoss.com/

Since 2010... new cool stuff

GIS on USB sticks

Mobile GIS

GIS today... beyond Desktop GIS

GIS Cloud computing

WebGIS

Since 2010...

Since 2010...

Markus Neteler 2013, CC-BY-SA