8/8/2019 Web 2.0 and Estate Maps
1/87
Web 2.0 and estateMaps
Final Year Project
BEng Software Engineering with Industrial Experience
The University of Manchester
Author: Imran Sarwar
Supervisor: Sean Bechhofer
April, 2008
8/8/2019 Web 2.0 and Estate Maps
2/87
Imran Sarwar 2 SEwI.E Final year Project
Disclaimer
No portion of the work referred to in the dissertation has been submitted in support of an application
for another degree or qualification of this or any other university or other institution of learning.
8/8/2019 Web 2.0 and Estate Maps
3/87
Imran Sarwar 3 SEwI.E Final year Project
Abstract
The World Wide Web (WWW) is a well known phenomenon which enables billions of users to
connect to an information pool used and updated by users across the globe. Closely linked with the
WWW is a fairly new and loosely defined concept called Web 2.0 which is defined as the second
generation of the WWW.
This report details the development of my hybrid application which exploits a combination of freely
available online services, to create a lightweight mashup application. The application also provides
information concerning the surrounding area of the property, creating a rich and informative
experience for the end user.
Included in this report are the project requirements, design, implementation, testing and the
background surrounding Web 2.0 and the different technologies which are associated with it.
8/8/2019 Web 2.0 and Estate Maps
4/87
Imran Sarwar 4 SEwI.E Final year Project
Web 2.0 is a set of economic, social, and technology trends that collectively form the basis for
the next generation of the Interneta more mature ,distinctive medium characterized by user
participation, openness, and network effects.
~ Tim O'Reilly
8/8/2019 Web 2.0 and Estate Maps
5/87
Imran Sarwar 5 SEwI.E Final year Project
Acknowledgements
I would like to thank my supervisor Sean Bechhofer for his guidance and direction throughout the
project. I would also like to thank my parents and family members for their support and advice during
the project.
I would like to say special thanks to my friends Asif, Michael and Shehzad for their support during the
project and keeping me in focus during the final year.
8/8/2019 Web 2.0 and Estate Maps
6/87
Imran Sarwar 6 SEwI.E Final year Project
Table of ContentsList of figures.........................................................................................................................................9
1. Introduction..................................................................................................................................11
1.1 Aims and Objectives ................................................................................................................ 11
1.2 Deliverables ............................................................................................................................. 12
1.3 Report Structure ....................................................................................................................... 12
2. Background ..................................................................................................................................13
2.1 Existing services ...................................................................................................................... 13
2.2 Putting Web 2.0 into context.................................................................................................... 14
2.3 Related Technologies............................................................................................................... 15
2.3.1 Web Services ..................................................................................................................... 15
2.3.2 AJAX ................................................................................................................................. 17
2.3.3 Web feeds.......................................................................................................................... 19
2.3.4 XML.................................................................................................................................. 20
2.3.5 Mashup.............................................................................................................................. 20
2.3.6 RSS and GeoRSS............................................................................................................... 22
2.3.7 XSLT.................................................................................................................................. 23
2.3.7 Google Maps ...................................................................................................................... 24
2.3.8 Nestoria .............................................................................................................................. 24
2.3.9 Flickr .................................................................................................................................. 25
2.3.10 PlanningAlerts................................................................................................................... 253. Design............................................................................................................................................27
3.1 Requirements ........................................................................................................................... 27
3.1.1 Functional requirements..................................................................................................... 27
3.1.2 Non-functional Requirements ........................................................................................... 28
3.2 Design Methodology................................................................................................................ 28
3.3 System Architecture................................................................................................................. 303.3.1 Presentation logic............................................................................................................... 30
8/8/2019 Web 2.0 and Estate Maps
7/87
Imran Sarwar 7 SEwI.E Final year Project
3.3.2 Processing Logic ................................................................................................................ 32
3.3.3 Data Logic.......................................................................................................................... 32
3.3.4 Architecture design decisions ............................................................................................ 32
3.4 Problems .................................................................................................................................. 33
3.4.1 JavaScript security restrictions........................................................................................... 33
3.4.2 Location and crime statistics data ..................................................................................... 34
4. Implementation ............................................................................................................................36
4.1 Programming Language - The Microsoft vs. Java syndrome .................................................. 36
4.2 Development Environment ....................................................................................................... 36
4.3 Development tools .................................................................................................................... 37
4.4 Server Side Implementation...................................................................................................... 37
4.5 Server Code exploration ........................................................................................................... 40
4.5.1 Reading HTTP Requests.................................................................................................... 40
4.5.2 Processing a search request................................................................................................ 40
4.5.3 Processing a Graph Request............................................................................................... 45
4.5.4 Locating Flickr images....................................................................................................... 47
4.4.6 Processing the search bar ................................................................................................... 47
4.5 Client side code......................................................................................................................... 48
4.5.1 Google Maps instillation................................................................................................... 48
4.5.2 AJAX Requests and Response Caching............................................................................ 49
4.5.3 Plotting houses on the Google Map ................................................................................. 50
Predictive search bar ..................................................................................................................... 53
5. Testing...........................................................................................................................................54
5.1 Debugging................................................................................................................................. 54
5.2 White Box Testing .................................................................................................................... 54
5.3 Black box Testing ..................................................................................................................... 55
6. The Result .....................................................................................................................................57
6.1 A typical search request ............................................................................................................. 57
8/8/2019 Web 2.0 and Estate Maps
8/87
Imran Sarwar 8 SEwI.E Final year Project
7. Conclusion ....................................................................................................................................62
7.1 Evaluation of the Implementation............................................................................................. 62
7.2 The future and extra functionality............................................................................................ 63
7.3 My Learning.............................................................................................................................. 65
8. Appendices....................................................................................................................................66
Appendix A - Development Plan ...................................................................................................... 67
Appendix B - Initial Requirements ................................................................................................... 68
Appendix C - XSLT file used to transform a house search request.................................................. 70
Appendix D - A typical XML file generated by the XSLT engine................................................... 71
Appendix E - A typical XML file for a house search request which is returned to the client .......... 75
Appendix F Mock-up interfaces .................................................................................................... 81
Mock-up 1..................................................................................................................................... 81
Mock-up 2..................................................................................................................................... 82
Appendix G Zillow test class ......................................................................................................... 83
References............................................................................................................................................84
8/8/2019 Web 2.0 and Estate Maps
9/87
Imran Sarwar 9 SEwI.E Final year Project
List of figures
FIGURE 1 - Trends in mashups [1].......................................................................................................13
FIGURE 2 Nestoria interface [11]......................................................................................................14
FIGURE 3 - Web services architecture [4] .........................................................................................16
FIGURE 4 - Simple REST example. ...................................................................................................17
FIGURE 5 - The traditional model vs. the AJAX approach [5]. ..................................................18
FIGURE 6 - XML syntax Example [20]..............................................................................................20
FIGURE 7 - Mashup programming inputs and outputs [12]................................................................21
FIGURE 8 - RSS 2.0 specification [13].................................................................................................22
FIGURE 9 - GeoRSS specification [14]................................................................................................23
FIGURE 10 - Template processor [16]..................................................................................................23
FIGURE 11 - XSL code to convert GeoRSS to XML...........................................................................24
FIGURE 12 - Iterative design approach [26].........................................................................................29
FIGURE 13 - EstateMaps architecture overview. .................................................................................30
FIGURE 14 - Predictive search bar. ......................................................................................................31
FIGURE 15 - Possible client side architecture. .....................................................................................33
FIGURE 16 - Accessing external services.............................................................................................34
FIGURE 17 - EstateMaps database structure.........................................................................................35
FIGURE 18 - UML class interaction diagram.......................................................................................38
FIGURE 19 - Reading a transformed house XML file..........................................................................42
FIGURE 20 - Process graph for reading a transformed house XML file...............................................43
FIGURE 21 - Double Array graph structure..........................................................................................46
FIGURE 22 - Process graph for the client side caching.........................................................................49
8/8/2019 Web 2.0 and Estate Maps
10/87
Imran Sarwar 10 SEwI.E Final year Project
FIGURE 23 - Use case for plotting items on the Google map...............................................................51
FIGURE 24 - Tabbed Google Bubble....................................................................................................52
FIGURE 25 - Control flow graph for processing a House XML file on the client side........................55
FIGURE 26 - Black box testing results..................................................................................................56
FIGURE 27 - EstateMaps initial GUI....................................................................................................57
FIGURE 28 - EstateMaps loading screen..............................................................................................58
FIGURE 29 - House information displayed on the estateMaps GUI.....................................................58
FIGURE 30 - Crime statistics displayed on the estateMaps GUI..........................................................59
FIGURE 31 Flickr images displayed on the estateMaps GUI............................................................59
FIGURE 32 Searching the Google map..............................................................................................60
FIGURE 33 Displaying planning alerts on the estateMaps GUI........................................................61
FIGURE 34 Average house prices on the estateMaps GUI................................................................61
8/8/2019 Web 2.0 and Estate Maps
11/87
Imran Sarwar 11 SEwI.E Final year Project
1. Introduction
The World Wide Web (WWW) is a fast and ever expanding market enabling users to communicate
and collaborate ideas on a global scale. A new and loosely defined concept has recently become the de
facto standard in creating web applications and websites. This new era of the internet is termed as
Web 2.0 where the user is a potential stakeholder in the website which was first conceptualised with
the increased trends in social networking and information sharing.
Many organisations such as Microsoft, Yahoo and Google have adopted the Web 2.0 principle by
making their data and services freely available to the public. These services are accessed via APIs
(Application Programming Interface) using a combination of technologies such as AJAX and XML
which are discussed in sections 2.3.2 and 2.3.4 respectively.
This report details the development of my hybrid/mashup application called estateMaps which uses a
number of freely available online services to create a rich and informative experience for the end user.
The report also describes the basic background and concepts of Web 2.0 and the technologies which
are associated with estateMaps such as REST, SOAP, XML and AJAX.
1.1 Aims and Objectives
The objectives of my project are to exploit the concepts of Web 2.0 and create an application which is
able to display saleable real estate geographically on a world map. The first objective identified for
estateMaps was to understand the different APIs and how data would have to be manipulated and
extracted from the different information sources. Many of the third party services use XML as their
primary language to manipulate and dispatch data when a request is generated from the service. This
required research into the structure of XML and GeoRSS documents which are discussed in sections
2.3.4 and 2.3.6 respectively.
The second objective was to join the different information sources into one single lightweight
application which as mentioned is termed a hybrid/mashup application. This required further research
into the different technologies available such as JavaScript and AJAX which have both been discussed
in the Background section of this report.
The high level objectives can be seen as;
Develop a Java servlet which can explicitly call third party services and generate response
based on the users request.
Create an application which is loosely coupled and easily upgradable.
8/8/2019 Web 2.0 and Estate Maps
12/87
Imran Sarwar 12 SEwI.E Final year Project
Investigate the concepts of Web 2.0 and the technologies which surround this subject.
Develop an application which meets the requirements defined in section 3.1.
1.2 DeliverablesThe developed artefact will provide users with the ability to search for saleable real estate across the
UK with information about the surrounding area. This information will include crime statistics about
the region, average house prices, planning requests in the area and images relevant to the location.
The application will be formed as a fully operational web based application allowing the user to
navigate the features in a simplistic and familiar environment. Real estate will be displayed on a
geographical map with markers to represent the location of each property.
1.3 Report Structure
The report consists of several distinct chapters, each chapter builds upon the last which is geared
towards the implementation and evaluation of the project.
Chapter 2 discuses the background of some of the technologies which are used within the context of
Web 2.0 and are used in the implementation of the project.
The Design and the implementation of the application are discussed in Chapter 3 and 4. Chapter 3
describes the challenges associated with estateMaps and the architectural design. Chapter 4 describesthe implementation of the application and how the core components of the application are
implemented into the system.
Chapter 5 tests the system against the initial system requirements and other various testing techniques
such as Black and White Box testing. Chapter 6 ends the implementation and aims to evaluate the
effectiveness of my application against other products available on the market.
Finally Chapter 7 outlines the future of Web 2.0 and the possibilities of Web 3.0 already on the
horizon. Chapter 7 also outlines the direction and manifestation which the internet is taking.
8/8/2019 Web 2.0 and Estate Maps
13/87
Imran Sarwar 13 SEwI.E Final year Project
2. BackgroundThis chapter is split into three distinct sections. The first section takes a brief look at some of the
existing services which are already available and how estateMaps differentiates from them. The
second section defines this idea of web 2.0 and how it affects the WWW in its current state. Finally
the last section lays out the technologies which are linked to the application domain of Web 2.0 and
those which are potentially used during the implementation phase of the project.
2.1 Existing services
With the increased trends in Web 2.0, there are thousands of existing mashups and many more being
created at an exceptional rate to create a new experience for the end user. Figure 1 shows the vogue in
mashup programming, these statistics are provided from ProgrammableWeb [1] a site which lists thelatest mashups and web services.
Figure 1 Trends in mashups [1].
Looking at the statistics shown in figure 1, there seems to be an increased interest in using mapping
services at this moment in time. Real estate is a relatively niche market and has not be been exploited
to its full potential. Combining real estate and a mapping service seems be a good basis for mashup
service hence the idea of estateMaps was formed.
There are many services which are able to list real estate geographically on a mapping service one of
which is the Nestoria [11] web service. Nestoria is of particular interest to estateMaps because it is the
main data driving force for the application. The Nestoria web service is mentioned in greater detail
during the third section of this chapter. Figure 2 shows the interface of the Nestoria web application it
provides the user with the key functionality of estateMaps but lacks information such as information
relevant to the search location and recent planning requests in the area.
8/8/2019 Web 2.0 and Estate Maps
14/87
Imran Sarwar 14 SEwI.E Final year Project
Figure 2 Nestoria interface [11].
I also researched a number of real estate agents to see if they could offer a service which was as rich
and interactive as estateMaps, but to no prevail. Main&Main [10] which is a popular real estate agent
in the UK offers a similar service but has less functionality than both Nestoria and estateMaps.
Main&Main do not provide any information in regards to the surrounding area, their primary goal is to
inform the user of the location of the property utilising a Google map.
No examples could be found at the time of writing this report which uses AJAX to dynamically update
the map when the users make a new search request based in the UK. This keeps in context with the
idea of Web 2.0 to create applications which is interactive for the user and feels as if though it is
running seamlessly on their local machine.
2.2 Putting Web 2.0 into context
Web 2.0 is perceived to be the second generation of the internet, which aims to get the end user
involved in the web experience. The idea of Web 2.0 is not to change the current infrastructure of the
WWW, but to change the way developers and users interact with WebPages.
There are no governing specifications which classify a Web 2.0 application instead a number of
publishers such as Tim OReilly president of O'Reilly Media have published their views on what
characteristics classify a Web 2.0 application.
OReilly published an interesting article titled What Is Web 2.0 [2]. In this article OReilly mentions
the seven key competences of web 2.0 which have been summarised below.
8/8/2019 Web 2.0 and Estate Maps
15/87
Imran Sarwar 15 SEwI.E Final year Project
The Web As Platform - Using the browser to run rich and interactive applications just as
though they are running natively on the users machine.
Harnessing Collective Intelligence Allowing users to become potential stakeholders in the
website by enabling them to add and manipulate the data such as blogs and Wikipedia.
Data is the Next Intel Inside The data is the key driving force of the application not from
one source but from many.
End of the Software Release Cycle Using the Internet as base operating system to run
applications so users dont have to install or update the software. The person/company
providing the service will update the application which separates the concern from the user.
Lightweight Programming Models The program should adopt a loosely coupled and flexibleapproach which allows other components and even different applications to utilise its
functionality. Using lightweight programming models means we can reach more people to
enable data collection and a more intelligent web [2].
Software Above the Level of a Single Device The application should not be restricted to one
platform it should be able to take advantage of all technologies available today and in the
future.
Rich User Experiences Allowing the user to take control over the web site and providing theuser with an experience which is interesting, innovative and informative in order to create ICT
as effective tool.
Looking at these characteristics we can already identify that there are number of practical uses for
Web 2.0 applications such as forums, blogs and even price comparison websites amongst a host of
others. EstateMaps adheres to the ideas put forward by OReilly, but the key feature can be seen as
integrating data from multiple information sources.
2.3 Related Technologies
2.3.1 Web Services
Web services are at the focal point of the web 2.0 concept; they enable developers to create
Lightweight Programming Models which as mentioned is one of the fundamental concepts of web
2.0. The World Wide Web Consortium (W3C) [3] defines a web service as
....A Web service is a software system designed to support interoperable machine-
to-machine interaction over a network. It has an interface described in a machine-
8/8/2019 Web 2.0 and Estate Maps
16/87
Imran Sarwar 16 SEwI.E Final year Project
processable format (specifically WSDL). Other systems interact with the Web service
in a manner prescribed by its description using SOAP-messages, typically conveyed
using HTTP with an XML serialization in conjunction with other Web-related
standards. [3]
In simple terms a web service can be described as self-contained and self-describing components that
can be published, discovered, and invoked across the network.
A web service is normally provided in the form of an API, an interface which allows a web browser or
server to interact with another application. The computational logic is usually calculated on the host
which is providing the web service which explicitly separates the two applications. The main
protocols or architectures used for web services are SOAP and REST.
SOAP (Simple Object Access Protocol) is a computer protocol used for exchanging XML basedmessages over the internet. SOAP allows computer programs to communicate on different operating
systems, different technologies and programming languages. WSDL (Web Service Definition
Language) is used to describe the methods and the location of a web service. [3].
Figure 3 - Web services architecture [4].
The typical SOAP architecture is shown in figure 4, it captures how data is transmitted from the
requester to the provider. The service requester sends a SOAP message in the XML format to the
provider which instantiates a WSDL binding (connection) which the requester can use to invoke the
methods or functions of the provider.
Representational State Transfer (REST) is not a protocol instead it is termed as an architectural style
[6]. It exploits the current HTTP /HTTPS protocol in order to send requests to a web server in the
8/8/2019 Web 2.0 and Estate Maps
17/87
Imran Sarwar 17 SEwI.E Final year Project
form of a URL (Uniform Resource Locator). HTTP is the set of rules for exchanging files (text,
graphic images, sound, video, and other multimedia files) on the WWW [10].
Each element on the internet is seen a resource and the URL is the resource locator [5]. The
representation is the information that the requester is interested in. The resource or data is sent back from the web server in a number of different standards which include XML, HTML, GIF etc.
Figure 4 highlights a simple GET request using the basic principles of REST. The starting portion of
the URL is the location of the resource i.e. the server. The second portion of the URL is the name of
the resource which the user wishes to retrieve, in this case an XML file called Text. The server
processes the second portion of the URL in order to send back the resource to the user. Both sending
and receiving the request exploits the HTTP protocol as illustrated in the example.
Figure 4 Simple REST example.
REST is of particular importance to estateMaps because a number of the third party services used
within the application use this architecture.
2.3.2 AJAX
Asynchronous JavaScript and XML (AJAX) is a relatively new concept which is able to transform the
way static WebPages are updated and refreshed in an intuitive manner. Following the principles of
Web 2.0 AJAX enables developers to create a new experience for the end user, WebPages with a
seamless flow of data exchange [7].
AJAX is not a new Scripting language instead it manipulates existing standards to create faster, more
reliably and friendlier user applications. AJAX allows portions of a website to be updated without the
need to update the entire page. This creates the perception that the application is actually running on
the local machine rather than a service over the WWW.
AJAX uses JavaScript as the scripting language of choice to instantiate AJAX function calls. XML is
the standardised way to retrieve data using the AJAX libraries. Data does not necessarily have to be in
HTTP GET / www.example.com/text
HTTP Response / XML
Resource
Resource
Location
http://www.example.com/texthttp://www.example.com/text8/8/2019 Web 2.0 and Estate Maps
18/87
Imran Sarwar 18 SEwI.E Final year Project
the XML format other formats are supported such as HTML (Hyper Text Mark-up Language, JSON
(JavaScript Object Notation) or even plain text.
Figure 5 - The traditional model vs. the AJAX approach [7].
Figure 1 illustrates the difference between the classic web application and the AJAX enabled
approach. The classic web application model generates a new HTML document each time the user
interface sends a request to the server. This method is expensive in regards to time and bandwidth
both to the client and the service provider as a whole page needs to be generated for the smallest of
changes to the client side [8].
The AJAX model uses JavaScript to call a new component which has been added to the model. The
AJAX engine acts as a channel between the web server and the JavaScript running on the client side.
The JavaScript is unable to call a web server directly so the AJAX engine is used to send a HTTP
request to the server which generates response based on the users request. The response is often in the
XML format which can then directly be interpreted by the JavaScript running on the client side. The
JavaScript is able to manipulate the page dynamically without the need for the page being refreshed
creating an effect that response has been generated by the users machine.
8/8/2019 Web 2.0 and Estate Maps
19/87
Imran Sarwar 19 SEwI.E Final year Project
While this interactivity is being carried out in the background the user still has access to the parts of
the page which are not being updated. This method often results in faster response times and less
bandwidth being utilised as all presentation details have been excluded from the response.
There are main advantages to using the AJAX approach, but every new technology also comes with itsassociated disadvantages, AJAX is also prone to this syndrome. The major disadvantage in using
AJAX is compatibility issues between the ways certain web browsers handle the AJAX libraries. In
internet explorer 6 and below the XMLHttpRequest object is not recognised as a valid JavaScript
function instead the ActiveXObject object has to be used in order intuitively call the web server [8].
This leaves the programmer with many cumbersome validation checks to carry out before the AJAX
functions are actually used. Fortunately Internet explorer 7 now includes the support for the
XMLHttpRequest as AJAX becomes more of standardised protocol but the implications still exist for
users using older browsers.
One of the other main concerns of AJAX is network latency between calls to the server. Without
proper consideration for the size of the data being sent it could potentially leave users waiting longer
than usual for the interface to be displayed due to bottlenecks and other network related issues. This
may be confusing for some users and in turn they may end the session before the data is actually
loaded. Another implication is the when data is loaded between server calls the page layout may be
adjusted while the user is still interacting with the data.
If AJAX is used in a thought out manner it can be used as powerful tool to create a fully fledged
application over the WWW. I see AJAX as a fundamental part of Web 2.0 and as its popularity
increases it will become a day to day experience for all Web users.
2.3.3 Web feeds
Web feeds are a data format to allow users to view regularly updated information in an effective and
simplistic manner [9]. Keeping in context with the web 2.0 paradigm web feeds allow users and
developers to utilise data from more than one information source to create a unique data resource.
Typical applications of web feeds are to provide up to date information of a service such news or
weather.
These feeds are often in the XML which in such means that they were never intended to be readable in
their raw format. This gives developers a great degree of flexibility as most modern programming
languages have XML readers/parsers which can be explicitly used to manipulate and interoperate the
feed. Each feed consists of one or more elements; an element is an object of interest which contains
such information as a title, description and a link to the information provider [9]. The two main web
feed formats are RSS and Atom which are discussed later in this section.
8/8/2019 Web 2.0 and Estate Maps
20/87
8/8/2019 Web 2.0 and Estate Maps
21/87
Imran Sarwar 21 SEwI.E Final year Project
reuse and extend them by using the web as a platform. The result of reusing and extending such
resources is known as the mashup concept [21].
Figure 5 shows the typical inputs and platforms which are associated with mashup applications. The
figure demonstrates how mashup applications are not only intended for developers it also involves thenormal daily users to interact and create their own mashups using such platforms as Facebook [22]
which allows users to create their own content and widgets [23]. It also emphasizes the division of
skill-level between the developers and normal users and how developers can reuse existing resources
to create new and interesting applications.
Figure 7 Mashup programming inputs and outputs [12].
The three key benefits of the mashup applications can be seen as;
Effective leverage of Web parts As mentioned Mashups are built on existing services,
adding code when it can't be sourced from internal or external suppliers or to provide
integration "glue" between the parts [12].
Simple, lightweight software models and services Mashups are typically built using bits of
code sourced from other vendors which usually develop all major libraries required and carry
out the computational logic. Typical examples include Google Maps or a YouTube video
player, such services originally required a massive investment from its creator [12].
8/8/2019 Web 2.0 and Estate Maps
22/87
Imran Sarwar 22 SEwI.E Final year Project
A focus on self-service and DIY Mashup development is not just geared towards developers
it enables normal web users to create their own applications that could never have been
justified on a build-vs.-buy perspective. These apps can now just be created by users and
groups of collaborating users on the fly as they need them [12].
2.3.6 RSS and GeoRSS
RSS (Really Simple Syndication) is a type of web feed used to display regularly updated information
such as news or weather updates. RSS utilises the XML syntax but it only uses a predefined number of
tags in order to record the data for each element within the feed. These tags can differ depending on
the specification of RSS used. The current standard is RSS 2.0 [13]; its Predecessors include Rich Site
Summary (RSS 0.91, RSS 1.0) and RDF Site Summary (RSS 0.9 and 1.0). Figure 8 shows an example
of a minimalistic RSS 2.0 feed.
Tag Name Description RSS 2.0 ExampleTitle The title of the item.
Example Channelhttp://example.com/My example
News for September the Secondhttp://example.com/2002/09/01other things
News for September the Firsthttp://example.com/2002/09/02
Link The URL of the item.Description The item synopsis.There are also a number of optional tags which canalso be used within the item tag these include; Author Category Comments Enclosure Guid pubDate source Image
Figure 8 RSS 2.0 specification [13].
GeoRSS [14] is essentially the same thing as RSS 2.0 except it contains an extra tag which defines the
geospatial location which is relevant to each item within the feed. It also has a few other predefined
tags which are not part of the RSS 2.0 specification. This information keeps in context with the Web
2.0, information with meaning. GeoRSS is of particular importance to estateMaps because the
information for the planning alerts is actually received in this format. The planning alerts element of
the system is discussed in section 2.3.10. Figure 9 shows the structure of a GeoRSS document, the
example is actually taken from a document which estateMaps is able to process.
8/8/2019 Web 2.0 and Estate Maps
23/87
Imran Sarwar 23 SEwI.E Final year Project
Tag Name Description GeoRSS Examplegeorss:point Contains the latitude and longitude
for the item in question. Thelatitude and longitude a separatedby a empty character.
....
98 Brantingham Road M16 8LZTue, 20 Nov 2007085029 ...53.44 2.25...........
...
georss:featurename Used to represent the name of theitem.
Figure 9 GeoRSS specification [14].
2.3.7 XSLT
XSLT (Extensible Stylesheet Language Transformations) is also another technology which utilises the
XML syntax. It enables one XML document to be transformed into another for example XML to RSS
or XML to HTML. XSLT can be considered to be a template processor. A template processor (also
known as a template engine or a template parser) is software or a software component that is designed
to combine one or more templates [16] to produce a single resulting document. The XSLT parser
reads the structure of the XML document to manipulate the document as specified in the XSLT file.
XSLT processing is a W3C [15] standard. All major web browsers incorporate a XSLT engine along
with all major programming languages such as Java and Microsoft C#. Figure 10 shows a high level
diagram of how template processing is actually carried out.
Figure 10 Template processor [16].
EstateMaps also incorporates a XSLT engine which is used to transform all XML documents from the
third party services into the estateMaps XML format. The design decision is discussed more during
the third section of this report. Figure 11 shows a snippet from a XSLT document which estateMaps
uses to transform GeoRSS to XML.
8/8/2019 Web 2.0 and Estate Maps
24/87
Imran Sarwar 24 SEwI.E Final year Project
Figure 11 XSL code to convert GeoRSS to XML.
2.3.7 Google Maps
Google Maps [17] is a free web service which enables developers to integrate a fully interactive world
map on their web page. This map can be manipulated and navigated by the user in real time. A Google
Map has a number of advanced features some of which include;
Search for businesses and attractions.
Generate driving directions between any pair of locations.
Integrate your own data on the map.
Display information using interactive markers.
Full Ajax support.
One of the most interesting features of Google Maps is that it is able to reverse geocode for a given
address or postcode. Reverse geocoding is the process whereby an address can be mapped to its
geospatial equivalent. Unfortunately this service is rather temperamental and does not always give an
exact location for the given address hence estateMaps has its own geocoded postal information which
is discussed in section 3.4.2.
Google Maps is an integral component of estateMaps as a majority of the processing and interactions
by the user is carried out on the map. Section 3 demonstrates how the Google Maps service is actuallyintegrated into estateMaps.
2.3.8 Nestoria
As mentioned during section 2.1 Nestoria [11] is the data driving force for estateMaps. Nestoria is a
free web service which is able to query over thirty estate agents across the UK and produce an
automated reply in the XML format. Nestoria utilises the REST architecture in order for the public to
access their data and services which is explained during section 2.2.1.
8/8/2019 Web 2.0 and Estate Maps
25/87
Imran Sarwar 25 SEwI.E Final year Project
Nestoria is in no way linked to selling property, they are purely a search engine and information
provider, in essence Nestoria is also a mashup application. Nestoria extracts their data from various
sources such as fish4homes [24] and pass on this information to the public. The information which
Nestoria pass on is limited for example the description of the property only contains the first two
hundred characters, but the most important data such has number of bed rooms and price are all
directly accessible. Nestoria limit the data in order to generate revenue from people accessing the
external links which are also accessible in the XML feed.
2.3.9 Flickr
Just like Google Maps and Nestoria, Flickr [18] is a freely accessible web API which enables the
public to access a library of pictures which are uploaded by its users. Flickr is considered to be one of
the earliest web 2.0 applications enabling users to share and distribute their pictures across the
internet. Unlike Nestoria, Flickr supports both the REST and SOAP architecture.
An interesting feature of Flickr and integral part of estateMaps is that flicker supports geocoding of
their images which means images can be searched by location. Previous search techniques involved
looking at the tags associated with the image which could be skewed by misinterpretations of the data.
Flickr is used in estateMaps to allow users to see scenic images which are relevant to their search
criteria by querying the entire Flickr library of images.
2.3.10 PlanningAlertsPlanningAlerts [19] is a web based API which is able to collate planning information from a number
of UK local authority planning websites. When a house developer or owner wishes to make drastic
improvements to their house they have to get approval from their local authority. Nearby residents are
able to view these planning requests and make appeals if they feel that request will affect them in any
way.
EstateMaps is able to use the information from PlanningAlerts and plot the information on a Google
Map. Users are able to see planning requests which are relevant to their search request which may
prove useful to see what type of improvements are being accepted in that area.
The information from PlanningAlerts is provided in the GeoRSS format, this means that each element
within the feed contains geospatial information as described in 2.2.6. PlanningAlerts uses the REST
architecture as described in 2.2.1.
PlanningAlerts receives its data by a process called Scraping. Scraping is the process whereby an
application reads the structure of a document i.e. HTML and extracts any relevant information.
PlanningAlerts has number of these scrapers which read the HTML of every local authority in the UK
8/8/2019 Web 2.0 and Estate Maps
26/87
Imran Sarwar 26 SEwI.E Final year Project
and extract the data on a periodic basis. PlanningAlerts is then able to dynamically generate GeoRSS
feeds based on the users request.
8/8/2019 Web 2.0 and Estate Maps
27/87
Imran Sarwar 27 SEwI.E Final year Project
3. DesignThis chapter details the design process of estateMaps. The first section defines the functional and non-
functional requirement which were defined for the system. The second section explains the design
methodology which was used to create estateMaps and the high level system architecture. This chapter
also details how the various third parties services are incorporated into estateMaps.
3.1 Requirements
In order to achieve the project objectives a list of functional and non-functional requirements were
drawn up. The requirements were formed from the preliminary project objectives and from the initial
research phase of the project. These requirements underline the fundamental operations which the end
application must fulfil.
3.1.1 Functional requirements
Functional requirements depict the internal operations and the behaviour of the application; they are
not concerned with the design or any performance related issues of the application.
FREQ1. The application must be able to utilise the Nestoria API in order to display properties
for sale geographically on a world map.
FREQ2. The application should allow users to search for propertys based on multiple searchconditions.
FREQ3. The house description, price and picture should be displayed where the information is
available.
FREQ4. Crime statistics about the area in which the property lies within must be displayed to
the user.
FREQ5. The application should allow multiple houses to be displayed on the map where
appropriate.
FREQ6. Caching techniques will be implemented into the application in order to reduce the
number of responses which will be generated by the server.
FREQ7. The application must be able to generate an informative response if the client does
not have all the required plug-ins into order to view the website.
FREQ8. The application should be able to display average house prices.
8/8/2019 Web 2.0 and Estate Maps
28/87
Imran Sarwar 28 SEwI.E Final year Project
3.1.2 Non-functional Requirements
Non-functional requirements express the constraints upon the implementation of the application such
as performance and security related issues.
NFREQ1. The application should follow good HCI principles which will make it easier for theuser to navigate the website.
NFREQ2. The application must be able to deal with the potential unreliability of third party
services.
NFREQ3. The server should be able to generate responses concurrently based on multiple client
sessions.
NFREQ4. The application should be built in a modular design to allow for extensibility in thefuture and to cater for unreliability of third party services.
NFREQ5. The performance of the application is dependent upon a number of factors which
include the bandwidth of the user, time to generate the response, speed of the third
party services and the bandwidth available to site is host.
Many psychology studies have concluded that eight seconds, plus or minus two is the
longest which a user a user will wait before they end the session [25]. Based on this
study the application must generate a response within the ten second barrier.
NFREQ6. The application must be able to cater for cross browser compatibility in order to reach
all potential users.
3.2 Design Methodology
In order to create estateMaps the iterative/incremental design cycle was chosen. The iterative design
cycle is best suited for creating fast prototype applications or better known as beta applications. The
iterative design cycle consists of one more iterations/cycles. Each iteration consists of five phaseswhich can be modified in order to meet the requirements of the application.
Requirements This process was used to gather the functional and non functional
requirements for estateMaps. The requirements can be found in section 3.1.
Analysis & Design The analysis and design phase was combined into one phase for
estateMaps. In this section the system architecture was agreed and mock-up interfaces were
created to see how the finished application would look. The mock up interfaces can be found
in Appendix F.
8/8/2019 Web 2.0 and Estate Maps
29/87
Imran Sarwar 29 SEwI.E Final year Project
Implementation Coding was initiated for estateMaps, the coding is discussed in chapter 4
Testing In this phase the application was put through black box testing. The application was
tested against a number of predefined tests. The testing is discussed in more detail in chapter
5.
Evaluation The program was evaluated and success of the application was measured against
the initial requirements.
It was decided during the preliminary planning of the project that only two iterations were needed in
order to create estateMaps because of the time limitations of the project. The development plan can be
found in appendix A.
The advantage of using the iterative approach is that incremental updates can be made to theapplication throughout its lifespan. Another advantage of using the methodology is that feedback is
received at the end of each iteration which means the application can be reassessed against its initial
requirements. At the end of each iteration the program is deployed to its intended market. In some
cases the application is actually deployed to the end users with minimal testing so that the users
become the testers. Figure 12 shows the iterative model which was the exact process which was used
to create estateMaps.
Managing and changing requirements is one of the most expensive characteristics in modern software
engineering. Fortunately the iterative approach allows for requirements to be changed or altered after
each iteration. During the second and final iteration of estateMaps it was decided that the application
should be able to display average house prices. Appendix B shows the initial requirements where as
the requirements in section 3.1 shows the requirements which were amended during the second
iteration.
Figure 12 Iterative design approach [26].
8/8/2019 Web 2.0 and Estate Maps
30/87
Imran Sarwar 30 SEwI.E Final year Project
Using feedback means that there is less risk for developers and organisations in creating applications
as all resources are not spent in one go. With the standard waterfall lifecycle every step within the
development process is done in a sequence and all effort is tied towards making one final application.
Many major originations have also adopted this beta approach such as Google with Google Maps and
Microsoft with Live search [46].
EstateMaps also used this notion of risk so the most complex components of the system were actually
made during the first iteration where as the second iteration was used for perfective programming.
EstateMaps can still be seen as a beta application and functionality can be added as requirements
change.
3.3 System Architecture
EstateMaps is effectively a web based application which utilises the basic three tier architecture. Thethree tier architecture consists of three main components which are the presentation logic, processing
logic and Data logic. Figure 13 demonstrates a high level overview of the architecture used within
estateMaps and the interactions between the major components.
Figure 13 EstateMaps architecture overview.
3.3.1 Presentation logic
A majority of the presentation logic is actually carried out locally on the users machine. The main
interface into the application is the HTML file called Index, as illustrated the diagram. Index is a static
HTML file which resides on the servlet. When the user interacts with the applications GUI (Graphic
User Interface) the Index.html file is updated, but the page is never refreshed. Components on the
page are dynamically removed or added as the user interacts with the application using JavaScript.
8/8/2019 Web 2.0 and Estate Maps
31/87
8/8/2019 Web 2.0 and Estate Maps
32/87
8/8/2019 Web 2.0 and Estate Maps
33/87
Imran Sarwar 33 SEwI.E Final year Project
It was decided that a fully fledged client side application would not be suitable for estateMaps mainly
because of the security restrictions of JavaScript and possible limitations of the client machines. Even
though these requests to the third party services are relatively small in size, the time required to
process them and fetch them is dependent on the clients machine and bandwidth. As mentioned in the
requirements many studies have found that eight seconds, plus or minus two is the longest a user
would wait for a typical website to load. Keeping this in mind it was decided that the three tier
architecture would be more suitable and stable for estateMaps as most of the processing and
bandwidth activities are passed to the server.
Carrying out a majority of the processing on the client side would have resulted in slow response
times and large memory requirements because of the amount of XML processing which is required.
Using the three tier architecture enables future developments to be carried out in a controlled
environment without any restrictions from the client. Figure 15 illustrates a possible client sidearchitecture for estateMaps.
Figure 15 Possible client side architecture.
3.4 Problems
3.4.1 JavaScript security restrictions
During the initial design phase and background research of estateMaps it was identified that it was not
possible to create the system as a client side application. The design decision behind this was the
security restrictions in JavaScript. JavaScript does not allow access to external servers except for that
server where the JavaScript files are hosted, which means it was not possible to send requests to the
services utilised by estateMaps. However because Google Maps use the SOAP architecture it was still
possible to send and receive requests because of the architectural style of SOAP. Hence it was decided
that a servlet was in fact needed and would be beneficial in other factors as discussed in section 4.1.Figure 16 shows a high level overview of the problem just discussed and how it was overcome.
8/8/2019 Web 2.0 and Estate Maps
34/87
8/8/2019 Web 2.0 and Estate Maps
35/87
Imran Sarwar 35 SEwI.E Final year Project
This data was provided as both a Microsoft Excel document and SQL statements, so it could be easily
transferred into a database. Given this data it was decided that a database was needed so that the
system could validate the post code or city inputted by the user. This data would also provide useful
for mapping grid references to the crime statistics data which discussed below.
3.4.2.2 Crime Statistics Data
The UK government publish crime statistics [30] data every year which record the number of
robberies and violent attacks in every major city. This data is provided in a Microsoft excel sheet and
is available freely to the public. The problem with the data is that it is not particularly friendly and
cannot be directly transferred into a database. The second problem with the data is that there is no
geospatial data which reflects the location of the city.
Thankfully the postcode and address data maps every city to a postcode and a latitude/longitude asdiscussed in section 3.4.2.1. This information is used to map crime statistics to each postcode and city
in the UK. The crime statistics data only provides information for cities so each postcode is actually
mapped to its nearest city
Crime_DataField Type Null
crime_ID Int(5) NoForce Text NoCity Text NoPopulation figures Int(11) No
Violence_against_person_06 Int(11) NoViolence_against_person_07 Int(11) NoSexual_offences_06 Int(11) NoSexual_offences_07 Int(11) NoRobbery_offences_06 Int(11) NoRobbery offences_07 Int(11) NoBurglar_ dwelling_06 Int(11) NoBurglary_dwelling_07 Int(11) NoTheft_of_a_motor_vehicle_06 Int(11) NoTheft_of_a_motor_vehicle_07 Int(11) NoTheft_from_a_vehicle_06 Int(11) NoTheft_from_a_vehicle_07 Int(11) No
Figure 17 EstateMaps database structure.
Post_Data
Field Type NullPostcode Text NoDistrict Text NoCity (FK) Text NoX_ref Int(11) NoY_ref Int(11) NoLatitude Double NoLongitude Double NoPostcode_ID Int(5) NoCrime_ID (FK) Int(5) No
1:1
1:N
1:1
1:N
8/8/2019 Web 2.0 and Estate Maps
36/87
Imran Sarwar 36 SEwI.E Final year Project
4. ImplementationThis chapter examines the implementation details of estateMaps. The various components of the
system are described in detail along with supporting diagrams which help further illustrate system
functionality. This chapter also details some of the tools used to create estateMaps.
4.1 Programming Language - The Microsoft vs. Java syndrome
In order to the implement the web server a number of different technologies could have been used
such as PHP or ASP. It was decided due to the time restrictions of the project that a technology which
I had prior experience should be considered amongst others. The two technologies which were being
considered were Microsoft C# with ASP and Java with Apache Tomcat [31]. Java was chosen for a
number of reasons, some of which are detailed below. Due to greater development experience with Java opposed to C# it was decided that more time
could be spent adding functionality and making the code more elegant rather than learning
the C# platform.
Java supports a number of free libraries which can be used to generate graphs which have to
be paid with its Microsoft equivalent.
Java supports cross platform compatibility where as Microsoft .net only has limited support in
Linux and no support in Max OSX.
PHP was also another interesting option but due to no prior scripting experience with the technology
the learning curve may have been too great.
4.2 Development Environment
Once the programming language was decided upon a suitable development environment was needed.
After some research into the different IDEs (Integrated development environment) the freely
available Netbeans [32] was chosen due to its integration with the Apache Tomcat servlet. This allows
the program to be easily tested and deployed with information and logs about the processing being
carried out by the servlet. Using the integrated server also allows the application to be deployed not
just in a test environment but also across the internet allowing multiple users to connect. Netbeans also
has a number of interesting tools which can assist with the testing of the program:
HTTP Monitor Allows the user to see information about the HTTP requests sent and
received and other useful information such as data about the processing servlet, performance
figures and the request headers.
http://.net/http://.net/8/8/2019 Web 2.0 and Estate Maps
37/87
Imran Sarwar 37 SEwI.E Final year Project
Code Refactoring Renames all variables, class location and class name if amended by the
user in the entire program.
Database Connection Monitor allows easy testing of a database connection and can be used
to see statistical information about a request such as performance ad number of requests.
4.3 Development tools
To aid in the development of estateMaps a number of tools were used. Below is a list of all the tools
used in the system and a brief description of its usage.
Netbeans As discussed in the section 4.1.2 Netbeans is used as the programming
environment for estateMaps. Netbeans is used to code the HTML, JavaScript and the server
side Java classes. Netbeans was also used to generate the UML diagrams used to explain someelements of the system in this report. Netbeans also assisted in testing the system as discussed
in chapter 5.
Apache Tomcat Tomcat is the server side container used to send and receive requests to the
servlet using the HTTP protocol.
MySQL MySQL is a local database used to store various information such postcode,
location and crime statistics data.
SQLYog SQLYog [33] is a frontend application (GUI) for MySQL which makes it easier to
interact with MySQL client.
4.4 Server Side Implementation
Figure 18 shows a UML dependency diagram for the Java server side code, which illustrates how each
class interacts with the other classes within the program. The main decision making class and entry
point into the application is the getServerResponse class. Each of the classes and their functions are
detailed on the following pages.
8/8/2019 Web 2.0 and Estate Maps
38/87
8/8/2019 Web 2.0 and Estate Maps
39/87
Imran Sarwar 39 SEwI.E Final year Project
getServerResponse - As mentioned during the introduction to this chapter this class is the
main entry point and decision making class in the program. Its main purpose it to decide what
response needs to be generated based on the headers sent from the HTTP request.
House This class is used to store information about each house which has been found fromthe Nestoria web Service. It contains a simple constructor which sets the various variables
within the class. There are many GET methods which can be used to retrieve a particular
piece of information for a house object.
HouseXMLParse The request which the Nestoria web service generates is in the XML
format. This class process the generated XML request from the Nestoria web service and
generates a list of House objects.
XsltEngine XsltEngine is used to communicate with the external third party services. It
takes a XSL file and URL as its inputs. The URL is the location of the resource which needs
to be fetched. The URL is streamed across the internet and the response is saved. The
response is then translated into the estateMaps XML format using the input XSL file.
ImageFinder This class is used to retrieve a locationID from the Flickr web service to
retrieve images relevant to a particular area. Flickr uses a unique ID for each city or postcode
in the UK. For example the postcode M16 corresponds to keGop2.YA5qp3lp39g and
Manchester Corresponds to cTto9E.bCZ7g_w". The XML response is also parsed by thisclass. The imageFinder functionality is discussed in more detail during this chapter.
GraphGenerator This class is used to generate a dynamic graph for the crime statistics data.
The class utilises the freely available JOpenChart Toolkit [39].
HouseXMLGeneration When a new house search request is sent to the servlet a response is
expected in the XML format. This class is used to generate a XML file from a list of house
objects which are processed by the HouseXMLParse class.
PlanLocater The PlanLocater class is used to retrieve planning Information for a particular
area. It utilises the XsltEngine to convert a XML stream into the estateMaps XML planning
format.
DatabaseLink The database link class is used to create JDBC MySQL connection to access
the estateMaps database. The class creates a connection to the database and is also able to
destroy a connection once it is no longer needed.
8/8/2019 Web 2.0 and Estate Maps
40/87
Imran Sarwar 40 SEwI.E Final year Project
BulidAutoCompXML This classes primary function is to query the estateMaps database in
order to find locations which could match the input text. The class is able to generate a XML
file which contains the possible matches in order to populate the predictive search bar on the
GUI.
4.5 Server Code exploration
This section explores how the various functions of the server side code are implemented and actually
work.
4.5.1 Reading HTTP Requests
In order to determine what request needs to be generated the servlet needs to be able to read the HTTP
request String. When the client side JavaScript sends a request to the server it adds a number of
parameters to the end of the URL. The request string can contain parameters with an associated value.
EstateMaps has four key requests which the server is able to process. These requests are illustrated
below with typical parameters which may be contained in the request.
..........
reqType=getHouse&locIn=Manchester&bedNum=Any&numRes=10&saleType=buy&priceVal=AnyreqType=goWidget&uIn=Manchester&saleType=buyreqType=goGraph&vi07=5891&rob07=718&sexu07=reqType=goWidget&uIn=Manchester&saleType=buyreqType=goPlan&uIn=Manchester
..........
The server side code which reads the result string is actually directly accessible using built in methods
of the Apache Tomcat servlet. The code below illustrated how the server side code is able to read the
location of the search and the request type.
.......... //Gets the location Parameter from the HTTP call associated with everycall from the JSString location = spaceRemove(request.getParameter("uIn"));
//HTTP request typeString reqType = request.getParameter("reqType");
..........
4.5.2 Processing a search request
When a user makes a new search request to locate new homes a number of events occur. Once the
request string has been processed and the relevant parameters associated with the request have been
stored, the Nestoria REST URL is manipulated with these parameters to retrieve the XML resource.
The next step is to retrieve the data from the Nestoria web service and convert the result it to the
estateMaps XML format. This is done by creating a new instance of the XsltEngine.
8/8/2019 Web 2.0 and Estate Maps
41/87
Imran Sarwar 41 SEwI.E Final year Project
.................String xslIn = " estateMaps/estate_assets/house.xsl";XsltEngine getXML = new XsltEngine(xmlIn,xslIn);StringWriter convXML = getXML.transformXslt();
.................
The XSLT engine takes a URL and XSL file as its input. The XSL file is a static file which resides on
the servlet which is used to transform the document into the estateMaps XML format. The XsltEngine
streams the URL across the internet and stores the response in a temporary character stream
(StringWriter) . This stream is then converted using the inbuilt Java XSLT transformer which is also
known as TransformerFactory.
.................public StringWriter transformXslt(){
try{
xmlSource = new StreamSource(xmlUrl.openStream());
xsltSource = new StreamSource(xsltUrl.openStream());trans = transFact.newTransformer(xsltSource);trans.transform(xmlSource, result);
}catch (TransformerException ex){
ex.printStackTrace();}catch (IOException ex){
ex.printStackTrace();}return resultString;
}.................
The reason why I decided that a XSLT engine would be useful is that even if any of the third partyservices decide to change the structure of their XML feeds it means the internal operations of the
program will not have to be changed. The only part of the application which will have to be changed
is the XSL file. This means that the program will not have to be recompiled which us usually means
that the service has to be stopped. A snippet of the house XSL file can be seen below and the full
version can be found in Appendix C.
.................
.................
Select each listing in theNestoria XML file
Set the output type
Select title tag for thecurrent item
Write these tags to the file
Read the URL
Read the XSL file(local loop back)
Transform the file
Exception handling in case of errors
generated from the transformation
Location of the XSL file
New instance of the XsltEngine
Transform the file
http://www.w3.org/1999/XSL/Transformhttp://www.w3.org/1999/XSL/Transform8/8/2019 Web 2.0 and Estate Maps
42/87
Imran Sarwar 42 SEwI.E Final year Project
The result which is generated from the transform XSLT method is then sent back to the
getServerResponse class. Now that the Nestoria XML has been transformed into the estateMaps
format it needs to be parsed so each element within XML feed can be processed. There are a number
of different implementations in Java for XML parsers some of which include SAX [34], Xerces [36]
and JDOM [35]. SAX is particularly good at processing smaller sized documents without too many
layers in the document hence SAX is used within estateMaps. There is a dedicated class for parsing
this XML feed which is called HouseXmlParse.
HouseXmlParse has one specific purpose which is to read the XML feed and add each house element
into an Array List. The SAX parser is instantiated by creating a new instance of the
DocumentBuilderFactory. The DocumentBuilderFactory Defines a factory API that enables
applications to obtain a parser that produces DOM object trees from XML documents [37]. DOM is a
W3C standard which enables such documents as XML and HTML to be dynamically accessed andupdate the content, structure and style of the document [38]. The code below illustrates how the
DocumentBuilderFactory is instantiated.
................DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();DocumentBuilder db = dbf.newDocumentBuilder();dom = db.parse(new InputSource(new StringReader(xmlIn)));
.................
Once the DocumentBuilderFactory has a DOM representation of the document you are able to
directly access tags within the document. The pseudo code for reading the estateMaps converted XMLdocument is shown below. A full listing of a typical XML document generated by the XSLT engine
can be found in Appendix D.
Pseudo Code Actual code
Read all the elements of the fileGet the nodes of the Channel tag
If (the channel element has something inside){
For each element within Channel{
Get the first element of the channel tagIf (the item is not empty){
For each element within the item{
Process the current itemCreate a house objectAdd the house to the array
}}
}}
Element docEle = dom.getDocumentElement();NodeList channel = docEle.getElementsByTagName("channel");
if(channel != null && channel.getLength() > 0){
for(int i = 0 ; i < channel.getLength();i++){
Element el = (Element)channel.item(i);
NodeList p = el.getChildNodes();if(channel != null && channel.getLength() > 0){
//Iterates the channel tag find each item.for(int j = 0 ; j < p.getLength();j++){
/*Process the item in the XML document and append* the arrayList*/ Element currentItem = (Element)p.item(j);
listOfHouse.add(createHouseObj(currentItem.getChildNodes()));}
}}
}
Figure 19 Reading a transformed house XML file.
8/8/2019 Web 2.0 and Estate Maps
43/87
Imran Sarwar 43 SEwI.E Final year Project
When the house element has been found in the document it is actually passed to another method
called createHouseObj. This method is used to read that particular house element in the document and
create a House object out of the data. A snippet of the code which examines the house element and
creates the house object is shown below. Figure 20 also illustrates this process diagrammatically.
.................private House createHouseObj(NodeList nodeIn){
//Iterate the XML itemfor(int g = 0 ; g < nodeIn.getLength();g++){
Element e = (Element)nodeIn.item(g);String nodeName = e.getNodeName();
//Check the node name and set the data which is applicable.if(nodeName.equalsIgnoreCase("title")){
try{
//get the title of the housetitle = e.getTextContent();
}}
//Create a house object from the data parsed.House currHouse = new House(title, description, bed, bath, price, type, type_price, link,
image, lat, longi); //return the house object.return currHouse;
.................
Figure 20 Process graph for reading a transformed house XML file.
The house class does not just store the information from the feed it also adds the crime statistics data
relevant to that property when the house object is created. This is done by retrieving the location of
that property and querying the estateMaps database in order to find the nearest city to that location.
Once the result set is retrieved from the database this information is also stored alongside the
information from the feed. The house class has a number of getter and setter methods which makes it
Gets the Title attributes value and stores it in a
temporary variable. This is done for each attribute in the
feed, until are attributes are found.
Attributes received from the XML feed are made into a
House object
8/8/2019 Web 2.0 and Estate Maps
44/87
Imran Sarwar 44 SEwI.E Final year Project
easy to retrieve information from the object. The code below shows the method in the house which
retrieves the crime statics data from the database.
................ //Method to set crime stats.public void setCrimeStats(String postcodeIn){
connectionThis = new DataBaseLink(); //creates new database connectionStatement statement = null;ResultSet rs = null;Try{
Connection newMysqlCon = connectionThis.connection;statement = newMysqlCon.createStatement(); //get the crime databased on postcoders = statement.executeQuery("SELECT *` FROM estatemaps`.post_data p,`estatemaps`.crime_data c WHERE c.crime_ID = p.crime_ID and p.postcode=\""+postcodeIn+"\"");
................
Now that the Array List has been populated with the house objects from the XML feed a new XML
file must be generated with the added crime statistics data to be sent back as the response. This is
handled by the class called HouseXmlGeneration.
................ //getServerResponse class instantiating the HouseXmlGeneration to build the new XML Feed.HouseXmlGeneration sd = new HouseXmlGeneration(HouseXmlObj.getHouses())String res = sd.bulidXML();
................
HouseXmlGeneration accepts a generic Array List with the type set as House. It iterates through each
house element in the Array and builds a XML file as string. Once the XML is generated it is sentback to the waiting client side code.
................ //Public default constructor.public HouseXmlGeneration(ArrayList listOfHouse){
listToGen = new ArrayList(); //Initiate arrayif (listOfHouse.size() > 0){
//Copy the input array to the class array.listToGen = listOfHouse;
} //Initiate the result string with the XML starting tag.
xmlRepresentation = "";}
//Method to build the string based on the estataMaps schema.public String bulidXML(){
xmlRepresentation =xmlRepresentation + "";xmlRepresentation =xmlRepresentation + "";
/*iterate through each of the house objects in the array, get the house*Data via the accessor methods and append the result string.*/
for (int i=0;i
8/8/2019 Web 2.0 and Estate Maps
45/87
Imran Sarwar 45 SEwI.E Final year Project
different XSL files are used transform the XML feeds. Also there is no need for the encapsulating
class such as the House class because no additional information is added for these services.
Essentially the resource is retrieved via the XsltEngine and transformed then sent back to the client
for processing.
4.5.3 Processing a Graph Request
One of the challenges in creating estateMaps is how would the crime statistics data be displayed to the
user. One of the options was to display them as raw text but because there are so many figures it
looked confusing and unreadable because they contained both the 2006 and 2007 comparisons. It was
decided that a graph would be the best possible way to display this data.
When a graph request is sent to the server side code, the crime statistics data is actually stored in the
HTTP request string as discussed in section 4.5.1. As discussed in section 4.5.2 the crime statisticsdata are actually sent to the client side when a house search request is generated. These crime
statistics data once again are actually sent back to the server side in order to generate the graph.
The reason why this design decision was taken is because instead of re-querying the database it was
actually much faster to send the statistics back in the request String. The second reason behind this is
that a graph is only generated when the user selects to see these figures for that particular property, so
they are not generated when a house search request is processed. These graphs could have been
generated when the house search request was being processed but because of the time required to
generate each graph it often consumed too many resources on the server and took too long.
The graph request is actually handled to a similar fashion as the house search request. The get
ServerResponse first processes the HTTP request which contains the crime statistics data. This data is
then passed to a dedicated class to create graphs called GraphGenerator.
.................
//Instance to the GraphGenerator classGraphGenerator graphIns = new GraphGenerator();
/*Creates a graph and stores the filename and location to the
*"filename" String*/ String filename = graphIns.graphGen(violence_against_person06, robbery_offences06,sexual_offences06, burglary_dwelling06, theft_motor06,theft_from_motor06, violence_against_person07,robbery_offences07, sexual_offences07,burglary_dwelling07, theft_motor07, theft_from_motor07);
.................
Once the GraphGenerator class receives these figures it populates a double array which is used by the
freely JOpenChart Toolkit to generate a graph. This graph is actually generated as a JPG image.
Figure 21 illustrates the structure of the double array.
Passes the variables to the GrapgGenerator,which sends back a filename to the newly
created graph.
8/8/2019 Web 2.0 and Estate Maps
46/87
Imran Sarwar 46 SEwI.E Final year Project
Figure 21 Double Array graph structure.
In order to create the graph, a new ObjectChartDataModel must be instantiated. The
ObjectChartDataModel is actually the entry point into the JOpenChart Toolkit. It takes a Double
Array, Array of columns, Array of rows and a chart title as its arguments. The graph is actually stored
in a temporary location on the servlet ready to be sent to the client. The code below illustrates how the
chart is initialised.
.................
int[][] model ={
{violence_against_person06, robbery_offences06, sexual_offences06,burglary_dwelling06,theft_motor06, theft_from_motor06
},{
violence_against_person07,robbery_offences07,sexual_offences07,burglary_dwelling07,theft_motor07,theft_from_motor07
}};
String[] columns = {"Robbery offences", "Burglary dwelling offences", "Sexual offences", "Theft from avehicle", "Theft of a motor vehicle offences", "Violence against the person offences"};
String[] rows = {"2006", "2007"};
String Graphtitle = "Crime statistics 2006 vs 2007";
ObjectChartDataModel data = new ObjectChartDataModel(model, columns, rowstry{
//Export the graph to a JPGFileOutputStream storeLocation = new FileOutputStream(("\\graph_store\\"+fiName));ChartEncoder.createJPEG((storeLocation), chartObject);
//Return the filename of the graphreturn fiName;
}.................
Now that graph is has been stored in a temporary location the graph is sent back to the waiting client
byte by byte as it is not possible to send a jpg file back directly. The file is read in chunks of 1024
bytes which is sent to the client in pieces and is regenerated by the clients browser.
.................
FileInputStream in = new FileInputStream(file);byte[] buf = new byte[1024];
int count = 0;while ((count = in.read(buf)) >= 0){
out.write(buf, 0, count);}in.close();
.................
New Double Array to store both the 2006and 2007 crime statistics.
Array used to represent the columns thegraph.
Creates the new graph model.
Exports the graph to JPG image file.
Reads the file 1024 bytes at a time and sendseach piece to the client where it is
regenerated.
8/8/2019 Web 2.0 and Estate Maps
47/87
Imran Sarwar 47 SEwI.E Final year Project
4.5.4 Locating Flickr images
As mentioned during section 2.3.9 the Flickr web service is able to geocode its images. This means
that images can be searched for in a specific geospatial location. Flickr has a unique ID for each city
and suburb in the UK. In order to find this ID you have to query the Flickr web service. The
ImageFinder class is used to locate this ID.
The ImageFinder class take a location as its argument. This location can either be a city or postcode.
This location is sent to the Flicker web service in a similar fashion to that used in the XSLT engine as
discussed in section 4.5.2. The response generated from the Flickr web service is a XML file which is
parsed using the SAX java parser.
.................
//GetServerResponse classImageFinder getImages = new ImageFinder();//new instance to ImageFinder
//Find the flickr location ID relevant to the locationString locationId = getImages.getLocation(spaceRemove(location));
.................
................. //ImageFinder classpublic String getLocation(String locationIn){
try{
/*URL to the flickr REST service, string is manipulated to include* location for which images are needed.*/
String flickRest = (" http://api.flickr.com/services/rest/?method=flickr.places.find&api_key=060b55702b4417d02a7b1338fc904f17&query="+locationIn+" UK");
//Converts the String type URL to a Java URL type.URL xmlUrl = new URL(flickRest); //Opens the URL to retrieve a XML file with location information.
.................
Once the response from the Flickr web service has been processed the location ID is inserted into the
Flickr image retriever URL, which is different to the location retriever. The rest of the process is
similar to that of the house search request which is discussed in section 4.5.2.
.................String xslIn = " estateMaps/estate_assets/image.xsl";
//String for flickr REST service manipulated with locationString xmlIn =(" http://api.flickr.com/services/rest/?method=flickr.photos.search&api_key=060b55702b441
7d02a7b1338fc904f17&place_id="+locationId+"&tags=city%2C+landmark&accuracy=11&safe_search=&per_page=55"); //Convert both XMLXsltEngine getGeoRSS = new XsltEngine(xmlIn,xslIn);
.................
4.4.6 Processing the search bar
As discussed during the design phase of this report the search bar on the GUI of estateMaps is able to
produce predictive results as the user inputs characters. In order to achieve this process the response
has to be generated as quickly as possible so that there is no lag between the user inserting
Top Related