CoSECiVi'14 - E-Cecilia. Implementation of a music game
-
Upload
sociedad-espanola-para-las-ciencias-del-videojuego -
Category
Education
-
view
61 -
download
2
Transcript of CoSECiVi'14 - E-Cecilia. Implementation of a music game
D’ IMATGE I GRÀFICS!T
EC
NO
LO
GIE
S!
D’ IMATGE I GRÀFICS!
TE
CN
OL
OG
IES!
http://gilab.udg.edu
E-cecilia: implementation of a music game
R. García1,5, I. Barbancho2, L. Tardón2, J. Arambarri3, M. Magdics1,4, M. Sbert1
1University of Girona 2University of Malaga
3Virtualware 4Budapest University of Technology and Economics
5iMinds / University of Hasselt
D’ IMATGE I GRÀFICS!T
EC
NO
LO
GIE
S!
• Introduction • Game description
• Architecture, client-server model
• Server
• Music composition & analysis
• Client • Graphics, non-photorealistic rendering
• Natural interaction
• Immersive devices
Contents
D’ IMATGE I GRÀFICS!T
EC
NO
LO
GIE
S!
Introduc-on
• Learning to sing requires long practising hours
• The early stages use very simple songs → boring
• Good singers start very young, so they tire more easily
• Solution: Make the practising into a game
• Related work:
• SingStar: full songs, too complex for starting young children
• Other games:
• focused on keyboard / mouse interaction
• don't do audio analysis
D’ IMATGE I GRÀFICS!T
EC
NO
LO
GIE
S!
• Serious game to practice singing and rhythm
• The user is presented with a small village to explore
• The houses are small, and can be enhanced by the user:
• After clicking a house, a song is presented
• The user sings to the best of his abilities
• The house evolves in response to the user singing
• The user can continue exploration and singing until he has practiced enough.
Game descrip-on
D’ IMATGE I GRÀFICS!T
EC
NO
LO
GIE
S!
Gameplay video
D’ IMATGE I GRÀFICS!T
EC
NO
LO
GIE
S!
System architecture
• Three main components: • Game client (Unity)
• Assets (on a web server)
• Recommendation and Evaluation server
• Audio and rhythm evaluation (matlab)
• Recommendation system (C++ library)
• Interconnection:
• TCP sockets (game-server)
• HTML (game-webserver)
D’ IMATGE I GRÀFICS!T
EC
NO
LO
GIE
S!
Game client
• Object oriented, asynchronous architecture using the Unity game engine. Object behaviour uses C# scripts.
• The main objects are:
• World Creation (terrain, house creation and evolution)
• World View: Camera, NPR effects, Dome transforms, GUI
• World control: Kinect, mouse interaction
• Asset downloading, communication with web server.
• Microphone: song recording
• ServerConnect: Communication with the server, evolving the GameState
D’ IMATGE I GRÀFICS!T
EC
NO
LO
GIE
S!
Music server
• The game server is responsible for the automatic generation of musical excercises (songs) and the assessment of the user performance
Server
D’ IMATGE I GRÀFICS!T
EC
NO
LO
GIE
S!
Music synthesis
• A set of training songs is used and analyzed
• Created songs replicate the selected style and complexity of that of the corresponding exemplars
Training Generation
D’ IMATGE I GRÀFICS!T
EC
NO
LO
GIE
S!
Music assessment
• Tempo, rythm and melody is compared against the reference
• Works with inexpensive microphones too
D’ IMATGE I GRÀFICS!T
EC
NO
LO
GIE
S!
Graphics, Non-‐photorealis-c Rendering (NPR)
• Post processing stylization
– State-of-the-art image processing (on the color and depth)
• Flexible, a wide range of effects
• Commercial Unity package under finalization
Original rendering (“Photo-realistic”)
Color Comic: Image simplification + edges
D’ IMATGE I GRÀFICS!T
EC
NO
LO
GIE
S!
NPR: edge effects
Intensity + depth-normal edges Colored painting: edge color is driven by the (original) rendered color
D’ IMATGE I GRÀFICS!T
EC
NO
LO
GIE
S!
• Replacement of the shadowed pixels by another color
NPR: Shadow effects
Complementary shadows: Shadow color is the complementary color
of the shadowed surface
D’ IMATGE I GRÀFICS!T
EC
NO
LO
GIE
S!
• Increasing the level of abstraction by depth
NPR: depth sensa-on
Depth-based desaturation Depth-based simplification: Texture details are removed
in the background
D’ IMATGE I GRÀFICS!T
EC
NO
LO
GIE
S!
NPR demo
D’ IMATGE I GRÀFICS!T
EC
NO
LO
GIE
S!
• Augmented reality: mobile device, augment camera image with controls (e.g. buttons)
• Kinect: exergame (imitating real movements, like walking or turning) or control by hand movements
Natural interac-on
AR button Exergame: walk
D’ IMATGE I GRÀFICS!T
EC
NO
LO
GIE
S!
• Different projection/camera settings
– E.g. stereo: 2 cameras, dome: 360º projection with cube-map
Stereo, top-bottom Stereo, side-by-side
Immersive visualiza-on
D’ IMATGE I GRÀFICS!T
EC
NO
LO
GIE
S!
Immersive visualiza-on
Spherical dome
source: immersivedisplay.co.uk
Immersapod
D’ IMATGE I GRÀFICS!T
EC
NO
LO
GIE
S!
Immersive visualiza-on
• NPR+immersive: NPR on the final image
– Regular 2D image processing
• No added computational complexity for using immersive displays w.r.t. NPR
Dome+color comic Immersive+B&W comic
D’ IMATGE I GRÀFICS!T
EC
NO
LO
GIE
S!
Conclusions
• Music teaching game • State-of-the-art automatic music synthesis and
singing assessment
• Controllable graphics style (NPR) as post-processing
• Support for natural interaction, immersive visualization
D’ IMATGE I GRÀFICS!T
EC
NO
LO
GIE
S!
Thank you!
Questions?
D’ IMATGE I GRÀFICS!
TE
CN
OL
OG
IES!