Eldad Perahia, Robert Stacey - Next Generation Wireless LANs
Touch Seen Visual Board Game Platform Computer Vision Workshop, Fall 2010 Eldad Zinger, Oren Kishon,...
-
Upload
roland-booker -
Category
Documents
-
view
213 -
download
0
Transcript of Touch Seen Visual Board Game Platform Computer Vision Workshop, Fall 2010 Eldad Zinger, Oren Kishon,...
Touch SeenVisual Board Game Platform
Computer Vision Workshop, Fall 2010Eldad Zinger, Oren Kishon, Ehud Fisher, Allon Freedman
Agenda• Overview• Architecture
– High Level– Server– Client– Camera Module
• Network Protocol• Game Plugin Platform• Client-Camera Interaction• Game move detection algorithm• Demonstration
2Thursday, April 20, 2023 Touch Seen
Touch Seen
• Application framework for playing board games over a computer network using a real physical board
• Moves are detected by a camera and transmitted over the network
• Provides a simple API for development of 3rd party games which can be uploaded to the platform in the form of plugins
4Thursday, April 20, 2023 Touch Seen
Features
• Input using camera or any other standard input (keyboard, mouse etc.)
• Multiple clients connected simultaneously• Multiple games played simultaneously• All currently connected users displayed in
client• Option to challenge another user to a game• Textual chat between players
5Thursday, April 20, 2023 Touch Seen
Features
• Open API for game development• Pluggable games• Simple text based protocol• Network based on ActiveMQ which supports
multi language clients
6Thursday, April 20, 2023 Touch Seen
Server
Two logical functions:• Messaging – distributing Move and Chat
messages between users• User & Game Management:
– User DB– User presence– Game state
Thursday, April 20, 2023 9Touch Seen
Server
• Messaging server based on Apache ActiveMQ• ActiveMQ – powerful open source Messaging
Server supporting JMS 1.1, J2EE 1.4, multiple cross language clients and protocols and morehttp://activemq.apache.org/
• An ActiveMQ server is run as an embedded broker within the Touch Seen Server
Thursday, April 20, 2023 10Touch Seen
Server
JMS• A specification that describes a common way for Java
programs to create, send, receive and read distributed enterprise messages
• loosely coupled communication• Synchronous & Asynchronous messaging• Reliable delivery
– A message is guaranteed to be delivered once and only once• Outside the specification
– Security services– Management services
Thursday, April 20, 2023 11Touch Seen
Server
JMS Messaging Domains• Point-to-Point (PTP)
Message Queues - each message has only one consumer
• Publish-Subscribe systems Message Topics - each message has multiple consumers
Thursday, April 20, 2023 12Touch Seen
Server
JMS Message Consumption• Synchronous
– A subscriber or a receiver explicitly fetches the message from the destination by calling the receive method
– The receive method can block until a message arrives or can time out if a message does not arrive within a specified time limit
• Asynchronous– A client can register a message listener with a consumer– Whenever a message arrives at the destination, the JMS provider
delivers the message by calling the listener's onMessage() method
• All messages in Touch Seen use the asynchronous method
Thursday, April 20, 2023 13Touch Seen
Server
JMS Messaging Domains – Queues
Queuesends
MsgMsg
acknowledges
consumes
MsgMsg
Client1 Client2
Thursday, April 20, 2023 14Touch Seen
Server
JMS Messaging Domains – Topics
publishessubscribes
subscribes
MsgMsg
delivers
delivers
Client1
Client2
Client3
Topi
c
Thursday, April 20, 2023 15Touch Seen
Server
JMS Model
ConnectionConnection
creates creates
creates
MsgMsg
receives fromsends to
ConnectionFactory
ConnectionFactory
MessageConsumerMessage
ConsumerSessionSessionMessageProducerMessageProducer
creates
Destination DestinationThursday, April 20, 2023 16Touch Seen
ServerThreads• ServerController & NetworkController each operate
in separate threads• This ensures that server operations never block the
network thread so it can always be available for sending/receiving messages
Management• The server has no UI• Management is done using JMX (e.g. with jconsole)
utilizing both APIs provided by ActiveMQ and custom API’s designed specifically for this server
Thursday, April 20, 2023 18Touch Seen
Client• Strict modular design based on MVC• Comprised of several independent modules• Each module has a controller class• ClientController acts as central controller for
entire application• All communication to and from a module is done
through its controller• The central controller connects to each of the
module controller (module controllers don’t interact directly)
Thursday, April 20, 2023 19Touch Seen
Client
• The advantages of this design:– Modules are entirely decoupled from each other– Each module can be developed and tested
independently– Communication between modules is done in one
place – easy to maintain and debug– Modules can easily be replaced without affecting
the rest of the application – only the central controller “knows” all the modules
Thursday, April 20, 2023 20Touch Seen
Threads
• The ClientController, NetworkController and CameraController all operate in separate threads
• This ensures that the Network and Camera threads are never blocked by other modules within the client
• All Objects in the View operate within Swing’s EventDispatcher thread
• The ClientController and Model also operate within the EventDispatcher thread to avoid synchronization problems with the View
Thursday, April 20, 2023 22Touch Seen
Logi
n Q
ueue
Login
usernamepassword
success
notify online
request players
players
Play
ers
Requ
est
Que
ue
Play
ers
Topi
c
Thursday, April 20, 2023 24Touch Seen
Chal
leng
e Q
ueue
Game
challenge
accept
notify
challenge
accept
notify
Play
ers
Topi
c
New Game
Thursday, April 20, 2023 25Touch Seen
Gam
e To
pic
Game
move
move
move
Game PlayingG
ame
Chat
Top
ic message
message
message
Thursday, April 20, 2023 26Touch Seen
Gam
e To
pic
Game
quit
quit
quit
Game EndPl
ayer
s To
pic
notify online
Thursday, April 20, 2023 27Touch Seen
Game SDK
• Games are developed using a provided SDK• Developers must implement only two abstract
classes:– GameInfo
• Flyweight which acts as a factory for creating instances of the game
• Provides common properties: name, icon etc.
– Game• Represents the game itself
Thursday, April 20, 2023 30Touch Seen
Game SDK
• The server and client controllers interact with the game using the Game class
• It’s recommended (although optional) that this class include the game’s logic
• Game must provide a JPanel which acts as the game’s UI
Thursday, April 20, 2023 31Touch Seen
Game Installation
• A game should be packaged as a .jar file and placed within the lib/games folder of the client and server
• A configuration file must be edited with the fully qualified name of the GameInfo class
• At startup, the client and server use a custom ClassLoader to load all the games dynamically
• A future release could include automatic deployment of new games to clients and installation/uninstallation without restart
• A simple (yet incomplete) example of Chess is included
Thursday, April 20, 2023 32Touch Seen
Client-Camera Interaction• The Touch Seen desktop client is written in Java• The camera module is based on OpenCV and written in C• Interaction between the two is done via a local TCP socket• The protocol is a simple text based protocol which simplifies debugging• When the user starts a game, the client starts the camera process and
waits for a socket connection for a specified timeout (externally configurable)
• If the initialization sequence completes successfully the camera can be used to play
• Alternative input devices (e.g. a mouse) can be used if errors occur during the initialization sequence or if moves aren’t detected successfully
Thursday, April 20, 2023 34Touch Seen
Detection algorithm (1)• Initialization:
– Detect chessboard– Find prospective transformation (3 x 3): rectify– Rotate coords to user’s point of view
• For each new image captured, do:– Apply prospective transformation (rectify)– Subtract image from previous image – to detect who
moved / captured by opponent player– Subtract image from empty board image – to detect who
actually cleared his location
Thursday, April 20, 2023 37Touch Seen
Detection algorithm (2) Dealing with noise
• Noise: intensity only gets higher values• Filter1: image is a MIN of 20 frame captures• Filter2: Time-adaptive noise threshold:
– Capture two images each time, 1 sec apart– Subtract to get noise-only image– Calculate average value for square– Find max among squares (empirically ~5)– Thresh = max + const (=2, heuristics)
Thursday, April 20, 2023 38Touch Seen
Detection algorithm (3) Square change detection
• Less sensitive change detection:– Among squares who passed threshold, take max.– <Used for move detection>
• More sensitive change detection:– For each square, sum value of pixels, regardless of
threshold. Take Max.– <Used for finding captured piece>
Thursday, April 20, 2023 39Touch Seen
Detection algorithm (4) Evolution of development
• At first – simple subtraction algorithm (gray): hard-coded thresh, one background image
• Use RGB space – track each piece by its color. Each piece physically had a unique color
• Use HSV space – not good when dealing with B&W backgrounds
• Back to gray – smarter algorithm: min filter, multiple backgrounds, adaptive threshold
Thursday, April 20, 2023 40Touch Seen
DemonstrationIn addition to the live demonstration we have provided the following documentation:•System Requirements Specification describing planned features and architecture•This presentation which provides the final design implemented•User Manual•Complete Javadoc for the Client, Server and Game Development SDK including UML diagrams (embedded within the Javadoc)•Fully documented code•Source code for Client, Server, SDK, Camera module and example game (Chess)•Installation instructions
Thursday, April 20, 2023 41Touch Seen