LED-Lux

Project Proposal and Feasibility Study

Team 06

Anna Little, Melanie Fox, Evan Hill, and Reuben Lewis

ENGR 339 Senior Design Project

Calvin College

December 11, 2017

© 2017, Team 06 and Calvin College

1

Executive Summary This document describes the LED-Lux project by Team 06. The project is designed to create a healthier environment in hospital rooms by providing a spectrum of light that adjusts throughout the day according to a patient’s circadian clock. This system will allow patients to get the most and best sleep possible, and allow nurses and doctors to stay attentive longer by adjusting the light spectrum to fit their respective needs. The lights in this system would be part of a node-mesh network where each light is a node. The network would be controlled via machine vision, which will be able to “see” the users in the room. The system will differentiate humans from other objects, and turn the light on and off automatically when they enter and leave the room. Other features will be added to the improve the machine vision as the project continues, such as gesture control and location tracking.

2

Table of Contents

1. Introduction 5 2. Project Management 6

2.1. Team Organization 6 2.2. Schedule 6 2.3. Budget 7 2.4. Method of Approach 7

3. Requirements 8 3.1. Spectrum and Light Requirements 8 3.2. Node-Mesh Network Requirements 8 3.3. Machine Vision Requirements 9

4. Task Specifications and Schedule 10 5. System Architecture 12

5.1. System Communication 12 5.2. Machine Vision Architecture 13

6. Design Criteria 14 6.1. Openness and Communication 14 6.2. Caring 14 6.3. Delightful Harmony 14

7. Design Alternatives 15 7.1. Fixture Design 15 7.2. LED Design and Implications 15 7.3. ZigBee Communication and Module 15 7.4. Machine Vision and Webapp Control 16

8. Design Decisions 17 8.1. Project 17 8.2. ZigBee 17 8.3. Machine Vision 17

9. Integration, Test, Debug 18 9.1. ZigBee Network 18 9.2. Web Application 18 9.3. Machine Vision 18

10. Conclusion 19 11. Acknowledgements 20 12. References 20

Appendix A 21

3

Table of Figures Figure 1: Organizational chart for Team 06 …………………………………………………………… 06 Figure 2: Task Specification Graph ………………………………….……………………………….. 10 Figure 3: Individual Task Diagram ………………………………… ………………………………... 11 Figure 4: Block diagram of the system communication of LED-Lux ………………………………… 12 Figure 5: Discerning different figures via machine vision…………………………………………….. 13

4

1. Introduction Calvin College is a Christian liberal arts college located in Grand Rapids, Michigan where students are encouraged to “think deeply, act justly, [and] live wholeheartedly as Christ’s agents of renewal in the world.” Engineering students, as future designers, optimizers, and managers, are especially taught to take this motto to heart. This, in addition to all the other skills gained during the engineering program, is showcased in the student’s senior design projects. For the projects, students form small groups that will each work on a subject of personal interest and address that subject in a culturally appropriate, just, transparent, and caring way. This year, Senior Design Team 06 decided to address the shortcomings of indoor lighting within hospitals, calling their project LED-Lux.

Though typical indoor lighting is sufficient for illuminating dark spaces, it does not properly stimulate the human circadian clock. The human circadian clock aligns with changes in sunlight throughout the day and helps regulates hormone production, body temperature, heart rate, immune responses, sleep schedules, and more [1]. Within a hospital, both night-shift workers and bedridden patients may not have access to sunlight or special supplementary lighting to help maintain their inner clocks. This conceivably results in lesser productivity, lower morale, and longer healing periods.

To combat these issues, LED-Lux plans to create a scalable, smart lighting system that works with the human circadian clock to improve user mood, health, and productivity. The system will be made up of a ZigBee node-mesh network and will use machine vision to monitor user activity and control the lighting system. The LED-Lux team is made up of three electrical engineering students and one mechanical engineering student. Each student brings unique skills to the team. Evan Hill studies Electrical/Computer Engineering, works as a Resident Assistant at Calvin College, and has interests in LED implementations and lighting spectral systems. Anna Little is a Electrical/Computer Engineering major with a minor in mathematics who specializes in the mechanics of ZigBee communication and the software programming of ZigBee modules. Reuben Lewis studies Electrical/Computer Engineering at Calvin College with minors in German and Computer Science. One of his interests is capabilities and applications of machine learning and vision. And finally, Melanie Fox, the Mechanical Engineering major, has a biotechnology minor and is interested in the effects sleep and circadian clocks have on the human body.

5

2. Project Management 2.1. Team Organization

As mentioned above, the team consists of four members: Anna Little, Reuben Lewis, Evan Hill, and Melanie Fox. Anna fills the role of the communication liaison by handling emails and setting up meetings with consultants and mentors. Reuben runs team meetings and keeps the team on track during discussions. Evan keeps minutes during meetings and handles the follow-up email to make sure everyone is on the same page. Melanie juggles schedules to plan meeting times and helps to assign tasks to the people with the best ability to complete them in the allotted time. Figure 1 gives a visual representation of how the team is organized.

Figure 1. Organizational chart for Team 06

Team meetings occur weekly and are used to discuss the next steps in the project, the timetable of current and future tasks, and any new problems or questions that have arisen. If any issue arises that the team feels it cannot solve without further information, they will contact their advisor Mark Michmerhuizen. The minutes and various documents are kept in Google Drive for ease of access.

2.2. Schedule The team schedule is managed using a calendar in Google Drive and Taiga software. The calendar shows due dates and specific benchmarks required by the Senior Design class, while the Taiga Kanban is used for task management.

The team has struggled with scheduling when due dates for the Senior Design class were changed close to the expected due date and when team members were suddenly unable to complete tasks due to unforeseen circumstances. These challenges improved the flexibility of the team and showed how adaptable the members are.

6

The overall projected hours per week per person (hrs/wk/person) is 5 for the first semester. For second semester, 7 to 10 hrs/wk/person is projected. These numbers are rough estimates. It is difficult to project how many hours each team member will be able to contribute each week due to other classes and responsibilities, but these numbers are an approximated average. The schedule itself is in Appendix A, and gives the team a set of deadlines to meet in order to ensure that all the requirements of the project are met.

2.3. Budget The budget was put together by estimating the cost for the different components of the project prototype and other parts needed for development. The team keeps track of expenses and attributes each expense to the categories that were defined. The budget is tracked and updated through this process. As more information on cost is gathered, the amount allotted to different categories is changed (though the total budget is kept constant). DornerWorks has also contributed $500 for the integration of machine vision into the project.

2.4. Method of Approach Once the project was defined, the next step was exploring how to design the system. Because of the scope of the project, it is not feasible to put the entire burden of a single overarching task on a single member. Instead, each member is leader of an overarching task. Each leader is tasked with focusing on the design of their area, and, when prototyping, tasked with organizing the work on their area. For example, the lighting designer would be tasked with researching and creating the proper spectrum for each time of day and then leading the design of the lighting controller. As some of these parts rely on other parts, it is up to the area leader to ensure that the work on their part is done in a timely and efficient fashion. It is up to the area leader how to go about the research in their area; however, some resources used are the library, online databases including Google Scholar, professionals in the specific area, and online resources such as Stack Overflow. Communication is done via Facebook Messenger and weekly meetings in the first semester and twice weekly in the second. Documentation and version control are done using Google Drive for documents and GitHub for software files. One of the pieces of scripture used to guide the team is Ephesians 4:2: “Be always humble, gentle, and patient. Show your love by being tolerant with one another.” As the project goes on, the team will have to have patience with each other as other commitments and homework are added onto the workload and people may not be able to focus on the project for as much time as is expected. This verse reminds the team that they need not only have patience but be humble as well, which is important as an engineer. Using this verse as a guide, the team hopes to work together well throughout the project.

7

3. Requirements The main goal of the project is to create a better indoor environment by not only adjusting the color and type of lighting to ensure the best type of lighting for humans, but integrating the system seamlessly into the indoor environment. In order to refine the scope of the project and ensure a correct area of focus, the following requirements were developed. The first set of requirements outline the system in general. The system shall be designed so that the lights change color temperature and spectrum as the time passes to give the optimal type of light for humans at that specific time of day. The system shall be a node-mesh network of light fixtures, with each light fixture being a single node in the system. Finally, the system shall be controlled via machine vision. This system shall differentiate between humans and other things such as animals, controlling the lights depending on what it sees. The goal of the machine vision is to reduce the number of purposeful interactions the user has with the system, creating a more seamless lighting environment. These requirements will be broken down further in the following sections.

3.1. Spectrum and Light Requirements In order to properly create the right lighting environment for the user, the lighting system must meet the following requirements. First, the lighting system shall adjust the spectrum in real time based on an internet connected clock and the user’s approximate location. To keep general anonymity, the user’s zip code will be used to calibrate the system accordingly. When the lights change, they shall change slowly and unobtrusively, allowing users eyes to adjust easily and without shock. The lights shall also be manually configurable by the user, so that any standard temperature or light can be set at any time, as well as the intensity of the lights. They shall be responsive, changing soon after the input was validated. The lights shall also be turned on and off or dimmed when a person enters or leaves the room, as sensed via machine vision. The lights shall also produce enough light that their designated area is adequately lit by using LEDs. The lights shall be designed for maximum efficiency, aiming for the maximum life of the LEDs. This means optimal current control, heat dissipation, and voltage control. There are a set of other options that, time and resources permitting, would improve the system. Being a node-mesh network, the nodes could be grouped together and controlled as one set. A manual scheduler may also be added to allow users to set their own unique schedule and light alarms. The light may also be adjusted via a light sensor, reading in the intensity of the light outside and then dimming the lights to preserve energy. Finally, the system may include a fail-safe system to detect if and when lights may be failing, and help identify the problem in that node.

3.2. Node-Mesh Network Requirements In order to change the lights effectively, the nodes in the node-mesh network shall communicate data properly and securely, ensuring that only the right nodes get the data

8

and commands they need. They shall also communicate in a timely manner, ensuring that system adjustments and changes begin within 5 seconds. The commands to change each node shall come from the gateway, which shall be connected to the machine vision system. Each node shall have the hardware capable to run the lights effectively, including a power supply to power the LEDs, the ability to adjust the intensity of each LED, and hardware for the node-mesh network. The hardware shall be designed to consumer electronics specifications, and shall be for indoor use only. This is just a base requirement, and as the project progresses, will be revisited and updated based on the specifications needed in a hospital environment. Should a node need to be replaced, the network shall easily integrate and categorize the new node. The gateway shall be a small but powerful computer with the ability to tap into the node network and run the machine vision software need for recognizing the users and potentially their actions. The gateway shall be connected via Ethernet for a reliable connection to the internet to ensure the accuracy of the time. Given time and resources, the network may also be able to self-organize. This means that once the lights are installed, they will know where in the building they are, as well as where they are with respect to other nodes. They may also be able to self-heal any issues that might arise. For example, if when the meshes are being brought up two networks are formed, they should automatically merge into one network.

3.3. Machine Vision Requirements The machine vision shall use an IP infrared security camera to recognize objects entering and leaving the room. The objects shall be classified, with actions only occurring when a human is recognized. The threshold for human recognition will be set to 85%, with reading less than that discarded. The machine vision shall control the systems actions by turning the system on when it detects a human in the room, and turn or dim the lights after a set time once the human leaves. Data, if needed for training or improving the model, shall be stored in a secure, encrypted manner, restricting access to only the machine vision software. Any data not needed will be discarded after the time of recognition. The IP camera shall connect securely to the processing gateway, sending data in an encrypted format. The camera shall have enough resolution to ensure a correct recognition can occur. The machine vision system can be extended in many different directions, depending on the time available. It may be trained to detect what activities people are doing in the room. For example, the system may notice that the user is opening a book, and adjust the lights to help the user read, or see that the user is sitting down to watch a movie and dim lights for a more entraining feel. It may be trained to recognize users and adjust lighting preferences based on who is in the room. It may be able to track users throughout the house, dimming lights in one part of the room while turning sets of lights up. It may also recognize key gestures, giving the user the ability to use manual turn off, dimming, or

9

brightening commands. These actions and ideas can be extended and developed based on the development time available.

4. Task Specifications and Schedule

Figure 2: Task Specification Graph

10

Figure 3: Individual Task Diagram

11

5. System Architecture 5.1. System Communication

The basic design of the system begins with the communication between the machine vision gateway and the ZigBee light network. First, the camera will use its infrared sensors to get a basic black and white image of the room. Once movement is detected, the camera will capture visuals of the movement and create contours of the image. Then, the contours will be fed to the neural network, which will classify the different people and objects in the field of view. If the classification returns an object classification of a human that is 85% sure or higher, the system will send out commands to turn the lights on or off. The ZigBee gateway will then distribute the instruction to the correct nodes via the ZigBee communication protocol. Each node and its location will be stored on the gateway so that it will be able to selectively send data to whichever node is specified. The nodes will be in the same connection group, sending the encrypted data between nodes with each node having a key. Since only the nodes in the connection group have valid keys, this will prevent the sniffing of data. If the gateway is not connected directly to one of the nodes, the data will be passed from node to node until it reaches the proper node. The nodes will then use the received data to change the light as specified. The nodes will then send an “OK” back to the gateway, which will send the signal out again until an “OK” is received or a timeout occurs. A block diagram showing this system can be seen in Figure 4.

Figure 4. Block diagram of the system communication of LED-Lux

12

5.2. Machine Vision Architecture A machine vision system will allow the system to automatically turn on and off depending on if any person is in the room. The general system is that there will be a infrared camera in a part of the room that allows a good view of the entire room. The infrared would capture data at around five to ten frames per second. The data will be sent encrypted over WiFi. Then, using a computer vision program such as openCV or simpleCV, the data will be processed to create a set of contours. The contours will be processed then using a program such as TensorFlow. The images captured will be removed after processing unless the system needs more training, in which case the data will securely stored. The processing would occur on the gateway, which would issue the commands to dim the lights or toggle them. The system would search for human figures, ignoring other movement or objects in the room, as seen in Figure 3.

Figure 5. Discerning different figures via machine vision

13

6. Design Criteria As a part of the Calvin engineering curriculum, students are made aware of how their Christian faith sets criteria for their work. There are seven basic Christian design criteria. Cultural appropriateness deals with ensuring that the design does not insult the traditions of its intended users. Openness and communication ensures that all important information relating to the project is easily available. This norm fits with trust which details design dependability. Stewardship deals with protecting the environment and effectively using resources. Smoothly integrating into people’s lives is described by the delightful harmony norm. Justice involves improving equal opportunities and quality of life between groups of people. Caring ensures that the design helps those in need. Though each norm applies to every project, openness and communication, caring, and delighlightful harmony are especially employed by LED-Lux.

6.1. Openness and Communication Transparency is important for LED-Lux because the system will be processing data gathered through machine vision. The intentions and means of these features shall be made clear to soothe those who are leery of seemingly intrusive technology. One of the ways this will be done is through good documentation and using GitHub so that public has access to the source code of the project. Though the team shall be upfront with the details of the project, it is foreseeable that some information may become the property of DornerWorks, the team’s sponsor. In this event, the team shall ensure that this censored information cannot be used against the good of the user.

6.2. Caring Caring for other people is at the heart of LED-Lux. Built to improve the indoor lighting in hospitals, care for patients and medical professionals is at the forefront of our design. In discussing the project with friends and loved ones, each team member received comments that time spent in hospitals would be improved immensely by better lighting. However, the team is only able to provide what is predicted to be better lighting. Objective studies must be done on the whole LED-Lux system before any claims can be made. This is a part of openness and communication.

6.3. Delightful Harmony Delightful harmony is integral to LED-Lux. The overall goal of the system is to seamlessly fit into its users’ lives. LED-Lux will simplify spaces by combining supplementary lights, like night-lights and therapy lighting, into overhead lighting that requires minimal user input by adjusting automatically with the help of the machine vision. The ideal system will adjust itself automatically, creating the perfect indoor lighting environment without the user having to lift a finger. Unfortunately, delightful

14

harmony will compete with both time and cost restraints. The team will not be able to build a perfectly functional system in its entirety within the given time period. With cost, delightful harmony will conflict with itself in balancing component quality and affordability.

7. Design Alternatives 7.1. Fixture Design:

When choosing a fixture, it is vital to consider not only the look of the frame that holds the lighting but the type of material that encases it. The shape and size of the fixture are also crucial to creating the optimal lighting experience. The material of the fixture is important because it directly affects the weight, durability, and reflection of light. Additionally, the materials used will have an impact on the environment and shall be selected to fit the the stewardship design norm. Anxieties and other light induced traumas are often caused by exposure to harsh lighting, and so, the shape of a lighting fixture, which controls light exposure, can have positive or negative effects on human functions. To fit with the major design norm of caring, the team will focus on ensuring the design helps, not hinders. Not only that, but the brightness must be able to fill a whole room with light so that there are no unnecessary light fixtures, ensuring the delightful harmony of the room. All of these factors have a direct influence on the cost and optimization quality of the lighting system. Some designs that are being explored are pendant lights, sconces, and recessed lighting. Depending on the hardware and research done, a viable design will be made.

7.2. LED Design and Implications: The LED market is vast with many types and brands with many different hues and colors of lights. The goal of the project is to be able to design a full solar lighting spectrum that emulates a more natural light. Since there are only a limited amount of LEDs one can put in each node before it becomes over budget, the team hopes make the most they can get out of each LED. LEDs may come in multivariable white lighting (cool vs. warm); however, they also come in the RGB scale which adds the ability to create a full solar spectrum via tuning the intensities of each LED. In short, there are many design alternative with the combination of LED colors, which will be chosen after research not only into the spectrums, but into industry standards as well.

7.3. ZigBee Communication and Module: ZigBee is a type of communication protocol between devices. Although ZigBee will offer all around flexibility by creating easy connections between the controlling gateway and the lighting nodes, there are other options of node-mesh communication protocols that could be used.

15

Wireless connection range, types of modules, and compatibility with other systems are also variables to be aware with when deciding which node-mesh communication protocol to use. Alternatives to using the ZigBee included the use of a Wifi Direct, but difficult security and discussions of connection issues eliminated that option. The new Bluetooth 5.0 was an alternative as well. However, being a new technology, it is not yet proven. The resources and help behind ZigBee are much greater, allowing for more ease of development.

7.4. Machine Vision and Webapp Control Part of the decision when imagining what the system would look like included weather the focus would be on the Machine Visioning component or the Webapp Control. By focusing on the Webapp Control, in result the team would be able to make a more user friendly interface with many different modes of operation. On the other hand, if the team focuses on the Machine Visioning Control then there would be time to do precise sensing techniques within the design, as well as exploring an up and coming field. With both of these alternatives, the openness and communication design norm is key. In both cases, data will be collected and used to analyze the user’s patterns and activities in order to optimize the system. Being open about what data the team collects and analyzes is key to this project.

16

8. Design Decisions 8.1. Project

During the summer of 2016, Reuben did research on high-power LEDs with Professor Yoon Kim at Calvin College. This inspired Reuben to work with LEDs for his senior design project. He was interested in finding an implementation for the range of colors and temperatures low-power LEDs can produce. The team brainstormed options and came up with a service-oriented use and design, hospital settings, that excited each member of the team. This was cemented when Anna visited her grandpa in the hospital and the thing he complained about most was the harsh lighting.

8.2. ZigBee A ZigBee network was chosen to control the lights in the project. ZigBee is a low-cost, low-power, scalable wireless network that fits the team goal of delightful harmony. There are many resources available that will make the implementation and integration of ZigBee simpler than other wireless networks. ZigBee can also be configured in a mesh network that is self-healing and has a greater range than point-to-point options like Bluetooth (BLE).

8.3. Machine Vision The team decided to incorporate machine learning because of their connection with DornerWorks. DornerWorks agreed to invest in the project if the team provided innovation in the machine learning realm and implemented it in their design. Because of this addition, the user interface will have fewer features, being used for more debugging than user control. The system will now act more “automatic” by sensing when someone enters and leaves a room and keeping track of occupancy (for the minimum viable product). This will increase the delightful harmony element of LED-Lux. If there is time, activity recognition will be integrated. This would include recognizing if someone is watching TV or reading vs sleeping. The machine vision part of the system would then communicate with the network and adjust the lighting accordingly.

17

9. Integration, Test, Debug LED-Lux has many components, each of which need to be tested exhaustively. The ZigBee nodes, web app, and machine vision are the main components.

9.1. ZigBee Network There are multiple tests that the ZigBee network must pass. The nodes must connect and be assigned addresses automatically. The hardware will be tested for range both with and without line of sight (through-wall). After connection is tested and debugged, the output will be tested by forcing inputs and seeing the change in light. The final ZigBee requirement is communication with the web app.

9.2. Web Application The web application will be used for system integration and will give the user a low level of control over the system. The communication between the web app and the ZigBee will be tested with simple output from the app to the network that prompts a simple change in the LEDs. It will also be helpful when machine vision is being integrated into the system. The web app will be able to show what the vision system sees and the values that the system will pass to the ZigBee network. This intermediate step will make it easier to track if there are inconsistencies in the system.

9.3. Machine Vision The machine vision component of the project will be supported by DornerWorks. The system should be able to learn a room using a camera. The tests it will need to pass are storing data correctly, using the data to impact the behavior of the nodes, learning a room, detecting the correct number of people in a room, and dimming the lights correctly according to the input. Testing methods include using a test room to see how the system learns a space, entering the room in the camera’s line of vision and monitoring the input, making sure the data is stored correctly using the web interface, and making sure the output of the machine vision controls the ZigBee network in the correct way.

18

10. Conclusion A prototype for a smart, scalable lighting system that meets all the criteria will be made and tested. The system will be used as means for exploratory research in machine vision, looking into the areas of figure recognition, gesture control, and location tracking. With the help of DornerWorks, the teams advisors and other industry contacts, the team believes that this project is not only viable but innovative, changing the way lighting design will be done into the future.

19

11. Acknowledgements We have worked very hard on this project, however it would not have been possible without the assistance of multiple people. We would like to thank our advisor Mark Michmerhuizen for the guidance he has provided to us throughout this project. The people at DornerWorks and the University of Michigan have aided us in the research and design phase of our project, and we look forward to working with them further as our project develops. We would also like to express our gratitude towards the staff at Calvin College for assisting us in design through information on vendors.

We know we would not be at Calvin College or able to take on a project like this without the support of our families and friends. The support we have received has been instrumental in our success.

12. References [1] “NHBL Workshop: “Circadian Clock at the Interface of Lung Health and Disease” 28-29 April 2014 Executive Summary”. National Heart, Lung, and Blood Institute. September 2014.

20

Appendix A Team 06 project schedule Each date marked shows what should be completed by that time

21