CSE Capstone Students Use Machine Learning and Haptics to Further Technology

On Wednesday, December 10, the Computer Science and Engineering senior capstone design course presented their semester projects to classmates and industry representatives alike in the Harvey R. Bright Building at Texas A&M University. Each of the five senior design teams was mentored by members of Dr. Tracy Hammond’s Sketch Recognition Lab. The projects included iErgonomics, VR, InMotion, C.A.N.E., and CourseSketch.

Team iErgonomics
The first group created iErgonomics, a system that attempts to use eye tracking to tell computer users when they are least busy to take a break in order to avoid computer vision syndrome. The group was comprised of students Aaron Moore, Karan Khatter, Michael Gyarmathy, and Zach Cannon. The group was supported by Dr. Hammond, SRL members Cassandra Oduola and Folami Alamudun, and Dr. Mark Benden, an Associate Professor in Environmental and Occupational Health.

Computer vision syndrome (CVS) refers to the pains 132 million people in the U.S. experience from looking at computer screens for extended periods of time. These pains include eyestrain, headaches, dry eyes, neck pain, and shoulder pain. Other programs have been created to help prevent CVS, but the group argued that none are user friendly, as they interrupt productivity. Therefore, the group created iErgonomics, which uses economical eye tracking to decide when a user is not busy and, thus, decide the best time to interrupt their computer usage for the recommended time to avoid CVS.

iErgonomics uses Eye Tribe, an eye tracking product from Denmark that detects eye direction and pupil dilation. When a user is considered “not busy,” a notification box appears on screen alerting the user to begin their rest period. If the user’s eyes return to the screen during the break, the eye tracking product recognizes their pupils and the break time starts to increase again, until the user looks away for the appropriate amount of time. The user can choose the 20-20-20 option, which requires the user to look away from the screen every twenty minutes for twenty seconds, ideally looking at an object at least twenty feet away. Or the user can choose the 60-5 option, which requires the user to look away from the screen every hour for five minutes. Both options are supported by research on CVS.

In the future, as per Dr. Mark Benden’s suggestion, the iErgonomics group plans to consider how to prove avoiding CVS improves person or employee productivity. With a focus on productivity, the group believes businesses would show interest in the product.

Karan Khatter

Aaron Moore

Zach Cannon

Michael Gyarmathy

Team VR
The second group created VR, a program that combines motion tracking and virtual reality to provide an immersive education system. The group was comprised of students Patrick Knauth, Jose Manriquez, Adrien Mombo-Caristan, and Hao Sun. The group was supported by Dr. Hammond, SRL members Cassandra Oduola and Raniero Lara-Garduno, and Dr. Debra Fowler, from the Center for Teaching Excellence at Texas A&M.

VR provides student users a physics learning application and an astronomy learning application. For physics, the program uses motion tracking, via Leap Motion, and virtual reality, via the Oculus Rift, to allow users to do learning tasks such as lifting and dropping balls in with varying gravitational pulls or rolling balls down surfaces of different levels of friction. In the astronomy application, users can go to each planet of our solar system and interact with each planet and celestial objects that have varying gravitational pulls as well.

In order to consider the economic impact these learning technologies could have on users, the team separated VR into modules. Through this, depending on their economic accessibility, users can simply use a mouse to interact with the educational program, or use the optional motion tracking and virtual reality components to make the experience more immersive.

In the future, the group sees VR serving different learning preferences as an easily distributed platform to many users.

Team InMotion

The third group created the Therapeutic Exercise Haptic Monitor, an at-home physical therapy program that uses haptic feedback, visual feedback, and motion tracking to ensure the patient is executing the exercises correctly. The group was comprised of students Jerry Barth, Patrick Vo, Trevor Gray, and Matthew Mjelde. The group was supported by Dr. Hammond and SRL members Cassandra Oduola and Vijay Rajanna.

The Therapeutic Exercise Haptic Monitor uses motion tracking to ensure a patient completes their physical therapy exercises correctly and completely. Physical therapy can be costly and time consuming, and the group hopes this program will help reduce the hours physicians are needed to help patients in executing basic movements. Physical therapists can create exercise routines and export the routine to the user’s home program. Then, the user can do the instructed exercises as per the program’s instruction. Once the user has reached the correct position in their exercise, such as lifting their arm 50 degrees, a visual circle on the screen turns red and a vibration is sent through a felt armband with sensors embedded in the material. The user feels the vibration, and lowers their arm to the original position.

In the future, the group sees the Therapeutic Exercise Haptic Monitor benefiting athletes in their training, with the primary example being weight lifters. The Therapeutic Exercise Haptic Monitor could be used to ensure these lifters are not overextending their limbs, and the program would vibrate when the athlete reaches the ideal angle.

Jerry Barth

Patrick Vo

Trevor Gray

Matthew Mjelde

Team C.A.N.E.
The fourth group created C.A.N.E., a vibrating belt that uses sonar and GPS to allow blind users to navigate their world without a conspicuous cane or Seeing Eye dog. The group was comprised of Grant Hendley, Kodi Tapie, Jeff Harrison, and Matt Harper. The group was supported by Dr. Hammond and SRL members Cassandra Oduola and Larry Powell.

The group carefully chose haptic, or vibrating, feedback, as they were told by visually impaired students that audio feedback would interrupt their natural ability to use sound to navigate their surroundings. The belt uses ultrasonic sensors and GPS triangulation to detect where the wearer is located and warn them of upcoming obstacles. If the belt vibrates on the right side of the body, the user needs to go to the left to avoid an obstacle.

The group is passionate about using human-computer interaction to improve the well-being of the disadvantaged. In the future they may extend this technology to be useful to firefighters, who are navigating usually dark, smoky, and dangerous situations with little to no guidance.

Team CourseSketch
The fifth group created CourseSketch, an educational program that allows student users to draw or sketch images that are recognized by the program. The group was comprised of Khoa Bui, James Granger, Andrew King, Angel Lozano, Matthew Runyon, Antonio Sanchez, Devin Tuchsen, Michael Turner, and Joshua Zodda. The group was supported by Dr. Tracy Hammond, SRL members Cassandra Oduola and David Turner, and Dr. Dwayne Raymond, who used CourseSketch in his Intro to Logic class last semester.

CourseSketch is a sketch recognition program. It allows instructors of online, or in-person classes with online components, to do more than multiple choice questions online. Sketch recognition means students can draw and sketch images and equations that can even be automatically graded by the system. If the instructor prefers the system to give automatic feedback, the program allows for instant feedback on open-ended problems. Later, too, the instructor can watch a student’s drawing of the image, and give them feedback in real time.

The program is intuitive for student and instructor users alike, the group argued, and interactive tutorials are already built into the system as well. In the future, the group hopes that CourseSketch will continue to be developed and allow for full lectures to be conducted successfully using the program. It will be used in the classroom next semester for Phil240.

Andrew King III

Antonio Sanchez

Joshua Zodda

Angel Lozano

Devin Tuchsen

James Granger

Khoa Bui

Matthew Runyon

Michael Turner

Jess Gantt can be reached at jessicalgantt@gmail.com

Clint Brown's Reflections on his Visit to the SRL

As discussed in last week's blog post, Clint Brown, the Director of Product Engineering for Esri, a company that is a leading innovator in developing geographic information systems (GIS), visited Dr. Tracy Hammond's Sketch Recognition Lab on Tuesday, November 18. Brown visited during the three-day Texas A&M University GIS Day 2014 event organized by Dr. Daniel Goldberg, Assistant Professor of Geography. 

Brown offered his thoughts on his visit to the SRL, and graciously allowed us to share them with you here! Brown found many similarities in the processes for SRL research and Esri product development. Both entities use small collaborative teams to engineer projects on the cutting edge of computer science. 

Clint Brown, Monday, December 1:

I very much enjoyed meeting the students in the lab and having each of them show their work. This reminded me of a lot of how our development teams work at Esri. At Esri, we are always searching to find for more true talent. We continually seek out clever software engineers who have what it takes to be creative, to work on teams, and who pay close attention to the user experience.

We like to work on real customer problems, very much like the ones that the students are working on in the lab. And a key aspect of that work is to imagine new kinds of experiences for software users. The best experiences are graphic and interactive and fun to use. And they do something useful, but are also cool. They are delightful in their own unique ways. That’s the vibe I picked up on in the Lab. You definitely have a talented team of very smart, creative people collaborating to solve some incredibly complex problems. And those team members are coming up with some inventive, and fun, approaches. It’s a talent that the software industry (and Esri in particular) is always looking for.

In addition, I believe that leadership and collaboration are incredibly important. Leaders who can motivate and guide designs and navigate the team through critically important creative processes are incredibly valuable. Dr. Hammond, I think, is providing strong vision and leadership. And I liked the collegial atmosphere. You work as a team.

At Esri, on our software development teams, we have similar meetings where our small project teams show their latest work to their peers and to our leadership. The process of having to introduce and explain, and ultimately -- to present your results through demonstrations is a very effective way to develop strong, robust solutions to the problems you are trying to solve. And this process needs to be repeated over the life of the project; you are iterating and evolving your designs and implementations. 

Great ideas are hard enough to create, but to implement elegant solutions for those ideas requires the genius of teams working together combined with a willingness to adapt and evolve your designs to move them forward. I really liked that about the lab. I really liked the creativity and openness of the students. I felt like feedback and collaboration is a big part of your success.

It’s fantastic to see these applied projects. I believe that add so much more depth and understanding about how new computing approaches and technology are evolving. The empirical experience that the students are gaining is incredibly valuable.

Here are a few other observations about how your work is like our product development here at Esri:

- We have small teams, and we expect each team to articulate their goals and to demonstrate their software in short iterations. We apply agile methods like Scrum.

- We expect that each new version of the software does something interesting and important to address key aspects of the overall problem for which we are trying to build solutions. We demo at each iteration and in between.

- Our work is very graphic – cartographic. In other words, maps and information layers are at the heart of our work. Your work is as well.

- The broad adoption of smart phones and tablets in the past decade is causing great disruption and is transforming computing. The designs for commercial software and apps are now incredibly graphic in nature. This is a great point of departure (compared to traditional computer science approaches) for the innovative work being done in the lab. It’s not that computer science concepts are invalid or are going away. It’s how they are being applied in this new paradigm.

- Today’s computing platforms are communications networks enabling a lot of web interaction and the use of smart devices and development tools like JavaScript, JSON, the web, CSS, iOS, Android, etc. And focused apps like you are building.

- Your projects are collaborative in nature. It’s about the work being done by and the talent and ethics of the team.

- Social coding and shared source are fundamental to how new software will be developed. This is about people collaborating and sharing great ideas and implementations that others can leverage in their own work.

So, I very much enjoyed my visit to the lab. And as an added bonus, it was in the Teague building in the same location as the “Institute” of Statistics was back in 1976 to 1978 when I was a graduate student in the Statistics program for my Masters of Science degree!

Thanks for the wonderful experience on my visit.






Jess Gantt can be contacted at jessicalgantt@gmail.com.

GIS Day 2014 - Esri Director of Software Products Clint Brown Visits the SRL

Clint Brown, the Director of Product Engineering for Esri, a company that is a leading innovator in developing geographic information systems (GIS), visited Dr. Tracy Hammond's Sketch Recognition Lab on Tuesday, November 18, during the three-day Texas A&M University GIS Day 2014 event organized by Dr. Daniel Goldberg, Assistant Professor of Geography. 

Brown, who received his Master's in Statistics and Computer Science from Texas A&M in 1978, was a distinguished guest at GIS Day and spent an hour at the SRL learning about the lab's innovative yet practical research and projects.

Miriam Olivares, the GIS Coordinator for Texas A&M University, noted that she "personally enjoyed witnessing how Clint Brown, who has seen it all, got very impressed with the SRL work and immediately opened the door for collaboration."

SRL member David Turner demonstrated Mechanix and CourseSketch for Brown, exhibiting each program's sketch recognition capabilities as developed in the lab. Raniero Lara-Garduño of the SRL demonstrated the SmartStrokes project for Brown, which computerizes cognitive psychology tests and analyzes the patients' sketches. Lara-Garduño noted that Brown helped him recognize that "sketch analysis in this domain had potential to uncover new and interesting information about a patient's cognitive abilities."

"It is always inspiring to see someone as esteemed as Clint Brown react positively with our projects!" says Lara-Garduño.

Vijay Rajanna of the SRL demonstrated Step Up Life, a program that suggests contextually suitable exercise via sending information from a watch to the user's phone, by having Brown himself use the system while discussing its potential. Brown noticed how the program integrates information from various contexts when making its decisions, and shared his ideas on how this system can be further extended to make smart decisions using geospatial information.

Rajanna states, "It was our privilege to demo and discuss on the various contemporary research topics that we conduct at SRL with Clint Brown. He contrasted our work at the SRL with conventional research conducted at other labs, and he clearly saw the relevance and applicability of our work in solving practical real world problems."

Jaideep Ray of the SRL found that Brown's ideas on the various applications of sketch recognition in areas such as indoor mapping and path planning could be new areas of research in the SRL. Stephanie Valentine, a senior SRL research member, felt Brown offered a unique and invaluable perspective on the applicability of SRL research and how it might grow in the future. 

"It was energizing to see his excitement at our projects and results," Valentine noted.

Olivares, TAMU's GIS Coordinator, summed up her experience with Brown at the SRL by stating: "The sense of ownership and passion each SRL developer transmitted while showcasing their latest invention to Clint Brown is unparalleled and, I guess, one of the secrets of this group’s continuous success."

Brown noted he was overwhelmed by the people and great work he saw at the SRL and TAMU GIS Day.

Dr. Tracy Hammond and SRL members demonstrating the lightness of the components of the HaptiMoto vest for Esri's Clint Brown.

Vijay Rajanna demonstrating the Step Up Life program, with Clint Brown wearing a watch sending contextual exercise recommendations to a smart phone.

 

Jess Gantt can be contacted at jessicalgantt@gmail.com

SRL Project Presentations at the Farewell to Zachry Engineering Center

On Friday, November 14, after 42 years serving as the home to Texas A&M engineering courses, distinguished guests, pioneer donors, and current and former students gathered on the lawn of the Zachry Engineering Center to say goodbye to the building. The Sketch Recognition Lab (SRL), directed by Dr. Tracy Hammond, was one of eight labs from the Dwight Look College of Engineering chosen to present recent research to event attendees.

Attendees not only paid respects to the Zachry building, but also witnessed the groundbreaking of the incoming Engineering Education Complex, which is set to open in 2017. The renovation will begin this winter break. Due to the scale of the construction project, speakers included distinguished former students such as Astronaut Mike Fossum and the current TAMU Student Body President Kyle Kelly. According to the College of Engineering's news page, additional remarks were made by Chairman of The Texas A&M University System Board of Regents Phil Adams, Chancellor John Sharp, and Vice Chancellor and Dean of Engineering Dr. M. Katherine Banks.

For the groundbreaking, SRL members Trevor Nelligan, Cassandra Oduola, and David Turner represented the lab, along with students in Dr. Hammond's senior-level Aggie Challenge or Capstone Design courses. The SRL and other labs were asked to set up posters and demonstrations to showcase current undergraduate research efforts in TAMU engineering.

SRL members and senior undergraduate students presented projects developed recently in part by SRL undergraduates. Nelligan and Turner used a touchscreen tablet and poster presentation to demonstrate Mechanix, which is a program that recognizes sketched diagrams drawn by users and operates as a sketch-based tutoring system. Mechanix automatically corrects and provides feedback to hand drawn statics homework problems. Mechanix is currently in use at TAMU, Georgia Tech, and LeTourneau University. 

Dr. Hammond's AggieChallenge students Michael Turner, Devin Tuchsen and Khao Bui, who are also mentored by David Turner, used a laptop and poster presentation to demonstrate CourseSketch. CourseSketch is another sketch-based education program similar to Mechanix that focuses on interactive learning and will soon be used to create entire sketch-based lecture programs that will pose questions and route students depending on their sketch responses.

Dr. Hammond's Capstone senior students Kodi Tapie and Patrick Vo, mentored respectively by Larry Powell and Vijay Rajanna of the SRL, demonstrated the HaptiMoto vest, the project that inspired their current haptic project. Haptimoto is designed to allow touch sensors within the vest to give motorcyclists hand-free turn-by-turn route guidance for enhanced safety on the road. HaptiMoto was developed and tested by a large number of undergraduate students in conjunction with recent SRL graduate Dr. Manoj Prasad.  

Additionally, Tapie and Vo showcased vibrating plush toys developed last spring in Hammond's 2014 undergraduate capstone course. Tapie and Vo demonstrated the toys in games of hide-and-seek, allowing the person holding the toy to know via vibrations how close or far they are from their target.

Feedback to the SRL presentations at the groundbreaking was overwhelmingly positive, with many distinguished donors and guests interested in the research products. Tapie even noted, "Some donors wanted to purchase the plush toys to play hide-and-seek with their grandchildren!" 

Below: David Turner and Michael Turner showcasing the CourseSketch poster.

Jess Gantt can be reached at jessicalgantt@gmail.com.

Folami Alamudun - The 21st Institute on Teaching and Mentoring in Atlanta, GA

On October 30, over 1000 invited minority doctoral students, new faculty members, and leaders in education attended the 21st Annual Institute on Teaching and Mentoring in Atlanta, Georgia. One of these invited doctoral students was the Sketch Recognition Lab's own Folami Alamudun. Alamudun is a Computer Science & Engineering PhD student at the TAMU Sketch Recognition Lab, directed by Dr. Tracy Hammond. He also serves as the Vice President of the Computer Science and Engineering Graduate Student Association. 

Alamudun was able to attend the Institute through generous support and a travel fellowship from the TAMU System-wide Alliance for Graduate Education and the Professoriate, or AGEP. The National Science Foundation (NSF)-funded AGEP exists to "support underrepresented students working to complete their doctorates in energy and sustainability and/or STEM related fields." 

The Institute on Teaching and Mentoring lasted from October 30 until Sunday, November 2, at the Omni CNN Center in Atlanta. The Institute is supported by the Compact for Faculty Diversity, which is a partnership comprised of regional, federal, and foundation programs. The Compact for Faculty Diversity focuses on minority participation in higher education and aims to increase the number of ethnic and racial minority faculty members on college and university campuses. 

The Institute is recognized as "a four-day conference that has become the largest gathering of minority doctoral scholars in the country." Attendance is by invitation only. Attendees include scholars from prestigious groups funded by renowned entities such as the United States Department of Education, the National Institutes of Health, and the National Science Foundation.

Throughout the four-day Institute, Alamudun and other attendees are given the opportunity to attend seminars and workshops on career development, as well as network with other graduate students, faculty members, and leaders in higher education from underrepresented minority groups. The Institute focuses on "mentoring and teaching preparation, community insights and scholar networking," while also putting a national spotlight on diversity in higher education. 

For these minority members in higher education, the Institute offers a chance to "share in the success of recent graduates and encourage one another to push through the challenges ahead," Alamudun states. The Institute offers attendees insight into overcoming career barriers and motivates them for the coming year.

"The benefit and value in attending this event include measurable values such as networking, development workshops, interaction with recruiters actively looking for minority candidates, and seminars," says Alamudun. "But it also includes less measurable values such as camaraderie, motivation, inspiration, and the wow factor."

Jess Gantt can be reached at jessicalgantt@gmail.com.

Stephanie Valentine - KidGab with the Girl Scouts of Central Texas

Stephanie Valentine, a Computer Science & Engineering PhD student at the TAMU Sketch Recognition Lab, conducts research pertaining to her passion for bringing awareness to and educating youth on social media safe practices. Valentine is the president and founder of Wired Youth Inc., a nonprofit started in June of 2014 aimed to “foster an understanding among children and families of the permanent, public, and context-free nature of internet communications.” Valentine's co-founders include her sister Catherine Miller and Dr. Tracy Hammond who serves as the Director of the Sketch Recognition Lab at TAMU, Valentine's graduate advisor, as well as the vice-president and treasurer of Wired Youth, Inc.

The Daily Mail noted in 2014 that more than half of children use social media by the age of 10. Just over 40 percent of these children using social media had messaged a stranger by the age of 12. With these and other startling statistics, concerning cyberbullying and the 32 percent of children on social media who receive messages from online strangers, research like Valentine’s focused on teaching children safe social media skills is gaining increased attention.

In addition to Wired Youth Inc., Valentine is also the creator of KidGab, a social network designed with kids in mind that is appropriate for children ages 8 – 10. These children are young, but typically are already tech savvy and often circumvent age restrictions on social media like Facebook on their own or with parental consent. KidGab, as Valentine states, is a “social network with training wheels.” KidGab offers a closed social media, where members are verified and approved by a proctor before being able to join and interact with other members. Valentine notes that the program “mentors kids through their first social networking experience online, and they can come to understand what is good and what might be inappropriate through the help of adult supervision.” Additionally, each child user account is connected to their guardian’s, who must log in once a week to monitor their user’s activity and allow continued participation.

Valentine created a short teaser video to showcase some the features KidGab offers, particularly reflecting the fun and artistic expression the program allows its users.

On Saturday, Oct. 11, Valentine met with seven Girl Scouts and their guardians in the Caldwell Civic Center to begin an eight week trial of KidGab. She was joined by Hammond and fellow SRL PhD student Cassandra Oduola, along with Emily Magnotta, the Program and Travel Specialist for the Bryan/College Station branch of Girl Scouts of Central Texas. The program was supported through joint collaboration. The TAMU Department of Computer Science and Engineering donated drawstring bags, engineering inspired coloring books and other treats for the participants. SRL and TAMU provided tablets for use at the event. Wired Youth Inc. supplied the event space and necessary logistical and financial assistance. In addition, the Girl Scouts of Central Texas provided volunteers, additional craft supplies, and support prior to and throughout the event.

The Girl Scouts and their guardians first participated in interactive group discussions, presentations, and crafts pertaining to friendship, communication, cyberbullying, and online interactions. After these activities, the families were introduced to the KidGab software and given hands-on training on the provided tablets using the program. In regards to the educational element being included before using the program, Valentine says understanding the nature of friendships versus “digital friendships,” knowing how to detect cyberbullies and online predators, and knowing who to talk to about these issues are necessary tools for children to learn before engaging in social media.  

“Most research is aimed at what to do once cyberbullies or predators are already present,” Valentine notes. “But kids shouldn’t have to be uncomfortable or even in danger. I’m concerned with education and hands-on practice to prevent these interactions from ever happening in the first place.” Valentine compared social media usage to sports, stating that practice is required to perfect the skill. KidGab can provide this needed practice in a safe yet fun environment.

“Valentine is passionate about protecting kids in an online setting,” adds Magnotta of Girl Scouts of Central Texas. “Girl Scouts of Central Texas and KidGab have developed this amazing partnership and we plan to work together for a long time.”

In the future, Valentine would like to see KidGab and projects with similar missions concerning protecting kids in online settings incorporated into schools and children’s programs akin to Girls Scouts across the country. For her current research, she plans to continue work with Girl Scout troops in the Bryan/College Station, Austin, and Houston areas. Eventually, Valentine plans to extend her research to other youth organizations such as youth groups, sports clubs, and even school curriculums.

Jess Gantt can be reached at jessicalgantt@gmail.com.