1. Macbook With Touch Screen
2. Macbook Pro Touch Screen

Macbook With Touch Screen

We turned a MacBook into a touchscreen using only $1 of hardware and a littlebit of computer vision. The proof-of-concept, dubbed “Project Sistine” afterour recreation of the famous painting in the Sistine Chapel, was prototypedby me, Kevin, Guillermo, and Logan in about 16 hours.

The basic principle behind Sistine is simple. Surfaces viewed from an angletend to look shiny, and you can tell if a finger is touching the surface bychecking if it’s touching its own reflection.

First, align the bar along the bottom bezel of the screen (this model is specifically sized for the 13-inch MacBook Air), then attach a couple of small round adhesive magnets. After that, the bar. Here's how you can easily enable or disable 'Silent clicking' on 12-inch MacBook, MacBook Pro or Magic Trackpad 2 with Force Touch. Turn off the Fake 'Click' Sound on Force Touch Trackpad. It's possible to connect a second (or third) display to your Mac. Here's how to use a second screen with your Mac or MacBook, including finding the right adapter and cable. Be it a Mac mini, iMac, MacBook Air, or a MacBook Pro, it could boot into a black or blank screen. It's known as the black screen of death. In some worse cases, the Mac won't turn on at all. You might be pressed for time to finish a client's work on the MacBook Pro or Air. Or you just want to wake your iMac up to have fun.

Our idea was to retrofit a small mirror in front of a MacBook’s built-in webcam, so that the webcam would be looking down at the computer screen at a sharp angle. The camera would be able to see fingers hovering over or touching the screen, and we’d be able to translate the video feed into touch events using computer vision.

Kevin, back in middle school, noticed this phenomenon and built ShinyTouch,utilizing an external webcam to build a touch input system requiring virtuallyno setup. We wanted to see if we could miniaturize the idea and make it workwithout an external webcam. Our idea was to retrofit a small mirror in front ofa MacBook’s built-in webcam, so that the webcam would be looking down at thecomputer screen at a sharp angle. The camera would be able to see fingershovering over or touching the screen, and we’d be able to translate the videofeed into touch events using computer vision.

Our hardware setup was simple. All we needed was to position a mirror at theappropriate angle in front of the webcam. Here is our bill of materials:

• Small mirror
• Rigid paper plate
• Door hinge
• Hot glue

After some iteration, we settled on a design that could be assembled in minutesusing a knife and a hot glue gun.

Here’s the finished product:

The first step in processing video frames is detecting the finger. Here’s atypical example of what the webcam sees:

The finger detection algorithm needs to find the touch/hover point for furtherprocessing. Our current approach uses classical computer vision techniques. Theprocessing pipeline consists of the following steps:

1. Filter for skin colors and binary threshold
2. Find contours
3. Find the two largest contours and ensure that the contours overlap in thehorizontal direction and the smaller one is above the larger one
4. Identify the touch/hover point as the midpoint of the line connecting thetop of the bottom contour and the bottom of the top contour
5. Distinguish between touch and hover based on the vertical distance betweenthe two contours

Shown above is the result of applying this process to a frame from the webcam.The finger and reflection (contours) are outlined in green, the bounding box isshown in red, and the touch point is shown in magenta.

The final step in processing the input is mapping the touch/hover point fromwebcam coordinates to on-screen coordinates. The two are related by ahomography. We compute the homography matrix through a calibration processwhere the user is prompted to touch specific points on the screen. After wecollect data matching webcam coordinates with on-screen coordinates, we canestimate the homography robustly using RANSAC. This gives us a projectionmatrix that maps webcam coordinates to on-screen coordinates.

The video above demonstrates the calibration process, where the user has tofollow a green dot around the screen. The video includes some debuginformation, overlaid on live video from the webcam. The touch point in webcamcoordinates is shown in magenta. After the calibration process is complete, theprojection matrix is visualized with red lines, and the software switches to amode where the estimated touch point is shown as a blue dot.

In the current prototype, we translate hover and touch into mouse events,making existing applications instantly touch-enabled.

If we were writing our own touch-enabled apps, we could directly make use oftouch data, including information such as hover height.

Project Sistine is a proof-of-concept that turns a laptop into a touchscreenusing only $1 of hardware, and for a prototype, it works pretty well! With somesimple modifications such as a higher resolution webcam (ours was 480p) and acurved mirror that allows the webcam to capture the entire screen, Sistinecould become a practical low-cost touchscreen system.

Our Sistine prototype is open source, released under the MIT License.

Recent Posts

MacBook Pro (15-inch, 2016) and later and MacBook Pro (13-inch, 2016, Four Thunderbolt 3 Ports) and later have accessibility features that help users who are blind or have low vision enjoy the features of Touch Bar.

Use VoiceOver with Touch Bar

VoiceOver tells you what’s on your screen, and walks you through actions like selecting a menu option or activating a button using your keyboard or trackpad. It can also tell you what's on your Touch Bar.

To turn VoiceOver on or off, hold the Command key and triple-press the Touch ID button, which is on the right side of Touch Bar at the top of your keyboard:

Macbook Pro Touch Screen

After you turn on VoiceOver, you can use these gestures with Touch Bar:

• Move one finger over the Touch Bar to change the Touch Bar focus and have VoiceOver announce the element under your finger.
• Swipe left or right with one finger to move the Touch Bar focus to the previous or next Touch Bar element.
• Double-tap anywhere on the Touch Bar to activate the element under the Touch Bar focus.
• Split-tap (touch an item with one finger, then tap the Touch Bar with another) to activate the element under the first finger you use.
• Double-tap and hold to enter direct touch mode for the element under the Touch Bar focus. This allows you to adjust sliders.

Use Touch Bar Zoom

If you use the Zoom feature on your Mac, you can also turn on Touch Bar Zoom.

Select Apple () menu > System Preferences. Then click on Accessibility, select Zoom, and turn on Enable Touch Bar Zoom.

Here's what you can do after you turn on Touch Bar Zoom:

• Touch and drag with one finger on the Touch Bar to see a zoomed view of the Touch Bar on your display.
• Change the magnification level by holding down the Command key and using a two-finger pinch gesture.
• While panning with one finger, quickly tap with a second finger to synthesize a tap where your first finger is. Hold the second finger down and move both fingers together to synthesize a tap down and drag where your first finger is.
• Hold your finger still in one location to enter direct-touch mode, which allows you to interact directly with the control under your finger.

Use Switch Control with Touch Bar

You can use Switch Control to display Touch Bar on your MacBook Pro screen. This lets you access Touch Bar elements with standard pointer controls.

First, turn on Switch Control:

1. Choose Apple menu > System Preferences, click Accessibility, then click Switch Control.
2. Click General, then select Enable Switch Control. The Switch Control Home Panel appears on your desktop.

Next, toggle Touch Bar:

1. In the Switch Control Home Panel, click System.
2. Click Toggle Touch Bar to show or hide Touch Bar.

Learn more about using pointer controls.

Use Accessibility Options to turn on other features

macOS features an Accessibility Options window that lets you quickly turn on or off common accessibility features like Zoom, VoiceOver, Sticky Keys, and more.

To bring up this window on your MacBook Pro, triple-press the Touch ID button.