WEBINAR: Real Time AI Assistance with Smart Glasses and Live Video

Introduction and Housekeeping

Tim: Elena, if you could make sure the live transcription is on, that would be great.

I’d like to welcome everyone to our webinar today. We’re so glad you joined us for this month’s webinar, called Real-Time AI Assistance with Smart Glasses and Live Video.

Today’s webinar is going to last a little over an hour. We’ve got a lot of content today, and there are several points where we’re going to give you a chance to ask questions or comment.

Just some housekeeping so this can be as smooth as possible: we’re expecting a large turnout today, so everybody is going to be muted until the question-and-answer sections.

If you have a question, please raise your hand at any time by pressing Alt + Y if you’re using Zoom on a PC. This will make it easy for our moderator to see you at the top of the list. It puts you in order, and she’ll be able to call on you when the time comes. To lower your hand, press Alt + Y again.

Don’t forget: if you’re called on, you must be sure to unmute yourself before speaking, so please try to find that unmute button relatively quickly if you’re called on.

This webinar is brought to you once again by The Lighthouse for the Blind, Incorporated. There are a number of Lighthouses around the country, but ours is headquartered in Seattle, Washington, and we’re called The Lighthouse for the Blind, Incorporated.

We have manufacturing facilities in Seattle, Washington; Spokane, Washington; and Summerville, South Carolina, where we make aerospace parts and all kinds of other products for the U.S. government. We own and operate AbilityOne Base Supply Centers located on military bases up and down the West Coast, and we also provide contract closeout services for the military at several other locations around the country.

Our mission is my favorite part: our mission is to empower people who are blind, DeafBlind, and blind with other disabilities through sustainable employment and transformative services for independence and career advancement.

We’re currently hiring, so if you or somebody you know is blind or visually impaired, please scope out our open positions at LHBlind.org/jobs to see if there’s a job for you. Elena is going to put that link in the chat for everybody.

Today’s webinar is one of several community-based programs we’re excited to offer at The Lighthouse. We’re thrilled about the progress we’ve made so far, and we’re so grateful for your ongoing support and for helping us spread the word. Stay tuned for more updates, and feel free to reach out any time to questions@LHBlind.org.

One final note: tomorrow you’re going to receive an informational email with detailed notes and relevant links that we covered today. We’ve made some updates to the content of this email.

Guest Introduction: Carrie on Accessibility

Tim: Carrie, thank you so much for joining us. Can you tell everybody a little about yourself, where they can find you, and the work you do?

Carrie: Absolutely. You can find me on YouTube at Carry On Accessibility. I also have a podcast where I do deep-dive interviews with founders, CEOs, and other people who are blind and innovating or making waves in the space. It’s called the Epic Ally Podcast.

You can ask the A-lady to play the Carry On Accessibility podcast, watch it on YouTube, or find it in the podcast player of your choice.

I’ve been visually impaired with aniridia all my life. A lot of my family is also visually impaired, including my son. I used to work at a low vision center where there was a lot of tech, and I loved my time there helping clients, customers, and patients find the tools they needed. I wanted to continue that through the internet and social media, and that’s how we got here today.

I love technology. Some people say I have a bit of an addiction, but it’s okay.

Tim: Right. It’s a positive thing.

Carrie: Exactly. Accessibility is incredibly important for me. One thing I like to capture is that assistive technology and accessibility are not things you can just walk into a Best Buy and try. A lot of the time, in the content I produce, I try to capture that feeling of actually sitting down, playing with the tech, and seeing what it’s like to use it.

Tim: We’re going to be doing some of that today too.

Overview: AI Live Video Features

Tim: Today, everybody, we’re going to demonstrate some of the live video features that are available right now from three of the leading AI large language models.

We’re going to cover live video features from:

ChatGPT from OpenAI,
Meta live video using Meta smart glasses, and
Gemini multimodal live video from Google.

About a year ago, some of you may have attended a webinar we did when OpenAI released a preview of this live video feature. OpenAI, the folks behind ChatGPT, also worked with Andy Lane from Be My Eyes. They put out a video that made this live video seem absolutely amazing. I was just blown away by what they showed it could do.

We’re going to look at that video from one year ago, and then we’re going to show you where this live video stuff is at now.

Video Clip: OpenAI and Be My Eyes Preview

Video Narrator: Be My Eyes and I are about to take a journey that may change lives around the world.

Vision is such a dominant sense for most of us. Our eyes feed us an endless stream of information. It is somewhat different for me. I’m blind, so I’ve had to learn new skills and use workarounds to live my life without sight.

I still think visually, but I have to build the images in my mind through other means. Today, though, I’m experiencing something new. I’m using OpenAI’s newest model and voice-first interface to imagine the world through a conversation with an AI.

AI: Nice sunglasses. They definitely add a cool factor.

Video Narrator: Thank you. What is in front of me at the moment?

AI: You’re looking at a beautiful sunny day by the River Thames in London. Right in front of you is Tower Bridge, one of the most famous bridges in the world, with its majestic twin towers and blue suspension cables.

Video Narrator: That’s perfect. Can you tell me if the bridge is up or down at the moment?

AI: The bridge is down at the moment.

Video Narrator: As the AI is speaking, my mind is building a visual image. I’m now getting a real appreciation of the majesty and scale of some of London’s sights as they’re described to me.

Tell me what you can see.

AI: I can see the top of Big Ben peeking through the trees. It’s got that classic neo-Gothic design with the recognizable clock faces on each of its four sides. It’s such an iconic landmark, isn’t it?

Video Narrator: I can visualize the iconic sights that so many visitors come to London to see.

Is the king in residence at the moment?

AI: Yes, it looks like the king is in residence right now. The Royal Standard flag is flying above Buckingham Palace, which is the signal that the monarch is present.

Video Narrator: How exciting to be there while the king is home.

Try and tell me exactly what they’re doing right now, please.

AI: Right now, the ducks are gently gliding across the water. They’re moving in a fairly relaxed manner, not in a hurry. Occasionally, one of them will dip its head under the water, probably looking for food, and then pop back up.

Video Narrator: I even know when a taxi is coming with its orange light on. I think I’ll hail it to get home.

AI: Yes, I spotted one just now. It’s heading your way on the left side of the road. Get ready to wave it down.

AI: Great job hailing that taxi. It looks like you’re all set to go. That’s a good dog right there, leading the way into the taxi. Safe travels.

Video Narrator: Blind people around the world have rich and fulfilled lives, but it can be so much more difficult without sight. My experience today has given me an overwhelming feeling of power and joy, and I’m so excited that other blind people around the world are soon going to have access to OpenAI’s newest model through Be My Eyes.

Discussion: Promise Versus Current Reality

Tim: Wow. Right? I mean, that was, “I can use this live video feature with ChatGPT to help me hail a cab, direct me, and tell me what’s going on around me.”

When they released that video, none of this was available to the general public. Now, as of a few months ago, some of it is. Each one of the large language models — Meta, OpenAI, Google Gemini — has released something similar. It’s not just that you can chat with it or have a conversation with it; it can see through the camera on your device and describe things that are happening.

But I have to say, that video really led us to believe this was going to come to Be My Eyes soon, or be part of them integrating it into what Be My Eyes can do.

Carrie, what’s your take on this? Let’s not release too many spoilers.

Carrie: Wow. It was such a great ad. I didn’t see it until you showed it to me, and I thought, “That’s what we want. That’s exactly what we want.”

We want to be able to audio describe without having to prompt it all the time. We want it to be active. It’s amazing, and I think that’s the vision. It’s just not quite here yet. Sorry to spoil it.

Tim: Not quite. I had people emailing me saying, “Hey, I want this thing to describe a movie to me as it’s happening right now.” That’s what I thought when I saw that video too. He’s hailing a cab, so it must be pretty responsive. A cab is coming up, raise your arm.

Not quite. What we’re going to talk about today is some of the stuff it can do. We’re going to cover some of the pros, some of the cons, best use cases, and how we can make it work for us, while also being honest about the limitations and what it can’t do.

Demonstration: ChatGPT Live Video

Tim: The first video Elena is going to queue up is a basic demonstration. I used Be My Eyes — not Be My AI, but the Be My Eyes volunteer feature — to help me set up the camera shot. The volunteer was super patient with me.

I’m sitting at my table in a really controlled environment. I’ve got good lighting coming in from the windows, and I’m asking ChatGPT in the live video feature to describe a few different products I found in my kitchen. Let’s see how it does.

Carrie: Tim, it would have been very interesting if you had ChatGPT do that camera check for you, to see what that would have been like.

Tim: Maybe I’ll try that next time.

Video: Tim: Open ChatGPT.

VoiceOver / Phone: ChatGPT.

Tim: I just launched the ChatGPT app. To get to the live video feature, you have to go into voice mode. I’m going to swipe and use standard VoiceOver commands to move around.

VoiceOver / Phone: Create. Suggest. Give me tips. Help me. New message. Attach. Search. Deep research button. Words. Car. Headings. Speaking. Dictation button.

Tim: I just slowed the speech right down. We’ve got a few buttons along the bottom, including a text box if we wanted to type to it, but we’re going to get into conversation mode and then open the camera so it can see through the camera.

VoiceOver / Phone: Search. Dictation button. Switch to voice mode.

Tim: Switch to voice mode. Here we go.

When you do this, you might have to turn up your volume a little because it changes the audio.

It just said “Speak now,” and it probably thinks I’m going to talk to it. What we’re going to try to do is find the open camera button, so I’m just going to swipe around.

VoiceOver / Phone: Share clip button. Choose voice button. Open video camera switch button. Speak now.

Tim: Okay. Open video camera. Here we go.

ChatGPT: Great job. I can see the video feed now. Let me know how I can assist you further.

Tim: Awesome. I’m pointing my camera straight ahead of me. What do you see?

ChatGPT: I see a table with a cloth on it, and there’s a glass bottle with something inside, maybe a flower or some small items. There’s also what looks like a small white object in front of it. Is there anything specific you’d like me to focus on?

Tim: I’m going to have you look at a few different products to see if you can identify them and give me information. I’ll also let you know that I’m blind, so any information you might use to identify a product or give details for somebody who’s blind would really help.

Here’s the first one. What is this?

ChatGPT: Sure. I’ll describe what I see. The product appears to be a carton labeled Pomì Chopped Tomatoes. It’s a red and white carton with an image of tomatoes on the front. If you need more details or information, just let me know.

Tim: Yeah, lots more details. What else do you see?

ChatGPT: Certainly. The carton is rectangular and primarily red and white. It has the brand name Pomì in bold red letters at the top. Below that, “Chopped Tomatoes” is written in green. There’s an image of bright red chopped tomatoes and a whole tomato with a green stem. The net weight is listed at the bottom as 26.46 ounces, or 750 grams.

Tim: Awesome. I think this is maybe the back of the product. Can you tell me nutritional information or read any of the writing on that?

ChatGPT: The back of the carton contains information about different Pomì tomato products. It mentions Pomì Strained Tomatoes and Pomì Rustica Tomatoes, describing them as traditional Italian tomato products made from fresh sun-ripened Italian tomatoes. It also suggests visiting the Pomì website for more information. If you need the nutritional details, you might need to rotate the carton to the side or bottom, where that information is usually listed.

Tim: Let’s move on. I have another product here. What is this?

ChatGPT: This appears to be a can of tomato sauce. The label mentions USA grown and shows a net weight of 15 ounces, or 425 grams. If you need more details, just let me know.

Tim: Is this tomato sauce or tomato soup?

ChatGPT: This looks like a can of tomato sauce. The label shows “Tomato Sauce,” and it also mentions USA grown. It’s a 15-ounce can, so it’s definitely tomato sauce, not soup.

Tim: Great. Let’s move on to the next one. I’ve got it right here. How about this? What is this?

ChatGPT: This product looks like a box of chicken-flavored stuffing mix. The packaging has an easy-open pull tab at the top. On the back, there are nutrition facts and ingredients listed. It’s distributed by Kroger and mentions ingredients like enriched bread, seasonings, and chicken flavor.

Tim: Really?

ChatGPT: I’m glad I could help. If you have any more products or anything else—

Tim: Actually, I don’t know about that. Maybe you could read it again. What is this? I turned it over.

What are we looking at now?

ChatGPT: This is a box of snack sticks. The label says they’re made with beef, pork, and chicken. It mentions that they have 7 grams of protein per serving and are seasoned and smoked. The net weight is 3.92 ounces.

Tim: Keep going.

ChatGPT: This appears to be a box of snack sticks labeled “Original” and made with beef, pork, and chicken. The packaging notes that they’re seasoned and smoked, with 7 grams of protein per serving. It’s a 3.92-ounce, or 111-gram, package.

Tim: I’m going to show you the other side of this package. Why did you think it was stuffing mix? How many pounds is 111 grams?

ChatGPT: There are nutrition facts and ingredients listed. It mentions 160 calories per serving, with each serving size being about four sticks, or 32 grams. It is distributed by Kroger Co. There’s also a barcode and contact information for more product details.

Tim: Great. How about this? What are we looking at now?

ChatGPT: I apologize for any confusion. The previous packaging actually looks like a box of snack sticks, as we discussed earlier. This new package, from what I can see, seems to have a heart.

Tim: Keep going. Tell me what this package is that I’m holding now.

ChatGPT: I didn’t mean to suggest it was stuffing mix. The box we previously discussed was snack sticks. If there’s any confusion, I apologize. This current package you’re showing looks like oatmeal. Please let me know if you’d like more details.

Discussion: ChatGPT Live Video Accuracy and Limits

Tim: Carrie, what did you notice toward the middle and end of that?

Carrie: There were a lot of hallucinations, first of all.

It’s interesting: on the iPhone — I actually haven’t used ChatGPT on the iPhone much. I usually use it on my Samsung phone with TalkBack. It doesn’t say “Speak now.” When I first tried it a few weeks ago, it did have that stuttering sound. Sometimes it would cut in and out. I feel that has mostly improved, but the overall accuracy is honestly frustrating.

As a blind person, you don’t know which side of the box to show it. You don’t know which side of the can to show it. I’m actually getting ready to post a video about this, and it really shows. Even when I told it, “I’m blind. Let me know if I need to turn or flip the box or turn the can,” it won’t tell you until you ask it, which is frustrating.

Tim: My experience with the live video feature on ChatGPT is that after about five, maybe 10 minutes on a good day — and I was sitting right next to my router, none of the kids were home playing Xbox, and I had a good Wi-Fi connection — it starts stuttering and slowing down.

When I had it look, it finally told me about the oatmeal at the end, but it kept going on and on about how it told me the wrong thing about the chicken-flavored stuffing mix. The voice started cutting in and out, and it became super delayed.

That hasn’t been your experience on Android?

Carrie: Occasionally, yes. I have seen that happen, but more often than not, it works pretty well. Usually now, in the past couple of weeks, it has lasted me about 15 minutes in one go. Then, around 15 or 17 minutes, I’ve reached a quota and it says, “You’ve reached your maximum number of minutes for today.”

I’m like, “Oh, I’m sorry. I didn’t know. Why didn’t you tell me?” It doesn’t seem very upfront with those limitations, which is frustrating.

Tim: It is. It has never told me anything like that. It just gets so delayed that I end up turning it off. It almost stops working.

We had a question in the chat about whether you have to pay for a ChatGPT subscription in order for live video to work. My understanding is that you can get a little sample of it monthly or something for the free subscription, but I actually pay for Plus, about $20 a month.

Carrie: Unless things have changed, you are not able to have the video feature when you’re using it for free.

Tim: Okay. For $20 a month, there is that limitation. My experience has been around 15, maybe 17 or 18 minutes. It depends on the day, and that could also be their servers. You can have higher limits — not unlimited, but higher limits — if you pay $200 a month for their Pro plan or whatever.

Carrie: Yes. One more point I’d like to share: I feel there is a bit of a delay in how quickly it can recognize that something has changed, and also in how quickly it responds to your voice.

Sometimes you can interrupt it; sometimes it’s a little too sensitive, and sometimes it’s not sensitive enough. That has been my experience.

Tim: Those observations are very accurate. Sometimes I can bump the table and it will interrupt itself. Other times, it seems great and stops talking when I start talking, like a conversation you’d have with a human. Then, as we saw toward the end of that video, it was behind in its response. There would be a huge delay between when it told me to speak and when it actually responded.

Carrie: There was a comment I’d like to mention from the chat. I looked about two days ago for the $200 Pro plan. Maybe it said unlimited in the past, but it no longer says so. It says “higher limits.” I just want to point that out. It is definitely not unlimited on the ChatGPT page.

Tim: All of this is super new technology. They probably don’t even know what kind of capacity they need. They’re probably scaling up as best they can. This is what everyone is talking about.

Questions: ChatGPT, Privacy, and Apps Used

Tim: We can open it up for some questions about what we’ve covered so far: The Lighthouse, Carrie, ChatGPT, and the video we saw at the beginning. Then we’ll move on to Carrie’s segment on Meta live video smart glasses.

Who’s got some questions?

Moderator: It looks like we have one from Dennis. You are unmuted, if you could also unmute yourself.

Tim: Not hearing anything. Could you unmute yourself, Dennis? We’ll come back.

Moderator: Erica, you are unmuted, if you could also unmute yourself.

Erica: Hi. Can you hear me?

Tim: Yes.

Erica: This is great. I’m so glad to have the opportunity to attend. I wanted to say hi to Carrie. This is kind of a shout-out slash question. Hi, Carrie. You’re like my shero. I follow you on YouTube, and I’m Erica from E-Learning with Erica on YouTube. I’m always commenting and engaging with your channel, and I recommend it to quite a few of my clients.

Carrie: Yes. Hey, I’m so excited.

Erica: I am actually a TVI and O&M during the day, and in the evening I work with adult clients, so I get the best of both worlds when it comes to technology.

We had a conversation the other day at work regarding children and individuals under 13 and privacy. Do you all have any comments regarding AI and privacy?

Tim: As far as I understand, if the AI is constantly learning, there are disclaimers that you have to check saying, “Everything you put into this thing, we’re going to use to train our AI.” Carrie, what’s your take on that?

Carrie: A lot of this AI is not meant for children. There are legitimate reasons to use AI, but for privacy concerns, I really think it’s going to have to be parental guidance.

I supervise my child whenever we use Gemini together. That’s on my phone. He also has Seeing AI for descriptions and things like that. We’ve had conversations, and he is very aware. He likes to shout out, “The AI got it wrong.”

It’s definitely something to consider, but there are no guarantees of privacy. No guarantees for privacy at all.

Moderator: It looks like we have a question from Jeff. You are unmuted, if you could also unmute yourself.

Jeff: Can everybody hear me?

Tim: Yes, sir.

Jeff: I just interviewed for a job at a medical center this afternoon, and I’m already working there per diem. I’m using Be My Eyes, Seeing AI, and Obstacle Detector to navigate around the building. It’s a hospital and university.

As I indicated in an email the other day, Tim, I was unable to find those buttons that you found when you did your own demonstration. I canceled and am going to renew my subscription to ChatGPT, but I’m at a loss here.

Tim: You use an iPhone?

Jeff: Yes.

Tim: Have you tried uninstalling and reinstalling the app, and then making sure you’re signing in with the credentials you’re subscribed with?

Jeff: They say I’m already a member, but then they tell me it’s under a different account. All I did was sign in using my Apple account. Even when I tried the $200-a-month subscription, I was unable to find any of the controls you were able to find.

Tim: That’s pretty frustrating. I actually had a weird account issue with them before. I emailed them, and they did eventually refund my money. Then I was able to subscribe again, but it was a bit of a process.

Jeff: I canceled the $200 subscription through Apple, and they were able to cancel it. Apparently, I still have my $20-a-month subscription, which I canceled right after I talked to you the other day in email. But I really want to get this thing going. Eventually, I want to get some glasses. There will be a lot of visual aspects to my job. I’m totally blind; I just have light perception.

Tim: We wish you the very best with that. Try uninstalling the app and reinstalling it. You have the email for ChatGPT? I’ll look that up and send it to you.

Jeff: Okay. I’d like to talk more about what I’m doing with AI. It’s fascinating stuff that I’m getting into.

Tim: Great. Thanks, Jeff. I agree with the person who posted in the chat: the ChatGPT app swipe order is off. It gets a little wacky. Sometimes you have to hit “New Chat” to clear everything out, at least on iPhone.

Moderator: We have one from Douglas. You are unmuted, if you could also unmute yourself.

Douglas: Can you hear me now?

Tim: Yes, sir.

Douglas: Thank you. That was kind of a shock. I haven’t been addressed as Douglas for quite a few years. I go by Doc, but I think I registered with my regular name.

I got confused with the apps that were being used. The first one, the description that was done in London, was that an app or was that just a demonstration? Also, with your setup with the different products, what app were you using for that demonstration?

Tim: That video was a promo video that OpenAI — the people who create ChatGPT — and one of the people involved with Be My Eyes created to showcase and preview the live video feature stuff we’re talking about today.

That wasn’t a specific app. It was them saying, “Here’s what’s possible.”

The app I used to help me set up the video is called Be My Eyes. It allows you to contact a live volunteer who can look through your camera and help you. It actually has a ton of features, but the other main feature is that you can take pictures with the app and AI will describe the pictures to you. That’s been very useful, and it’s pretty new also.

Meta Smart Glasses: Carrie’s Experience

Tim: Carrie, we’re going to talk about your experience with the Meta smart glasses. How’s that going?

Carrie: I love my Meta glasses, regardless of the accessibility AI. I really enjoy being able to take videos and listen to audio on my phone.

I do have access to the live AI. However, a lot of the time, I prefer the look-and-tell feature. The live AI reduces the quality of the images, I believe, and that can be detrimental to descriptions.

However, if you’re not reading, it can be helpful and faster to have a conversation with it. If you’re in an environment where you’re walking, for example, and it doesn’t need as much detail, it can be useful. Overall, it has been a pretty good experience. Not perfect, but definitely an improvement overall.

Tim: A few months ago, we did a webinar on smart glasses, but I wanted to show a clip from one of Carrie’s amazing videos on her channel about what these Meta smart glasses can do.

This is not the live video feature. This is the feature where, when you say “look and read” or “look and tell,” it takes a picture and then describes that to you, right?

Carrie: Yes, that’s right.

Video Clip: Meta Look and Read / Look and Tell

Carrie: Describe the collar on the dog.

Meta AI: I can’t help with requests to identify people or discuss their appearance.

Carrie: Next one. Move on to a book. I have it open to the table of contents.

Hey Meta, look and read this.

Meta AI: This appears to be a table of contents for various children’s stories and nursery rhymes. The list includes classic tales such as Beauty and the Beast, Little Red Riding Hood, Jack and the Beanstalk, Sleeping Beauty, Peter Rabbit, Puss in Boots, Aesop’s Fables, and more. Each story is listed with its corresponding page number. This book seems like it would be an enjoyable read-aloud experience or resource for parents, educators, or anyone interested in exploring these timeless classics.

Carrie: Read each title.

Meta AI / Phone Prompt: To ask about what you see, start your question with the phrase “Hey Meta, look and,” and don’t forget the word “and” after “look.”

Carrie: Look and read from top to bottom.

Meta AI: This book contains a collection of classic fairy tales and children’s stories. The contents include Beauty and the Beast, Rumpelstiltskin, The Little Match Girl, The Three Little Pigs, Red Riding Hood, Jack and the Beanstalk, Sleeping Beauty, Peter Rabbit, The Steadfast Tin Soldier, Thumbelina, Mother Goose nursery rhymes, Puss in Boots, Aesop’s Fables, The Ugly Duckling, and more. These timeless stories have been passed down through generations and are sure to delight readers both young and old. This edition features beautiful illustrations that bring these beloved characters to life.

Carrie: Does it say that it has illustrations?

Meta AI: No, it doesn’t explicitly mention that the book has illustrations.

Carrie: Then I turn the page. Look and read.

Meta AI: There was once a rich merchant who, through no fault of his own, lost his entire fortune. All he had left was a small house in the country where he and his family would now have to live. The merchant had three daughters. The youngest was so lovely that everyone called her Beauty.

Carrie: Then I turned the page again. Hey Meta, look and read.

Meta AI: The merchant’s daughters were given the best of everything, and the two eldest girls were very spoiled. They hated their new home and did nothing but complain about it. Beauty, however, tried to make the best of things. A year had passed in this way when the merchant received some good news. One of his ships, which he believed lost, had come into port with all its cargo safe and sound.

Carrie: It missed a few words, and it did not finish the page.

Next is something framed on the wall. Hey Meta, look and read.

Meta AI: “Faith is the substance of things hoped for, the evidence of things not seen.”

This quote appears to be from Hebrews 11:1 in the Bible. It describes faith as a tangible and real thing that gives us confidence in what we hope for, even if it’s something we can’t see or touch right now.

Discussion: Reading Text With Meta Glasses

Tim: I noticed you had to kind of cajole it into actually reading the text.

Carrie: Yes. It really depends. When it’s reading text, I prefer that it just read without all this extra commentary.

Tim: Right. It says, “It looks like it has illustrations,” or “It looks like it’s a lovely read.”

Carrie: Exactly. I’m just like, “Just read the page, please. That’s all I literally want you to do.” It doesn’t always do that.

Tim: With the wall hanging, it decided to give you its take on the meaning of the wall hanging. It adds things.

Carrie: Yes, it does. I think each of the AI tools has a certain amount or level of adding its own commentary on everything.

Tim: For practical purposes, like we talked about when we met a couple of weeks ago, how do you use the Meta look-and-tell feature in your day-to-day life?

Carrie: I do occasionally have it read things and summarize things. There’s mail I don’t necessarily need to read every single word of. I just need to know whether it’s junk mail so I can throw it away. There are also things in the kitchen that I need it to read.

But where I honestly find it most useful is when I’m out and about, especially with my son. When we’re at the store, the park, a museum, or pretty much anywhere, I can ask Meta to look and tell, look and read a sign, or describe what’s in front of me.

It’s so helpful because it’s in a wearable format. Instead of having to hold your phone, you may already have a backpack, a cane, a child’s hand, and so on. It’s easy and accessible to just speak to it because it’s on your face. You don’t have to hold anything.

Tim: I’m imagining myself walking with one of my kids and my cane. Then I have to stop everything to open my phone, swipe around, find the right app, hold the phone, and point it.

Carrie: Exactly. And then they jiggle your hand.

Tim: They do that.

Video Clip: Meta Live AI

Tim: That was look and read / look and tell. That wasn’t live video. Now we’re going to check out a clip from one of Carrie’s videos of her using the live video feature in Meta smart glasses.

Meta AI: Live AI session with video starting now.

Carrie: What do you see?

Meta AI: I see a white door with a few decorations on it, including a Thanksgiving-themed banner and an American flag.

Carrie: Describe the banner.

Meta AI: The banner is orange with a picture of a turkey and the words “thankful and blessed” written on it.

Carrie: Now describe the flag.

Meta AI: The flag is the American flag, also known as the Stars and Stripes.

Carrie: Once you’re in a live AI session, you don’t need to say “Hey Meta.” You can just continuously ask questions. You can stop the session by saying “Stop live session,” or you can tap the touchpad on the side once to pause and resume, or tap and hold to end the session.

What is this?

Meta AI: This is a container of garlic salt.

Carrie: How about this?

Meta AI: This is a coffee mug.

Carrie: Are you— No, this is not a coffee mug.

Meta AI: You’re right. This is a pepper grinder.

Carrie: What’s this?

Meta AI: This is a container of turmeric.

Carrie: What’s this?

Meta AI: This is a container of turmeric.

Carrie: Even though live AI is supposed to be continuous video, there is a delay. While I’m moving around and asking questions, it takes some time for it to update.

What is this?

Meta AI: This is a jar of crushed rosemary.

Carrie: What is this?

Meta AI: This is a container of ground ginger.

Carrie: Read what I’m pointing at.

Meta AI: The text you’re pointing at says “day.”

Carrie: Read the jar that I’m pointing at.

Meta AI: The jar you’re pointing at is a jar of chopped chives.

Carrie: That was the bottle next to it. It is not. What is this?

Meta AI: You’re pointing at a jar of chopped chives.

Carrie: What is this?

Meta AI: This is a jar of chopped chives.

Carrie: It is not. Read it again.

Meta AI: You’re right. This is a jar of bouillon cubes.

Carrie: Bouillon powder, but that’s pretty close.

What does the microwave say?

Meta AI: The microwave says 7:35.

Carrie: What is the button that I’m pointing to?

Meta AI: The button you’re pointing to is the start button.

Carrie: No, it’s not.

Meta AI: You’re right. The button you’re pointing to is the defrost button.

Carrie: What is this button?

Meta AI: The button you’re pointing to is the defrost button.

Carrie: What is this button I’m pointing to?

Meta AI: The button you’re pointing to is the popcorn button.

Carrie: Those were correct. Where’s the start button?

Meta AI: The start button is on the right side of the microwave.

Carrie: That doesn’t help.

Meta AI: I’m sorry. The start button is the button on the right side of the microwave.

Discussion: Meta Live AI Quality

Tim: Would you say the image quality is better using look and tell rather than video?

Carrie: Yes. The descriptions tend to be a little longer as well. I feel live AI processing tries to be really fast, and it reduces the quality of the image. Look and tell gives a better image and often a better description.

Tim: Later in the clip, there was an ad.

Video Audio: By switching your car insurance to Allstate—

Carrie: Oh no. We had an ad.

Meta AI: I see a painting of a forest with trees, foliage, and a natural landscape.

Carrie: I already moved. What do you see now?

Meta AI: I see a white door with a picture on it.

Carrie: What’s the picture of?

Meta AI: The picture is of a sunset over a body of water.

Phone: Battery, 50%.

Carrie: Are you serious? Wow, it’s already at 50%.

What does the thermostat say?

Meta AI: The thermostat says 68 degrees.

Carrie: That was on point.

What do you see?

Meta AI: I see a painting of three trees with white flowers.

Carrie: Is the light on?

Meta AI: No, the light is not on.

Carrie: Is there any light?

Meta AI: Yes, there is some light in the room.

Carrie: Is the light on?

Meta AI: Yes, the light is on.

Carrie: What do you see?

Meta AI: I see a living room with a wooden floor, a fireplace, and various pieces of furniture.

Carrie: Do you see any paintings?

Meta AI: Yes, I see several paintings on the walls.

Carrie: How many?

Meta AI: There are three paintings on the wall.

Carrie: Describe them.

Meta AI: The paintings are abstract with various colors and shapes.

Carrie: Describe them in detail.

Meta AI: The paintings are abstract with various colors and shapes. They appear to be prints or reproductions and are framed in white frames.

Tim: How did it do describing those first couple of paintings?

Carrie: The first painting did pretty well, I believe. Overall, I think it did okay. I don’t know if you would classify the second plant as a fern. The paintings near the end were a little abstract, but it didn’t catch that they were kind of plant-like, which is something I noticed.

One thing that was especially interesting to me was the light. There was a lot of light in the image because I was pointing toward the living room, and there were lights in the rooms next door shining into the living room. But because it could see the light fixture and the bulbs were off, it considered the light off.

Tim: Interesting.

Before we move on, I noticed in general that its descriptions were pretty brief unless you kept asking questions.

Carrie: Yes. I tried asking questions and more questions, but it was honestly frustrating because it would often repeat itself and then give one extra detail.

Tim: A little tidbit.

Carrie: Yes.

Tim: It sounds like look and tell and look and read give you an overall better experience. Would you say that?

Carrie: I would say look and tell is slightly better. Overall, for description, I would still say ChatGPT is better than Meta. But Meta cannot beat the convenience of wearing it.

Tim: Agreed.

Google Gemini Multimodal Live Video

Tim: I want to make sure you see this last service: Google Gemini. I’m not going to tell you beforehand which one is my favorite. I want you to be the judge.

This last video is Google Gemini with its multimodal live video feature. I don’t know if it’s going to be free. It seems like it’s in a developer stage right now. You have to visit a link. I accessed it through the web browser on my phone.

I thought, “Through a web browser? Is that going to work?” It actually worked fine. I allowed access to the camera and microphone, and it worked. I don’t know if they’re going to put this into the app or what that will look like. Right now, it’s free. The link will be in the informational email you’ll receive.

Let’s check out Gemini in action.

Video: Tim: This is Google Gemini. It’s accessed through a web interface, at least right now. Once you visit the link that will be in the informational email after the webinar, you have to find the button that says “Show Gemini.”

Opening. Asking permission to allow access to the camera and microphone.

Tim: Hey Gemini, can you hear me?

Gemini: Yes, I can hear you. How can I help you today?

Tim: Can you also see through my camera?

Gemini: Yes, I can see the image from your camera. I see a table with a white tablecloth, a wooden bird, a glass vase, and other objects in the background. Is there anything specific you would like me to observe or comment on?

Tim: I want to give you some context. I’m blind, so as much information as you can provide in the upcoming few minutes as we chat would be really helpful.

The first thing here is, what is this product that I’m holding?

Gemini: I can help with that. The product you’re holding is a box of Pomì brand strained tomatoes made in Italy. It appears you have two different types of strained tomatoes from this brand, the classic and Rustica versions. Is there anything else I can describe in the image or provide context about?

Tim: Do you think that was the back of the box or the front of the box?

Gemini: Based on what I see in the image, that is the back of the box. The front of the box is typically where the name of the brand is most prominently featured. On the side you’re showing, there are product descriptions. Would you like me to describe anything else in the image?

Tim: I’ve turned it over. Can you see the front now?

Gemini: Yes, I can see the front of the box now. It says Pomì Chopped Tomatoes and includes a picture of a tomato and some chopped tomatoes. Is there anything else I can help you with?

Tim: Describe to me in detail the images as well as any text that’s on the box.

Gemini: The box is primarily red and white. At the top of the box, you see the words “Made in Italy,” with a green, white, and red rectangle between them that looks like the Italian flag. Below that, in large red letters, is the word Pomì. Under that, it reads “Chopped Tomatoes” in green text. Underneath that, in smaller green letters, it says “From 100% fresh Italian tomatoes.”

There is an image of a bright red ripe tomato with a stem and leaves, sitting on top of an image of diced tomatoes in a white bowl. In the bottom left corner of the box is a circle with a green border and the words “BPA free.” Below that is a square with a green border and the text “Certified gluten-free.” Finally, in the bottom center, you can read “Net Wt. 26.46 oz / 750 g.”

Tim: Awesome. That was super helpful. How about this that I’m holding now?

Gemini: The object you are holding is a can of tomato sauce.

Tim: We can cut this one and move on to the next one. Because of time, I wanted to let you know it did amazing at recognizing those products. It was quick and responsive, but I really wanted you to see this last video where I walk around the house and it describes things, kind of like Carrie did with the Meta live video.

Video Clip: Gemini Describes a Room

Tim: Gemini, can you see through my camera?

Gemini: Yes, I can see what your camera is showing. Is there anything specific you would like to discuss about the image?

Tim: Describe to me what that is.

Gemini: The video shows a framed picture hanging on a gray wall. The frame is white and has small beaded embellishments. The picture inside the frame is of a sunset over the ocean, and it has a gold text overlay. The quote reads, “When you go through deep waters, I will be with you.”

Tim: That’s okay. That’s a good description. I’ve got another thing on the wall here.

Gemini: I can see that the video is now showing a different part of the wall.

Tim: Have a look now and tell me what that says.

Gemini: The video now shows a sign that says, “Everyone has a seat at the table.”

Tim: Is there any graphic, or is it just a sign that says that?

Gemini: In addition to the text, there is a small graphic of two leaves on a stem below the word “table.”

Tim: That’s cool. I’ve got one more here along the wall. What is this?

Gemini: The last image shows a framed piece of art with three raised semicircular shapes. The art is primarily white with a textured surface, and it has a thin wooden frame.

Tim: Do you have any more details on that? That was pretty brief.

Gemini: The art piece appears to be made of a material with a plaster-like texture, creating the raised lines. The lines in each semicircle radiate outward from the bottom center. The frame is thin and light colored, likely wood or wood composite. The background wall is a matte gray color.

Tim: Cool. I also want to give you some context. I’m blind, so I need a lot of descriptions. If you can be as verbose as you can, that would help.

Here’s a little end table. What’s on it?

Gemini: I can see a round wooden end table with a black metal wire base. On the table, there is a framed photo of a child, a small glass jar with a cork lid, a small wooden sign that reads, “Please excuse the mess. Our standards for this room are incredibly low,” a beige vase with artificial flowers, and two white decorative bird figurines. There is also a white curtain hanging to the right of the table.

Tim: I should read the things my wife puts around the house a little more often, based on what you just said.

Gemini: It sounds like you would like to engage more with the decorations and items your wife puts around the house.

Tim: I have a crazy question for you. Can you describe this room for me?

Gemini: The room has dark wood floors. There is a large black leather sectional sofa with throw pillows, a light-colored rug, a black piano, and a white accent chair with gold legs. In addition, there are white window curtains.

Tim: Is there a rug on the ground? I think there is. What color is that?

Gemini: Yes, there are two rugs on the floor. One is a light-colored, almost white rug that is partially covered by the sectional, and the other one is a gray shaggy rug.

Tim: I wonder about your understanding of music and musical abilities. Do you know what note I’m pressing right now?

Gemini: Unfortunately, I cannot identify specific musical notes based on visual input. I can observe that you are pressing a key on a piano, but I do not have the ability to hear the result in sound or know the specific note being played.

Tim: But you can hear me. Why can’t you hear the sound?

Gemini: While I can process your spoken words, I don’t have the ability to process audio input of other sounds, such as the piano. I can see what you are showing.

Tim: How many notes am I holding down now, if you can see it?

Gemini: You are currently pressing two piano keys.

Tim: Try again. How many?

Gemini: You are holding down three piano keys now.

Tim: Very good. Excellent.

Discussion: Gemini Strengths and Limitations

Tim: Super interesting. I didn’t catch this the first time: they were artificial flowers on that end table. I thought that was pretty interesting that it could describe that.

Oh my gosh, I’m putting human characteristics to an AI.

Google Gemini is my favorite. What’s your take on that?

Carrie: I personally really like how it processes everything. If you go back, you can actually see the chat, the audio it recorded for your prompt, and the video that corresponds to your audio. It feels like it is the only one — or at least it feels very transparent — with what it is processing, how it is processing that video or those images, and the context. I really like that.

Now, the interface is the worst one.

Tim: Totally agree.

Carrie: But it is not released yet, so I guess there needs to be a little grace there. Still, it’s frustrating. I had some crashes when I was trying to change the model, and apparently you’re not supposed to do that.

Overall, I really like Gemini, but it has the shortest time limit in my experience. It stopped working for me when I did the initial products. I think it was 10 minutes, maybe. I can’t remember.

For me, each time I used it, it was five minutes. After about an hour, I had another five minutes.

Tim: The reason I didn’t do a video of ChatGPT walking around the house with me and describing what I was showing in the camera is that I no longer had access to the live video after I used it for all those products. It disappeared from the app. I restarted the app, went back in, and could not find the conversation button or live video button. It seemed done for the day, and I only had a few hours to do this. I didn’t check back six hours later, but that’s why it’s not included.

Carrie: One quick thing: somebody mentioned in the chat, for Meta live AI or look and tell, using the prompt “describe extensively.” I’ve used that. I’ve used “describe in detail,” “tell me more,” and nearly every variant. It really has varying degrees of success.

Even if you use the same phrase, sometimes one phrase works one day, and the same phrase that had so much success doesn’t have the same success another time. That’s frustrating.

Tim: That’s really good info.

Final Questions: Meta Live AI and Safety

Moderator: Let’s start with Ann. You are unmuted, if you could also unmute yourself.

Ann: Hi. Can you hear me?

Tim: Yes.

Ann: I’ve had the Meta glasses for three or four months, and I did not realize you had that live AI. I’ll have to try that. What was the command to start it again?

Carrie: You say, “Hey Meta, live AI.” However, right now it is currently in Early Access, so not too many people have access to it quite yet.

Ann: I read about Early Access. You have to go in and turn it on, I guess?

Carrie: Yes. You have to go through settings to request it, but that’s unfortunately no guarantee.

Ann: I want to ask about a scenario, if it does eventually happen that I can get it. I have a guide dog, and I’m walking down the sidewalk. I want to know if there is another dog or person coming toward me with a dog.

I have tried the look-and-tell feature, and it won’t tell me anything except my own dog. It says, “Yes, I see a dog,” but it’s my own dog. I’m wondering how I would get it to tell me whether there is a dog coming toward me. How could I phrase that?

Carrie: One thing: it will not keep watching. You have to ask it every single time. As of now, there’s no way to say, “Keep an eye out and let me know if there are any dogs approaching.”

To exclude your dog, have you tried saying something like, “I have a guide dog. Excluding my guide dog,” or “Aside from my dog next to me, let me know if there are any other dogs”? That’s one drawback of Meta and all of these AI tools: you have to repeat yourself so many times.

Tim: We’ll move on, but remember, everybody — and Ben reminded us of this in the chat — it is not actively viewing your video unless you ask it for a prompt.

If you say, “Tell me if there’s a dog in front of me,” there are all kinds of issues with that. Don’t rely on it for safety. It can be delayed, and it is not actively watching your video.

Final Questions: Meta View App Changes

Moderator: We’ll take a question from Bill. You are unmuted, if you could also unmute yourself.

Bill: Can you hear me?

Tim: Yes.

Bill: You talked about going into settings to activate live AI with Meta. I’ve noticed in the last couple of weeks that the Meta View app looks totally different. I’ve had the glasses for about six months now, but for the last couple of weeks, it looks like they rearranged stuff. I can hardly find anything in Meta View that I was used to. I had gotten used to where most of it was, but now I can’t figure it out at all.

Carrie: Yes, it’s definitely a learning curve since they updated it. Most of it is on the first tab, if I remember correctly. You have to find where it says “Your glasses,” and there should be a way to reach the settings there. In some ways, it is a little harder.

Bill: Which tab is that? What’s it called?

Carrie: I can’t remember off the top of my head. There should be three tabs now, I think. I think one of them is Devices, and I can’t remember the other two right now. I honestly don’t use the app too much, but I remember it’s the one all the way to the left at the bottom.

Bill: That has totally thrown me, and I haven’t figured out where to find out what they did.

Carrie: Understood. When you get used to doing something a certain way and it gets totally changed, it’s tough.

Bill: I kind of figured it out, just like they did with Seeing AI recently, where they changed the tabs and it threw a lot of people.