Experiential Activations and AI [Part 2]
What are some challenges facing the use of AI for experiential activations?
Now that you have started to think about the potential of AI, let’s talk about some challenges facing all of the above. Don’t worry, then we’ll talk about how we might improve things moving forward so we can continue on the hype train. Choo choo.
Most of these challenges mentioned ahead will primarily look at experiential activations involving personalization, and this is likely more of a “from scratch” type of project versus AI solutions that come a bit more ready-made like image recognition and voice control. This is an admittedly limited scope of AI, but I’m limiting on purpose to keep things focussed. Also, a few of these can be applied more broadly to other forms of AI. Let’s get started.
A lot experiential activations these days revolve around a concept of unique experiences and personalization.
I know you deeply, I anticipate your wants and needs.
At our core, we all want to be acknowledged, listened to and understood.
This thing is understanding me. It knows me.
To really, truly make this connection, you need information about the person interacting. And lots of it. This is a huge challenge — are you usually deeply invested in some stranger you just met for a few minutes?
This brings us to the first big challenge of working with AI in experiential.
Collecting enough data from users in real time for personalized experiences:
Experiential activations are usually quick by necessity. Clients want as many people as possible to see and experience what you’ve made and throughput is usually very important. Even at larger events that have multiple stations of interactivity, you usually only have limited places to collect data from users. Pulling from my own memory, I don’t think I’ve participated in an branded experience that has been longer than 20 minutes start to finish, and even those are broken up with discrete 2–3 minute chunks of actual interactivity. Chances are those of you have done one of these transactional experiences:
Wave your arms around -> make some particles. Neat.
Tap a touch screen -> learn a thing. Cool.
Put on a VR headset -> contemplate life. Wow.
Enter your email-> get sent a weird photo of yourself. Awesome.
No one is typically standing around and interacting with a thing for hours, and honestly most experiences aren’t built to be that deep. This means you only have a small chance to capture data about your user and show them what is unique to them.
If you are going down a personalized or unique for everyone road for an activation and need to collect a long questionnaire of data about a user to get something interesting out of AI, you are going to have a bad time — it takes too long and users will potentially feel uncomfortable with such obvious info gathering. This is partially why AI is so useful to businesses that already have a ton of data about their users — they have collected it over many clicks/transactions/listens/behaviors in the background of the user going about their day and they can compare you to millions of other users. You can take cues from this method of collection and try to collect data from a user while they think they are doing something else. If you are too obvious in your intention of collection, the magic will be stripped away.
Even when hiding this data collection as best you can, you’re typically capturing only a small amount of data from a user in an experiential activation — you’re not watching them over days or weeks, you’re not getting all of their public social data (without their consent). this isn’t always enough to glean meaningful results. You have maybe 2 minutes max of their time in a standard activation. You might be able to have them:
- Answer some basic questions/select options on a screen (please, no)
- Interact with physical objects or play a game and draw some conclusions from how they interact
- Stand in front of a camera/be watched by a camera unknowingly
- Do facial detection to detect their emotional state, age, gender and a few other things
- Track their body
- Track their hands
- Eye tracking to check where their gaze goes
- Have them hold up an object that can be detected
- Use other sensors to collect info about them:
- Their heart rate, breathing pattern, galvanic skin response, weight, response time
- Say something out loud and do speech to text transcription
- Submit or send a photo from their personal library
- Enter their twitter/social handle and collect a bit more data from their public social profiles:
- With this, you risk someone not having a social account you need, or not wanting to consent to log in, or not knowing their password, or their account doesn’t represent them at all and only has a few tweets on it that aren’t retweets of random other accounts.
A side challenge to all of this is tracking an individual user across multiple locations if you have more than one thing at an event. This is one of the hardest problems to solve seamlessly and one that retail stores are working hard on. Things like Amazon Go are a pretty close embodiment of what tracking your every move and intention can become. At events, you have things like RFID bracelets, unique ID codes, user’s names, apps. Very soon, things like facial recognition might reduce friction a bit, but with a big tradeoff in terms of user privacy. Someday there may also be the potential for the blockchain or other technologies to empower users to control the transaction of their data a bit more without feeling like they are giving up their privacy.
Of course, sometimes you luck out and have a client who may already has a lot of interesting data about their user base. This leads us to our next challenge:
Getting access to client data
When you start pitching on a project you’re often talking with a company’s marketing team. That team may desire one outcome, but may not have run certain assumptions past their data science or technology teams. I’ve seen many projects hit a wall because a project would require access to client data or deep integration with their systems, and because this requirement wasn’t vetted beforehand the project enters a limbo state. Clients typically have very good reasons for this, and we’ll talk about how to improve things on both sides later.
Getting access to client’s user data, IP, or transactional data may be their core business, and letting that out and risking a leak could cause a lot of issues for them. Aside from leaks, their system may not be set up to be easily accessed from the outside, and that may mean that they would need to assign internal teams to make a tool for you to even get at it for an activation. A lot of companies don’t have programmers sitting around or aren’t set up to just dedicate a team to hacking at their core product so it can be used in a stunt.
Luckily, clients from certain industries will have more accessible (and plentiful) data than others. This means that you might be set up for success (or a long road ahead).
- Certain technology companies such as manufacturers, music and video streaming services, cloud services, social networks will have more readily accessible data, but may still be hesitant about handing it over
- Medical and Financial companies may have a lot of interesting data, but data privacy is a massive part of their business. They may face a lot of challenges with supplying this data to you, even from a legal and regulations standpoint. This applies to municipalities and organizations more closely tied to government agencies as well. Some of the companies may require a more lengthy procurement and assessment process before you can even start learning about the data they have, so be prepared for a few weeks or months of auditing in some cases.
- Fashion, Retail, Food and Beverage, and Auto may have data, but in more spread out formats that aren’t as standardized or broadly applicable
Unfortunately, during the pitch phase when you’re coming up with ideas, not knowing about the types of data you might be able to access can really cripple certain creative. Your team will be like “Surely the [XYZ corporation] has this data somewhere!” and certainly they might. Unfortunately this is a question you probably can’t answer by looking around online. You also definitely won’t know anything about whether they have an easy API to interact with and if it has the data in format that would suit your purpose. There may even be legal or regulatory restrictions to letting outside vendors even look at that data. This makes feasibility, timelines and budgets extremely hard to evaluate accurately. Let’s talk about that stuff next!
Comparatively short timelines and data processing challenges
Every experiential or creative code shop is different. Some specialize in projects that take 6 months to 2 years before they see the light of day. Some may have to turn something around in 2,4,6 weeks from concept, execution to live run. Some do one-offs, make products they use over and over, or a mix of the two. There are budget constraints, special events with deadlines, and other cycles at play here. A limited amount of time usually means reaching for something a little more baked and sure — in the worst case you get something boring that doesn’t even really work. Things like Spotify’s Discovery Weekly, Google Assistant and Amazon Echo are massive systems that poke at the usefulness of AI, but they have massive teams that worked for years on those products. Attempting to recreate anything close in a month or two just isn’t realistic.
In the very best case a client comes in with data that is ready to go, a well documented and functional API for accessing the data. They may even have some initial thoughts about how they think it can be used. However, you may get a client that has disorganized data spread out over many sources that require cleanup and standardization before you can even start working with it to see if there is something interesting in there. This step could be a manual process that could take a small team a few weeks by itself.
The step of data normalization and performing experiments with the data will typically be expertise that falls outside of the skillset of a lot of technologists that work at experiential agencies. Most shops don’t have data scientists sitting around waiting to smoosh datasets together for a hot activation at SXSW. If you want something new that doesn’t use off the shelf tools, you will need to find another company, research institution or university to partner with. Tracking them down can take considerable foresight, networks and probably top-level Googling skills.
Once you have the data and the personnel to do something interesting, you need to figure out how to make it matter.
Drawing meaningful conclusions
Around our office, we have a few shorthand terms for poor applications of AI and personalization. Buzzfeed quizzes and horoscopes are our go to. Things like Buzzfeed quizzes are completely arbitrary mappings of a user’s choices to an outcome. They are fun to a point, but we know we aren’t really a Hufflepuff (really more of a Ravenclaw). Horoscopes are similar — using your birthday to describe what will happen to you that week or the type of person you are have no real basis in fact. AI has the potential to create more factual, truthful and meaningful outcomes. However, if you don’t put the effort in, make something half assed that doesn’t work well, and call it AI, you’re contributing to the image of AI as useless parlor trick that means nothing. Truly meaningful output risks being lost in a sea of noise.
For years I spent a lot of time exploring the relationship between music and visuals. Color scales, music videos, live visuals at concerts, physical representations. You eventually realize that it’s all subjective and in a lot of situations there is no single truth of mapping one set of data to another — correlation≠causation and all that. Your brain will seek out patterns and its super fun to play with those patterns but you have to put in a lot of work to make something that is greater than just the music on its own. A lot of experiential activations out there currently that purport to be using AI are just arbitrarily mapping some of your provided info to some limited outcomes. After working on these for a while, they can feel less like they tap into the suspension of disbelief required for theater, and instead they start to feel much more hollow and toy-like. To really prove that those limited outcomes are truthful and make things feel impactful, it helps if you have the time to run things like a scientific experiment.
For example, let’s say a client has some really cool shoes they want to sell and they want to partner with a music service. Shoes are cool, and music is cool, the kids will love it. The client wants to suggest specific shoes to users based on the music they like. Of course there is nothing about the music itself that would map to a specific sneaker. If you have all the user’s music tastes, how are you supposed to decide that people who are really into Third Eye Blind like this particular style of footwear? Of course you could just make some educated guesses and lump people into groups that you connect to the limited set of shoes. This, however, is not AI. You would need to have both sets of data — the users music preferences AND their shoe collection data — ideally thousands or millions of data points. You could then train a model that looks at music taste and shoe preference and relates them together. Then, given a new user where you ONLY know their music preferences, you have a much more realistic chance of suggesting a shoe they might actually be interested in. If you can then prove that you’re getting more sales success by accurately suggesting shoes based on music preferences, you have really proven your project’s worth.
The above is a very simplified example and some may still consider this a parlor trick application — obviously there is a lot more to shoe preference than music taste. Thank god I’m not planning on becoming a marketing expert.
That demonstrates that these concepts do get harder and harder to relate to various teams, and that brings us to the challenge of…
Educating your team and client about AI
So your client, coworker or another team saw a cool project involving AI, and now they want to know if the same idea can be recreated in-house. AI is very complicated, and it is not always easy to take one idea and translate it to another. The trouble is that it’s not always intuitive about what might be a small step versus a giant leap.
That project that analyzed and identified cat faces? Probably won’t work on dogs without a massive dog face dataset to train on.
It’s also important to consider the source of the project and how long it likely took. That project that took your face and mapped it to faces in famous paintings? Google made that. A small shop probably can’t take a few weeks and make a photo booth that has the same level of quality for comparing users to celebrity faces or cat faces. Even getting up to speed with pre-existing tools can take a few weeks depending on what it is.
Give presentations to your internal teams, walk them through other projects — everyone will be better off the more you can teach everyone else about what AI can and should be used for.
Ethical concerns, privacy and AI in experiential activations
This could probably be its own article, but we’ll keep things brief. A very important component of incorporating data collection based AI into your project is really sticking to your morals and following good ethics. There are things to consider like AI bias, data privacy, and manipulation.
A lot of pre-existing tools have inherent biases that must be considered. Facial recognition has a history of issues with gender and race discrimination. Translation algorithms can be trained to have gender biases ingrained in them. Classification software can wrongly group certain individuals as more likely to commit crimes. A lot of AI is still in a state of reflecting societies own biases, and this is very important to consider when moving forward with a certain application. You probably don’t want to be making Microsoft’s Tay.
There are also tons of privacy concerns about collecting all of this personally identifiable data, and you have to treat their data with the same respect you would expect of a company dealing with your own personal data. Solid security protocols also take time to implement and can occasionally get left on the side when you’re moving fast and breaking things to reach a deadline for SXSW. Things like GDPR in the EU are going to make this step of data collection much much more complex in the future. Make sure you’re following some kind of action list about data collection and that you’re doing the basic stuff like getting and recording user consent, that you’re storing the data securely, and that you’re responsibly deleting the data after it is no longer needed.
There is a potential future where this line becomes much safer for the user and the tension isn’t as present. Things like the blockchain may eventually empower consumers to “share” a profile for an experience but don’t allow the vendor to collect or store the info after the transaction. Vendors have a lot to gain by embracing these kinds of mechanics as early as possible because public sentiment for data privacy will likely catch up with this industry sooner than later. The flip side of when users can trust that their data is safe and secure is that they will then feel more safe sharing even more intimate details about themselves, allowing for even deeper experiences. Further reading on these last points here here and here(and thanks Gene Kogan for your tips here!)
Finally, there may come a time where AI involved in experiences starts to feel too good and too effective. Something to always keep in the back of your mind is whether it starts to feel like its crossing a line of manipulation. Of course, all advertising feels like manipulation at some level, and it’s up to each person to decide if something is going too far. It’s probably going to be years before this kind of thing is a true concern. No one any time soon is going to say: “I can make a consumer feel so connected to a product with this interactive activation and their personal data that I can make them purchase something 99% of the time.”
And now that we’ve talked about a number of challenges facing the use of AI, we’ll now briefly get into ways to improve.
How can we improve?
This is still an ongoing learning process for me and the team, but I think there are some good lessons to take away, particularly about the challenges discussed in the other sections. It is going to take improvements from both the client side and the vendor side to push these relationships further and find new opportunities.
How can clients make this an easier process and help make more meaningful activations with AI and their data?
- Gather as much information as possible about data you have available to you, how vendors can access it, if there is an existing API, documentation for the API, etc etc.
- Be up front about how your data can be supplied to vendors, it will help massively in the pitch process. Coming up with protocols ahead of time will streamline the production process later on.
- Talk to your internal teams about interesting conclusions that could be drawn from their data. They live and breathe that data, they might already have some good ideas about how it could be used.
- This is obvious, but I’ll say it anyway — finding bigger budgets and approaching projects well ahead of time will dramatically improve the quality of work that comes out of your vendor. Huge payoffs in the end require risk, risk requires experimentation, and experimentation requires time.
- Support and get to know artists and research institutions that are doing interesting work. Chances are the interesting new directions will come from somewhere outside of your industry.
- Build a relationship with your vendor — some things might take multiple projects to really dig into something that results in a good return. Trying something once and tossing it out without analyzing what worked and what didn’t doesn’t give you a good platform for improvement.
Conversely, how can vendors do a better job of helping clients work with their data for AI applications?
- Clients may not have data for you to work with, so start thinking about pre-trained models and other off the shelf AI products or research that may be more broadly applicable to different needs. This is only getting easier and easier.
- Teach your team, teach the client. Run workshops and invite clients to learn more about the whole process involved with working with AI. Come up with some guidelines of how you’d like to work with some client’s data
- Practice — start making internal projects now that can be applied to activations later. If you’re waiting for the right AI project to come along from a client before you really dig into this stuff, you’re going to be running behind.
- Do your best to raise the bar of how this stuff is being used in this space — don’t always reach for the easy option, try to push yourselves and your client. Make fun things, silly things, really serious things — they will all come in handy for a given client and need.
- Work with your client to collect data about how effective these activations are. The more you can prove that using AI provides a larger return on investment versus other types of advertising, the more budget you’ll get next time.
- Data security is huge — having a proven track record or documentation on how you protect client’s data will make their teams feel more at ease with sharing sensitive information with you. Larger experiential shops who have more weight behind data management and security may have an edge here. Be prepared for audits and come up with internal protocols for data protection. Also start considering decentralized AIand blockchain and encryption for personal data — things like the basic attention tokenare another direction.
There is a lot of work to be done on both sides of the table and we’re definitely getting there. I think its important to remember that AI won’t be good for everything — it will be better for some things. Just like other emerging technologies, AI will find its niche. I think it’s all in service of helping to create more impactful stories, to draw people in, and to show them something new. Until the next shiny thing comes along :)
Thanks for reading!
Other interesting things I came across while researching.
http://proceedings.mlr.press/v81/buolamwini18a/buolamwini18a.pdf
