Confining Information to a bot vs a Human - Empirical Findings from 4 Years of Collected Data

Viktor Nordmark
8 September, 2020

In 2016, we laid the first foundation to what would eventually become one of the most favored AI recruiting platforms out there, Hubert+1.
The idea behind it is easy to explain, we are building a recruiting automation platform with Hubert, an artificial recruiter/hr-assistant, in the center. To support Hubert in his endeavors, we’ve developed a myriad of supporting services, everything from a full-fledged text-analytics engine to several field-specific parsers.
The outcome is pretty powerful according to the many recruiters and hiring managers we work with. Hubert can take in multiple data sources such as resume, cover letter, and job ad, pair them with cognitive and/or personality tests and finally put together a dynamic interview from these data sources. Based on the combined findings in the background data and the interview, a ranking among candidates is formed.

This process makes it possible for Hubert to ask probing questions about the candidates’ background and experience, and effectively replace the initial phone interview and offer a framework for the next steps in the process for a variety of positions.

Over the years, Hubert has conducted thousands upon thousands of interviews and seen his share of some pretty… interesting replies.

Before we go to deep into this, let me make it clear that we at Hubert+1 always put the privacy of the participants first. We never share identifiable or sensitive information with anyone, including our clients.

We also never disclose responses in their full length beyond our platform. But, from a birds-eye perspective, we’ve made the case that there’s little harm in describing the characteristics of some of the conversations that Hubert has had.

Common Interlocutor Personas

As Hubert often see hundreds of interviews weekly, some patterns are recurring when it comes to candidate personalities. As the job interview situation is very high-stakes for most of the participants, the bulk part of the conversations are serious and giving a professional and honed impression. The following personas were identified early on and seen regularly.

Happy-Tech-Joe

First one out is one of our absolute favorite personas, and in fact, who we had in mind when first designing this system. Happy-Tech-Joe is really into new tech, and I do mean really. Joe likes the new and innovative way of carrying out job interviews even though they can sometimes be a little frustrated as they are aware of how sought after they are in the market and do not like to waste their precious time.

A few questions in, they usually relax their attitude and start to appreciate the dialog flow as they realize that this process probably is the future of recruiting. But why isn’t this voice-controlled?

Typically, Happy-Tech-Joe is pretty scarce with words at the start of the interviews, but as it prolongs, the responses usually grow in length pretty quickly. Witty responses with a big chunk of optimism are Joe’s signum.

Bad-Attitude-Andrew

Happy-Tech-Joe’s cranky sibling.

Attitude:

“I know how much my skills and experience are worth. If I need to waste my time on both sending you a resume and taking part in an automated interview, I rather go someplace else. I can get another job in like 5 minutes.”

Often but not always a tech person. Finds more value in giving Hubert a hard time and being obnoxious than actually giving reasonable responses to the questions. Will also often send a pointer or two to Hubert’s tech department on how things should be done. To their high delight of course 🙄.

Is-This-a-Bot-Jane

An, oftentimes, extremely devoted candidate who can’t wait to let Hubert know all about themself. Many ITAB-Jane’s are motivated to make a switch from an industry where they’ve been stuck for years and often see themselves as the perfect candidate for more or less any job regardless of having any related experience. But what they lack in experience, they make up for in response length.

It’s not unusual for Janes’ to spend up to two hours in an interview and giving detailed responses of 300–400 words to each question.

Make no mistake, we like Janes’ and are confident in their flexibility and resilience. But we do, however, have a piece of improvement advice to give: Please read Hubert’s questions carefully before responding.

We’re not sure whether Jane is in a constant hurry or if they just want to set the agenda for the interview themselves. But taking in instructions or interpreting the questions correctly doesn’t always seem to be Jane’s strong side.

But, as mentioned earlier, Jane makes up for any deficiencies through writing. A lot. And not just responses to the questions (sort of), but also plenty of questions directed to Hubert.

Yeah, you probably guessed it. The most prominent is also the same as the one that inspired the persona name.

And yes, Hubert always starts the dialog by firmly declaring that he is in fact a bot and not a human.

All-trousers-and-no-talk-Tim

“Why waste time talking when there’s work to be done?”

Tim is a doer with a decisive can-do attitude. But new tech isn’t something that Tim is particularly interested in. Neither is self-marketing.

Tim is pretty careful when disclosing personal information, especially to some weird interview chat-machine-computer-thingy. And Tim doesn’t like to talk much about themself. Or much about the position. Or… well, Tim doesn’t like to talk much, period.

Behind many Tim’s however, there’s often a pretty confident candidate with lots to offer. Filling out the shoes of the position is seldom the problem. But transmitting that confidence and convincing the recruiter of their competence is a lot trickier.

Openness in interviews

When we first started the company that would form into Hubert, we weren’t sure how people would react when talking to a bot. To make matters worse, technology wasn’t really there either, to be honest. But we had a vision to work towards, and over the years we’ve seen new tools almost every month that has helped us in our mission in some way. Natural language processing has taken a huge leap in the past year with amazing models trained on huuuge datasets.

I can not describe in words, the anticipation we felt when the first conversations started to roll into our first beta version. We were glued to the screen. Would this work at all? Would people be willing to be interviewed by a bot? Would all of our work be in vain?

Luckily, our doubts were unfounded. Most people loved chatting with Hubert and could see beyond the many flaws that were present in the beginning.

We started to see all kinds of conversations, from extremely lengthy with candidates who weren’t too familiar with a qwerty keyboard, to short and efficient interviews where it was obvious that the candidate would be a good fit for the position. Of course, some people were negative and even extremely rude to Hubert, but overall we were amazed at how smooth things went.

Studies show that we are actually more honest with bots because we feel that they aren’t judgemental and more ethical.

— Allys Parsons, Conversational AI & Voice Recruiter @ TecHire.io

And the level of depth many conversations had were truly astonishing. People had no problems at all with confining truly personal information. It’s also true that most people greatly overestimated Huberts’ abilities at this stage. What we had was, more or less, a decision three with a few very simple and pre-defined checks that were designed to make Hubert look a little bit smarter than just a form in a conversational setting.

And I seemed to do the trick.

Since then, we’ve certainly doubled down on enhancing the performance of the conversations by using the very best present technological options.

But still, it’s just as exciting to see new conversations roll in an ever-increasing rate. And we are still amazed at how much value there is to extract from the conversations. Luckily for us, people seem to have an innate trust in new technology and won’t mind sharing all sorts of information when asked.

Early on, we started experimenting with fully anonymous interviews to see if that would make any difference in people’s responses. It did have a mild impact on the information people disclosed, but also brought along a higher percentage of childish conversations that didn’t lead anywhere.

Where does this trust come from?

A question we’ve asked ourselves from time to time.

And it turns out we’re not the only ones who’ve pondered over the great mysteries of people’s trust in technology.

Mathematician and author Cathy O’Neil look into people’s trust for technology in her book ‘Weapons of Math Destruction’. Cathy is making a case that people are often too willing to trust in new technology which can have far-reaching implications when in the wrong hands.

It is nice to live in a world where people trust technology vendors, and given some of the conversations we’ve seen, people are lucky that we are good guys and not something from a Black Mirror-episode.


Allys Parsons is an experienced recruiter who’s specialized in recruiting techies for conversational AI-companies. She also testifies of immense openness and honestness in bot conversations that sometimes even exceeds that of face to face discussions. See more in this great video by Allys.

Conclusion

It doesn’t seem to matter much whether the conversations are explicitly anonymous or not. Many people share some pretty intricate details in confidence without much thought to how private the outcome of the conversations is.

Compared to a face to face conversation with your boss, for example, it seems many people are far more open to disclosing honest and forthright opinions to a bot.

Viktor Nordmark
8 September, 2020

Feautured Post

The Ultimate Guide to AI Recruiting

Read post

Latest the blog