5 Questions Employers Should Ask Before Buying An AI Tool

Not all artificial intelligence tools are created equal, and employers should be sure to ask a few crucial questions before buying AI tools for resume screening or other decision-making in order to get the most out of the technology and avoid violating antidiscrimination law, experts say.

Employers considering buying an AI tool must be mindful of the Americans with Disabilities Act, the Age Discrimination in Employment Act, Title VII of the Civil Rights Act of 1964, long-standing guidance from the U.S. Equal Employment Opportunity Commission on employee selection procedures, and state and local laws regulating AI use. Though such tools are sometimes touted as being able to help companies avoid bias in their resume screening, hiring or other decision-making processes, a faulty tool or one that's used the wrong way can actually compound discrimination, experts warn.

The first step in a thoughtful AI purchase is going back to basics, said Rachel See, senior counsel at Seyfarth Shaw LLP.

"Really understanding, why are we using this? What benefit are we getting out of it? And how do we know it's working?" is key, she said. "However attitudes about AI regulation or AI safety change, getting answers to those fundamental questions is still critical."

Here are five questions employers should ask before buying AI tools for the workplace.

What problems does the tool solve?

Before employers direct any questions at a purveyor of AI systems, they need to look inward, See said.

It's easy to be drawn to "something shiny," See said, a new tool AI developers say will solve employers' problems, like an impulse buy in the checkout line. But she advised employers to have a fleshed-out understanding of what exactly they want to get from the tool and what they see as the value of that benefit.

Nick Sarokhanian, a partner at Barnes & Thornburg LLP, said that while he sees vendors pitching a variety of AI tools to employers, the most commonly adopted tools so far help with screening or sorting applicant resumes, to save time and money in narrowing down a candidate pool. Or employers may use tools that assess applicants' skills or personality traits through video interviews or games.

See emphasized that envisioning the use of the tool is a critical first step, though it may seem basic.

Ridhi Shetty, senior policy counsel at the Center for Democracy and Technology, said that both the vendor and employer should know how a tool was designed to be used. For example, some tools are developed for certain job positions or industries, or are meant to be used alongside human judgment rather than in place of it.

"So understanding what the vendor or developer's intentions are with respect to how those tools should be used will also help inform how employers go about analyzing the outlook that they're looking at," she said.

How do we know the tool works?

Any AI vendor worth its salt will be able to prove the product works as advertised, experts said.

Lindsey Zuloaga is chief data scientist at HireVue, which provides AI-based human resources tools like video interview platforms. As one of the older AI tech companies in the game — having been founded in 2004 — HireVue learned how to hand over that proof, Zuloaga said.

"We have the data to show that how people do on our assessments is very highly correlated with how they do at the job," she said.

Management-side lawyers have expressed frustration that AI tech vendors tend to be reluctant to turn over their training data, algorithms or other information they consider proprietary.

"There's, generally speaking, with a very broad brush, a bit of reluctance to open up the hood and show you exactly what they're doing. And frankly, sometimes the people you're dealing with may not even know the answers, so you may need to get kind of a committee approach of people to tell you what they're doing," Sarokhanian said.

Zuloaga agreed, but she said there are ways to confirm the product works without knowing exactly what's under the hood. She calls them "AI explainability statements." Essentially, they include documentation on how the models work and how they're trained, she said.

"I mean, obviously, we're not going to give our algorithms away," she said. "We don't want those out in the world, or people could somehow figure out how to game them. But I think explainability is interesting."

By that, Zuloaga said she means being able to summarize data points or responses the model marked as "good" or "bad," or share themes that appear in aggregate data.

"We train our models with a rubric of, 'Here's what adaptability means, and here's thousands of people answering this question about adaptability. And here's what a score of one, two, three, four, five looks like.' We want to see that when we separately do some explainability work, it should reflect what was in that rubric, right? So if it didn't, that would be a kind of a red flag."

AI models "are pretty hard to decipher" in terms of "what's going on in the inside," she added. "So you kind of have to poke and prod and see, what is it looking for?"

Has the developer conducted bias audits?

Bias audits of AI technology are now required by law in New York City, though Zuloaga said there's been a lot of confusion about what an AI audit should encompass. It's not like in the finance industry, where "audit" has a clear meaning, she said.

The expertise of industrial-organizational psychologists, who study workplace behavior, can be helpful in a bias audit, experts said.

Richard Landers, a professor of industrial-organizational psychology at the University of Minnesota, said such psychologists have studied employee selection processes since the 1930s.

"A lot of the same rules still apply," Landers said, but just need to be translated for the AI era.

Technology companies don't tend to be steeped in the ins and outs of antidiscrimination law, he said, so part of his work is to address that.

"In my auditing work, it's often about raising awareness, especially with Silicon Valley startups, about what they need to worry about," he said. "It can make them aware of the issues they need to worry about, which they often don't."

An audit, ideally conducted by an independent third party, would test a screening or interviewing tool to ensure that it doesn't have a disparate impact on any particular group — for example, that an algorithm doesn't learn to toss out resumes with certain names or zip codes.

Hiring third-party auditors is expensive, though, and can be a challenge for smaller employers, said Shetty of the Center for Democracy and Technology.

"But regardless of size, employers still have obligations under civil rights laws, so there should be at least some inquiry into these kinds of questions for them to have a better sense of whether there may be gaps in their own compliance," she said.

How does the developer test for and mitigate bias?

Engineers might look at data that shows differential scoring rates for different genders or races and try to fix that problem by overcorrecting, Landers warned.

"Their natural inclination, from an engineering perspective, is, 'Oh, we'll just fix it. So we'll just give extra points to people in different groups. … they make very explicit decisions on the basis of these group memberships," he said.

That might make sense to an engineer correcting an algorithm, but making employment decisions based on someone's race or gender or age is unlawful, Landers pointed out. That's the kind of misunderstanding an audit could catch. There are practices vendors might call "bias reduction" that might "make the models work better" but also "closely mimic disparate treatment in a lot of cases," Landers said.

Zuloaga acknowledged it's important to keep a close eye on algorithms once they're released from the lab into "the wild."

"We build our algorithms in the lab, we test them, we validate them, but then we put them out into the world, and we might have customers that are using it on a specific population that's different," she said.

The employer and vendor should check in to ensure the tool is working as expected and not skewing toward an adverse impact on any particular group, she said. Developers can check in on their algorithms "almost continuously" these days, she said, and tweak them as necessary.

Landers noted that a lot of newer AI startups don't have a long history of testing their selection procedures extensively and working their way up to real-life situations.

"They should have a comprehensive set of documentation about how they have evaluated bias," Landers said. "And if you don't see [a] competently done bias assessment across populations and lots of different diverse groups and different kinds of customers, then I would be nervous about that."

Shetty said it's also crucial to ensure that an applicant being evaluated by an AI tool has access to reasonable accommodations, regardless of whether they're built into the tool. For example, a tool may allow some applicants to take extra time on a personality assessment.

"Having alternatives available, I think, is a good requirement to have," Shetty said. "And it may also be easier to implement than trying to change the tool itself, which employers may have a lot less control over."

At what point are humans involved?

Shetty said she thinks humans should remain involved in AI-led employee selection processes. She said there's a "finger-pointing problem," where vendors say their tools are meant to be used with human discretion while employers say the tools were represented to them as not needing human oversight.

"But it really requires a more holistic analysis of the job applicant when you are just relying on particular outputs of a tool," she said.

Sarokhanian said some companies are already advertising AI tools created without "human-reinforced learning," when a human provides feedback during the training of a tool. Whether or not those tools are just as good, employers need to know what they're dealing with, he said.

"You want to know, who did the training? What data were they trained on? Who did the human-reinforced learning, if that did happen?" he said.

"You just want to have some sort of assurance that there was some thought given to this."

See agreed, urging employers to dig in on questions of testing and validation.

"What human processes do we have to see if this is wrong?" she said.

Zuloaga said the human interaction involved in developing a tool should be laid out in the AI explainability statement — for example, the process of signing off on various steps of development.

And she emphasized that humans should still be involved in the decision-making process once the tool is being used in the real world.

"Humans and AI working together are more powerful than either one alone," she said. "You still need humans."

--Editing by Aaron Pelc and Amy Rowe.

For a reprint of this article, please contact reprints@law360.com.