BERT: Clever but not Smart

Is BERT a computational versions of Clever Hans, the early 20th-century horse who seemed smart enough to do arithmetic, but who was actually just following unconscious cues from people around?

But is AI actually starting to understand our language — or is it just getting better at gaming our systems?

After BERT took the world of NLP by storm last year, displaying impressive results on notoriously difficult benchmarks, this year's is about deconstructing its inner working and realizing we're far from human-level natural language understanding. BERT was right, but not for the right reasons: namely, being very good at taking on spurious cues in the training set.

A perfect follow-up to the Data Skeptic episode we featured last week.

Chess felt like a serious test of intelligence until we figured out how to write a chess program.


Want to receive more content like this in your inbox?