A behaviour tree is basically a hierarchical state machine with a very strict set of states where each state is represented by a node in a tree. A behaviour tree combines decision making and execution in one structure which allows it to be used almost like visual scripting.
An introduction to behaviour trees can be found here:
Overview,
Video Part 1,
Video Part 2,
Video Part 3,
A detailed video presentation can be found here.