How is psychology the study of behavior?
Psychology has its beginnings in tangential traces of early philosophy, but the psychological practices and theories as we know them today, and as they are practiced today, are relatively recent. Some of the founders of modern philosophy are Wilhelm Wundt and William James, both working and theorizing in the late 19th century.
Psychology is the study of mental function and the reciprocal relationship between the mind and the external world. There are many different branches of psychology which attempt to describe the functions of the mind and the mind/world interaction and most of them have something to say about behavior.
Freud’s psychoanalysis focused on the patient’s unconscious to resolve conflicts in the patient’s mental function/behavior.
Behaviorism treats all actions, even thinking, as behavior. Therefore, behaviorists focused on treating all behavior (all observable action) objectively. There are many different branches of behaviorism, but in general, they did not differentiate between mental and external. Their aim was to explain all behavior in terms of science; not so much affected by inner (mental) states. Some radical behaviorists, B.F. Skinner, treat an organism (i.e. human) almost like a machine; functioning and reacting only to prompts and empirical interaction.
There are other branches; functionalism, cognitive psychology, existentialism (as applied). And psychology is not just limited to the study of individual psyches. Carl Jung’s famous “collective unconscious” is the theorization of how social groups mentally function together.
A recurrent debate in psychology and philosophy of mind, regarding behavior or any mental function, is the debate of mental and physical states. If mental states do exist, how do they affect the person in acting in the external world? How does the external world affect the mind? And so on. So, the study of psychology involves more than just study of mental function. You could include study of history, neurology, physiology, social mores, cultural traditions, sex and gender roles, philosophy, etc.