One point of view you rarely see is the perspective of sex differences from an evolutionary standpoint. Never mind that this is the reason why the differences exist in the first place if they do at all.
But the short answer is that Human Male and Female brains are unquestionably selected to be different (whether this means they are actually different requires a modern side by side comparison). But to build on this point, think of male and females in terms of evolution. Male and females inherently fill different roles in history and prehistory (If you want to argue that men and women didn't or don't have different roles then show me the man who can give birth to a baby). Given that there were different biological pressures behind the evolution and success of each sex, the brain has been selected (as have other parts of human physiology) to better accommodate the demands of its respective sex as it pertains to one's own reproduction and survival. This much is beyond any reasonable dispute.
Unfortunately this fact cant be used as definitive evidence of sex differences, because even though we can prove that for all of human existence the sexes have done different things what we really need is a side by side comparison of how they are today to come to any conclusion. And given that the current political climate is ridiculously politically correct, it is unlikely any study will be brave enough to publish anything anti PC or else lose funding.
Ref: (Just random references)
https://pdfs.semanticscholar.org/4510/e98234d30ea146f523039ba353520a04be3a.pdf
http://www.annualreviews.org/doi/abs/10.1146/annurev.an.15.100186.000325?journalCode=anthro
https://www.eurekalert.org/pub_releases/2015-10/nuos-wam100715.php
https://stanmed.stanford.edu/2017spring/how-mens-and-womens-brains-are-different.html
Edit:
This extension to address '@Elise van Looij' comment that this is a circular argument.
This is history. I have often heard the 'circular argument' position used to try and defend against evolution, aka which came first the chicken or the egg, but it is a logical fallacy. So for the sake of flushing out the position a bit better lets go through the points.
This appears to be a circular argument. Males and females have inherited different roles and therefor they have evolved differently
to fill those different roles. How come they 'inherently' have
different roles? Because they have evolved that way.
(1) Males and females have inherited different roles? I don't think this is true. Males and females have inherited different potential roles (and maybe an inclination for, or biology best suited for those roles). However no one personally has to fulfill any role. And the roles evolution selects for (namely what is best suited for reproduction) changes over time and between environments. However it is true that those that excel at certain roles best suited for reproduction and survival, will possess traits that will propagate to the larger population by virtue of exponential growth of those genes in the population.
(2) We have evolved differently [solely] because of these roles? No. We have evolved only to reproduce effectively. And this statement isn't limited to just things that reproduce sexually. Biological "roles" are a consequence of the environment. Right now we are still the product of ancient environments because not enough time has passed between then and now for that biological legacy to have been erased. And this is why I put forth the position that the pressures human populations faced for millions(?) of years which unquestionably shaped our biology, can be used in the modern day to make educated hypothesis about our modern (still relatively ancient) physiology.
(3) How come they 'inherently' have different roles? Because in each generation the creatures that did those 'roles' made more babies than the ones that didn't. This is simple math now. 2 A and 2B. 2A reproduce and make 4A. 2B make 1 baby B. Old generation dies, new generation is now 4A 1B. Repeat until B = 0. Now if you want to go all the way back to when there were no roles, you can do that. Go back in history until we reproduced asexually, were hermaphrodites, or no life existed, or some other condition was met. There is no circular-ness in my opinion because you can follow the path back until all starting conditions are different (and thus the circle ends).
But to reiterate the important point because it might be buried in the rest of the text. We have evolved to be a certain way. This way may change in the future and is probably changing right now. But as we exist right now, we are the product of time in the distance past. That passed has shaped us to be a certain way. And we haven't evolved to be radically different from that way yet because not enough time has passed since then for our fundamental makeup to have changed. Think about it; when did the social roles of Male and Female change? 50 years ago? 100 years ago? 1000 years ago? 10,000 years ago? We took millions of years to evolve into this variant of ape that we are. Our ancestral legacy is still strong with us.
But this turned into a discussion of evolution. Instead of what we can learn from evolution. Which was the real purpose of this post.