Short Answer
The words extension and flexion actually mean to increase or decrease the angle of a joint, respectively. In other words, the terms are directly related to the joints themselves. Bones can only move relative to each other at joints, so when the angle between two bones increases or decreases, that action is occurring at the joint. In other words, the joint acts as a vertex of the angle of movement.
Long answer
In general, one should think of extension as "increasing the angle of a joint" and flexion as "decreasing the angle of a joint" (see here, here, here, here, here, here or here for reference). From Saladin's 2015 Anatomy textbook$^1$:
Flexion = a joint movement that , in most cases, decreases the angle between two bones.
Put differently,
Flexion refers to a movement that decreases the angle between two body parts.
I think where you are confused is that you do not understand what a joint is. A joint is anywhere where two bones articulate (or touch). Although various types of joints exist, those that readily move well enough to receive movement terms such as flexion/extension are called synovial joints.
At any given synovial joint, two bones are able to move relative to each other due to a cavity that provides space for movement -- the synovial cavity. Put differently, bones (which are solid, unbending objects) move relative to each other during our body movements, and this relative movement occurs at joints.
Using your example of arms and legs:
When your arm flexes/extends, what's happening is that the head of your humerus is moving inside the glenoid cavity of the scapula. This movement at the "glenohumeral joint" results in the entire arm (humerus and all distal components) either swinging forward or backward due to the angle of the glenohumeral joint increasing or decreasing (though, actually, other movements are possible as well). The same applies to the hip.
Another example is the "elbow" joint -- which actually consists of 2 joints: the humeroradial and humeroulnar joints. Consider when you "flex your arm" as in showing off your bicep. Yes, your lower arm (forearm) moves, but the movement all occurs at the point of the elbow at these two joints. In other words, the elbow joints form the vertex of the angle between your upper and lower arm bones. When you "flex" your arm, the angle of these joints decreases. When you return your elbow to being straight (i.e., "extend" your elbow), your are increasing the angle of the humeroradial and humeroulnar joints.
$^1$ Saladin, K. S. 2015. Anatomy & Physiology: The Unity of Form and Function. Seventh ed., McGraw-Hill, New York, NY. 1248pp.