This is a neophyte that has been recorded in Europe since 1979:
Sceliphron curvatum, in German also known as orientalische Mörtelwespe. Unusually large considering what else is buzzing around, but mostly harmless.
The nests, or rather individual larvae cells, are made of mud and while there are about thirty species in the genus, only Sceliphron curvatum is found in Germany - there‘s no other wasp or other insect in Germany that builds similar nest cells, especially the size is a good indicator. Your description of nest location in a window frame is typical, curtain folds or (vertical) nooks in furniture, e.g. shelves are also welcome. Our specimen picked a hiding spot behind a few books a few years ago.
Because of their size, there are usually some articles about them in the low-news summer media lull, describing baffled citizens and their “discovery”. A few less sensational descriptions, photos and further information can be found here (sorry, in German).