Transcription factors usually initiate the cellular changes, but it's mostly the chromatin state of the nucleus that maintains the state of a cell and stops it from changing back and forth.
Embryonic stem cells start out with open chromatin ('open' means 'accessible'). This means low DNA methylation and overall epigenetic modifications that enable transcription. This can be interpreted as "a cell with full potential", since many regions on the DNA are available for transcription.
The further a cell differentiates, the more DNA regions assume a closed chromatin state, meaning higher DNA methylation and repressive epigenetic marks that lead to DNA condensation. It's as if the 'potential' decreases, since huge regions on the DNA become 'locked up'. These huge regions on the DNA are partitioned into 'TADs' (Topologically Associating Domains) and their epigenetic state can be determined by ATAC-Seq
See this review for reading about chromatin state and stem cells.
As mentioned by Bryan, specific transcription factors are unique to a cell type. Some transcription factors bind to multiple promoter regions and initiate a complex differentiation program that orchestrates cellular/biochemical/morphological changes needed to fulfill specialized function.
The promoters involved in cell differentiation can be in a 'poised' (aka 'bivalent') state, meaning that they got both transciption -inducing (H3K27me3) and -repressing (H3K4me3) epigenetic marks, so it's as if these promoters are "ready and waiting" for a signal to decide whether to transcribe or lock up a DNA-region. And once a DNA region is locked up, the cell knows that it has lost some potential and "knows" what cell type it is. Lesch & Page 2014
Such poised promoters are not just present in embryos, but also in adult tissues, like mesenchymal stem/stromal cells (MSCs) in the bone marrow. The RUNX2 Promoter is poised and is waiting for signals to turn the MSC into an osteoblast (or chondrocyte or adipocyte). Wu et al 2017