Markov decision processes provide powerful tools for adaptive management of computing and communication in cyber-physical systems. However, efficient solvers are required to provide these capabilities on embedded computing platforms. This paper describes two new MDP solvers for embedded applications: Sparse Value Iteration (SVI) uses sparse matrix methods and runs on small, single-threaded CPU platforms; Sparse Parallel Value Iteration (SPVI) extends this approach to leverage the parallelism of embedded graphics processing units (GPUs) to further improve performance on more sophisticated embedded platforms. Both solvers improve running time and reduce power consumption.