Spacecraft relative on-off control via reinforcement learning

Abstract

The article investigates the task of spacecraft relative control using reactive actuators, the output of which has two states "on" or "off". For cases where the resolution of the thrusters does not provide an accurate approximation of linear control laws using a pulse-width thrust modulator, the possibility of applying reinforcement learning methods for direct finding of control laws that map the state vector and the on-off thruster commands has been investigated. To implement such an approach, a model of controlled relative motion of two satellites in the form of a Markov decision process was obtained. The intelligent agent is presented in the form of "actor" and "critic" neural networks, and the architecture of these modules is defined. It is proposed to use a cost function with variable weights of control actions, which allows optimizing the number thruster firings explicitly.      To improve the control performance, it is proposed to use an extended input vector for "actor" and "critic" neural networks of the intelligent agent, which, in addition to the state vector, also includes information about the control action on the previous control step and the control step number. To reduce the training time, the agent was pretrained on the data obtained using conventional control algorithms. Numerical results demonstrate that the reinforcement learning methodology allows the agent to outperform the results provided by the linear controller with pulse-width modulator in terms of control accuracy, response time, and numbers of thruster firings.