Spacecraft relative on-off control via reinforcement learning

Authors

DOI:

https://doi.org/10.15407/knit2024.02.003

Keywords:

actor, critic, neu-ral network, on-off control, reinforcement learning, spacecraft relative control, thruster firing

Abstract

The article investigates the task of spacecraft relative control using reactive actuators, the output of which has two states "on" or "off". For cases where the resolution of the thrusters does not provide an accurate approximation of linear control laws using a pulse-width thrust modulator, the possibility of applying reinforcement learning methods for direct finding of control laws that map the state vector and the on-off thruster commands has been investigated. To implement such an approach, a model of controlled relative motion of two satellites in the form of a Markov decision process was obtained. The intelligent agent is presented in the form of "actor" and "critic" neural networks, and the architecture of these modules is defined. It is proposed to use a cost function with variable weights of control actions, which allows optimizing the number thruster firings explicitly.      To improve the control performance, it is proposed to use an extended input vector for "actor" and "critic" neural networks of the intelligent agent, which, in addition to the state vector, also includes information about the control action on the previous control step and the control step number. To reduce the training time, the agent was pretrained on the data obtained using conventional control algorithms. Numerical results demonstrate that the reinforcement learning methodology allows the agent to outperform the results provided by the linear controller with pulse-width modulator in terms of control accuracy, response time, and numbers of thruster firings.

References

Alpatov A. P., Cichocki F., Fokov A. A., Khoroshylov S. V., Merino M., Zakrzhevskii A. E. (2015). Algorithm for determination of force transmitted by plume of ion thruster to orbital object using photo camera. 66th Int. Astronautical Congress, Jerusalem, Israel, 2239-2247.

Alpatov A., Khoroshylov S., Lapkhanov E. (2020). Synthesizing an Algo-rithm to Control the Angular Motion of Spacecraft Equipped with an Aeromagnetic Deorbiting System. Eastern-European Journal of Enterprise Technologies. 5 (103), 37-46.

https://doi.org/10.15587/1729-4061.2020.192813

Anthony T., Wie B., Carroll S. (1989). Pulse-Modulated Control Synthesis for a Flexible Spacecraft. Journal of Guidance, Control, and Dynamics. Vol 13 (6), 1014-1022.

https://doi.org/10.2514/6.1989-3433

Artificial intelligence: a modern approach (2010). Eds. S. J. Russell, P. Norvig. Pearson education. Inc. ISBN-13: 978-0134610993.

Bernelli-Zazzera F., Mantegazza P., Nurzia V. (1998). Multi-Pulse-Width Modulated Control of Linear Systems. Journal of Guidance, Control, and Dynam-ics. Vol 21 (1), 64-70.

https://doi.org/10.2514/2.4198

Deep Learning (2016). Eds. I. Goodfellow, Y. Bengio, A. Courville. The MIT Press. ISBN 978-0262035613.

Gaudet B., Linares R., Furfaro R. (2020). Adaptive guidance and integrated navigation with reinforcement meta-learning. Acta Astronautica, 169, 180-190.

https://doi.org/10.1016/j.actaastro.2020.01.007

Gaudet B., Linares R., Furfaro R. (2020). Seeker based adaptive guidance via reinforcement meta-learning applied to asteroid close proximity operations. Acta Astronautica, 171, 1-13.

https://doi.org/10.1016/j.actaastro.2020.02.036

Golubek A. V., Dron M. M., Petrenko O. M. (2023). Estimation of the pos-sibility of using electric propulsion systems for large-sized orbital debris post-mission disposal. Space Science and Technology., 29, № 3 (142), 34-46.

https://doi.org/10.15407/knit2023.03.034

Hovell K., Ulrich S. (2020). On deep reinforcement learning for spacecraft guidance. AIAA SciTech Forum, 6-10 January 2020, Orlando, FL.

https://doi.org/10.2514/6.2020-1600

Ieko T., Ochi Y., Kanai K. (1997) A New Digital Redesign Method for Pulse-Width Modulation Control Systems. AIAA proceedings AIAA-97, 3700.

https://doi.org/10.2514/6.1997-3770

Izzo D., Märtens M., Pan B. (2019). A survey on artificial intelligence trends in spacecraft guidance dynamics and control. Astrodyn., 3, 287-299.

https://doi.org/10.1007/s42064-018-0053-6

Khoroshylov S. V. (2018). Relative motion control system of spacecraft for contactless space debris removal. Nauka innov., 14, № 4, 5-16.

https://doi.org/10.15407/scin14.04.005

Khoroshylov S. V., Redka M. O. (2019). Relative control of an underactuat-ed spacecraft using reinforcement learning. Тechnical Mechanics, 4, 43-54.

https://doi.org/10.15407/itm2020.04.043

Khoroshylov S. V., Redka M. O. (2021). Deep learning for space guidance, navigation, and control. Space Science and Technology. Vol. 27, № 6 (133), 38-52.

https://doi.org/10.15407/knit2021.06.038

Khosravi A., Sarhadi P. (2016). Tuning of pulse-width pulse-frequency modulator using PSO: An engineering approach to spacecraft attitude controller de-sign. Automatika. № 57, 212-220.

https://doi.org/10.7305/automatika.2016.07.618

Lapkhanov, E., Khoroshylov, S. (2019). Development of the aeromagnetic space debris deorbiting system. Eastern-European Journal of Enterprise Technolo-gies. 5 (101), 30-37.

https://doi.org/10.15587/1729-4061.2019.179382

Lewis F. L., Vrabie D., Syrmos V.L., Optimal Control, 3rd Edition. John Wiley & Sons, Inc., New York, USA (2012).

https://doi.org/10.1002/9781118122631

Li W., Cheng D., Liu X., at al. (2019). On-orbit service (OOS) of spacecraft: A review of engineering developments, Progress in Aerospace Sciences, Volume 108, 32-120.

https://doi.org/10.1016/j.paerosci.2019.01.004

Machine Learning (1997). Ed. T. Mitchell. New York: McGraw Hill. ISBN 0070428077.

Mnih V., Badia A., Mirza M., Graves A., Lillicrap T., Harley T., Silver D. (2016). Asynchronous Methods for Deep Reinforcement Learning. arXiv preprint, ArXiv:1602.01783.

Oestreich C.E., Linaresy R., Gondhalekarz R. (2021). Autonomous six-degree-of-freedom spacecraft docking maneuvers via reinforcement learning. J. Aerospace Inform. Syst., 18, № 7.

https://doi.org/10.2514/1.I010914

Redka M. O., Khoroshylov S. V. (2022). Determination of the force impact of an ion thruster plume on an orbital object via deep learning // Space Science and Technology. 28, № 5 (138), 15-26.

https://doi.org/10.15407/knit2022.05.015

Reinforcement learning: an introduction (1998). Eds. R. S. Sutton, A. G. Barto. MIT press. ISBN 978-0262193986.

Robinett R. D., Parker G. G., Schaub H., Junkins J. (1997). Lyapunov Opti-mal Saturated Control for Nonlinear Systems. Journal of Guidance, Control, and Dynamics. Vol 20 (6), 1083-1088.

https://doi.org/10.2514/2.4189

Schulman J., Wolski F., Dhariwal P., Radford A., Klimov O. (2017). Proxi-mal policy optimization algorithms. arXiv preprint, arXiv:1707.06347.

Silver D., Schrittwieser J., Simonyan K. (2017). Mastering the game of Go without human knowledge. Nature, 550, 354-359.

https://doi.org/10.1038/nature24270

Song G., Buck N. V., Agrawal B. N. (1999). Spacecraft Vibration reduction using pulse-width pulse-frequency modulated input shaper. Journal of Guidance, Control, and Dynamics. Vol 22 (6), 433-440.

https://doi.org/10.2514/2.4415

Yamanaka K., Ankersen F. (2002). New State Transition Matrix for Relative Motion on an Arbitrary Elliptical Orbit. Journal of Guidance, Control, and Dynamics. 25 (1), 60-66.

https://doi.org/10.2514/2.4875

Downloads

Published

2024-06-01

How to Cite

KHOROSHYLOV, S. V., & WANG, C. (2024). Spacecraft relative on-off control via reinforcement learning. Space Science and Technology, 30(2), 03–14. https://doi.org/10.15407/knit2024.02.003

Issue

Section

Spacecraft Dynamics and Control