This paper introduces a deep reinforcement learning (DRL) based approach for autonomous planning of low-thrust spacecraft trajectories, while addressing the intricate challenges of orbital dynamics and mission design. We propose a single-agent attention actor-critic (SA3C) algorithm, which integrates an attention mechanism to significantly enhance sample efficiency and decision-making capabilities in complex trajectory optimization tasks. Our research extends the application of DRL beyond traditional geocentric transfers, incorporating cislunar mission scenarios where strong third-body gravitational influences play a critical role. The SA3C algorithm provides better results compared to existing automated approaches, demonstrating its effectiveness in optimizing transfers to GEO and near-rectilinear halo orbit (NRHO). We offer a comprehensive comparison of three algorithmic frameworks—sequential, DRL-based, and optimization-based—in terms of optimality and potential for autonomy in spacecraft trajectory planning. Through rigorous evaluation, we demonstrate that attention-based modifications in RL enhance the adaptability and efficiency of low-thrust spacecraft trajectory planning, offering a promising avenue for autonomous and effective mission designs in multi-body gravitational environments. This work contributes toward advancing spacecraft autonomy and optimizing complex orbital maneuvers by introducing the SA3C algorithm, which demonstrates the potential to achieve near-optimal transfer times in geocentric and cislunar missions. © 1965-2011 IEEE.