In both cases به or با , the whole expression means thinking via applying language in one's mind. If someone is actually / vocally talking to himself, we often say: زیر لب می گفت / زمزمه می کرد or something like these.

As far as I am concerned ,seems to be reflection rather than murmuring

another option

This does not necessarily mean he was talking out loud.

The sentence-
با خودش میگفت، مسلم است که یکتنه از پس آنها بر نمیآیم
====
با خود and به خود mean the same. The term با خود می گفت triggers the presence of monologue. So what is for sure is that there was sort of communication with self. This communication can be through the application of speech (whispering, talking, ...) or it may be a mental process without the application of the articulatory device. No option could be ruled out.
In case we need a safe term that refers only to the non-verbal monologue we may use با خودش فکر می کرد

It is not finally correct to use past continuous tense here in which case there was a need for the continuous marker داشت in the phrase and hence
با خودش داشت می گفت or داشت با خود می گفت
In fact the term با خودش می گفت stresses the repetition rather than the continuity of the phenomenon and hence can better be expressed by "s/he kept telling himself/herself..." and even "s/he told himself/herself"

Translations:
1- S/he told himself/herself, ""For sure I can't take on them by my own".
2- S/he kept telling himself/herself, "No doubt, I won't be able to take on them by my own".

and if the context supports that it was not a verbal communication
3- S/he thought, "No doubt, I won't be able to take on them alone/by my own".