Python Blues
2024-08-12 14:46:08.5889+02 by
ebwolf
5 comments
I've been writing Python for more than 20 years now. It's a very versatile language but I'm currently stumped by something that should be simple:
I am trying to read out-of-order packets from a socket, reorder them and write them to a file. I am seeing one, if not two problems:
- socket.recv doesn't have a good way to ensure you have all the data. Data length in python is always len(data). But Len(data) counts bytes to the first NUL (0x00). I think recv is only delivering truncated packets.
- file.write(data) on a binary file only writes to the first NUL (0x00).
I don't have time to write a custom C/C++ module to handle these situations. Which seems to be that "standard" answer or use a third party module which is also not an option.
On a related note, why does GoLang exist when Rust is at least a little faster, been around longer, and has built-in testing capability (not unlike pytest)?
[ related topics:
Interactive Drama Writing Monty Python Python
]
comments in ascending chronological order (reverse):
#Comment Re: Python Blues made: 2024-08-12 14:52:00.219059+02 by:
ebwolf
Hmm:
The key difference between Rust and Go: you think you can understand Go code, but you’re wrong. On the other hand, you think you CAN’T understand Rust code, and you’re right.
Found here
#Comment Re: Python Blues made: 2024-08-12 16:04:51.365726+02 by:
Dan Lyke
I think the reason Python exists is the community, and community tends to form around expressive languages. Go is the pidgin of computer languages.
Alas, I don't have the answers to your networking questions.
#Comment Re: Python Blues made: 2024-08-12 17:56:57.397854+02 by:
ebwolf
That last quote has me wondering if Rust is the new Perl whereas Go tries to be a better Python.
#Comment Re: Python Blues made: 2024-08-13 01:14:30.470627+02 by:
SnoopJ
[edit history]
socket.recv doesn't have a good way to ensure you have all the data…
I may be misunderstanding (especially re: what is "good"), but the usual way would be to call `recv()` in a loop until you're sure that you got all the data. Either because the socket was closed and you got `b""` back to indicate EOF, or because you've accumulated "enough" data to make sense of it and determine that you have enough to do [something else] even if there are more bytes waiting on the wire.
But Len(data) counts bytes to the first NUL (0x00)
First NUL in the `bytes` object, or in the stream on the wire? The former would be a bug for sure. The latter probably indicates that something up the stack from you thinks that NUL is a good place to treat like a boundary. That sounds pretty weird, though.
I think recv is only delivering truncated packets.
Have you tried enabling the `MSG_WAITALL` flag to block until the full request size is available? It's the only thing I can think of here aside from calling `recv()` multiple times.
#Comment Re: Python Blues made: 2024-08-13 04:49:20.065828+02 by:
spc476
What type of socket it is? If it's a UDP-based socket, then recv() will only return the number of bytes requested, and if it's less than the packet size actually received, it will discard the rest. For TCP, it will work as expected (at the system call level---return consecutive bytes from the stream, waiting for more to come in if there isn't enough to satisfy the call).