This sounds like something which could be connected to the fact that Stdio.File()->write(Stdio.Buffer()) does not release the interpreter lock. This makes it less useful for situations where write is not called from within one backend thread. This makes me think that in your situation using a temporary string would be better. I think this version is thread-safe, allows other threads to add to the buffer in parallel and avoids output_to:
string tmp = buffer->peek(SIZE); int bytes = fd->write(tmp);
if (bytes < 0) { // handle error } buffer->consume(bytes);
It would be possible to make Stdio.File()->write(Stdio.Buffer()) release the interpreter lock at the additional cost of adding synchronization to Stdio.Buffer(). However, the main objective of this API (for me at least) was to make the common use-case more efficient in which you have one single backend doing all the writing on non-blocking sockets. In that situation I think it often makes more sense to have the IO thread complete one loop and release the interpreter lock when calling epoll. Other threads can then run while the backend thread is waiting for new IO events.
Arne
On Mon, 9 Mar 2020, Stephen R. van den Berg wrote:
Tobias S. Josefowitz @ Pike developers forum wrote:
As to what exactly goes wrong... What I can report is that using that commit, it mostly works, until the scheduling gets tight. I.e. under intense pgsql query load, things go wrong. So that suggests that there might be issues that occur when
Hmm, curious. If we tried to reproduce this without pgsql though, what would we be looking for? Data going out on the Stdio.File being not what one would expect it to be? I.e. not a crash or so?
It did not crash. What happened was that the communication with the database got stuck. I.e. most likely cause would be that some of the add()s get lost.
Theoretically, something like this could reproduce it:
Stdio.File thesocket; Stdio.Buffer thebuf;
Thread A: Tight loop with: thesocket->write(thebuf);
Thread B: Tight loop with: thebuf->add()
Some of the add()s if thread B might get lost (race condition).
Stephen.