Generally speaking, buffers help enormously when reading and parsing from bytestreams into pike, but they do not help all that much when writing to bytestreams. The writev() system call handles the fragmented writes quite nicely (see the pgsql driver) and more efficient than an IOBuffer which would copy the data at least once.
That is only true if you do not include the overhead you get from making the strings. It is also only even close to true if you can always write the whole buffer.
The very common case where you have nonblocking output means you always have to chop off data from the strings, if you have them, and keep track of the array members when you use writev.
The actual gain when I switched the communication pipe in one of subprocesses in the storage system we have at Opera to BinaryBuffer, which is basically a predecessor to IOBuffer, when communicating encode_value:ed values in arrays was about 98% speed gain in the buffer handling. It went from 80% to less than 2% of the overall CPU time.
Meanwhile the number of lines needed to do it went from 20 to 10. And that buffer did not have write_to
Basically:
Before, pseudo code of course: array outbuf; output(mixed value) { outbut += ({ sprintf("%4H", encode_value(value)) });}
write_callback() { int num = fd->write(outbuf[..100]); // if 100 is left out this is... rather slow.. for large buffers // remove entries from outbuf until sizeof(outbuf[0]) < num.. // this was of course done in one step outbuf[0] = outbuf[0][num..]; // << 99.99% of the CPU time goes here. }
After: BinaryBuffer outbuf; output: outbuf->add_hstring(encode_value(value),4);
write_callback() { string chunk = outbuf->read(4096); int num = fd->write( outbuf ); outbuf->unread( 4096-num ); }
Things like subbuffers is unfortunately actually impossible when using a stringbuilder.
Well, it's not, really. Unless I made some rather striking mistakes.
How do you add the trailing 0, and how do you avoid the fact that the string_builder_x functions can at any time reallocate the 's' member to point to another memory location, making the sub-object totally invalid, and how do you avoid the fact that the str member is actually _part of_ the struct pike-string structure, so you at least temporarily have to overwrite the data in the main buffer?
perf( String.Buffer() );
Result 2: 325250971 Compilation: 624ns, Execution: 144.86s
Erm, how do you get this nice looking timings? Do I have to turn on a profiling flag on Hilfe somewhere?
set format bench
(did you note the factor of 1k difference in speed in the test?