Hi,
Here is another bug with the Shuffler.
If I create a big file with the Shuffler (more than 2.1GB) and test this program:
#!/usr/local/bin/pike
Stdio.File dest_fd;
void shuffle_filename(string filename) { object shuffler; object shuffle; shuffler = Shuffler.Shuffler(); shuffle = shuffler->shuffle(dest_fd); shuffle->set_done_callback(lambda() { destruct(dest_fd);}); shuffle->add_source(Stdio.File(filename, "r")); shuffle->start(); }
int main(int argc, array argv) { object port = Stdio.Port(8080); dest_fd = port->accept(); shuffle_filename(argv[1]); return -1; }
Then if call this program with a 2.1GB file as argument and connect to port 8080, I get no data (while I should get 2.1GB of data) and the done callback is called immediatly. The same program works perfectly for files with "normal" size. The limit seem to be just above 2.0GB. Tested on Linux 2.4 with reseirfs and Pike 7.6 CVS.
/ David
Which version of ResierFS?
/ Marcus Agehall (PacketFront)
Previous text:
2004-08-27 01:21: Subject: Shuffler can't handle big file
Hi,
Here is another bug with the Shuffler.
If I create a big file with the Shuffler (more than 2.1GB) and test this program:
#!/usr/local/bin/pike
Stdio.File dest_fd;
void shuffle_filename(string filename) { object shuffler; object shuffle; shuffler = Shuffler.Shuffler(); shuffle = shuffler->shuffle(dest_fd); shuffle->set_done_callback(lambda() { destruct(dest_fd);}); shuffle->add_source(Stdio.File(filename, "r")); shuffle->start(); }
int main(int argc, array argv) { object port = Stdio.Port(8080); dest_fd = port->accept(); shuffle_filename(argv[1]); return -1; }
Then if call this program with a 2.1GB file as argument and connect to port 8080, I get no data (while I should get 2.1GB of data) and the done callback is called immediatly. The same program works perfectly for files with "normal" size. The limit seem to be just above 2.0GB. Tested on Linux 2.4 with reseirfs and Pike 7.6 CVS.
/ David
/ Brevbäraren
3.6.25 but it's nearly impossible it would be a bug in reiserfs since other application can read this file including pike's Stdio.File with a call to read().
Marcus Agehall (PacketFront) @ Pike (-) developers forum wrote:
Which version of ResierFS?
/ Marcus Agehall (PacketFront)
Previous text:
2004-08-27 01:21: Subject: Shuffler can't handle big file
Hi,
Here is another bug with the Shuffler.
If I create a big file with the Shuffler (more than 2.1GB) and test this program:
#!/usr/local/bin/pike
Stdio.File dest_fd;
void shuffle_filename(string filename) { object shuffler; object shuffle; shuffler = Shuffler.Shuffler(); shuffle = shuffler->shuffle(dest_fd); shuffle->set_done_callback(lambda() { destruct(dest_fd);}); shuffle->add_source(Stdio.File(filename, "r")); shuffle->start(); }
int main(int argc, array argv) { object port = Stdio.Port(8080); dest_fd = port->accept(); shuffle_filename(argv[1]); return -1; }
Then if call this program with a 2.1GB file as argument and connect to port 8080, I get no data (while I should get 2.1GB of data) and the done callback is called immediatly. The same program works perfectly for files with "normal" size. The limit seem to be just above 2.0GB. Tested on Linux 2.4 with reseirfs and Pike 7.6 CVS.
/ David
/ Brevbäraren
Sounds too me like there is a unsigned/signed bug in the shuffler.
/ Marcus Agehall (PacketFront)
Previous text:
2004-08-27 08:56: Subject: Re: Shuffler can't handle big file
3.6.25 but it's nearly impossible it would be a bug in reiserfs since other application can read this file including pike's Stdio.File with a call to read().
Marcus Agehall (PacketFront) @ Pike (-) developers forum wrote:
Which version of ResierFS?
/ Marcus Agehall (PacketFront)
Previous text:
2004-08-27 01:21: Subject: Shuffler can't handle big file
Hi,
Here is another bug with the Shuffler.
If I create a big file with the Shuffler (more than 2.1GB) and test this program:
#!/usr/local/bin/pike
Stdio.File dest_fd;
void shuffle_filename(string filename) { object shuffler; object shuffle; shuffler = Shuffler.Shuffler(); shuffle = shuffler->shuffle(dest_fd); shuffle->set_done_callback(lambda() { destruct(dest_fd);}); shuffle->add_source(Stdio.File(filename, "r")); shuffle->start(); }
int main(int argc, array argv) { object port = Stdio.Port(8080); dest_fd = port->accept(); shuffle_filename(argv[1]); return -1; }
Then if call this program with a 2.1GB file as argument and connect to port 8080, I get no data (while I should get 2.1GB of data) and the done callback is called immediatly. The same program works perfectly for files with "normal" size. The limit seem to be just above 2.0GB. Tested on Linux 2.4 with reseirfs and Pike 7.6 CVS.
/ David
/ Brevbäraren
/ Brevbäraren
It seems like the datastructure in the shuffler uses an int to keep track of the length. (See shuffler.h in the data struct)
This would cause problems with large files on small[*] systems.
The quick fix, would be to replace the int len with unsigned int len. That would allow 4gb files.
[*] Small systems being systems with int size of 32bits or less.
/ Marcus Agehall (PacketFront)
Previous text:
2004-08-27 09:38: Subject: Re: Shuffler can't handle big file
Sounds too me like there is a unsigned/signed bug in the shuffler.
/ Marcus Agehall (PacketFront)
And why not putting an unsigned double directly?
Marcus Agehall (PacketFront) @ Pike (-) developers forum wrote:
It seems like the datastructure in the shuffler uses an int to keep track of the length. (See shuffler.h in the data struct)
This would cause problems with large files on small[*] systems.
The quick fix, would be to replace the int len with unsigned int len. That would allow 4gb files.
[*] Small systems being systems with int size of 32bits or less.
/ Marcus Agehall (PacketFront)
Previous text:
2004-08-27 09:38: Subject: Re: Shuffler can't handle big file
Sounds too me like there is a unsigned/signed bug in the shuffler.
/ Marcus Agehall (PacketFront)
Well, one could do that as well.
I did some testing. Changing len to unsigned does not solve the problem. Doing that, I get nothing. No output and no done-callback.
/ Marcus Agehall (PacketFront)
Previous text:
2004-08-27 10:26: Subject: Re: Shuffler can't handle big file
And why not putting an unsigned double directly?
Marcus Agehall (PacketFront) @ Pike (-) developers forum wrote:
It seems like the datastructure in the shuffler uses an int to keep track of the length. (See shuffler.h in the data struct)
This would cause problems with large files on small[*] systems.
The quick fix, would be to replace the int len with unsigned int len. That would allow 4gb files.
[*] Small systems being systems with int size of 32bits or less.
/ Marcus Agehall (PacketFront)
Previous text:
2004-08-27 09:38: Subject: Re: Shuffler can't handle big file
Sounds too me like there is a unsigned/signed bug in the shuffler.
/ Marcus Agehall (PacketFront)
/ Brevbäraren
Fix in 7.7 cvs. Please test it. If it works, it ought to be backported to 7.6.
/ Marcus Agehall (PacketFront)
Previous text:
2004-08-27 10:28: Subject: Re: Shuffler can't handle big file
Well, one could do that as well.
I did some testing. Changing len to unsigned does not solve the problem. Doing that, I get nothing. No output and no done-callback.
/ Marcus Agehall (PacketFront)
No, len can be -1.
/ Martin Nilsson (DivX Networks)
Previous text:
2004-08-27 09:51: Subject: Re: Shuffler can't handle big file
It seems like the datastructure in the shuffler uses an int to keep track of the length. (See shuffler.h in the data struct)
This would cause problems with large files on small[*] systems.
The quick fix, would be to replace the int len with unsigned int len. That would allow 4gb files.
[*] Small systems being systems with int size of 32bits or less.
/ Marcus Agehall (PacketFront)
But then again, perhaps not. As far as I can tell, off_t is a signed datatype, so it should work.
/ Marcus Agehall (PacketFront)
Previous text:
2004-08-27 12:50: Subject: Re: Shuffler can't handle big file
Hmm, that signaling might break my current fix too. Why can len be <0?
/ Marcus Agehall (PacketFront)
I don't know, but that source obviously checks for that case in several places (but perhaps it doesn't apply in your case).
/ Martin Nilsson (DivX Networks)
Previous text:
2004-08-27 12:50: Subject: Re: Shuffler can't handle big file
Hmm, that signaling might break my current fix too. Why can len be <0?
/ Marcus Agehall (PacketFront)
There was something that tested for negative values. I was aiming to get the length up to 64bits, but apparently I failed. off_t is 4 bytes on my machine, but still, the shuffler seems to work.
/ Marcus Agehall (PacketFront)
Previous text:
2004-08-27 12:54: Subject: Re: Shuffler can't handle big file
I don't know, but that source obviously checks for that case in several places (but perhaps it doesn't apply in your case).
/ Martin Nilsson (DivX Networks)
Works for me now. I've backported the bugfix to 7.6.
/ David
Marcus Agehall (PacketFront) @ Pike (-) developers forum wrote:
There was something that tested for negative values. I was aiming to get the length up to 64bits, but apparently I failed. off_t is 4 bytes on my machine, but still, the shuffler seems to work.
/ Marcus Agehall (PacketFront)
Previous text:
2004-08-27 12:54: Subject: Re: Shuffler can't handle big file
I don't know, but that source obviously checks for that case in several places (but perhaps it doesn't apply in your case).
/ Martin Nilsson (DivX Networks)
pike-devel@lists.lysator.liu.se
-
David Gourdelier -
Marcus Agehall (PacketFront) ＠ Pike (-) developers forum -
Martin Nilsson (DivX Networks) ＠ Pike (-) developers forum