13.11.04 22:35:03, Martin Stjernholm wrote:

...

buf->st_mtime = (time_t)((((INT64)findbuf.ftLastWriteTime.dwHighDateTime << 32)

• findbuf.ftLastWriteTime.dwLowDateTime - 116444736000000000) / 10000000);

Does that work in the transition between DST and non-DST time?

Yes.

FILETIME holds the number of 100-nanosecond intervals since January 1, 1601 00:00 UTC.

time_t holds the number of seconds since January 1, 1970 00:00 UTC.

The reference points are absolute points in time - the notion of days and time-of-day doesn't come into it at all; thus timezones, DST and all that becomes irrelevant. You simply have to compensate the different reference points in time (the subtraction) and go from 100-nanoseconds to seconds (the division).

With this in mind I doubt that SystemTimeToTzSpecificLocalTime only does that simple calculation, at least on post-W9x system.

The calculation it does is actually even simpler. That function works on SYSTEMTIME structures, which hold a date and time split into year, month, day, hour and so on. This representation is dependent on the timezone and DST, in contrast to FILETIME or time_t.

If you use FileTimeToSystemTime to convert a FILETIME into the SYSTEMTIME representation, you get GMT time. Now if you want to know what time that was in *your* timezone, say GMT+02:00, you feed that SYSTEMTIME to SystemTimeToTzSpecificLocalTime. That function will now simply add 2 to the "hour" member of SYSTEMTIME and cope with overflows into the day, month and year members. Nothing more.

Now __loctotime_t does exactly the opposite - take a SYSTEMTIME specifying the date and time in the local timezone, adjust the time and maybe date to get GMT time, and then convert that into the time_t representation.

So SystemTimeToTzSpecificLocalTime and the first part of __loctotime_t exactly cancel each other out! I'd be quite surprised if that was indeed part of a KB article or coding sample, especially considering that http://msdn.microsoft.com/library/en-us/sysinfo/base/converting_a_time_t_val... describes the right way to do it.

Axel

14.11.04 02:55:01, Martin Stjernholm wrote:

So I gather that older windowsen store local time in the timestamps, as opposed to NT, and that FileTimeToLocalFileTime is a nop on the old OS's.

No. The FILETIME values in the WIN32_FIND_DATA structure are defined as the number of 100-nanosecond intervals since January 1, 1601 00:00 UTC on *all* Windows versions. Likewise, the behaviour of FileTimeToLocalFileTime, and the bug, is identical on NT and W9x.

It all depends on the underlying *file system*.

On NTFS volumes, the time is stored as UTC, so it's easy to calculate the difference to January 1, 1601 00:00 UTC directly.

On FAT volumes, the time is stored as local time, as a heritage from MS-DOS days. This means that this local time must be converted to UTC first before calculating the FILETIME. During this conversion, all Windows versions exhibit the same bug, using the *current* DST status instead of the DST status at that time. So the FILETIME value can be off by one hour for FAT volumes.

Now the CRT _stat implementation uses FileTimeToLocalFileTime to convert this value back to local time - which suffers from the same DST bug. Then the undocumented function __loctotime_t is used to convert this local time to a time_t, and this function correctly accounts for DST at that time.

On FAT volumes, the conversion error while retrieving the FILETIME, and the conversion error of FileTimeToLocalFileTime exactly cancel each other out, so the final result of the _stat implementation is always correct.

On NTFS volumes however, there is no conversion error while retrieving the FILETIME because the file system already stores it as UTC, so the conversion error of FileTimeToLocalFileTime can cause the final result to be 1 hour off.

The current Pike implementation does it exactly the other way round - the result is always correct on NTFS volumes, but can be 1 hour off on FAT volumes.

The simple calculation I proposed has exactly the same properties as the current Pike implementation - the result is always correct on NTFS volumes, but can be 1 hour off on FAT volumes. What's more, it's considerably less bloated and does not completely break Pike on W9x.

If you want to always get correct results for NTFS *and* FAT volumes, you'd have to detect which kind of file system the file is in, and then use my proposed calculation for NTFS and the standard _stat implementation for FAT. However this would be quite a hassle just because of an occasional 1 hour off on FAT volumes.

Summing up, I renew my pledge to replace the current Pike implementation with my proposed calculation.

Cheers, Axel

I wonder how many of these comments are still there from Windows 3.0. Or earlier.

15.11.04 00:50:02, Martin Stjernholm wrote:

...

No. The FILETIME values in the WIN32_FIND_DATA structure are defined as the number of 100-nanosecond intervals since January 1, 1601 00:00 UTC on *all* Windows versions. Likewise, the behaviour of FileTimeToLocalFileTime, and the bug, is identical on NT and W9x.

Doesn't that contradict the comment in the CRT source? (Not that I necessarily believe that the comment must be correct.)

The comment is very old, and what it refers to, albeit in a quite obscure way, is that Win32s incorrectly implements FILETIME based on local times. However this has not been carried over to Win32c aka W9x, I have tested and confirmed that W9x implements FILETIME exactly like NT/2K/XP. So the comment is not relevant for Pike anymore, which would never run on Win32s anyway.

But if it isn't cheap to check the filesystem type, we probably can't afford to do it anyway.

Well, it's not THAT expensive:

GetVolumeInformation( "C:", NULL, 0, NULL, NULL, NULL, szFs, 8);

For the first parameter, the root directory must be extracted from the file name. If the file name doesn't contain a drive specification, that parameter should be chosen as NULL, then GetVolumeInformation uses the current drive.

Then, if "NTFS" or "HPFS" is returned in szFs, the Pike implementation low_stat (changed to use my proposed calculation) should be used, otherwise the standard _stat function.

Btw, speaking of DST bugs, it seems GetFileTime, which is used by Stdio.File.stat, has an even more nasty one:

FAT records times on disk in local time. GetFileTime retrieves cached UTC times from FAT. When it becomes daylight saving time, the time retrieved by GetFileTime will be off an hour, because the cache has not been updated. When you restart the machine, the cached time retrieved by GetFileTime will be correct.

Firstly, I don't see why it would get incorrect; it's afterall cached in UTC where there's no such thing as DST.

Indeed, it seems that Micro$oft is very, very confused by DST. By "incorrect" they probably mean "incorrect in comparison to our already incorrect implementation of FILETIME values in WIN32_FIND_DATA", and not "incorrect" in the absolute sense of the word :)

Regarding Stdio.File.stat, it should probably be changed to use my proposed combination of _stat/_low_stat as well - avoiding redundancy and more bugs. BTW, I just noticed the function "convert_filetime_to_time_t" in that context - which exactly implements my proposed calculation, albeit a little more expensive by using floats instead of INT64...

Axel

15.11.04 13:40:03, Marcus Comstedt wrote:

...

GetVolumeInformation( "C:", NULL, 0, NULL, NULL, NULL, szFs, 8);

For the first parameter, the root directory must be extracted from the file name. If the file name doesn't contain a drive specification, that parameter should be chosen as NULL, then GetVolumeInformation uses the current drive.

What about paths starting with \?

For UNC paths, one has no option but to trust the corresponding network resource to return the correct FILETIME. So yes, UNC paths should be detected and the same calculation as for NTFS should be used.

Axel

15.11.04 23:40:02, Martin Stjernholm wrote:

Ok then. Do you already have a patch?

No, because the proposed change has evolved quite a bit during this thread. I think I'll better leave it to an experienced Pike developer to make the final implementation.

...

Indeed, it seems that Micro$oft is very, very confused by DST. By "incorrect" they probably mean "incorrect in comparison to our already incorrect implementation of FILETIME values in WIN32_FIND_DATA", and not "incorrect" in the absolute sense of the word :)

Even with that quirky interpretation of "incorrect", it doesn't really explain how a reboot should change matters.

After a reboot, GetFileTime will recalculate the cached UTC times according to the current DST status, so they will once again be consistent with the values returned in WIN32_FIND_DATA, and thus by M$ definition "correct".

Axel

An attempt at fixing it is committed to 7.7. It's not tested at all, not even compiled. It's actually rather unfortunate that there's no NT system in xenofarm anymore.

I gave up on fixing a workaround for GetFileTime since I couldn't find any reasonably simple way to get the volume information for a file handle.

We need to finish the MinGW port, so that it's easy to compile pike nativly under Win32.