This stands on top of exporting both implementations on all systems, i.e. the largefile variant with 64bit offset values and a traditional variant using a "long" offset value. The rest ist just an ifdef-else in the headers for a third function without suffix while the binary has only two functions func32 and func64.
Thomas Orgis has pointed me to < ahref="http://bugs.winehq.org/show_bug.cgi?id=22132"> http://bugs.winehq.org/show_bug.cgi?id=22132 which did show a problem with twinlibs with respect to dynaloading the instance. So, in order to make a binary compatible variant of a libary that is not as bad to maintain one can try to care for a compat32 lib variant. This is now considered the best practice.
I changed the python style for match/replace to my new style using classes Match/MatchReplace. While going over the source I did find some minor glitches like missing "r" on regex pattersn. Additionally I added an "ignore:" when the `ldd` did resolve to a non-library artifact as an import (which might need to be expanded on).
Renewed intest has shown that there was a small bug in the testscript. It is fixed it and additionally I was reformatting the output of the testscript run to make it better for postprocessing with third party tools. Another report did point out problems when using config.h based on a general problem space described on the broken-builds page.
Created a sitefile and rebuilding the website with the formatter. Here we get a sitemap as well as a printer-friendly mode for free. In the course some markup was modified, e.g. all paragraphs should now be justified and indented slightly.
To get an easy overview how to support largefile with a library these pages tell the necessary changes - step by step. It's been given from memory but let's see if someone notes me of minor problems and buglets. Atleast it should give people an easier start.
during porting the script to python, I did also change the perl script to keep them in sync. Well, it did turn out have a char missing in the end. Tobias Brausen was kind to send a patch that fixes a bit more and also adds "-w" in the header, that's a good choice I think.
Otherwise, a little discussion about largefile in win32 was there on on the autoconf mailinglist. I did copy some parts to a new page on win32libc.html, and may be you want to read the full thread here: mail.gnu.org/archive/html/autoconf/2003-11/msg00039.html
and a few updates to the website were added as well. Likewise, the build system was redone to allow proper snapshot building plus the ability to install the scripts and docs and to build rpm packages. That should allow for even wider distribution.
Using a new makefile setup, the website is given a uniform look for sure. The last refresh of 08-02 was wrong since code2html was installed, now we accept gnu source-highlight as an alternative.
While checking newer distros I became aware that libgnomevfs-2 is not a dualmode library - the perl script did think it was since it did saw a pair of *_read_int/*_read_int64. We exclude that pair now. Furthermore I did notice that some routines reference both the "fopen" and "fopen64" routine. This can not be anything else than an error, a miscompiled binary. We already have been marking these libraries internally, now we show a list classified as "DANGER". Finally, we add six other symbols to the list of known imported symbols that make a library to be classified.
From problems of zziplib on solaris-8, I have learned that the -D_LARGEFILE_SOURCE define does only select that the program sources want to use the XOpen extension of `off_t` instead of the old `long`. As per default however there is a `typedef long off_t` so it does not make a difference but for one thing: solaris (and glibc) do only export the symbol `fseeko` then which happens to be introduced with the `off_t` type as well. The only correct way in Unix98 to shift a 32bit default off_t into a 64bit off_t happens to use a setting of the -D_FILE_OFFSET_BITS=64 and everything else might just not work. Here I need to check all webpages here to adapt to this.
From the experiences, I am reworking the off_t_headers page with a reference to ac_sys_largefile_sensitive, some proper example source, and a hint on howto fix an existing off_t-mismatch problem quickly and incrementally. I'm going to add a hint on the kde bug id below.
Stephan Kulow (coolo) has looked into the matter and simply comments: "I guess this needs to be fixed asap". The bug id is now "assigned" to him, so perhaps this will be ironed out very soon now.
The fseeko() function is not defined with AC_SYS_LARGEFILE alone, you need to add _LARGEFILE_SOURCE somewhere. The AC_FUNC_FSEEKO will do that - however, it makes it somewhat impossible to write software that runs in both modes, non-LFS and LFS in a system. (and what about mixed-mode libraries?). The link above tells of a glibc bug around it: the off_t will be made 64bit wide but your library will link to 32bit fseeko() instead of fseek64(). Note that glibc is not alone with this: a quick check into the solaris8 headers gives me another horror. It seems the LFS spec is non-realistic in this very situation - and probably needs to be fixed.
The LSB ships a tool `appchk` to test conformance of binaries including its imports. It would be natural to add a check for largefile-mismatch in there. Similar could be done for `libchk` and the sampel rpm scripts shipped by linuxbase.org.
I did also send an mail to the lsb-test mailing list regarding some action items for largefile support. It does basically list the three basic steps to get away from 64on32 completely within a few years.
The talkback leads me to add two new pages:
win32/other lists behaviour of the
win32-API and some pre-unix98 systems. The
environ changes tells about
possibilities in fixing the compile-time environment of
systems in the future to get rid of the problems of the
64on32 world.
So, to get some awareness for the largefile problem, I had been preparing an editorial ready to be posted to freshmeat. The policy requires to contact Jeff Covey which I did with sending an e-mail on the 13th. Today he did reply along with a html version of the editorial text and a request to send some little bio information that should go with it. Well, let's see when it gets posted, looking forward to that day :-)
clean up the code so that it does document itself - may be some people would be afraid to just run some perl script on their system. Furthermore, a "--noncleanall" option was added to show all the symbols that could be offending - and looking closer, I do see now that some more offendings must be also silenced down. The thing starts to get smart...
It took about a day to create the website, and after that I've been looking agtain into the testscript - it did seem to me that sometimes the largefile mismatch is not important. Therefore, I did add a routine that can detect common binding that are simply acceptable - the are marked as "(clean)" in the output list and running the script as "--smart" or even "--silent" will suppress these entries from the final output list altogether. Furthermore, using "--nonclean" can help to detect which symbols were offending (it only shows the first imported symbol that is not in the good-list).
Creating a perl script that checks binaries/libraries installed in system whether they have mismatching off_t types. That may be later used by system vendors to extend they post-rpm-build scripts to check for mismatches.
Starting to create a whole website around the topic and placing all relevant info just there. It's also the start of this history document that you just read.
The discussion on the autoconf mailinglist comes to the conclusion that it should be part of the distro maker and packaging process to ensure that binaries are not mixed up between 32bit-off_t mode and 64bit-off_t mode.
That is also from a note from the largefile support group that states there are holes in the mixed-style support for libc filedescriptors. After all, it is not only a callframe-mismatch problem but also about file-descriptor inheritance and passing file-descriptors from one library/plugin to the next (e.g. zlib's gzdopen).
Starting a diary entry on advogato.org that summarizes the problemspace a bit in the way it did exhibit itself up to this. The discussion on mailinglist has heatened up and it makes me more and more worry that it is actually a very bad idea to mix binaries of 32bit-off_t with those of 64bit-off_t.
Filing a bug report for KDE3 - the <kio/jobclasses.h>
header makes use of off_t without guards. It seems that even an
experienced programmer like Stephan Kulow himself has missed the
problemspace about off_t.
Note that most of the /usr/lib/kde3/* plugins are actually compiled as 32bit while some are compiled as 64bit-off_t - that brings the kde plugin system at risk.
Grepping a bit through header files, I get to note that zlib itself uses off_t for its function pair of gztell/gzseek. The zlib on my system is compiled as 32bit-off_t while the gzip utility is compiled as 64bit-off_t - only by luck the gzip does not use the function pair that got affected. So... sending a bug report to zlib-at-gzip-org.
Furthermore, I notice that in mandrake9 about all files have some dependency on libaudiofile which makes heavy use of off_t in its header files but has not guarded against largefile-sensitivy. Again, sending a bug report to the oss.sgi.com mailinglist for it.
Sending a bug-report mail to the maintainers of these projects that they have a problem - these frontends are compiled as 32bit off_t.
See below that Samba has already implemented 64bit SMB extensions, and even the libsmbclient software is compiled as 64bit-off_t with it.
Making the LARGEFILE_SENSITIVE macro available on the ac-archive.sf.net site, and starting a thread on the autoconf Mailinglist about largefile support.
Note that samba uses a much more elaborate autoconf check to enable
64bit-off_t support in the various systems that goes beyond what the
original autoconf macro AC_SYS_LARGEFILE does provide.
http://cvs.samba.org/cgi-bin/cvsweb/samba/source/configure.in?rev=1.383
Making zziplib 0.10.77 into a dualmode library offering both 32bit and 64bit off_t interfaces.
Checking a bit deeper, I learn that Unix98-system are actually
required to support largefile system - it just happens that
some systems still use a 32bit off_t default.
http://unix.org/version2/whatsnew/lfs.html
more links:
http://unix.org/whitepapers/64bit.html
http://www.google.de/search?q=_XBS5_ILP32_OFFBIG
http://docsrv.caldera.com/cgi-bin/man/man?Intro+2#LargeFileSupport
http://www.llnl.gov/asci/platforms/bluepac/large_file_prog.html
http://www.google.de/search?q=_LARGEFILE_SOURCE
The original incident that made me think about the problemspace - a user of LinNeighborhood had problems to copy files to his local computer when they are slightly bigger than 2GiB - in fact, this graphical frontend did show the filesize numbers all incorrect, and even that copying started, it broke after somewhat beyond 2GiB.