Largefile Problems - A Summary

The following has been sent out as email to some people to hint them about the largefile problems. It gives a quick overview of what the problems might be and what's that about the testscript.

Summary
  The Unix98 standard requires largefile support and in fact
  many of the latest operating systems do. However some
  systems chose to still make it not the default resulting
  in two models where some parts of the system use the
  traditional 32bit off_t while others already compiled
  with a largefile 64bit off_t. Mixing libraries and
  plugins is not a good idea.

The 64on32
  While systems like FreeBSD and Darwin do simply use
  a 64bit off_t as the default, there are also systems
  like Linux and Solaris that do not. Instead they do
  implement what's called the `transitional API` in
  the largefile specifications - many calls are given
  a 64-cousin, so that there are both "open" and "open64"
  in the C library, and also "lseek" and "lseek64".

  Using a define like -D_FILE_OFFSET_BITS=64 will make
  for some magic that traditional calls are remapped
  to the transitional API - so that your source code
  might read `open(...)` and `lseek(...)` but it will
  get linked to the symbols "open64" and "lseek64" for
  real. Also, it will make off_t a 64bit entity.

In headers
  As a result however it is highly dangerous to use
  off_t in header files in largefile sensitive systems.
  Most software writers do not expect that an integral
  type like off_t can change its size, it is just used
  in the exported interface like in making up a new
  call `off_t my_lseek(int,off_t,int)`.

  In reality we get a scent from old DOS-modes with
  a "small" mode and "large" mode to compile source
  code - the library code might be compiled with a
  64bit off_t while the application code using the
  library is compiled with 32bit off_t - possibly
  ending up in a callframe mismatch.

  A similar problem arrives when using off_t in
  exported structures as these can have differeng
  sizes and offsets for the member variables therein.
  A library maker should take measures to defend
  against improper off_t size, possibly making up
  dualmode func/func64 like the C library does, but
  in reality many software writers have been not
  aware of the actual problem so far.

seek problem
  Another problem is described in the largefile
  documents in the section about holes in the
  protection system - it stems from the fact that
  some file descriptors might be opened in largefile
  mode while others are not, and they can even be
  transferred from a non-largefile application into
  largefile libraries and vice versa.

  The 64on32 transitional API is trying to support
  this scheme, mostly by introducing a new error
  code EOVERFLOW that will be returned when a
  "small"file application access a file that has
  grown beyond the 2GiB limit due to calls from
  other software parts compiled as "large"file.

  However, most "small"file software does not
  expect this error-code and it is known that many
  software writers do not check the return value
  of lseek - it can easily lead to data corruption
  when the file-pointer is not actually moved.

mixing up
  Most of the software problems arise on the side
  of "small"file applications. As a general advice
  one should compile all software as largefile as
  soon as the system provides these interfaces.
  That is pretty easy, AC_SYS_LARGEFILE in autoconf'ed
  software can do it, or just some -D_FILE_OFFSET_BITS=64
  to be defined somewhere.

  A lot of software however is not aware of a need
  to enable largefile mode somewhere, hundreds of
  opensource applications are compiled with 32bit
  off_t by default. It's been simply forgotten, and
  it would create a lot of work and publicity to
  make everyone aware - with the only result that
  the next new developer misses it just again.

  As a result, we better use technical support
  tools to track the problem area of mixing up
  compiled code from largefile sides and those
  which do not yet do so.

check mismatch
  The perl script to do that can be fetched from
  http://ac-archive.sf.net/largefile - it will
  try to classify binaries and libraries whether
  they are using "-32-" or "-64-" modes, it does
  so by looking for fopen() .vs. fopen64() to
  present in the list of dynamic symbols. Each
  argument binary is checked plus the dynamic
  dependencies it has - and if there are
  mismatches then it prints a list of them.

  Furthermore, the script can detect when a
  library is trying to exhibit itself as dualmode
  exporting both func() and func64() calls, e.g.
  libgstreamer does so. For these it is okay that
  software may be in either of -32- or -64- mode
  when linking to these. So actually, only three
  combinations are rejected: -64- depends on -32-,
  -32- depends on -64-, and 3264 dualmode libraries
  to depend on simple -32- libraries.

distro problem
  When the script is run in a contemporary Linux
  system on /usr/lib/*.so or just /usr/bin then
  it will detect a lot of largefile mismatches.
  The common user will not experience any
  problems with that as long no files are handled
  being possibly larger than 2 GibiByte. And note
  that the Unix98 mandates base utilities like
  `cat` and `cp` to be compiled as largefile ones
  for sure.

  Real opensource OS distributions however carry
  a lot more code from very different sides, and
  particularly there is a number of graphical
  frontend of the filemanager type that is not
  compiled in largefile mode. Sooner or later,
  the problem will come up. It would be best if
  no rpm/deb/whatever binary package to exist
  that has a largefile mismatch in the first place.

  That can be helped if the packagers and distro
  makers would check the binary packages while
  making them. That could be easy when integrating
  the check-routine into the post-%files tools
  (as it is called in rpm) which needs to check
  the libraries and binaries anyway for dependent
  libraries as to do a `chrpath` on them since
  they have been relinked in the DESTDIR staging area.

Future
  The future should be to see all packages get
  compiled in largefile mode thereby eliminating
  any problems from mixing up libraries from
  different sides. A distro maker can ensure that,
  and if that needs a few patches then it is good
  anyway as it makes the very same software more
  portable to freebsd/darwin.

  And a little more beyond, one should really
  think about dropping the 32bit off_t default
  altogether - as it was done on FreeBSD
  obviously. The Linux 2.4 and glibc 2.2 should
  be all ready fine to do the step, and leaving
  the old times of "small"files behind.

links
   http://ac-archive.sourceforge.net/largefile
   http://ftp.sas.com/standards/large.file/
   http://unix.org/version2/whatsnew/lfs.html
   http://www.suse.de/~aj/linux_lfs.html

P.S.
  If you are a programmer then be noted that it does not 
  hurt to compile with AC_SYS_LARGEFILE by default - all 
  the traditional 32bit calls of the C library are just 
  wrappers to forward the arguments to the 64bit calls.
  That's simply because the internals and the OS kernel
  are using 64bit off_t anyway. In effect, your program
  might run even slightly faster, it will need to cope
  with less errorcode, and gets more portable to systems
  like freebsd that use a 64bit off_t anyway.