Callframe Mismatch

Suppose you have call declared as

      MY_FILE* my_fseek (MY_FILE* file, off_t offset, int whence); 

That is actually an interesting one as it contains an "off_t" somewhere in the middle. Now, let's suppose further this is an ILP32 system with 32bit pointers and 32bit integers, and a classic default of 32bit for off_t as well.

The callframe for this function will therefore consume 3x32bit being 12 bytes. An application calling this function will build a callframe on the stack (or in the cpu) using that amount of memory, placing "whence" at a distance of 8 bytes from the top of the callframe.

       +------------------+  0
       | MY_FILE* file    |
       +------------------+  4
       | off_t    offset  |
       +------------------+  8
       | int      whence  |
       +------------------+ 12 

Now let's suppose the library is compiled as 32bit-off_t which makes the called-function to expect that form of the callframe on the stack and picking arguments from it. However, the very same header line declaring my_fseek() can be included also from a different place - an application that wants to call the my_fseek function. With bad luck, the application is compiled as 64bit-off_t, and therefore it will build the following callframe:

       +------------------+  0
       | MY_FILE* file    |
       +------------------+  4
       | off_t    offset  |
       |                  |
       +------------------+ 12
       | int      whence  |
       +------------------+ 16 

Note that the C linker will not complain about a callframe mismatch - the C linker does only know the symbol name that it shall bind, so it will see the (unresolved) symbol in the application code and the (exported) symbol in the library, and happily bind the two. As soon as the application runs, it will build a callframe as shown in picture 2. The called library routine will expect the other callframe and happily pick something else than expected. Of course, that can have desastrous results.

That's the reason why the unix98 largefile group proposed a transitional API for systems that have been using 32bit off_t all along as the default - where the exported symbols are actually renamed. So, there is something in the header files that reads like:

   #if defined LARGEFILE_SENSITIVE && _FILE_OFFSET_BITS+0 == 64
      MY_FILE* my_fseek64 (MY_FILE* file, off_t offset, int whence);
   #define my_fseek my_fseek64
   #else
      MY_FILE* my_fseek (MY_FILE* file, off_t offset, int whence);
   #endif

and which will make the 64bit application to be linked with a different symbol that is simply there to catch a call with a callframe using a 64bit-off_t. So in the latter case, the 64bit-off_t callframe will be handled through my_fseek64 while the 32bit-off_t callframe gets handled in my_fseek - and the C linker will make it right by linking 32bit-off_t on 32bit-off_t symbol (my_fseek), and also linking 64bit-off_t on 64bit-off_t symbol (my_fseek64).