The Boston Diaries

The ongoing saga of a programmer who doesn't live in Boston, nor does he even like Boston, but yet named his weblog/journal “The Boston Diaries.”

Go figure.

Tuesday, July 08, 2008

A quick dip back into assembly with some curious results …

Speaking of assembly

One of the instructions of the x86 architecture I've been curious about is ENTER. Oh, I know it's there to support higher level languages like C and Pascal that use stack frames for local variables. It even supposedly supports nested function definitions (ala Pascal) using the second operand as a kind of “nesting level.”

But I've never seen an actual instance of ENTER used with a “nesting level” greater than 0. The only instance I've ever seen used has been

ENTER	n,0

Which is equivilent to

PUSH	EBP	; or BP if 16-bit code
MOV	EBP,ESP
SUB	ESP,n

(And in fact, that sequence is generated by GCC as it's actually faster than ENTER n,0 and C doesn't allow nested functions to begin with.)

But being curious about what ENTER actually does, I decided to play around with it. I wrote some simple code:

		bits	32
		global	sub0
		extern	pmem

		section	.text

sub0		enter	8,0
		mov	eax,0DEADBEEFh
		mov	[ebp-4],eax
		mov	eax,0CAFEBABEh
		mov	[ebp-8],eax
		lea	ebx,[ebp+4]
		push	dword 0c0000001h
		call	sub1
		leave	
		ret

sub1		enter	8,1
		mov	eax,0DEADBEEFh
		mov	[ebp-4],eax
		mov	eax,0CAFEBABEh
		mov	[ebp-8],eax
		push	dword 0c0000002h
		call	sub2
		leave
		ret

sub2		enter	8,2
		mov	eax,0DEADBEEFh
		mov	[ebp-4],eax
		mov	eax,0CAFEBABEh
		mov	[ebp-8],eax
		push	dword 0c0000003h
		call	sub3
		leave
		ret

sub3		enter	8,3
		mov	eax,0DEADBEEFh
		mov	[ebp-4],eax
		mov	eax,0CAFEBABEh
		mov	[ebp-8],eax
		push	dword 0c0000004h
		call	sub4
		leave
		ret

sub4		enter	8,4
		mov	eax,0DEADBEEFh
		mov	[ebp-4],eax
		mov	eax,0CAFEBABEh
		mov	[ebp-8],eax
		push	dword 0
		push	dword 0
		push	ebx
		push	esp
		call	pmem
		add	esp,16
		leave
		ret

And the following C code:

#include <stdio.h>
#include <stdlib.h>

extern void sub0(void);

void pmem(unsigned long *pl,unsigned long *ph)
{
  assert(pl < ph);

  while(ph >= pl - 2)
  {
    printf("\t%08lX: %08lX\n",(unsigned long)ph,*ph);
    ph--;
  }  
}

int main(void)
{
  sub0();
  return EXIT_SUCCESS;
}

Nothing horribly complicated here. pmem() just dumps the stack, and the various sub*() routines create deeper nestings of stack activation records while creating enough space to store two four-byte values. The results though?

Curious (comments added by me after the run) …

BFFFFD1C: 0804853C		return addr to main()
BFFFFD18: BFFFFD20	stack frame sub0
BFFFFD14: DEADBEEF		local0
BFFFFD10: CAFEBABE		local1
BFFFFD0C: C0000001		marker for calling sub1
BFFFFD08: 08048591		return addr to sub0
BFFFFD04: BFFFFD18	stack frame sub1
BFFFFD00: DEADBEEF		local0
BFFFFCFC: CAFEBABE		local1
BFFFFCF8: 08049708		?
BFFFFCF4: C0000002		marker for calling sub2
BFFFFCF0: 080485B1		return addr to sub1
BFFFFCEC: BFFFFD04	stack frame sub2
BFFFFCE8: DEADBEEF		local0
BFFFFCE4: CAFEBABE		local1
BFFFFCE0: 00000002		?
BFFFFCDC: 400079D4		?
BFFFFCD8: C0000003		marker for calling sub3
BFFFFCD4: 080485D1		return addr to sub2
BFFFFCD0: BFFFFCEC	stack frame sub3
BFFFFCCC: DEADBEEF		local0
BFFFFCC8: CAFEBABE		local1
BFFFFCC4: BFFFFCD0		? sf3
BFFFFCC0: 4000F000		?
BFFFFCBC: 02ADAE54		?
BFFFFCB8: C0000004		marker for calling sub4
BFFFFCB4: 080485F1		return addr to sub3
BFFFFCB0: BFFFFCD0	stack frame sub4
BFFFFCAC: DEADBEEF		local0
BFFFFCA8: CAFEBABE		local0
BFFFFCA4: BFFFFCD0		? sf3
BFFFFCA0: BFFFFCB0		? sf4
BFFFFC9C: 40011FE0		?
BFFFFC98: 00000001		?
BFFFFC94: 00000000		push dword 0
BFFFFC90: 00000000		push dword 0
BFFFFC8C: BFFFFC8C		? supposed to be ebx
BFFFFC88: BFFFFC8C		? supposed to be esp
BFFFFC84: 08048618		return addr to sub4

From my understanding of what ENTER does, each “level” creates a type of nested stack activation record with pointers to each previous “level's” stack record. And while each level has the required number of additional entries, the actual contents don't make sense.

Running this on a different Linux system produced similarly confusing results. I'm not sure if ENTER is horribly broken these days (I wonder how often the instruction is actually used), or perhaps, it is indeed a Linux problem? Not that I'm going to be using assembly any time soon … I'm just curious.

Obligatory Picture

An abstract representation of where you're coming from]

Obligatory Contact Info

Obligatory Feeds

Obligatory Links

Obligatory Miscellaneous

Obligatory AI Disclaimer

No AI was used in the making of this site, unless otherwise noted.

You have my permission to link freely to any entry here. Go ahead, I won't bite. I promise.

The dates are the permanent links to that day's entries (or entry, if there is only one entry). The titles are the permanent links to that entry only. The format for the links are simple: Start with the base link for this site: https://boston.conman.org/, then add the date you are interested in, say 2000/08/01, so that would make the final URL:

https://boston.conman.org/2000/08/01

You can also specify the entire month by leaving off the day portion. You can even select an arbitrary portion of time.

You may also note subtle shading of the links and that's intentional: the “closer” the link is (relative to the page) the “brighter” it appears. It's an experiment in using color shading to denote the distance a link is from here. If you don't notice it, don't worry; it's not all that important.

It is assumed that every brand name, slogan, corporate name, symbol, design element, et cetera mentioned in these pages is a protected and/or trademarked entity, the sole property of its owner(s), and acknowledgement of this status is implied.

Copyright © 1999-2024 by Sean Conner. All Rights Reserved.