A toilet is composed of two main pieces -- the Tank and the Bowl.  The working parts of a toilet are contained in the water reservoir called the Tank, shown ghosted in the following illustration.  The part you sit on is called the Bowl.

When the Flush Lever is pushed it pulls up on the Chain, which in turn pulls up the Flush Valve.  This exposes the large discharge hole at the bottom of the tank allowing water to rush quickly into the toilet bowl.  There are two types of Flush Valves.  Older Flush Valves were rubber Balls (see circular insert in figure).  Modern Flush Valves are called Flappers.  The Ball or Flapper Bulb is filled with air, so it wants to float, but the weight of the water on top of it clamps/seals it tightly against discharge hole.  But, once it is pulled away from the hole, water flows under it and it floats.  This holds it open, away from the discharge hole. 

Immediately after a flush begins the water level in the tank starts to go down, so the Float starts to go down.  [It's called a Float because it's a hollow ball floating on the surface of the water.]  The Float, by way of the rod attached to it, opens the Fill Valve, and water from the house water pipe begins to flow into the tank.  So, water's coming in and going out at the same time.  But, it's going out much faster than it's coming in, so the tank empties into the bowl quickly.  When it's almost empty the Flapper can no longer float and falls onto the discharge hole sealing/closing it again.  Now that the discharge hole is closed the water coming in through the Fill Valve starts refilling the tank.  While the tank is refilling a small amount of water is also going from the Fill Valve through the Refill Tube into the Overflow Tube.  The Overflow Tube empties directly into the toilet bowl.  This water raises the water level in (refills) the bowl.  If it weren't for the Refill Tube there would be only a small amount of water in the bottom of the bowl. 

When the water level in the tank has returned to it's original level the Float closes the Fill Valve and the cycle is complete. 

Because something could go wrong during this cycle causing refill water to continue to pour into the Tank flooding your house, the Overflow Tube is provided to direct this water into the bowl after it reaches a certain level. 

The most common problem with toilets is a leaking Flush Valve (Flapper or Ball) causing the Fill Valve to cycle on-and-off periodically.  Sometimes there is just some foreign matter on the Flapper or the discharge hole where they seal, and after you clean them they'll stop leaking.  But, for various reasons, as Flappers get old they stop sealing well and allow water to leak into the bowl.  Flappers are cheap and pretty easy to install.

The diagram above is the only one I could find and it shows an older toilet tank.  Today they no longer have "float" or a rubber ball flush valve, instead they have a combination fill valve and float and a flapper. 

Speedy Jim from alt.home.repair recommends one of the red Fluidmaster replacement Flappers such as this one:

Below is a modern style combination Fill Valve and Float preferred by many on alt.home.repair.  I use these Fluidmaster Fill Valves.  Occasionally it can be a little tricky to get the height just right during installation.