Getting started with electronics design takes buying some equiptment, but it can be confusing what to buy. I'm going to list here some of the items that I throught were helpful getting started.
Breadboard with power supply
Most people start with electronics by connecting up circuits on a breadboard. A breadboard is great for trying out circuits before moving to more permanent construction techniques such as perfboard or custom PCBs.
Spools of wire
Wire is available as stranded or solid. Solid works best with boardboards and perfboards. Stranded works best when creating cables for interboard connections.
Radio Shack is a good source of cheap multimeters.
An oscilloscope is critical for debugging analog circuits or communication protocols. I prefer PC based oscilloscopes since they are cheaper then dedicated equiptment and are more automated then a traditional phosphor oscilloscope.
As more components are bought, it becomes necessary to store all the components in an organized way and storage compartments are very helpful. These can be bought at the local home improvement or hardware store.
It's important to have alot of different resistor values around as it's almost impossible to buy the correct values in advance. Digikey sells resistors in kits that have a small number of all the standard resistor values.
Capacitors are like resistors. You need a large number of values. Digikey also sells capacitor kits in addition to the resistor kits.
Microcontrollers are a great way to get started in digital electronics. They are essentially a little computer on a single chip. Unlike large computers these processors usually don't run a full blown operating system so dedicated hardware is used for programming an debugging. In Circuit Debugging are the prefered way to go for development since it's a huge time saver to be able to program and debug software without needing to pull the chip from the circuit. Beleve me, spend the extra money it is worth it.
If you are willing to spend some money development boards are a quick way to get started. They typically have a microcontroller, voltage regulator, some LEDs, and some sensors all on one board. The downside is that if you break anything on the board you have to buy a whole new board. With descrete components only the broken part needs to be replaced. On the other hand, more and more components are only available in surface mount packages which almost require a custom PCB to use at all. ARM processors are relatively powerfull microcontroller but at present are not available in through hole versions.